2024-04-13T21:02:33.628 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-13T21:02:33.628 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822 --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/orig.config.yaml 2024-04-13T21:02:33.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T21:02:33.900 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822 branch: reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} email: ceph-qa@ceph.com first_in_suite: false job_id: '7654822' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 7d36322c373c73b184fd252cd881b122d8c51a3b wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7d36322c373c73b184fd252cd881b122d8c51a3b rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: 7d36322c373c73b184fd252cd881b122d8c51a3b owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 3741 sha1: 7d36322c373c73b184fd252cd881b122d8c51a3b sleep_before_teardown: 0 subset: 0/1 suite: rgw suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7d36322c373c73b184fd252cd881b122d8c51a3b targets: smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA/N/FKLBquRPdl60M0y1Az5c6woju1lkGZ2FTsAqgZWA3bDiyOusiaFVnYZpk0uy/nC9ouvTL4sTO6i7eJUTrk= smithi175.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9iF1EUT4zEZ4kQHllVdkLfUpKC05i7vMlbtsRrcaTRcRWNj2GSkNeTg3+iKQ9jFBfrQGq3nEZemVeieYd/EBw= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-12_22:40:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-13T21:02:33.900 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa; will attempt to use it 2024-04-13T21:02:33.900 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks 2024-04-13T21:02:33.901 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-13T21:02:33.902 INFO:teuthology.task.internal:Checking packages... 2024-04-13T21:02:33.926 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '7d36322c373c73b184fd252cd881b122d8c51a3b' 2024-04-13T21:02:33.926 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-13T21:02:33.926 INFO:teuthology.packaging:ref: None 2024-04-13T21:02:33.926 INFO:teuthology.packaging:tag: None 2024-04-13T21:02:33.926 INFO:teuthology.packaging:branch: reef 2024-04-13T21:02:33.926 INFO:teuthology.packaging:sha1: 7d36322c373c73b184fd252cd881b122d8c51a3b 2024-04-13T21:02:33.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-13T21:02:34.242 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1219.ge6cbcc46 2024-04-13T21:02:34.244 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-13T21:02:34.251 INFO:teuthology.task.internal:no buildpackages task found 2024-04-13T21:02:34.251 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-13T21:02:34.256 INFO:teuthology.task.internal:Saving configuration 2024-04-13T21:02:34.270 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-13T21:02:34.276 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T21:02:34.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 20:57:26.002487', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA/N/FKLBquRPdl60M0y1Az5c6woju1lkGZ2FTsAqgZWA3bDiyOusiaFVnYZpk0uy/nC9ouvTL4sTO6i7eJUTrk='} 2024-04-13T21:02:34.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 20:57:26.000943', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9iF1EUT4zEZ4kQHllVdkLfUpKC05i7vMlbtsRrcaTRcRWNj2GSkNeTg3+iKQ9jFBfrQGq3nEZemVeieYd/EBw='} 2024-04-13T21:02:34.325 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-13T21:02:34.331 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-04-13T21:02:34.331 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-04-13T21:02:34.332 INFO:teuthology.run_tasks:Running task console_log... 2024-04-13T21:02:34.437 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faac9f6c3a0>, signals=[15]) 2024-04-13T21:02:34.437 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-13T21:02:34.443 INFO:teuthology.task.internal:Opening connections... 2024-04-13T21:02:34.444 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T21:02:34.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T21:02:34.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T21:02:34.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T21:02:34.596 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-13T21:02:34.603 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-04-13T21:02:34.623 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-04-13T21:02:34.624 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-04-13T21:02:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="Red Hat Enterprise Linux" 2024-04-13T21:02:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8.6 (Ootpa)" 2024-04-13T21:02:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:ID="rhel" 2024-04-13T21:02:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="fedora" 2024-04-13T21:02:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8.6" 2024-04-13T21:02:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://www.redhat.com/" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-13T21:02:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-13T21:02:34.685 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-04-13T21:02:34.709 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2024-04-13T21:02:34.730 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2024-04-13T21:02:34.730 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2024-04-13T21:02:34.789 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="Red Hat Enterprise Linux" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="8.6 (Ootpa)" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:ID="rhel" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="fedora" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="8.6" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el8" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-13T21:02:34.790 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://www.redhat.com/" 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-13T21:02:34.791 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-13T21:02:34.791 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2024-04-13T21:02:34.817 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-13T21:02:34.826 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-13T21:02:34.861 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-13T21:02:34.861 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-04-13T21:02:34.864 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2024-04-13T21:02:34.882 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-13T21:02:34.889 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-13T21:02:34.889 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-04-13T21:02:34.923 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2024-04-13T21:02:34.953 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-13T21:02:35.002 INFO:teuthology.run_tasks:Running task kernel... 2024-04-13T21:02:35.017 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-13T21:02:35.018 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-13T21:02:35.018 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T21:02:35.018 DEBUG:teuthology.orchestra.run.smithi175:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T21:02:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T21:02:35.036 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-04-13T21:02:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T21:02:35.037 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-04-13T21:02:35.093 INFO:teuthology.orchestra.run.smithi175.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:35.093 INFO:teuthology.task.kernel:Running kernel on smithi175: 4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:35.093 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2024-04-13T21:02:35.098 INFO:teuthology.orchestra.run.smithi132.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:35.098 INFO:teuthology.task.kernel:Running kernel on smithi132: 4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:35.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2024-04-13T21:02:36.572 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:02:36.572 INFO:teuthology.orchestra.run.smithi132.stdout:Unable to read consumer identity 2024-04-13T21:02:36.572 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:02:36.573 INFO:teuthology.orchestra.run.smithi175.stdout:Unable to read consumer identity 2024-04-13T21:02:36.842 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:36.842 INFO:teuthology.orchestra.run.smithi175.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-13T21:02:36.842 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:36.853 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:36.854 INFO:teuthology.orchestra.run.smithi132.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-13T21:02:36.854 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:38.615 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-13T21:02:38.623 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-04-13T21:02:39.769 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-04-13T21:02:40.076 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-13T21:02:43.332 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 428 kB/s | 24 kB 00:00 2024-04-13T21:02:43.675 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 487 kB/s | 24 kB 00:00 2024-04-13T21:02:44.059 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-13T21:02:44.060 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-13T21:02:44.146 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:02:44.147 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:02:44.147 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:02:44.295 DEBUG:teuthology.orchestra.run.smithi175:> echo no | sudo yum reinstall kernel || true 2024-04-13T21:02:44.385 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-13T21:02:44.385 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-13T21:02:44.465 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:02:44.466 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:02:44.467 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:02:44.632 DEBUG:teuthology.orchestra.run.smithi132:> echo no | sudo yum reinstall kernel || true 2024-04-13T21:02:44.668 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:02:44.669 INFO:teuthology.orchestra.run.smithi175.stdout:Unable to read consumer identity 2024-04-13T21:02:44.924 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:44.924 INFO:teuthology.orchestra.run.smithi175.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-13T21:02:44.924 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:45.006 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:02:45.007 INFO:teuthology.orchestra.run.smithi132.stdout:Unable to read consumer identity 2024-04-13T21:02:45.232 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:45.233 INFO:teuthology.orchestra.run.smithi132.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-13T21:02:45.233 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:45.238 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Apr 2024 09:02:43 PM UTC. 2024-04-13T21:02:45.322 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-13T21:02:45.331 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-13T21:02:45.377 INFO:teuthology.orchestra.run.smithi175.stderr:Error: No packages marked for reinstall. 2024-04-13T21:02:45.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum reinstall -y kernel || true 2024-04-13T21:02:45.548 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Apr 2024 09:02:43 PM UTC. 2024-04-13T21:02:45.625 INFO:teuthology.orchestra.run.smithi132.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-13T21:02:45.627 INFO:teuthology.orchestra.run.smithi132.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-13T21:02:45.671 INFO:teuthology.orchestra.run.smithi132.stderr:Error: No packages marked for reinstall. 2024-04-13T21:02:45.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum reinstall -y kernel || true 2024-04-13T21:02:45.762 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:02:45.762 INFO:teuthology.orchestra.run.smithi175.stdout:Unable to read consumer identity 2024-04-13T21:02:45.895 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:45.895 INFO:teuthology.orchestra.run.smithi175.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-13T21:02:45.895 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:02:46.053 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:02:46.053 INFO:teuthology.orchestra.run.smithi132.stdout:Unable to read consumer identity 2024-04-13T21:02:46.202 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Apr 2024 09:02:43 PM UTC. 2024-04-13T21:02:46.265 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-13T21:02:46.265 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-13T21:02:46.273 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:46.273 INFO:teuthology.orchestra.run.smithi132.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-13T21:02:46.274 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:02:46.299 INFO:teuthology.orchestra.run.smithi175.stderr:Error: No packages marked for reinstall. 2024-04-13T21:02:46.347 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T21:02:46.508 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:46.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:46.508 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-13T21:02:46.508 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-04-13T21:02:46.508 DEBUG:teuthology.orchestra.run.smithi175:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-13T21:02:46.547 INFO:teuthology.orchestra.run.smithi175.stdout:ttyS1 2024-04-13T21:02:46.559 DEBUG:teuthology.parallel:result is None 2024-04-13T21:02:46.578 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Apr 2024 09:02:43 PM UTC. 2024-04-13T21:02:46.643 INFO:teuthology.orchestra.run.smithi132.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-13T21:02:46.643 INFO:teuthology.orchestra.run.smithi132.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-13T21:02:46.673 INFO:teuthology.orchestra.run.smithi132.stderr:Error: No packages marked for reinstall. 2024-04-13T21:02:46.726 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T21:02:46.884 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:46.884 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-13T21:02:46.884 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-13T21:02:46.884 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-04-13T21:02:46.884 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-13T21:02:46.914 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-04-13T21:02:46.930 DEBUG:teuthology.parallel:result is None 2024-04-13T21:02:46.931 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-13T21:02:46.938 INFO:teuthology.task.internal:Creating test directory... 2024-04-13T21:02:46.939 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T21:02:46.973 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T21:02:46.991 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-13T21:02:46.998 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-13T21:02:47.004 INFO:teuthology.task.internal:Creating archive directory... 2024-04-13T21:02:47.004 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T21:02:47.033 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T21:02:47.068 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-13T21:02:47.075 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-13T21:02:47.076 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T21:02:47.097 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T21:02:47.122 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T21:02:47.132 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T21:02:47.141 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T21:02:47.152 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T21:02:47.153 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-13T21:02:47.160 INFO:teuthology.task.internal:Configuring sudo... 2024-04-13T21:02:47.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T21:02:47.176 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T21:02:47.220 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-13T21:02:47.230 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-13T21:02:47.230 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T21:02:47.244 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T21:02:47.277 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T21:02:47.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T21:02:47.387 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T21:02:47.471 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T21:02:47.512 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:02:47.512 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T21:02:47.578 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T21:02:47.622 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T21:02:47.680 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T21:02:47.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T21:02:47.807 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:02:47.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T21:02:47.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-04-13T21:02:47.875 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2024-04-13T21:02:47.933 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T21:02:47.962 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T21:02:48.226 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-13T21:02:48.234 INFO:teuthology.task.internal:Starting timer... 2024-04-13T21:02:48.234 INFO:teuthology.run_tasks:Running task pcp... 2024-04-13T21:02:48.245 INFO:teuthology.run_tasks:Running task selinux... 2024-04-13T21:02:48.253 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-13T21:02:48.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-04-13T21:02:48.308 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-04-13T21:02:48.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2024-04-13T21:02:48.379 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: 2024-04-13T21:02:48.381 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-13T21:02:48.381 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-04-13T21:02:48.409 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-04-13T21:02:48.409 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2024-04-13T21:02:48.451 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2024-04-13T21:02:48.451 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi175.front.sepia.ceph.com': 'permissive'} 2024-04-13T21:02:48.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T21:02:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T21:02:48.479 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T21:02:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T21:02:48.518 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-13T21:02:48.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-04-13T21:02:48.556 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2024-04-13T21:02:48.586 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-13T21:02:48.596 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-13T21:02:48.602 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-13T21:02:48.728 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-13T21:02:48.744 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-13T21:02:48.745 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi132.front.sepia.ceph.com,smithi175.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-13T21:12:35.794 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi175.front.sepia.ceph.com')] 2024-04-13T21:12:35.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-13T21:12:35.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T21:12:35.905 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-04-13T21:12:35.971 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-13T21:12:35.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-13T21:12:35.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T21:12:36.044 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-04-13T21:12:36.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-13T21:12:36.114 INFO:teuthology.run_tasks:Running task clock... 2024-04-13T21:12:36.126 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-13T21:12:36.126 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T21:12:36.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T21:12:36.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T21:12:36.129 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T21:12:36.160 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T21:12:36.174 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T21:12:36.186 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T21:12:36.201 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T21:12:36.247 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-04-13T21:12:36.259 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-04-13T21:12:36.269 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2024-04-13T21:12:36.273 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T21:12:36.280 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2024-04-13T21:12:36.287 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T21:12:36.295 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T21:12:36.309 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T21:12:36.379 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2024-04-13T21:12:36.383 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T21:12:36.383 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-04-13T21:12:36.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:bash: ntpq: command not found 2024-04-13T21:12:36.389 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T21:12:36.389 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-04-13T21:12:36.389 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.389 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.390 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.390 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T21:12:36.390 INFO:teuthology.run_tasks:Running task install... 2024-04-13T21:12:36.398 DEBUG:teuthology.task.install:project ceph 2024-04-13T21:12:36.398 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b'}} 2024-04-13T21:12:36.398 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b'} 2024-04-13T21:12:36.398 INFO:teuthology.task.install:Using flavor: default 2024-04-13T21:12:36.404 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-13T21:12:36.404 INFO:teuthology.task.install:extra packages: [] 2024-04-13T21:12:36.404 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'tag': None, 'wait_for_package': False} 2024-04-13T21:12:36.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7d36322c373c73b184fd252cd881b122d8c51a3b 2024-04-13T21:12:36.407 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'tag': None, 'wait_for_package': False} 2024-04-13T21:12:36.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7d36322c373c73b184fd252cd881b122d8c51a3b 2024-04-13T21:12:36.633 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/ 2024-04-13T21:12:36.633 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1217.g7d36322c 2024-04-13T21:12:36.640 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/ 2024-04-13T21:12:36.640 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1217.g7d36322c 2024-04-13T21:12:36.806 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T21:12:36.806 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:12:36.806 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T21:12:36.812 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/7d36322c373c73b184fd252cd881b122d8c51a3b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T21:12:36.812 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:12:36.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T21:12:36.835 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-13T21:12:36.836 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d36322c373c73b184fd252cd881b122d8c51a3b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T21:12:36.841 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-13T21:12:36.841 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d36322c373c73b184fd252cd881b122d8c51a3b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T21:12:36.905 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T21:12:36.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T21:12:36.983 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T21:12:36.987 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T21:12:37.049 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2024-04-13T21:12:37.051 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2024-04-13T21:12:37.053 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-04-13T21:12:37.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-04-13T21:12:37.426 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:12:37.429 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:12:38.813 INFO:teuthology.orchestra.run.smithi175.stdout:63 files removed 2024-04-13T21:12:38.814 INFO:teuthology.orchestra.run.smithi132.stdout:63 files removed 2024-04-13T21:12:38.864 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw 2024-04-13T21:12:38.871 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2024-04-13T21:12:39.201 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:12:39.208 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:12:40.911 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 313 kB/s | 80 kB 00:00 2024-04-13T21:12:40.959 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 222 kB/s | 80 kB 00:00 2024-04-13T21:12:41.195 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 46 kB/s | 11 kB 00:00 2024-04-13T21:12:41.343 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-04-13T21:12:41.520 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2024-04-13T21:12:41.652 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 5.3 kB/s | 1.5 kB 00:00 2024-04-13T21:12:41.679 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-13T21:12:41.927 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-04-13T21:12:43.314 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-04-13T21:12:43.880 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 16 MB 00:02 2024-04-13T21:12:46.853 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 432 kB/s | 24 kB 00:00 2024-04-13T21:12:47.439 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 394 kB/s | 24 kB 00:00 2024-04-13T21:12:47.605 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-04-13T21:12:48.741 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 60 MB 00:01 2024-04-13T21:12:50.381 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 60 MB 00:01 2024-04-13T21:12:59.507 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-04-13T21:13:01.840 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 67 MB 00:01 2024-04-13T21:13:03.125 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 MB/s | 67 MB 00:02 2024-04-13T21:13:16.953 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:13:16.955 INFO:teuthology.orchestra.run.smithi175.stdout:==================================================================================================== 2024-04-13T21:13:16.955 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:13:16.955 INFO:teuthology.orchestra.run.smithi175.stdout:==================================================================================================== 2024-04-13T21:13:16.955 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 13 M 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 3.7 M 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 3.6 M 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 5.5 M 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 22 M 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 24 k 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 780 k 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-13T21:13:16.956 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 475 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 6.6 M 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 45 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 129 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 197 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 371 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 364 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 107 k 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-13T21:13:16.957 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout:==================================================================================================== 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout:Install 21 Packages 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 2 Packages 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 59 M 2024-04-13T21:13:16.958 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:13:17.980 INFO:teuthology.orchestra.run.smithi175.stdout:(1/23): ceph-base-18.2.2-1217.g7d36322c.el8.x86 8.1 MB/s | 5.5 MB 00:00 2024-04-13T21:13:18.046 INFO:teuthology.orchestra.run.smithi175.stdout:(2/23): ceph-selinux-18.2.2-1217.g7d36322c.el8. 366 kB/s | 24 kB 00:00 2024-04-13T21:13:18.147 INFO:teuthology.orchestra.run.smithi175.stdout:(3/23): libcephfs2-18.2.2-1217.g7d36322c.el8.x8 7.6 MB/s | 780 kB 00:00 2024-04-13T21:13:18.489 INFO:teuthology.orchestra.run.smithi175.stdout:(4/23): libradosstriper1-18.2.2-1217.g7d36322c. 1.4 MB/s | 475 kB 00:00 2024-04-13T21:13:18.509 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:13:18.511 INFO:teuthology.orchestra.run.smithi132.stdout:==================================================================================================== 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout:==================================================================================================== 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 13 M 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 3.7 M 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 3.6 M 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 5.5 M 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 22 M 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 24 k 2024-04-13T21:13:18.512 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 780 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 475 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 6.6 M 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-13T21:13:18.513 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 45 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 129 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 197 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 371 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 364 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 107 k 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout:==================================================================================================== 2024-04-13T21:13:18.514 INFO:teuthology.orchestra.run.smithi132.stdout:Install 21 Packages 2024-04-13T21:13:18.515 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-04-13T21:13:18.515 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:13:18.515 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 59 M 2024-04-13T21:13:18.515 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:13:18.631 INFO:teuthology.orchestra.run.smithi175.stdout:(5/23): ceph-radosgw-18.2.2-1217.g7d36322c.el8. 9.8 MB/s | 13 MB 00:01 2024-04-13T21:13:18.689 INFO:teuthology.orchestra.run.smithi175.stdout:(6/23): python3-ceph-argparse-18.2.2-1217.g7d36 779 kB/s | 45 kB 00:00 2024-04-13T21:13:18.748 INFO:teuthology.orchestra.run.smithi175.stdout:(7/23): python3-ceph-common-18.2.2-1217.g7d3632 2.2 MB/s | 129 kB 00:00 2024-04-13T21:13:18.815 INFO:teuthology.orchestra.run.smithi175.stdout:(8/23): python3-cephfs-18.2.2-1217.g7d36322c.el 2.9 MB/s | 197 kB 00:00 2024-04-13T21:13:18.915 INFO:teuthology.orchestra.run.smithi175.stdout:(9/23): librgw2-18.2.2-1217.g7d36322c.el8.x86_6 16 MB/s | 6.6 MB 00:00 2024-04-13T21:13:18.940 INFO:teuthology.orchestra.run.smithi175.stdout:(10/23): python3-rados-18.2.2-1217.g7d36322c.el 2.9 MB/s | 371 kB 00:00 2024-04-13T21:13:18.966 INFO:teuthology.orchestra.run.smithi175.stdout:(11/23): python3-rbd-18.2.2-1217.g7d36322c.el8. 7.1 MB/s | 364 kB 00:00 2024-04-13T21:13:19.008 INFO:teuthology.orchestra.run.smithi175.stdout:(12/23): python3-rgw-18.2.2-1217.g7d36322c.el8. 1.6 MB/s | 107 kB 00:00 2024-04-13T21:13:19.158 INFO:teuthology.orchestra.run.smithi175.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 404 kB/s | 59 kB 00:00 2024-04-13T21:13:19.184 INFO:teuthology.orchestra.run.smithi175.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-04-13T21:13:19.217 INFO:teuthology.orchestra.run.smithi175.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2024-04-13T21:13:19.351 INFO:teuthology.orchestra.run.smithi175.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 10 MB/s | 1.7 MB 00:00 2024-04-13T21:13:19.384 INFO:teuthology.orchestra.run.smithi175.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 263 kB/s | 44 kB 00:00 2024-04-13T21:13:19.526 INFO:teuthology.orchestra.run.smithi175.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-04-13T21:13:19.576 INFO:teuthology.orchestra.run.smithi175.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.9 MB/s | 354 kB 00:00 2024-04-13T21:13:19.702 INFO:teuthology.orchestra.run.smithi175.stdout:(20/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 269 kB/s | 47 kB 00:00 2024-04-13T21:13:19.791 INFO:teuthology.orchestra.run.smithi132.stdout:(1/23): ceph-radosgw-18.2.2-1217.g7d36322c.el8. 12 MB/s | 13 MB 00:01 2024-04-13T21:13:19.883 INFO:teuthology.orchestra.run.smithi132.stdout:(2/23): ceph-selinux-18.2.2-1217.g7d36322c.el8. 265 kB/s | 24 kB 00:00 2024-04-13T21:13:19.894 INFO:teuthology.orchestra.run.smithi175.stdout:(21/23): librados2-18.2.2-1217.g7d36322c.el8.x8 12 MB/s | 3.7 MB 00:00 2024-04-13T21:13:19.967 INFO:teuthology.orchestra.run.smithi132.stdout:(3/23): libcephfs2-18.2.2-1217.g7d36322c.el8.x8 9.2 MB/s | 780 kB 00:00 2024-04-13T21:13:20.169 INFO:teuthology.orchestra.run.smithi175.stdout:(22/23): librbd1-18.2.2-1217.g7d36322c.el8.x86_ 7.6 MB/s | 3.6 MB 00:00 2024-04-13T21:13:20.234 INFO:teuthology.orchestra.run.smithi132.stdout:(4/23): libradosstriper1-18.2.2-1217.g7d36322c. 1.7 MB/s | 475 kB 00:00 2024-04-13T21:13:20.334 INFO:teuthology.orchestra.run.smithi132.stdout:(5/23): ceph-base-18.2.2-1217.g7d36322c.el8.x86 3.4 MB/s | 5.5 MB 00:01 2024-04-13T21:13:21.018 INFO:teuthology.orchestra.run.smithi132.stdout:(6/23): python3-ceph-argparse-18.2.2-1217.g7d36 65 kB/s | 45 kB 00:00 2024-04-13T21:13:21.135 INFO:teuthology.orchestra.run.smithi132.stdout:(7/23): librgw2-18.2.2-1217.g7d36322c.el8.x86_6 7.3 MB/s | 6.6 MB 00:00 2024-04-13T21:13:21.160 INFO:teuthology.orchestra.run.smithi132.stdout:(8/23): python3-ceph-common-18.2.2-1217.g7d3632 910 kB/s | 129 kB 00:00 2024-04-13T21:13:21.185 INFO:teuthology.orchestra.run.smithi132.stdout:(9/23): python3-cephfs-18.2.2-1217.g7d36322c.el 3.9 MB/s | 197 kB 00:00 2024-04-13T21:13:21.244 INFO:teuthology.orchestra.run.smithi132.stdout:(10/23): python3-rbd-18.2.2-1217.g7d36322c.el8. 6.1 MB/s | 364 kB 00:00 2024-04-13T21:13:21.269 INFO:teuthology.orchestra.run.smithi132.stdout:(11/23): python3-rados-18.2.2-1217.g7d36322c.el 3.3 MB/s | 371 kB 00:00 2024-04-13T21:13:21.294 INFO:teuthology.orchestra.run.smithi132.stdout:(12/23): python3-rgw-18.2.2-1217.g7d36322c.el8. 2.1 MB/s | 107 kB 00:00 2024-04-13T21:13:21.495 INFO:teuthology.orchestra.run.smithi132.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-04-13T21:13:21.520 INFO:teuthology.orchestra.run.smithi132.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 266 kB/s | 59 kB 00:00 2024-04-13T21:13:21.562 INFO:teuthology.orchestra.run.smithi132.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-04-13T21:13:21.755 INFO:teuthology.orchestra.run.smithi132.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 230 kB/s | 44 kB 00:00 2024-04-13T21:13:21.796 INFO:teuthology.orchestra.run.smithi132.stdout:(17/23): thrift-0.13.0-2.el8.x86_64.rpm 6.3 MB/s | 1.7 MB 00:00 2024-04-13T21:13:21.921 INFO:teuthology.orchestra.run.smithi132.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-04-13T21:13:21.997 INFO:teuthology.orchestra.run.smithi132.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.7 MB/s | 354 kB 00:00 2024-04-13T21:13:22.063 INFO:teuthology.orchestra.run.smithi132.stdout:(20/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 332 kB/s | 47 kB 00:00 2024-04-13T21:13:22.505 INFO:teuthology.orchestra.run.smithi132.stdout:(21/23): librbd1-18.2.2-1217.g7d36322c.el8.x86_ 8.1 MB/s | 3.6 MB 00:00 2024-04-13T21:13:22.839 INFO:teuthology.orchestra.run.smithi132.stdout:(22/23): librados2-18.2.2-1217.g7d36322c.el8.x8 4.4 MB/s | 3.7 MB 00:00 2024-04-13T21:13:23.639 INFO:teuthology.orchestra.run.smithi132.stdout:(23/23): ceph-common-18.2.2-1217.g7d36322c.el8. 4.4 MB/s | 22 MB 00:04 2024-04-13T21:13:23.643 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:13:23.644 INFO:teuthology.orchestra.run.smithi132.stdout:Total 12 MB/s | 59 MB 00:05 2024-04-13T21:13:23.694 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:13:23.739 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:13:23.740 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:13:24.073 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:13:24.082 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:13:25.110 INFO:teuthology.orchestra.run.smithi175.stdout:(23/23): ceph-common-18.2.2-1217.g7d36322c.el8. 2.8 MB/s | 22 MB 00:07 2024-04-13T21:13:25.114 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:13:25.114 INFO:teuthology.orchestra.run.smithi175.stdout:Total 7.3 MB/s | 59 MB 00:08 2024-04-13T21:13:25.162 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:13:25.207 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:13:25.207 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:13:25.320 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:13:25.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-13T21:13:25.537 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:13:25.546 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:13:25.662 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-13T21:13:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-13T21:13:26.304 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 3/25 2024-04-13T21:13:26.388 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 3/25 2024-04-13T21:13:26.718 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 4/25 2024-04-13T21:13:26.820 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:13:26.952 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-13T21:13:27.011 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-13T21:13:27.162 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 6/25 2024-04-13T21:13:27.186 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-13T21:13:27.551 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 6/25 2024-04-13T21:13:27.586 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-13T21:13:27.770 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 7/25 2024-04-13T21:13:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 3/25 2024-04-13T21:13:27.836 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 7/25 2024-04-13T21:13:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 3/25 2024-04-13T21:13:28.019 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 8/25 2024-04-13T21:13:28.142 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 4/25 2024-04-13T21:13:28.204 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2024-04-13T21:13:28.273 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2024-04-13T21:13:28.406 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:18.2.2-1217.g7d36322c.el8. 10/25 2024-04-13T21:13:28.409 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-13T21:13:28.614 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 11/25 2024-04-13T21:13:28.628 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 6/25 2024-04-13T21:13:28.837 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 12/25 2024-04-13T21:13:29.022 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 6/25 2024-04-13T21:13:29.028 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_6 13/25 2024-04-13T21:13:29.085 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_6 13/25 2024-04-13T21:13:29.252 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-13T21:13:29.478 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-13T21:13:29.503 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 7/25 2024-04-13T21:13:29.574 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 7/25 2024-04-13T21:13:29.777 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 8/25 2024-04-13T21:13:29.961 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2024-04-13T21:13:30.028 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2024-04-13T21:13:30.198 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:18.2.2-1217.g7d36322c.el8. 10/25 2024-04-13T21:13:30.285 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-13T21:13:30.389 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 11/25 2024-04-13T21:13:30.487 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 17/25 2024-04-13T21:13:30.559 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 17/25 2024-04-13T21:13:30.637 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 12/25 2024-04-13T21:13:30.771 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 18/25 2024-04-13T21:13:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_6 13/25 2024-04-13T21:13:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_6 13/25 2024-04-13T21:13:30.945 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:18.2.2-1217.g7d36322c.el8.x8 19/25 2024-04-13T21:13:31.048 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-13T21:13:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-13T21:13:32.126 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-13T21:13:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 17/25 2024-04-13T21:13:32.404 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 17/25 2024-04-13T21:13:32.670 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 18/25 2024-04-13T21:13:32.836 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:18.2.2-1217.g7d36322c.el8.x8 19/25 2024-04-13T21:13:35.080 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 20/25 2024-04-13T21:13:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 20/25 2024-04-13T21:13:35.941 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 20/25 2024-04-13T21:13:36.204 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 21/25 2024-04-13T21:13:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 21/25 2024-04-13T21:13:36.470 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 22/25 2024-04-13T21:13:36.573 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 20/25 2024-04-13T21:13:36.802 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 20/25 2024-04-13T21:13:37.393 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 20/25 2024-04-13T21:13:37.681 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 21/25 2024-04-13T21:13:37.763 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 21/25 2024-04-13T21:13:37.943 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 22/25 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 22/25 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /run 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-04-13T21:13:56.030 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-04-13T21:13:56.031 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-04-13T21:13:56.031 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-04-13T21:13:56.031 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2024-04-13T21:13:56.031 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:13:57.318 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 22/25 2024-04-13T21:13:57.318 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2024-04-13T21:13:57.318 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2024-04-13T21:13:57.318 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /dev 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /run 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /dev 2024-04-13T21:13:57.319 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:13:57.384 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 23/25 2024-04-13T21:13:57.406 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 23/25 2024-04-13T21:13:57.406 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-13T21:13:57.610 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-13T21:13:57.611 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 1/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 2/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 3/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 4/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 5/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_6 6/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 7/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1217.g7d36322c.el8. 8/25 2024-04-13T21:13:58.301 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:18.2.2-1217.g7d36322c.el8.x8 9/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 10/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 11/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 12/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 13/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2024-04-13T21:13:58.302 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2024-04-13T21:13:58.303 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 21/25 2024-04-13T21:13:58.303 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 22/25 2024-04-13T21:13:58.303 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-13T21:13:58.303 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 24/25 2024-04-13T21:13:58.652 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 23/25 2024-04-13T21:13:58.673 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 23/25 2024-04-13T21:13:58.673 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-13T21:13:58.925 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-13T21:13:58.925 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-13T21:13:59.585 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 1/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 2/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 3/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 4/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 5/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_6 6/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 7/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1217.g7d36322c.el8. 8/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:18.2.2-1217.g7d36322c.el8.x8 9/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 10/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 11/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 12/25 2024-04-13T21:13:59.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 13/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 21/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 22/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-13T21:13:59.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 24/25 2024-04-13T21:14:06.106 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-13T21:14:06.106 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.652 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-13T21:14:06.653 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.654 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.654 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:06.654 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-13T21:14:06.654 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:06.654 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:14:06.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2024-04-13T21:14:07.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-13T21:14:07.204 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:14:07.208 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:14:07.754 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.755 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:07.756 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:14:07.958 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-test 2024-04-13T21:14:08.314 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:14:08.863 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-13T21:14:09.551 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-13T21:14:09.986 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-13T21:14:10.633 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-13T21:14:11.962 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 47 M 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 142 k 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-13T21:14:11.963 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-13T21:14:11.964 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:11.964 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:14:11.964 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-04-13T21:14:11.964 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-04-13T21:14:11.964 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:11.964 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 48 M 2024-04-13T21:14:11.965 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 213 M 2024-04-13T21:14:11.965 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:14:12.002 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.8 MB/s | 70 kB 00:00 2024-04-13T21:14:12.161 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-04-13T21:14:12.320 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-04-13T21:14:12.420 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): libcephsqlite-18.2.2-1217.g7d36322c.el8. 313 kB/s | 142 kB 00:00 2024-04-13T21:14:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-04-13T21:14:12.968 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:14:12.969 INFO:teuthology.orchestra.run.smithi175.stdout:============================================================================================ 2024-04-13T21:14:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:14:12.969 INFO:teuthology.orchestra.run.smithi175.stdout:============================================================================================ 2024-04-13T21:14:12.969 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:14:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 47 M 2024-04-13T21:14:12.969 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 142 k 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:12.970 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:14:12.971 INFO:teuthology.orchestra.run.smithi175.stdout:============================================================================================ 2024-04-13T21:14:12.971 INFO:teuthology.orchestra.run.smithi175.stdout:Install 6 Packages 2024-04-13T21:14:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:12.972 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 48 M 2024-04-13T21:14:12.972 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 213 M 2024-04-13T21:14:12.972 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:14:13.018 INFO:teuthology.orchestra.run.smithi175.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.5 MB/s | 70 kB 00:00 2024-04-13T21:14:13.237 INFO:teuthology.orchestra.run.smithi175.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2024-04-13T21:14:13.343 INFO:teuthology.orchestra.run.smithi175.stdout:(3/6): libcephsqlite-18.2.2-1217.g7d36322c.el8. 382 kB/s | 142 kB 00:00 2024-04-13T21:14:13.394 INFO:teuthology.orchestra.run.smithi175.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-04-13T21:14:13.511 INFO:teuthology.orchestra.run.smithi175.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-04-13T21:14:16.252 INFO:teuthology.orchestra.run.smithi175.stdout:(6/6): ceph-test-18.2.2-1217.g7d36322c.el8.x86_ 14 MB/s | 47 MB 00:03 2024-04-13T21:14:16.254 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:14:16.254 INFO:teuthology.orchestra.run.smithi175.stdout:Total 15 MB/s | 48 MB 00:03 2024-04-13T21:14:16.294 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:14:16.315 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:14:16.315 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:14:16.584 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:14:16.593 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:14:17.089 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:14:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-13T21:14:17.329 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-13T21:14:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-13T21:14:17.653 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-18.2.2-1217.g7d36322c.el8.x86_ 8.3 MB/s | 47 MB 00:05 2024-04-13T21:14:17.655 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:14:17.655 INFO:teuthology.orchestra.run.smithi132.stdout:Total 8.4 MB/s | 48 MB 00:05 2024-04-13T21:14:17.695 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:14:17.716 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:14:17.717 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:14:17.736 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-13T21:14:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-13T21:14:17.986 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:14:17.995 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:14:18.077 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 5/6 2024-04-13T21:14:18.492 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:14:18.629 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-13T21:14:18.714 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-13T21:14:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-13T21:14:19.121 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-13T21:14:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-13T21:14:19.487 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 5/6 2024-04-13T21:14:22.560 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 5/6 2024-04-13T21:14:22.815 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 6/6 2024-04-13T21:14:23.076 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 6/6 2024-04-13T21:14:23.076 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 1/6 2024-04-13T21:14:23.076 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 2/6 2024-04-13T21:14:23.076 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-13T21:14:23.076 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-13T21:14:23.076 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-13T21:14:23.977 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 5/6 2024-04-13T21:14:24.193 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 6/6 2024-04-13T21:14:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 6/6 2024-04-13T21:14:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 1/6 2024-04-13T21:14:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 2/6 2024-04-13T21:14:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-13T21:14:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-13T21:14:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-13T21:14:32.367 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-13T21:14:32.367 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:14:32.918 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:32.918 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:14:32.918 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout: jq-1.6-7.el8.x86_64 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:32.919 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:14:33.090 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph 2024-04-13T21:14:33.427 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:14:34.327 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-13T21:14:34.327 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:14:34.858 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-7.el8.x86_64 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:34.859 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:14:35.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2024-04-13T21:14:35.356 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:14:35.374 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 20 kB/s | 4.5 kB 00:00 2024-04-13T21:14:36.037 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-04-13T21:14:37.221 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-13T21:14:37.921 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-13T21:14:38.390 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================= 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================= 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 6.4 k 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 2.3 M 2024-04-13T21:14:38.393 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 1.6 M 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 245 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 4.8 M 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 18 M 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-13T21:14:38.394 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-13T21:14:38.395 INFO:teuthology.orchestra.run.smithi175.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================= 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout:Install 32 Packages 2024-04-13T21:14:38.396 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:14:38.397 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 31 M 2024-04-13T21:14:38.397 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 124 M 2024-04-13T21:14:38.397 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:14:38.867 INFO:teuthology.orchestra.run.smithi175.stdout:(1/32): ceph-18.2.2-1217.g7d36322c.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2024-04-13T21:14:39.018 INFO:teuthology.orchestra.run.smithi175.stdout:(2/32): ceph-mgr-18.2.2-1217.g7d36322c.el8.x86_ 5.6 MB/s | 1.6 MB 00:00 2024-04-13T21:14:39.876 INFO:teuthology.orchestra.run.smithi175.stdout:(3/32): ceph-mon-18.2.2-1217.g7d36322c.el8.x86_ 4.8 MB/s | 4.8 MB 00:01 2024-04-13T21:14:40.051 INFO:teuthology.orchestra.run.smithi175.stdout:(4/32): ceph-osd-18.2.2-1217.g7d36322c.el8.x86_ 18 MB/s | 18 MB 00:01 2024-04-13T21:14:40.077 INFO:teuthology.orchestra.run.smithi175.stdout:(5/32): ceph-mgr-modules-core-18.2.2-1217.g7d36 1.2 MB/s | 245 kB 00:00 2024-04-13T21:14:40.147 INFO:teuthology.orchestra.run.smithi175.stdout:(6/32): ceph-mds-18.2.2-1217.g7d36322c.el8.x86_ 1.6 MB/s | 2.3 MB 00:01 2024-04-13T21:14:40.201 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================= 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================= 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 6.4 k 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 2.3 M 2024-04-13T21:14:40.204 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 1.6 M 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 245 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 4.8 M 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 18 M 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-13T21:14:40.205 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-13T21:14:40.206 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================= 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout:Install 32 Packages 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:14:40.207 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 M 2024-04-13T21:14:40.208 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 124 M 2024-04-13T21:14:40.208 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:14:40.513 INFO:teuthology.orchestra.run.smithi132.stdout:(1/32): ceph-18.2.2-1217.g7d36322c.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2024-04-13T21:14:40.586 INFO:teuthology.orchestra.run.smithi175.stdout:(7/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 82 kB/s | 44 kB 00:00 2024-04-13T21:14:40.604 INFO:teuthology.orchestra.run.smithi132.stdout:(2/32): ceph-mgr-18.2.2-1217.g7d36322c.el8.x86_ 5.7 MB/s | 1.6 MB 00:00 2024-04-13T21:14:40.736 INFO:teuthology.orchestra.run.smithi175.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 281 kB/s | 185 kB 00:00 2024-04-13T21:14:40.836 INFO:teuthology.orchestra.run.smithi175.stdout:(9/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 252 kB/s | 173 kB 00:00 2024-04-13T21:14:40.862 INFO:teuthology.orchestra.run.smithi175.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 88 kB/s | 11 kB 00:00 2024-04-13T21:14:40.954 INFO:teuthology.orchestra.run.smithi175.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2024-04-13T21:14:41.046 INFO:teuthology.orchestra.run.smithi175.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 265 kB/s | 49 kB 00:00 2024-04-13T21:14:41.104 INFO:teuthology.orchestra.run.smithi175.stdout:(13/32): python3-cherrypy-18.4.0-1.el8.noarch.r 742 kB/s | 384 kB 00:00 2024-04-13T21:14:41.163 INFO:teuthology.orchestra.run.smithi132.stdout:(3/32): ceph-mds-18.2.2-1217.g7d36322c.el8.x86_ 2.7 MB/s | 2.3 MB 00:00 2024-04-13T21:14:41.163 INFO:teuthology.orchestra.run.smithi175.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 281 kB/s | 59 kB 00:00 2024-04-13T21:14:41.213 INFO:teuthology.orchestra.run.smithi175.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-04-13T21:14:41.247 INFO:teuthology.orchestra.run.smithi132.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1217.g7d36 2.9 MB/s | 245 kB 00:00 2024-04-13T21:14:41.263 INFO:teuthology.orchestra.run.smithi175.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-04-13T21:14:41.305 INFO:teuthology.orchestra.run.smithi132.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 752 kB/s | 44 kB 00:00 2024-04-13T21:14:41.322 INFO:teuthology.orchestra.run.smithi175.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-04-13T21:14:41.356 INFO:teuthology.orchestra.run.smithi132.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.6 MB/s | 185 kB 00:00 2024-04-13T21:14:41.363 INFO:teuthology.orchestra.run.smithi175.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 292 kB/s | 29 kB 00:00 2024-04-13T21:14:41.398 INFO:teuthology.orchestra.run.smithi132.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 4.0 MB/s | 173 kB 00:00 2024-04-13T21:14:41.430 INFO:teuthology.orchestra.run.smithi175.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 254 kB/s | 27 kB 00:00 2024-04-13T21:14:41.514 INFO:teuthology.orchestra.run.smithi175.stdout:(20/32): python3-pecan-1.3.2-9.el8.noarch.rpm 604 kB/s | 283 kB 00:00 2024-04-13T21:14:41.606 INFO:teuthology.orchestra.run.smithi132.stdout:(8/32): ceph-osd-18.2.2-1217.g7d36322c.el8.x86_ 18 MB/s | 18 MB 00:01 2024-04-13T21:14:41.632 INFO:teuthology.orchestra.run.smithi132.stdout:(9/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 1.6 MB/s | 384 kB 00:00 2024-04-13T21:14:41.674 INFO:teuthology.orchestra.run.smithi132.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-04-13T21:14:41.699 INFO:teuthology.orchestra.run.smithi132.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2024-04-13T21:14:41.706 INFO:teuthology.orchestra.run.smithi175.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 445 kB/s | 85 kB 00:00 2024-04-13T21:14:41.724 INFO:teuthology.orchestra.run.smithi132.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 973 kB/s | 49 kB 00:00 2024-04-13T21:14:41.731 INFO:teuthology.orchestra.run.smithi175.stdout:(22/32): python3-waitress-1.4.3-1.el8.noarch.rp 695 kB/s | 255 kB 00:00 2024-04-13T21:14:41.749 INFO:teuthology.orchestra.run.smithi132.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-04-13T21:14:41.775 INFO:teuthology.orchestra.run.smithi132.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-04-13T21:14:41.800 INFO:teuthology.orchestra.run.smithi132.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2024-04-13T21:14:41.812 INFO:teuthology.orchestra.run.smithi175.stdout:(23/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 660 kB/s | 251 kB 00:00 2024-04-13T21:14:41.825 INFO:teuthology.orchestra.run.smithi132.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2024-04-13T21:14:41.840 INFO:teuthology.orchestra.run.smithi175.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 172 kB/s | 23 kB 00:00 2024-04-13T21:14:41.850 INFO:teuthology.orchestra.run.smithi132.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2024-04-13T21:14:41.876 INFO:teuthology.orchestra.run.smithi132.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 580 kB/s | 29 kB 00:00 2024-04-13T21:14:41.901 INFO:teuthology.orchestra.run.smithi132.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2024-04-13T21:14:41.907 INFO:teuthology.orchestra.run.smithi175.stdout:(25/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.5 MB/s | 457 kB 00:00 2024-04-13T21:14:41.927 INFO:teuthology.orchestra.run.smithi132.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 5.0 MB/s | 255 kB 00:00 2024-04-13T21:14:41.994 INFO:teuthology.orchestra.run.smithi132.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-04-13T21:14:42.027 INFO:teuthology.orchestra.run.smithi132.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2024-04-13T21:14:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 453 kB/s | 23 kB 00:00 2024-04-13T21:14:42.118 INFO:teuthology.orchestra.run.smithi175.stdout:(26/32): python3-markupsafe-0.23-19.el8.x86_64. 128 kB/s | 39 kB 00:00 2024-04-13T21:14:42.166 INFO:teuthology.orchestra.run.smithi175.stdout:(27/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 324 kB/s | 103 kB 00:00 2024-04-13T21:14:42.183 INFO:teuthology.orchestra.run.smithi175.stdout:(28/32): python3-mako-1.0.6-14.el8.noarch.rpm 569 kB/s | 157 kB 00:00 2024-04-13T21:14:42.186 INFO:teuthology.orchestra.run.smithi132.stdout:(24/32): python3-markupsafe-0.23-19.el8.x86_64. 274 kB/s | 39 kB 00:00 2024-04-13T21:14:42.225 INFO:teuthology.orchestra.run.smithi175.stdout:(29/32): python3-pytz-2017.2-11.el8.noarch.rpm 499 kB/s | 54 kB 00:00 2024-04-13T21:14:42.245 INFO:teuthology.orchestra.run.smithi132.stdout:(25/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2024-04-13T21:14:42.250 INFO:teuthology.orchestra.run.smithi175.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-04-13T21:14:42.270 INFO:teuthology.orchestra.run.smithi132.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 1.2 MB/s | 103 kB 00:00 2024-04-13T21:14:42.387 INFO:teuthology.orchestra.run.smithi132.stdout:(27/32): ceph-mon-18.2.2-1217.g7d36322c.el8.x86 2.6 MB/s | 4.8 MB 00:01 2024-04-13T21:14:42.404 INFO:teuthology.orchestra.run.smithi132.stdout:(28/32): python3-pytz-2017.2-11.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2024-04-13T21:14:42.438 INFO:teuthology.orchestra.run.smithi132.stdout:(29/32): python3-mako-1.0.6-14.el8.noarch.rpm 816 kB/s | 157 kB 00:00 2024-04-13T21:14:42.485 INFO:teuthology.orchestra.run.smithi175.stdout:(31/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 787 kB/s | 238 kB 00:00 2024-04-13T21:14:42.567 INFO:teuthology.orchestra.run.smithi175.stdout:(32/32): python3-cryptography-3.2.1-7.el8_9.x86 1.6 MB/s | 559 kB 00:00 2024-04-13T21:14:42.571 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:14:42.572 INFO:teuthology.orchestra.run.smithi175.stdout:Total 7.4 MB/s | 31 MB 00:04 2024-04-13T21:14:42.621 INFO:teuthology.orchestra.run.smithi132.stdout:(30/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2024-04-13T21:14:42.646 INFO:teuthology.orchestra.run.smithi132.stdout:(31/32): python3-pycparser-2.14-14.el8.noarch.r 422 kB/s | 109 kB 00:00 2024-04-13T21:14:42.667 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:14:42.680 INFO:teuthology.orchestra.run.smithi132.stdout:(32/32): python3-cryptography-3.2.1-7.el8_9.x86 2.3 MB/s | 559 kB 00:00 2024-04-13T21:14:42.682 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:14:42.683 INFO:teuthology.orchestra.run.smithi132.stdout:Total 12 MB/s | 31 MB 00:02 2024-04-13T21:14:42.705 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:14:42.705 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:14:42.780 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:14:42.817 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:14:42.818 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:14:42.931 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:14:42.942 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:14:43.043 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:14:43.052 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:14:44.113 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:14:44.224 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:14:44.352 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-13T21:14:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-13T21:14:44.600 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-13T21:14:44.729 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-13T21:14:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2024-04-13T21:14:44.919 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2024-04-13T21:14:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2024-04-13T21:14:45.307 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2024-04-13T21:14:45.424 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 5/32 2024-04-13T21:14:45.594 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 5/32 2024-04-13T21:14:45.629 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2024-04-13T21:14:45.749 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2024-04-13T21:14:45.839 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2024-04-13T21:14:45.942 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2024-04-13T21:14:46.052 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2024-04-13T21:14:46.147 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2024-04-13T21:14:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2024-04-13T21:14:46.337 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2024-04-13T21:14:46.478 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2024-04-13T21:14:46.554 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2024-04-13T21:14:46.864 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2024-04-13T21:14:46.929 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2024-04-13T21:14:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2024-04-13T21:14:47.166 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2024-04-13T21:14:47.359 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-13T21:14:47.410 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-13T21:14:47.595 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-13T21:14:47.614 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-13T21:14:47.751 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-13T21:14:47.812 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-13T21:14:48.018 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-13T21:14:48.091 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-13T21:14:48.236 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-13T21:14:48.291 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-13T21:14:48.397 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-13T21:14:48.444 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-13T21:14:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-13T21:14:48.667 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-13T21:14:48.827 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-13T21:14:48.849 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-13T21:14:49.030 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-13T21:14:49.085 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-13T21:14:49.254 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-13T21:14:49.357 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-13T21:14:49.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-13T21:14:49.634 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-13T21:14:49.784 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-13T21:14:49.855 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-13T21:14:50.082 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-13T21:14:50.177 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-13T21:14:50.424 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-13T21:14:50.544 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-13T21:14:50.847 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1217.g7d36322c.el8. 27/32 2024-04-13T21:14:50.938 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1217.g7d36322c.el8. 27/32 2024-04-13T21:14:51.034 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 28/32 2024-04-13T21:14:51.133 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 28/32 2024-04-13T21:14:52.685 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 28/32 2024-04-13T21:14:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 28/32 2024-04-13T21:14:53.933 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 29/32 2024-04-13T21:14:54.077 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 29/32 2024-04-13T21:14:54.421 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 29/32 2024-04-13T21:14:54.565 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 29/32 2024-04-13T21:14:54.565 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 30/32 2024-04-13T21:14:54.779 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 30/32 2024-04-13T21:14:54.940 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 31/32 2024-04-13T21:14:54.959 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 31/32 2024-04-13T21:14:55.072 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 32/32 2024-04-13T21:14:55.178 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 30/32 2024-04-13T21:14:55.398 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 30/32 2024-04-13T21:14:55.491 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 32/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 1/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 2/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 3/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 4/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 5/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1217.g7d36322c.el8. 6/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-13T21:14:55.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2024-04-13T21:14:55.493 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2024-04-13T21:14:55.494 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2024-04-13T21:14:55.494 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2024-04-13T21:14:55.494 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2024-04-13T21:14:55.494 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2024-04-13T21:14:55.620 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 31/32 2024-04-13T21:14:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 31/32 2024-04-13T21:14:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 32/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 32/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 1/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 2/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 3/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 4/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 5/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1217.g7d36322c.el8. 6/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-13T21:14:56.129 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-13T21:14:56.130 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2024-04-13T21:14:56.131 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2024-04-13T21:15:03.378 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 32/32 2024-04-13T21:15:03.378 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:15:03.917 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:03.917 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:15:03.917 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-13T21:15:03.918 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-2.6-1.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-13T21:15:03.919 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:03.920 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:15:04.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2024-04-13T21:15:04.446 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:06.118 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-13T21:15:06.795 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-13T21:15:08.998 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:15:09.181 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:15:09.182 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:15:09.182 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:15:09.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2024-04-13T21:15:09.669 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 32/32 2024-04-13T21:15:09.725 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-13T21:15:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-2.6-1.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-13T21:15:10.278 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:10.279 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:15:10.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-base 2024-04-13T21:15:10.836 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:11.861 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-13T21:15:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-04-13T21:15:12.762 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-13T21:15:13.426 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-04-13T21:15:14.842 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:15:14.843 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:14.843 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:15:14.843 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:14.843 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 170 k 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 170 k 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 692 k 2024-04-13T21:15:14.844 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:15:15.529 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-18.2.2-1217.g7d36322c.el8.noarch.rpm 248 kB/s | 170 kB 00:00 2024-04-13T21:15:15.529 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:15:15.530 INFO:teuthology.orchestra.run.smithi132.stdout:Total 248 kB/s | 170 kB 00:00 2024-04-13T21:15:15.530 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:15:15.539 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:15:15.539 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:15:15.542 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:15:15.550 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:15:15.612 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:15:15.662 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-base-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:15:15.851 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:15:15.852 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:15:15.852 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:15:15.989 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install cephadm 2024-04-13T21:15:16.337 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:17.465 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:17.634 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:17.836 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:18.064 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-13T21:15:18.712 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-13T21:15:21.123 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:15:21.124 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:15:21.124 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:15:21.124 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 170 k 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 170 k 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 692 k 2024-04-13T21:15:21.125 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:15:21.433 INFO:teuthology.orchestra.run.smithi175.stdout:cephadm-18.2.2-1217.g7d36322c.el8.noarch.rpm 553 kB/s | 170 kB 00:00 2024-04-13T21:15:21.433 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:15:21.433 INFO:teuthology.orchestra.run.smithi175.stdout:Total 551 kB/s | 170 kB 00:00 2024-04-13T21:15:21.433 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:15:21.441 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:15:21.442 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:15:21.444 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:15:21.452 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:15:21.498 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:15:23.243 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:23.423 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:23.628 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:26.409 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:15:26.962 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:26.962 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:15:26.962 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:15:26.962 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:26.962 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:15:27.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2024-04-13T21:15:27.491 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:29.473 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 kB/s | 4.5 kB 00:00 2024-04-13T21:15:30.188 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-13T21:15:32.476 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:15:32.477 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:32.477 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-04-13T21:15:32.477 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:32.477 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:15:32.477 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 166 k 2024-04-13T21:15:32.477 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 166 k 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 503 k 2024-04-13T21:15:32.478 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:15:32.780 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-18.2.2-1217.g7d3632 550 kB/s | 166 kB 00:00 2024-04-13T21:15:32.781 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:15:32.781 INFO:teuthology.orchestra.run.smithi132.stdout:Total 547 kB/s | 166 kB 00:00 2024-04-13T21:15:32.781 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:15:32.792 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:15:32.792 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:15:32.846 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:15:32.854 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:15:32.970 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:15:33.134 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c. 1/1 2024-04-13T21:15:33.537 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c. 1/1 2024-04-13T21:15:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/1 2024-04-13T21:15:34.625 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:15:35.182 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:35.183 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:15:35.183 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:15:35.183 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:35.183 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:15:35.341 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-immutable-object-cache 2024-04-13T21:15:35.702 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:37.531 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-13T21:15:38.153 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-13T21:15:40.455 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:15:40.455 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:15:40.455 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-04-13T21:15:40.455 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:15:40.455 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:15:40.455 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 166 k 2024-04-13T21:15:40.456 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:40.456 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:15:40.456 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:15:40.456 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:15:40.456 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:40.456 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 166 k 2024-04-13T21:15:40.457 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 503 k 2024-04-13T21:15:40.457 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:15:40.637 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-immutable-object-cache-18.2.2-1217.g7d3632 917 kB/s | 166 kB 00:00 2024-04-13T21:15:40.638 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:15:40.638 INFO:teuthology.orchestra.run.smithi175.stdout:Total 909 kB/s | 166 kB 00:00 2024-04-13T21:15:40.639 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:15:40.650 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:15:40.650 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:15:40.705 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:15:40.713 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:15:40.832 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:15:41.028 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c. 1/1 2024-04-13T21:15:41.428 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c. 1/1 2024-04-13T21:15:41.929 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c. 1/1 2024-04-13T21:15:41.929 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:15:42.468 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:42.468 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:15:42.468 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:42.468 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:42.468 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:15:42.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2024-04-13T21:15:42.983 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:44.954 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-13T21:15:45.636 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-13T21:15:47.844 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:15:48.031 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:15:48.032 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:15:48.032 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:15:48.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2024-04-13T21:15:48.546 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:50.311 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-13T21:15:50.919 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c. 1/1 2024-04-13T21:15:50.919 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:15:50.959 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2024-04-13T21:15:51.489 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:51.490 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:15:51.490 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:15:51.490 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:15:51.490 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:15:51.652 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr 2024-04-13T21:15:52.003 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:53.235 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 3.5 M 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 24 k 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 15 k 2024-04-13T21:15:53.236 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout:Install 5 Packages 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:15:53.237 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.8 M 2024-04-13T21:15:53.238 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 87 M 2024-04-13T21:15:53.238 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:15:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1217.g7d36 102 kB/s | 15 kB 00:00 2024-04-13T21:15:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1217.g7d3 120 kB/s | 24 kB 00:00 2024-04-13T21:15:53.751 INFO:teuthology.orchestra.run.smithi132.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1217.g7d36322c 9.8 MB/s | 3.5 MB 00:00 2024-04-13T21:15:53.768 INFO:teuthology.orchestra.run.smithi132.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 144 kB/s | 34 kB 00:00 2024-04-13T21:15:53.876 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-13T21:15:53.985 INFO:teuthology.orchestra.run.smithi132.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 501 kB/s | 196 kB 00:00 2024-04-13T21:15:53.986 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:15:53.987 INFO:teuthology.orchestra.run.smithi132.stdout:Total 5.1 MB/s | 3.8 MB 00:00 2024-04-13T21:15:53.987 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:15:53.994 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:15:53.995 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:15:54.063 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:15:54.071 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:15:54.317 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:15:54.505 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-13T21:15:54.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-13T21:15:54.722 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-13T21:15:54.846 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1217.g7d36322c.el8.n 3/5 2024-04-13T21:15:56.047 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1217.g7d36322c.el8. 4/5 2024-04-13T21:15:56.306 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarc 5/5 2024-04-13T21:15:56.574 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarc 5/5 2024-04-13T21:15:56.574 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1217.g7d36322c.el8. 1/5 2024-04-13T21:15:56.574 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarc 2/5 2024-04-13T21:15:56.574 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1217.g7d36322c.el8.n 3/5 2024-04-13T21:15:56.574 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-13T21:15:56.620 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-mgr-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:15:56.808 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:15:56.809 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:15:56.809 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:15:56.978 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-dashboard 2024-04-13T21:15:57.323 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:15:59.154 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-13T21:15:59.782 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-13T21:16:02.147 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:16:02.148 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 3.5 M 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 24 k 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 15 k 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:02.149 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:16:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:16:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:Install 5 Packages 2024-04-13T21:16:02.150 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.8 M 2024-04-13T21:16:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 87 M 2024-04-13T21:16:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:16:02.598 INFO:teuthology.orchestra.run.smithi175.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1217.g7d36 68 kB/s | 15 kB 00:00 2024-04-13T21:16:02.749 INFO:teuthology.orchestra.run.smithi175.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1217.g7d3 66 kB/s | 24 kB 00:00 2024-04-13T21:16:03.015 INFO:teuthology.orchestra.run.smithi175.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 81 kB/s | 34 kB 00:00 2024-04-13T21:16:03.407 INFO:teuthology.orchestra.run.smithi175.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 298 kB/s | 196 kB 00:00 2024-04-13T21:16:04.733 INFO:teuthology.orchestra.run.smithi175.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1217.g7d36322c 1.5 MB/s | 3.5 MB 00:02 2024-04-13T21:16:04.736 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:16:04.736 INFO:teuthology.orchestra.run.smithi175.stdout:Total 1.5 MB/s | 3.8 MB 00:02 2024-04-13T21:16:04.737 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:16:04.746 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:16:04.746 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:16:04.821 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:16:04.829 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:16:05.062 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:16:05.304 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-13T21:16:05.553 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-13T21:16:05.671 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1217.g7d36322c.el8.n 3/5 2024-04-13T21:16:06.869 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1217.g7d36322c.el8. 4/5 2024-04-13T21:16:07.151 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarc 5/5 2024-04-13T21:16:07.403 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarc 5/5 2024-04-13T21:16:07.403 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1217.g7d36322c.el8. 1/5 2024-04-13T21:16:07.403 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarc 2/5 2024-04-13T21:16:07.403 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1217.g7d36322c.el8.n 3/5 2024-04-13T21:16:07.403 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-13T21:16:07.580 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-13T21:16:07.580 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:16:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:08.122 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:16:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-13T21:16:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-13T21:16:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:08.123 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:16:08.305 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-13T21:16:08.657 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:16:10.440 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 kB/s | 4.5 kB 00:00 2024-04-13T21:16:11.145 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-13T21:16:13.422 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================================= 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================================= 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 7.4 M 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-13T21:16:13.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================================= 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout:Install 8 Packages 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:13.425 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 36 M 2024-04-13T21:16:13.426 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 207 M 2024-04-13T21:16:13.426 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:16:13.642 INFO:teuthology.orchestra.run.smithi132.stdout:(1/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.0 MB/s | 225 kB 00:00 2024-04-13T21:16:13.759 INFO:teuthology.orchestra.run.smithi132.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-04-13T21:16:13.967 INFO:teuthology.orchestra.run.smithi132.stdout:(3/8): ceph-mgr-diskprediction-local-18.2.2-121 14 MB/s | 7.4 MB 00:00 2024-04-13T21:16:14.084 INFO:teuthology.orchestra.run.smithi132.stdout:(4/8): openblas-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2024-04-13T21:16:14.159 INFO:teuthology.orchestra.run.smithi132.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 10 MB/s | 5.2 MB 00:00 2024-04-13T21:16:14.318 INFO:teuthology.orchestra.run.smithi132.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 2.7 MB/s | 645 kB 00:00 2024-04-13T21:16:14.368 INFO:teuthology.orchestra.run.smithi132.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 827 kB/s | 172 kB 00:00 2024-04-13T21:16:14.560 INFO:teuthology.orchestra.run.smithi132.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 23 MB/s | 14 MB 00:00 2024-04-13T21:16:14.561 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:16:14.561 INFO:teuthology.orchestra.run.smithi132.stdout:Total 32 MB/s | 36 MB 00:01 2024-04-13T21:16:14.807 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:16:14.826 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:16:14.826 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:16:15.134 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:16:15.143 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:16:15.689 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:16:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-13T21:16:16.024 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-13T21:16:16.213 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-13T21:16:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-13T21:16:16.668 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:16:16.844 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-13T21:16:17.052 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-13T21:16:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:17.226 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:16:17.391 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-13T21:16:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-13T21:16:17.740 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:16:17.869 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-13T21:16:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-13T21:16:18.847 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-13T21:16:19.085 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-13T21:16:19.338 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-04-13T21:16:19.937 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-13T21:16:20.833 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-13T21:16:22.119 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-13T21:16:22.260 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================================= 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================================= 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 7.4 M 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-13T21:16:22.261 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================================= 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout:Install 8 Packages 2024-04-13T21:16:22.262 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:22.263 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 36 M 2024-04-13T21:16:22.263 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 207 M 2024-04-13T21:16:22.263 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:16:22.301 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322 8/8 2024-04-13T21:16:22.480 INFO:teuthology.orchestra.run.smithi175.stdout:(1/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 17 MB/s | 3.7 MB 00:00 2024-04-13T21:16:22.505 INFO:teuthology.orchestra.run.smithi175.stdout:(2/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 932 kB/s | 225 kB 00:00 2024-04-13T21:16:22.623 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322 8/8 2024-04-13T21:16:22.623 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322 1/8 2024-04-13T21:16:22.623 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2024-04-13T21:16:22.624 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2024-04-13T21:16:22.624 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2024-04-13T21:16:22.624 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2024-04-13T21:16:22.624 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2024-04-13T21:16:22.624 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2024-04-13T21:16:22.747 INFO:teuthology.orchestra.run.smithi175.stdout:(3/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.2 MB 00:00 2024-04-13T21:16:22.822 INFO:teuthology.orchestra.run.smithi175.stdout:(4/8): openblas-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2024-04-13T21:16:22.981 INFO:teuthology.orchestra.run.smithi175.stdout:(5/8): ceph-mgr-diskprediction-local-18.2.2-121 10 MB/s | 7.4 MB 00:00 2024-04-13T21:16:23.023 INFO:teuthology.orchestra.run.smithi175.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.1 MB/s | 645 kB 00:00 2024-04-13T21:16:23.250 INFO:teuthology.orchestra.run.smithi175.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 27 MB/s | 14 MB 00:00 2024-04-13T21:16:23.282 INFO:teuthology.orchestra.run.smithi175.stdout:(8/8): libquadmath-8.5.0-20.el8.x86_64.rpm 574 kB/s | 172 kB 00:00 2024-04-13T21:16:23.283 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:16:23.283 INFO:teuthology.orchestra.run.smithi175.stdout:Total 35 MB/s | 36 MB 00:01 2024-04-13T21:16:23.527 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:16:23.546 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:16:23.546 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:16:23.855 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:16:23.864 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:16:24.435 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:16:24.619 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-13T21:16:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-13T21:16:24.927 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-13T21:16:25.555 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-13T21:16:25.719 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-13T21:16:26.328 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-13T21:16:26.502 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-13T21:16:27.206 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-13T21:16:27.522 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-13T21:16:27.744 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-13T21:16:29.500 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-13T21:16:30.728 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-13T21:16:30.951 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322 8/8 2024-04-13T21:16:31.228 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322 8/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322 1/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2024-04-13T21:16:31.229 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2024-04-13T21:16:34.788 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2024-04-13T21:16:34.788 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:16:35.354 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:35.355 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:16:35.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2024-04-13T21:16:35.878 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:16:37.707 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-13T21:16:38.448 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:16:40.844 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:16:40.846 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================== 2024-04-13T21:16:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:16:40.846 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================== 2024-04-13T21:16:40.846 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:16:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 49 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-13T21:16:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================== 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout:Install 14 Packages 2024-04-13T21:16:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:40.849 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.0 M 2024-04-13T21:16:40.849 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 25 M 2024-04-13T21:16:40.849 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:16:41.079 INFO:teuthology.orchestra.run.smithi132.stdout:(1/14): ceph-mgr-rook-18.2.2-1217.g7d36322c.el8 294 kB/s | 49 kB 00:00 2024-04-13T21:16:41.271 INFO:teuthology.orchestra.run.smithi132.stdout:(2/14): python3-certifi-2018.10.15-7.el8.noarch 45 kB/s | 16 kB 00:00 2024-04-13T21:16:41.297 INFO:teuthology.orchestra.run.smithi132.stdout:(3/14): python3-cachetools-3.1.1-4.el8.noarch.r 86 kB/s | 33 kB 00:00 2024-04-13T21:16:41.497 INFO:teuthology.orchestra.run.smithi132.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 321 kB/s | 64 kB 00:00 2024-04-13T21:16:41.547 INFO:teuthology.orchestra.run.smithi132.stdout:(5/14): python3-google-auth-1.1.1-10.el8.noarch 174 kB/s | 81 kB 00:00 2024-04-13T21:16:41.648 INFO:teuthology.orchestra.run.smithi132.stdout:(6/14): python3-websocket-client-0.56.0-5.el8.n 407 kB/s | 61 kB 00:00 2024-04-13T21:16:41.790 INFO:teuthology.orchestra.run.smithi132.stdout:(7/14): python3-jsonpointer-1.10-11.el8.noarch. 83 kB/s | 20 kB 00:00 2024-04-13T21:16:41.849 INFO:teuthology.orchestra.run.smithi132.stdout:(8/14): python3-pyasn1-modules-0.3.7-6.el8.noar 550 kB/s | 110 kB 00:00 2024-04-13T21:16:41.882 INFO:teuthology.orchestra.run.smithi132.stdout:(9/14): python3-kubernetes-11.0.0-6.el8.noarch. 1.9 MB/s | 1.2 MB 00:00 2024-04-13T21:16:42.032 INFO:teuthology.orchestra.run.smithi132.stdout:(10/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 688 kB/s | 126 kB 00:00 2024-04-13T21:16:42.058 INFO:teuthology.orchestra.run.smithi132.stdout:(11/14): python3-jsonpatch-1.21-2.el8.noarch.rp 99 kB/s | 27 kB 00:00 2024-04-13T21:16:42.091 INFO:teuthology.orchestra.run.smithi132.stdout:(12/14): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-04-13T21:16:42.250 INFO:teuthology.orchestra.run.smithi132.stdout:(13/14): python3-requests-oauthlib-1.0.0-1.el8. 222 kB/s | 43 kB 00:00 2024-04-13T21:16:42.267 INFO:teuthology.orchestra.run.smithi132.stdout:(14/14): python3-oauthlib-2.1.0-1.el8.noarch.rp 662 kB/s | 155 kB 00:00 2024-04-13T21:16:42.270 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:16:42.270 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-04-13T21:16:42.349 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:16:42.365 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:16:42.365 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:16:42.447 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:16:42.456 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:16:42.559 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2024-04-13T21:16:42.560 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:16:42.991 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-13T21:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:43.157 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:16:43.189 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-13T21:16:43.326 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-rook 2024-04-13T21:16:43.408 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-13T21:16:43.590 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-13T21:16:43.678 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:16:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 4/14 2024-04-13T21:16:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 5/14 2024-04-13T21:16:44.755 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 6/14 2024-04-13T21:16:44.984 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 7/14 2024-04-13T21:16:45.160 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 8/14 2024-04-13T21:16:45.380 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-13T21:16:45.449 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-13T21:16:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-13T21:16:45.854 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-13T21:16:46.068 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-13T21:16:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-13T21:16:47.100 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-13T21:16:47.305 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 14/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 14/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 1/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/14 2024-04-13T21:16:47.583 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/14 2024-04-13T21:16:47.584 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 12/14 2024-04-13T21:16:47.584 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/14 2024-04-13T21:16:48.379 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================== 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================== 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 49 k 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-13T21:16:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================== 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout:Install 14 Packages 2024-04-13T21:16:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:16:48.383 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 2.0 M 2024-04-13T21:16:48.383 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 25 M 2024-04-13T21:16:48.383 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:16:48.937 INFO:teuthology.orchestra.run.smithi175.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 70 kB/s | 16 kB 00:00 2024-04-13T21:16:48.963 INFO:teuthology.orchestra.run.smithi175.stdout:(2/14): ceph-mgr-rook-18.2.2-1217.g7d36322c.el8 190 kB/s | 49 kB 00:00 2024-04-13T21:16:48.979 INFO:teuthology.orchestra.run.smithi175.stdout:(3/14): python3-cachetools-3.1.1-4.el8.noarch.r 120 kB/s | 33 kB 00:00 2024-04-13T21:16:49.113 INFO:teuthology.orchestra.run.smithi175.stdout:(4/14): python3-google-auth-1.1.1-10.el8.noarch 463 kB/s | 81 kB 00:00 2024-04-13T21:16:49.213 INFO:teuthology.orchestra.run.smithi175.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 612 kB/s | 61 kB 00:00 2024-04-13T21:16:49.238 INFO:teuthology.orchestra.run.smithi175.stdout:(6/14): python3-rsa-4.9-2.el8.noarch.rpm 248 kB/s | 64 kB 00:00 2024-04-13T21:16:49.372 INFO:teuthology.orchestra.run.smithi175.stdout:(7/14): python3-kubernetes-11.0.0-6.el8.noarch. 3.0 MB/s | 1.2 MB 00:00 2024-04-13T21:16:49.397 INFO:teuthology.orchestra.run.smithi175.stdout:(8/14): python3-jsonpointer-1.10-11.el8.noarch. 109 kB/s | 20 kB 00:00 2024-04-13T21:16:49.422 INFO:teuthology.orchestra.run.smithi175.stdout:(9/14): python3-pyasn1-modules-0.3.7-6.el8.noar 619 kB/s | 110 kB 00:00 2024-04-13T21:16:49.489 INFO:teuthology.orchestra.run.smithi175.stdout:(10/14): python3-jsonpatch-1.21-2.el8.noarch.rp 227 kB/s | 27 kB 00:00 2024-04-13T21:16:49.514 INFO:teuthology.orchestra.run.smithi175.stdout:(11/14): python3-jwt-1.6.1-2.el8.noarch.rpm 463 kB/s | 43 kB 00:00 2024-04-13T21:16:49.540 INFO:teuthology.orchestra.run.smithi175.stdout:(12/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 923 kB/s | 126 kB 00:00 2024-04-13T21:16:49.607 INFO:teuthology.orchestra.run.smithi175.stdout:(13/14): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.3 MB/s | 155 kB 00:00 2024-04-13T21:16:49.632 INFO:teuthology.orchestra.run.smithi175.stdout:(14/14): python3-requests-oauthlib-1.0.0-1.el8. 365 kB/s | 43 kB 00:00 2024-04-13T21:16:49.635 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:16:49.635 INFO:teuthology.orchestra.run.smithi175.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-04-13T21:16:49.714 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:16:49.730 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:16:49.730 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:16:49.813 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:16:49.822 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:16:50.368 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:16:50.572 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-13T21:16:50.800 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-13T21:16:50.990 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-13T21:16:51.258 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 4/14 2024-04-13T21:16:51.526 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 5/14 2024-04-13T21:16:51.753 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 6/14 2024-04-13T21:16:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 7/14 2024-04-13T21:16:52.117 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 8/14 2024-04-13T21:16:52.305 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-13T21:16:52.529 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-13T21:16:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-13T21:16:53.858 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-13T21:16:54.116 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-13T21:16:54.288 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 14/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 14/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 1/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-13T21:16:54.586 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-13T21:16:54.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/14 2024-04-13T21:16:54.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/14 2024-04-13T21:16:54.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/14 2024-04-13T21:16:54.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/14 2024-04-13T21:16:54.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 12/14 2024-04-13T21:16:54.587 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/14 2024-04-13T21:16:55.560 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 14/14 2024-04-13T21:16:55.560 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:16:56.194 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:56.194 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-13T21:16:56.195 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-13T21:16:56.196 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:16:56.196 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:16:56.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2024-04-13T21:16:56.968 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:16:58.745 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-13T21:16:59.363 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 56 kB/s | 4.1 kB 00:00 2024-04-13T21:17:01.772 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================================== 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================================== 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 138 k 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-13T21:17:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================================== 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout:Install 5 Packages 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.0 M 2024-04-13T21:17:01.774 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-04-13T21:17:01.775 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:17:02.041 INFO:teuthology.orchestra.run.smithi132.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 291 kB/s | 60 kB 00:00 2024-04-13T21:17:02.066 INFO:teuthology.orchestra.run.smithi132.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1217.g7d36322c.e 595 kB/s | 138 kB 00:00 2024-04-13T21:17:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 14/14 2024-04-13T21:17:02.100 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:17:02.158 INFO:teuthology.orchestra.run.smithi132.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2024-04-13T21:17:02.308 INFO:teuthology.orchestra.run.smithi132.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.2 MB/s | 538 kB 00:00 2024-04-13T21:17:02.467 INFO:teuthology.orchestra.run.smithi132.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-04-13T21:17:02.468 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:02.468 INFO:teuthology.orchestra.run.smithi132.stdout:Total 8.6 MB/s | 6.0 MB 00:00 2024-04-13T21:17:02.523 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:17:02.534 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:17:02.534 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:17:02.623 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:17:02.631 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:17:02.672 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:02.672 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-13T21:17:02.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-13T21:17:02.674 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-13T21:17:02.674 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:02.674 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:17:02.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-cephadm 2024-04-13T21:17:03.226 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:03.585 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:17:03.940 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-13T21:17:04.213 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-13T21:17:04.585 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-13T21:17:04.912 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-13T21:17:04.930 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-13T21:17:05.095 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 5/5 2024-04-13T21:17:05.352 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 5/5 2024-04-13T21:17:05.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/5 2024-04-13T21:17:05.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-13T21:17:05.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-13T21:17:05.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-13T21:17:05.510 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-13T21:17:07.851 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================================== 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================================== 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 138 k 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:17:07.852 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================================== 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout:Install 5 Packages 2024-04-13T21:17:07.853 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:07.854 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 6.0 M 2024-04-13T21:17:07.854 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 26 M 2024-04-13T21:17:07.854 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:17:08.143 INFO:teuthology.orchestra.run.smithi175.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 339 kB/s | 60 kB 00:00 2024-04-13T21:17:08.268 INFO:teuthology.orchestra.run.smithi175.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.5 MB/s | 476 kB 00:00 2024-04-13T21:17:08.293 INFO:teuthology.orchestra.run.smithi175.stdout:(3/5): ceph-mgr-cephadm-18.2.2-1217.g7d36322c.e 421 kB/s | 138 kB 00:00 2024-04-13T21:17:08.377 INFO:teuthology.orchestra.run.smithi175.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.8 MB/s | 538 kB 00:00 2024-04-13T21:17:08.460 INFO:teuthology.orchestra.run.smithi175.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2024-04-13T21:17:08.461 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:08.461 INFO:teuthology.orchestra.run.smithi175.stdout:Total 9.8 MB/s | 6.0 MB 00:00 2024-04-13T21:17:08.515 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:17:08.526 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:17:08.526 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:17:08.617 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:17:08.624 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:17:09.612 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:17:09.973 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-13T21:17:10.254 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-13T21:17:10.653 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-13T21:17:10.968 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-13T21:17:11.193 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 5/5 2024-04-13T21:17:11.472 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 5/5 2024-04-13T21:17:11.472 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 1/5 2024-04-13T21:17:11.473 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-13T21:17:11.473 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-13T21:17:11.473 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-13T21:17:12.813 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-13T21:17:12.813 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:17:13.373 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:13.373 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:17:13.373 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:17:13.373 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-13T21:17:13.374 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-13T21:17:13.374 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-13T21:17:13.374 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-13T21:17:13.374 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:13.374 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:17:13.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2024-04-13T21:17:13.899 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:15.596 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 kB/s | 4.5 kB 00:00 2024-04-13T21:17:16.210 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 55 kB/s | 4.1 kB 00:00 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse 2024-04-13T21:17:18.485 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 859 k 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-04-13T21:17:18.486 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:18.487 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 942 k 2024-04-13T21:17:18.487 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 3.0 M 2024-04-13T21:17:18.487 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:17:18.487 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-13T21:17:18.487 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:17:18.842 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 236 kB/s | 83 kB 00:00 2024-04-13T21:17:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:19.064 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:17:19.130 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-fuse-18.2.2-1217.g7d36322c.el8.x86_ 1.3 MB/s | 859 kB 00:00 2024-04-13T21:17:19.130 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:19.131 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.4 MB/s | 942 kB 00:00 2024-04-13T21:17:19.144 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:17:19.167 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:17:19.167 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:17:19.228 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-fuse 2024-04-13T21:17:19.256 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:17:19.265 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:17:19.428 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:17:19.587 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:19.676 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-13T21:17:19.850 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 2/2 2024-04-13T21:17:20.230 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 2/2 2024-04-13T21:17:20.230 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/2 2024-04-13T21:17:21.487 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-13T21:17:22.094 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-13T21:17:24.512 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:17:24.512 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:17:24.512 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:17:24.512 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:17:24.512 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout: x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 859 k 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2024-04-13T21:17:24.513 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:24.514 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 942 k 2024-04-13T21:17:24.514 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 3.0 M 2024-04-13T21:17:24.514 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:17:24.660 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 568 kB/s | 83 kB 00:00 2024-04-13T21:17:25.569 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-fuse-18.2.2-1217.g7d36322c.el8.x86_ 814 kB/s | 859 kB 00:01 2024-04-13T21:17:25.569 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:25.570 INFO:teuthology.orchestra.run.smithi175.stdout:Total 892 kB/s | 942 kB 00:01 2024-04-13T21:17:25.583 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:17:25.605 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:17:25.606 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:17:25.694 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:17:25.703 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:17:25.869 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:17:26.132 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-13T21:17:26.332 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 2/2 2024-04-13T21:17:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 2/2 2024-04-13T21:17:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/2 2024-04-13T21:17:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-13T21:17:27.441 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:17:27.997 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:27.998 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:17:27.998 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-13T21:17:27.998 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:27.998 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:17:28.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-volume 2024-04-13T21:17:28.519 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:30.277 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-13T21:17:30.981 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-13T21:17:33.298 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================== 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================== 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 263 k 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================================== 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-04-13T21:17:33.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:33.300 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 311 k 2024-04-13T21:17:33.300 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.4 M 2024-04-13T21:17:33.300 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:17:33.552 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 191 kB/s | 48 kB 00:00 2024-04-13T21:17:33.911 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-volume-18.2.2-1217.g7d36322c.el8.no 430 kB/s | 263 kB 00:00 2024-04-13T21:17:33.912 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:33.912 INFO:teuthology.orchestra.run.smithi132.stdout:Total 508 kB/s | 311 kB 00:00 2024-04-13T21:17:33.924 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:17:33.936 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:17:33.936 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:17:33.974 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:17:33.983 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:17:34.182 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:17:34.497 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-13T21:17:34.732 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 2/2 2024-04-13T21:17:35.157 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 2/2 2024-04-13T21:17:35.158 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 1/2 2024-04-13T21:17:37.921 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-13T21:17:37.921 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:17:38.474 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:38.474 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:17:38.474 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-13T21:17:38.474 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:38.474 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:17:38.641 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-volume 2024-04-13T21:17:39.002 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:40.865 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 4.5 kB 00:00 2024-04-13T21:17:41.506 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-13T21:17:43.861 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================== 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================== 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume noarch 2:18.2.2-1217.g7d36322c.el8 ceph-noarch 263 k 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:43.862 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:17:43.863 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================================== 2024-04-13T21:17:43.863 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2024-04-13T21:17:43.863 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:43.863 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 311 k 2024-04-13T21:17:43.863 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 1.4 M 2024-04-13T21:17:43.863 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:17:44.010 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 331 kB/s | 48 kB 00:00 2024-04-13T21:17:44.218 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-volume-18.2.2-1217.g7d36322c.el8.no 742 kB/s | 263 kB 00:00 2024-04-13T21:17:44.218 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:44.219 INFO:teuthology.orchestra.run.smithi175.stdout:Total 875 kB/s | 311 kB 00:00 2024-04-13T21:17:44.230 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:17:44.242 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:17:44.242 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:17:44.280 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:17:44.288 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:17:44.461 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:17:44.792 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-13T21:17:45.062 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 2/2 2024-04-13T21:17:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 2/2 2024-04-13T21:17:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 1/2 2024-04-13T21:17:45.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-13T21:17:45.492 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:17:46.052 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:46.052 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:17:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:17:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-13T21:17:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:46.053 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:17:46.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2024-04-13T21:17:46.568 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:48.507 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-13T21:17:49.219 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-13T21:17:51.515 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:17:51.516 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:17:51.516 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-04-13T21:17:51.516 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:17:51.516 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 139 k 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 139 k 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 513 k 2024-04-13T21:17:51.517 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:17:51.803 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-18.2.2-1217.g7d36322c.el8.x86_64 486 kB/s | 139 kB 00:00 2024-04-13T21:17:51.804 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:51.804 INFO:teuthology.orchestra.run.smithi132.stdout:Total 484 kB/s | 139 kB 00:00 2024-04-13T21:17:51.804 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:17:51.810 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:17:51.810 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:17:51.867 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:17:51.876 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:17:51.994 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:17:52.235 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:17:52.460 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-13T21:17:52.461 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:17:52.502 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:17:53.015 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:53.015 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:17:53.015 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume-2:18.2.2-1217.g7d36322c.el8.noarch 2024-04-13T21:17:53.015 INFO:teuthology.orchestra.run.smithi175.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-13T21:17:53.016 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:53.016 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:17:53.159 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados-devel 2024-04-13T21:17:53.520 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:17:55.312 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-13T21:17:55.910 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-13T21:17:58.214 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 139 k 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:58.215 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:17:58.216 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:17:58.216 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:17:58.216 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:17:58.216 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 139 k 2024-04-13T21:17:58.216 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 513 k 2024-04-13T21:17:58.216 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:17:58.398 INFO:teuthology.orchestra.run.smithi175.stdout:librados-devel-18.2.2-1217.g7d36322c.el8.x86_64 763 kB/s | 139 kB 00:00 2024-04-13T21:17:58.398 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:17:58.399 INFO:teuthology.orchestra.run.smithi175.stdout:Total 758 kB/s | 139 kB 00:00 2024-04-13T21:17:58.399 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:17:58.404 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:17:58.404 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:17:58.460 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:17:58.469 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:17:58.635 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:17:58.866 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:17:59.137 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:01.710 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:01.710 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:18:02.278 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:18:02.278 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:18:02.279 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:18:02.279 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:18:02.279 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:02.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2024-04-13T21:18:02.782 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:04.567 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-13T21:18:05.203 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-04-13T21:18:07.090 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:07.090 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:18:07.297 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:07.483 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:07.484 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:07.484 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:07.617 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2024-04-13T21:18:07.654 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:18:07.654 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:18:07.654 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:18:07.654 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:18:07.654 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:07.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs2 2024-04-13T21:18:07.977 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:08.171 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:09.784 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 kB/s | 4.5 kB 00:00 2024-04-13T21:18:09.971 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-13T21:18:10.451 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-13T21:18:10.584 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-13T21:18:12.743 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:12.743 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:18:12.743 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-04-13T21:18:12.743 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 31 k 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 k 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 139 k 2024-04-13T21:18:12.744 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:18:12.745 INFO:teuthology.orchestra.run.smithi175.stdout:Package libcephfs2-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:12.883 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-18.2.2-1217.g7d36322c.el8.x86_6 220 kB/s | 31 kB 00:00 2024-04-13T21:18:12.884 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:18:12.884 INFO:teuthology.orchestra.run.smithi132.stdout:Total 218 kB/s | 31 kB 00:00 2024-04-13T21:18:12.885 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:18:12.888 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:18:12.888 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:18:12.900 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:18:12.909 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:18:12.935 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:12.936 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:12.936 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:13.002 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:18:13.082 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs-devel 2024-04-13T21:18:13.182 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:13.436 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:13.440 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:15.402 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-13T21:18:16.046 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-13T21:18:18.343 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 31 k 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:18:18.344 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:18:18.345 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 31 k 2024-04-13T21:18:18.345 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 139 k 2024-04-13T21:18:18.345 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:18:18.568 INFO:teuthology.orchestra.run.smithi175.stdout:libcephfs-devel-18.2.2-1217.g7d36322c.el8.x86_6 139 kB/s | 31 kB 00:00 2024-04-13T21:18:18.568 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:18:18.569 INFO:teuthology.orchestra.run.smithi175.stdout:Total 138 kB/s | 31 kB 00:00 2024-04-13T21:18:18.569 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:18:18.572 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:18:18.572 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:18:18.585 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:18:18.593 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:18:18.659 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:18:18.879 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:19.129 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:24.040 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:24.040 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:18:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:18:24.604 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:18:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:18:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:18:24.605 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:24.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2024-04-13T21:18:25.131 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:26.963 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:18:26.964 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:18:27.009 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-13T21:18:27.537 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:18:27.537 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:18:27.537 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:18:27.537 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:18:27.537 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:27.711 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-13T21:18:27.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados2 2024-04-13T21:18:28.082 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:29.911 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:29.959 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-13T21:18:30.098 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:30.099 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:30.099 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:30.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2024-04-13T21:18:30.588 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-13T21:18:30.600 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:32.384 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-13T21:18:32.707 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:32.897 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:32.898 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:32.898 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:33.034 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librbd1 2024-04-13T21:18:33.075 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-13T21:18:33.393 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:35.182 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:35.267 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 kB/s | 4.5 kB 00:00 2024-04-13T21:18:35.372 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:35.373 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:35.373 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:35.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2024-04-13T21:18:35.846 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2024-04-13T21:18:35.859 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:37.740 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-13T21:18:37.976 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:38.162 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:38.163 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:38.163 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:38.302 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rados 2024-04-13T21:18:38.449 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:18:38.661 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:40.546 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:40.695 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-13T21:18:40.741 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:40.742 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:40.743 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:40.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2024-04-13T21:18:41.248 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:41.284 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-04-13T21:18:42.934 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-13T21:18:43.427 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rados-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:43.613 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:43.614 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:43.614 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:43.633 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:18:43.748 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rgw 2024-04-13T21:18:44.105 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:45.768 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:45.956 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:45.957 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:45.958 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:45.974 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-13T21:18:46.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2024-04-13T21:18:46.455 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:46.606 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:18:48.271 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 54 kB/s | 4.5 kB 00:00 2024-04-13T21:18:48.720 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rgw-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:48.913 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:48.914 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:48.914 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:48.989 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-13T21:18:49.045 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-cephfs 2024-04-13T21:18:49.401 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:51.106 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:51.144 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-13T21:18:51.301 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:51.302 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:51.302 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:51.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2024-04-13T21:18:51.805 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:51.820 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-04-13T21:18:53.612 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-13T21:18:53.926 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-cephfs-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:54.114 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:54.115 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:54.115 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:54.247 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rbd 2024-04-13T21:18:54.334 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-13T21:18:54.605 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:56.410 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-13T21:18:56.535 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:56.727 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:18:56.728 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-04-13T21:18:56.728 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:18:56.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2024-04-13T21:18:57.054 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-13T21:18:57.221 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:18:58.985 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-13T21:18:59.179 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rbd-2:18.2.2-1217.g7d36322c.el8.x86_64 is already installed. 2024-04-13T21:18:59.366 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:18:59.367 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-04-13T21:18:59.367 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:18:59.504 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-fuse 2024-04-13T21:18:59.696 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:18:59.860 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:19:01.709 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 95 k 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:02.085 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:19:02.086 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:02.086 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:19:02.086 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:02.086 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 95 k 2024-04-13T21:19:02.086 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 280 k 2024-04-13T21:19:02.086 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:19:02.273 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-18.2.2-1217.g7d36322c.el8.x86_64.rpm 510 kB/s | 95 kB 00:00 2024-04-13T21:19:02.273 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:19:02.274 INFO:teuthology.orchestra.run.smithi132.stdout:Total 507 kB/s | 95 kB 00:00 2024-04-13T21:19:02.274 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:19:02.284 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:19:02.284 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:19:02.339 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:19:02.344 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:19:02.348 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:19:02.456 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:19:02.632 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:02.903 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:04.675 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 95 k 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:19:04.676 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:04.677 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:19:04.677 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:04.677 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 95 k 2024-04-13T21:19:04.677 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 280 k 2024-04-13T21:19:04.677 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:19:04.863 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-fuse-18.2.2-1217.g7d36322c.el8.x86_64.rpm 509 kB/s | 95 kB 00:00 2024-04-13T21:19:04.864 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:19:04.864 INFO:teuthology.orchestra.run.smithi175.stdout:Total 506 kB/s | 95 kB 00:00 2024-04-13T21:19:04.864 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:19:04.874 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:19:04.875 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:19:04.930 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:19:04.939 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:19:05.051 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:19:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:05.531 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:13.277 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:13.277 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:19:13.849 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:13.849 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:19:13.849 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:19:13.849 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:13.849 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:19:13.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-04-13T21:19:14.357 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:19:15.771 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:15.771 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:19:16.131 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-13T21:19:16.364 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:16.364 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:19:16.364 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:19:16.364 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:16.364 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:19:16.518 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-mirror 2024-04-13T21:19:16.853 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-13T21:19:16.883 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:19:18.777 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-13T21:19:19.128 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:19:19.128 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:19.128 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-04-13T21:19:19.128 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:19.128 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:19:19.128 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 3.5 M 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.5 M 2024-04-13T21:19:19.129 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 16 M 2024-04-13T21:19:19.130 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:19:19.319 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 kB/s | 4.1 kB 00:00 2024-04-13T21:19:19.751 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-18.2.2-1217.g7d36322c.el8.x86_64.rpm 5.6 MB/s | 3.5 MB 00:00 2024-04-13T21:19:19.752 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:19:19.752 INFO:teuthology.orchestra.run.smithi132.stdout:Total 5.5 MB/s | 3.5 MB 00:00 2024-04-13T21:19:19.753 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:19:19.765 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:19:19.765 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:19:19.833 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:19:19.841 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:19:20.317 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:19:20.498 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:20.927 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:21.616 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 3.5 M 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:19:21.617 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:21.618 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.5 M 2024-04-13T21:19:21.618 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 16 M 2024-04-13T21:19:21.618 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:19:22.223 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-mirror-18.2.2-1217.g7d36322c.el8.x86_64.rpm 5.7 MB/s | 3.5 MB 00:00 2024-04-13T21:19:22.224 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:19:22.224 INFO:teuthology.orchestra.run.smithi175.stdout:Total 5.7 MB/s | 3.5 MB 00:00 2024-04-13T21:19:22.224 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:19:22.236 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:19:22.236 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:19:22.305 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:19:22.313 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:19:22.762 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:19:22.977 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:23.358 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:35.320 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:35.320 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:19:35.895 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:35.895 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:19:35.896 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:19:35.896 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:35.896 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:19:36.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2024-04-13T21:19:36.428 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-04-13T21:19:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:37.045 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:19:37.619 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:37.619 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:19:37.619 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:19:37.619 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:37.619 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:19:37.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-nbd 2024-04-13T21:19:38.151 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2024-04-13T21:19:38.219 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 23 kB/s | 4.5 kB 00:00 2024-04-13T21:19:38.952 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-13T21:19:40.144 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-13T21:19:40.736 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-13T21:19:41.235 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-13T21:19:41.235 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:41.235 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 181 k 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 181 k 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 572 k 2024-04-13T21:19:41.236 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-13T21:19:41.464 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-18.2.2-1217.g7d36322c.el8.x86_64.rpm 797 kB/s | 181 kB 00:00 2024-04-13T21:19:41.464 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:19:41.465 INFO:teuthology.orchestra.run.smithi132.stdout:Total 792 kB/s | 181 kB 00:00 2024-04-13T21:19:41.465 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-13T21:19:41.476 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-13T21:19:41.476 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-13T21:19:41.533 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-13T21:19:41.542 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-13T21:19:41.659 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-13T21:19:41.868 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:42.140 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:43.040 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-13T21:19:43.040 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:43.040 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-04-13T21:19:43.040 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:43.040 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-13T21:19:43.040 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:18.2.2-1217.g7d36322c.el8 ceph 181 k 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 181 k 2024-04-13T21:19:43.041 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 572 k 2024-04-13T21:19:43.042 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-13T21:19:43.467 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-nbd-18.2.2-1217.g7d36322c.el8.x86_64.rpm 426 kB/s | 181 kB 00:00 2024-04-13T21:19:43.468 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-13T21:19:43.468 INFO:teuthology.orchestra.run.smithi175.stdout:Total 424 kB/s | 181 kB 00:00 2024-04-13T21:19:43.468 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-13T21:19:43.480 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-13T21:19:43.480 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-13T21:19:43.538 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-13T21:19:43.547 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-13T21:19:43.670 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-13T21:19:43.879 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:44.148 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:53.188 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:53.189 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-04-13T21:19:53.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:53.778 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-13T21:19:53.778 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:19:53.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:19:53.778 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-13T21:19:53.975 DEBUG:teuthology.parallel:result is None 2024-04-13T21:19:57.660 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 1/1 2024-04-13T21:19:57.661 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2024-04-13T21:19:58.220 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:58.220 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-13T21:19:58.220 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:18.2.2-1217.g7d36322c.el8.x86_64 2024-04-13T21:19:58.221 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:19:58.221 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-13T21:19:58.389 DEBUG:teuthology.parallel:result is None 2024-04-13T21:19:58.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7d36322c373c73b184fd252cd881b122d8c51a3b 2024-04-13T21:19:58.536 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T21:19:58.563 INFO:teuthology.orchestra.run.smithi132.stdout:18.2.2-1217.g7d36322c.el8 2024-04-13T21:19:58.564 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1217.g7d36322c.el8 2024-04-13T21:19:58.564 INFO:teuthology.task.install:The correct ceph version 18.2.2-1217.g7d36322c is installed. 2024-04-13T21:19:58.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7d36322c373c73b184fd252cd881b122d8c51a3b 2024-04-13T21:19:58.797 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T21:19:58.821 INFO:teuthology.orchestra.run.smithi175.stdout:18.2.2-1217.g7d36322c.el8 2024-04-13T21:19:58.821 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1217.g7d36322c.el8 2024-04-13T21:19:58.821 INFO:teuthology.task.install:The correct ceph version 18.2.2-1217.g7d36322c is installed. 2024-04-13T21:19:58.823 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-13T21:19:58.823 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:19:58.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T21:19:58.856 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:19:58.856 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T21:19:58.889 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-13T21:19:58.890 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:19:58.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T21:19:58.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T21:19:58.997 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:19:58.997 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T21:19:59.024 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T21:19:59.091 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-13T21:19:59.091 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:19:59.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T21:19:59.120 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T21:19:59.189 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:19:59.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T21:19:59.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T21:19:59.282 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-13T21:19:59.283 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:19:59.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T21:19:59.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T21:19:59.377 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:19:59.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T21:19:59.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T21:19:59.471 INFO:teuthology.run_tasks:Running task ceph... 2024-04-13T21:19:59.559 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-13T21:19:59.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /var/log/ceph 2024-04-13T21:19:59.562 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /var/log/ceph 2024-04-13T21:19:59.588 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-13T21:19:59.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-13T21:19:59.630 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-13T21:19:59.656 INFO:tasks.ceph:Creating extra log directories... 2024-04-13T21:19:59.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-13T21:19:59.697 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-13T21:19:59.748 INFO:tasks.ceph:Creating ceph cluster c1... 2024-04-13T21:19:59.748 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-13T21:19:59.749 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7654822', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 3741, 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'targets': {'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA/N/FKLBquRPdl60M0y1Az5c6woju1lkGZ2FTsAqgZWA3bDiyOusiaFVnYZpk0uy/nC9ouvTL4sTO6i7eJUTrk=', 'smithi175.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9iF1EUT4zEZ4kQHllVdkLfUpKC05i7vMlbtsRrcaTRcRWNj2GSkNeTg3+iKQ9jFBfrQGq3nEZemVeieYd/EBw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-12_22:40:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-13T21:19:59.749 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-13T21:19:59.764 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-13T21:19:59.780 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T21:19:59.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T21:19:59.848 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:19:59.848 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T21:19:59.903 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T21:19:59.903 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 181712 Links: 1 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-13 21:19:20.698178517 +0000 2024-04-13T21:19:59.960 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-13 21:11:48.476497081 +0000 2024-04-13T21:19:59.961 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-13 21:11:48.476497081 +0000 2024-04-13T21:19:59.961 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-13T21:19:59.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T21:20:00.027 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-13T21:20:00.027 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-13T21:20:00.027 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000245513 s, 2.1 MB/s 2024-04-13T21:20:00.029 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T21:20:00.086 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-04-13T21:20:00.144 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T21:20:00.144 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:00.144 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 177151 Links: 1 2024-04-13T21:20:00.144 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:00.144 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:00.145 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-13 21:19:20.699178492 +0000 2024-04-13T21:20:00.145 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-13 21:11:48.840487753 +0000 2024-04-13T21:20:00.145 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-13 21:11:48.840487753 +0000 2024-04-13T21:20:00.145 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-13T21:20:00.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T21:20:00.210 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-13T21:20:00.210 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-13T21:20:00.210 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000233355 s, 2.2 MB/s 2024-04-13T21:20:00.211 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T21:20:00.269 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 180841 Links: 1 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-13 21:19:20.699178492 +0000 2024-04-13T21:20:00.327 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-13 21:11:49.214478167 +0000 2024-04-13T21:20:00.328 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-13 21:11:49.214478167 +0000 2024-04-13T21:20:00.328 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-13T21:20:00.328 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T21:20:00.393 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-13T21:20:00.393 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-13T21:20:00.393 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000186561 s, 2.7 MB/s 2024-04-13T21:20:00.394 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T21:20:00.452 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-04-13T21:20:00.509 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T21:20:00.509 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 180925 Links: 1 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-13 21:19:20.699178492 +0000 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-13 21:11:49.566469146 +0000 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-13 21:11:49.566469146 +0000 2024-04-13T21:20:00.510 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-13T21:20:00.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T21:20:00.574 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-13T21:20:00.575 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-13T21:20:00.575 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000238896 s, 2.1 MB/s 2024-04-13T21:20:00.576 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T21:20:00.633 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:00.633 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T21:20:00.634 INFO:tasks.ceph:Generating config... 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-13T21:20:00.635 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-13T21:20:00.636 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-13T21:20:00.636 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-13T21:20:00.636 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-13T21:20:00.636 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-13T21:20:00.636 INFO:tasks.ceph:[client] setuser = ceph 2024-04-13T21:20:00.636 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-13T21:20:00.636 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-13T21:20:00.636 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-13T21:20:00.636 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-13T21:20:00.636 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-13T21:20:00.636 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-13T21:20:00.636 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-13T21:20:00.636 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-13T21:20:00.636 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-13T21:20:00.637 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-13T21:20:00.637 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-13T21:20:00.637 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-13T21:20:00.637 INFO:tasks.ceph:Setting up c1.mon.a... 2024-04-13T21:20:00.637 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-04-13T21:20:00.721 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/c1.keyring 2024-04-13T21:20:00.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-04-13T21:20:00.769 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-13T21:20:00.795 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.132')] 2024-04-13T21:20:00.795 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.132'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-13T21:20:00.796 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:00.796 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T21:20:00.852 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.132 --print /home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:00.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:20:00.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: generated fsid 86f15865-1caf-41ce-80a5-8ccfb5344934 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:setting min_mon_release = pacific 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:epoch 0 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:fsid 86f15865-1caf-41ce-80a5-8ccfb5344934 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:last_changed 2024-04-13T21:20:00.942560+0000 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:created 2024-04-13T21:20:00.942560+0000 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:min_mon_release 16 (pacific) 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:election_strategy: 1 2024-04-13T21:20:00.935 INFO:teuthology.orchestra.run.smithi132.stdout:0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-04-13T21:20:00.936 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-04-13T21:20:00.937 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T21:20:00.994 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 86f15865-1caf-41ce-80a5-8ccfb5344934... 2024-04-13T21:20:00.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-13T21:20:01.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-13T21:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-04-13T21:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2024-04-13T21:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T21:20:01.079 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T21:20:01.080 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2024-04-13T21:20:01.081 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2024-04-13T21:20:01.082 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = 172.21.15.132 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = 86f15865-1caf-41ce-80a5-8ccfb5344934 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2024-04-13T21:20:01.083 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2024-04-13T21:20:01.084 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: rocksdb delete range threshold = 0 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2024-04-13T21:20:01.085 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2024-04-13T21:20:01.086 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T21:20:01.087 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2024-04-13T21:20:01.088 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: debug rgw = 20 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: debug rgw sync = 20 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw crypt require ssl = False 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw crypt s3 kms backend = testing 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw curl low speed time = 300 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw data log num shards = 4 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw md log max shards = 4 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync data inject err probability = 0.1 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync log trim interval = 0 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync meta inject err probability = 0.1 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync obj etag verify = True 2024-04-13T21:20:01.089 INFO:teuthology.orchestra.run.smithi132.stdout: setgroup = ceph 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout: setuser = ceph 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout: rgw data notify interval msec = 0 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout: osd_max_omap_entries_per_request = 10 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout: osd_max_omap_entries_per_request = 1000 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2024-04-13T21:20:01.090 INFO:teuthology.orchestra.run.smithi132.stdout: osd_max_omap_entries_per_request = 10000 2024-04-13T21:20:01.094 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2024-04-13T21:20:01.094 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.095 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2024-04-13T21:20:01.096 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.097 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = 172.21.15.132 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 86f15865-1caf-41ce-80a5-8ccfb5344934 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2024-04-13T21:20:01.098 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd mclock profile = high_recovery_ops 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2024-04-13T21:20:01.099 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-13T21:20:01.100 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 20 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: rocksdb delete range threshold = 0 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-13T21:20:01.101 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.102 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: # 1m isn't quite enough 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: mon_down_mkfs_grace = 2m 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_filestore_osds = false 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:01.103 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw = 20 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw sync = 20 2024-04-13T21:20:01.104 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt require ssl = False 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms backend = testing 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw curl low speed time = 300 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data log num shards = 4 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw md log max shards = 4 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync data inject err probability = 0.1 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync log trim interval = 0 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync meta inject err probability = 0.1 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync obj etag verify = True 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: setgroup = ceph 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: setuser = ceph 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2024-04-13T21:20:01.105 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data notify interval msec = 0 2024-04-13T21:20:01.106 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2024-04-13T21:20:01.106 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10 2024-04-13T21:20:01.106 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2024-04-13T21:20:01.106 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 1000 2024-04-13T21:20:01.106 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2024-04-13T21:20:01.106 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10000 2024-04-13T21:20:01.107 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-04-13T21:20:01.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-04-13T21:20:01.189 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-13T21:20:01.189 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:01.189 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-04-13T21:20:01.205 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:01.205 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-04-13T21:20:01.262 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T21:20:01.262 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:01.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/c1.keyring 2024-04-13T21:20:01.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-13T21:20:01.339 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:01.339 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:01.395 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T21:20:01.395 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:01.396 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/c1.keyring 2024-04-13T21:20:01.396 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-13T21:20:01.440 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:01.440 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:01.499 INFO:tasks.ceph:Setting up mon nodes... 2024-04-13T21:20:01.499 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-13T21:20:01.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-04-13T21:20:01.553 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-04-13T21:20:01.556 INFO:tasks.ceph:Setting up mds nodes... 2024-04-13T21:20:01.556 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-13T21:20:01.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-04-13T21:20:01.599 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/c1.client.0.keyring 2024-04-13T21:20:01.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-04-13T21:20:01.696 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/c1.client.1.keyring 2024-04-13T21:20:01.708 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-13T21:20:01.708 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T21:20:01.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-04-13T21:20:01.774 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:01.774 INFO:tasks.ceph:role: c1.osd.0 2024-04-13T21:20:01.774 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T21:20:01.775 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-13T21:20:01.848 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T21:20:01.849 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T21:20:02.283 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-04-13T21:20:02.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-04-13T21:20:02.285 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-04-13T21:20:02.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-04-13T21:20:02.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-04-13T21:20:02.569 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:02.569 INFO:tasks.ceph:role: c1.osd.1 2024-04-13T21:20:02.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T21:20:02.570 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-13T21:20:02.644 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T21:20:02.645 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T21:20:03.077 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-04-13T21:20:03.078 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-04-13T21:20:03.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-04-13T21:20:03.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-04-13T21:20:03.218 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-04-13T21:20:03.284 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:03.285 INFO:tasks.ceph:role: c1.osd.2 2024-04-13T21:20:03.285 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T21:20:03.285 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-13T21:20:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T21:20:03.360 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T21:20:03.820 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-04-13T21:20:03.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-04-13T21:20:03.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-04-13T21:20:03.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-04-13T21:20:04.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:04.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:04.116+0000 7fc4b5fd07c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-04-13T21:20:04.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:04.118+0000 7fc4b5fd07c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-04-13T21:20:04.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:04.118+0000 7fc4b5fd07c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-13T21:20:04.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:04.118+0000 7fc4b5fd07c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-13T21:20:04.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:04.118+0000 7fc4b5fd07c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-13T21:20:04.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:04.125+0000 7fc4b5fd07c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-04-13T21:20:06.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-04-13T21:20:06.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:06.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:06.334+0000 7f8653f1d7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-04-13T21:20:06.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:06.334+0000 7f8653f1d7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-04-13T21:20:06.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:06.334+0000 7f8653f1d7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-13T21:20:06.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:06.334+0000 7f8653f1d7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-13T21:20:06.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:06.334+0000 7f8653f1d7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-13T21:20:06.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:06.335+0000 7f8653f1d7c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-04-13T21:20:08.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-04-13T21:20:08.446 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:08.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:08.543+0000 7f5cb86737c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-04-13T21:20:08.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:08.543+0000 7f5cb86737c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-04-13T21:20:08.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:08.543+0000 7f5cb86737c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-13T21:20:08.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:08.543+0000 7f5cb86737c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-13T21:20:08.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:08.543+0000 7f5cb86737c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-13T21:20:08.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:20:08.544+0000 7f5cb86737c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-04-13T21:20:10.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-04-13T21:20:10.648 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-13T21:20:10.648 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:10.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-04-13T21:20:10.716 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:10.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-04-13T21:20:10.784 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:10.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-04-13T21:20:10.850 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:10.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-04-13T21:20:10.917 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:10.918 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-04-13T21:20:10.975 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:10.975 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-04-13T21:20:11.031 INFO:tasks.ceph:Adding keys to all mons... 2024-04-13T21:20:11.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo tee -a /etc/ceph/c1.keyring 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.x] 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAB9xpmDAt1IRAABZeEvva2rcEnozU0Ezxzbw== 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAE9xpm6kn0BhAAwUcwHnmsspONDJ9Mq7xAMg== 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAG9xpmCa/1ExAAiifT/fuUpBZ6NTp2cD0nGw== 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAI9xpm/YF3IBAAltTLncexUftcDwZxnrTW5A== 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-04-13T21:20:11.097 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAB9xpmklcvJBAA7AuRZyzAkK9GvXJ9SgmlMw== 2024-04-13T21:20:11.098 INFO:teuthology.orchestra.run.smithi132.stdout:[client.1] 2024-04-13T21:20:11.098 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAB9xpm+MzpKRAAJH2KgnRrqJRBkzSfoVnTxA== 2024-04-13T21:20:11.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-13T21:20:11.192 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T21:20:11.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T21:20:11.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T21:20:11.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T21:20:11.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T21:20:11.466 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-13T21:20:11.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-04-13T21:20:11.493 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-04-13T21:20:11.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-04-13T21:20:11.617 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-04-13T21:20:11.674 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-04-13T21:20:11.675 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-04-13T21:20:11.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-04-13T21:20:11.717 INFO:tasks.ceph.c1.mon.a:Started 2024-04-13T21:20:11.717 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-04-13T21:20:11.717 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-04-13T21:20:11.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-04-13T21:20:11.719 INFO:tasks.ceph.c1.mgr.x:Started 2024-04-13T21:20:11.720 INFO:tasks.ceph:Setting crush tunables to default 2024-04-13T21:20:11.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 osd crush tunables default 2024-04-13T21:20:11.946 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2024-04-13T21:20:11.957 INFO:tasks.ceph:check_enable_crimson: False 2024-04-13T21:20:11.957 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-04-13T21:20:11.958 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:11.958 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-04-13T21:20:11.984 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:11.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-04-13T21:20:12.055 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:12.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-04-13T21:20:12.123 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 osd new 3464daec-2e5b-4e82-ab01-75a5490342a8 0 2024-04-13T21:20:12.382 INFO:teuthology.orchestra.run.smithi132.stdout:0 2024-04-13T21:20:12.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 osd new 65ff9e68-7cc9-494d-9d2d-00af934b2a19 1 2024-04-13T21:20:12.459 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:12.465+0000 7f2631dd7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T21:20:12.565 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:12.571+0000 7f2631dd7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T21:20:12.635 INFO:teuthology.orchestra.run.smithi132.stdout:1 2024-04-13T21:20:12.646 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:12.652+0000 7f2631dd7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T21:20:12.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 osd new 32228de9-0d3a-42a2-88da-bc584c2f57d7 2 2024-04-13T21:20:12.722 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:12.729+0000 7f2631dd7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T21:20:12.865 INFO:teuthology.orchestra.run.smithi132.stdout:2 2024-04-13T21:20:12.882 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-04-13T21:20:12.882 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-04-13T21:20:12.884 INFO:tasks.ceph.c1.osd.0:Started 2024-04-13T21:20:12.884 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-04-13T21:20:12.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-04-13T21:20:12.886 INFO:tasks.ceph.c1.osd.1:Started 2024-04-13T21:20:12.887 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-04-13T21:20:12.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-04-13T21:20:12.888 INFO:tasks.ceph.c1.osd.2:Started 2024-04-13T21:20:12.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-13T21:20:12.924 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:12.930+0000 7f2631dd7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T21:20:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:13.100 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":5,"fsid":"86f15865-1caf-41ce-80a5-8ccfb5344934","created":"2024-04-13T21:20:11.855036+0000","modified":"2024-04-13T21:20:12.870062+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T21:20:12.870062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3464daec-2e5b-4e82-ab01-75a5490342a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"65ff9e68-7cc9-494d-9d2d-00af934b2a19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32228de9-0d3a-42a2-88da-bc584c2f57d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:13.113 INFO:tasks.ceph.ceph_manager.c1:[] 2024-04-13T21:20:13.113 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-13T21:20:13.119 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:13.126+0000 7f2631dd7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T21:20:13.230 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:13.237+0000 7f2631dd7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T21:20:13.326 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:13.332+0000 7f2631dd7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T21:20:13.401 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:13.407+0000 7f2631dd7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T21:20:13.415 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-13T21:20:13.495 INFO:tasks.ceph.c1.osd.2.smithi132.stderr:2024-04-13T21:20:13.501+0000 7fabf6a0a7c0 -1 Falling back to public interface 2024-04-13T21:20:13.496 INFO:tasks.ceph.c1.osd.0.smithi132.stderr:2024-04-13T21:20:13.501+0000 7f07032cd7c0 -1 Falling back to public interface 2024-04-13T21:20:13.519 INFO:tasks.ceph.c1.osd.1.smithi132.stderr:2024-04-13T21:20:13.525+0000 7f218ecb47c0 -1 Falling back to public interface 2024-04-13T21:20:13.640 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:13.646+0000 7f2631dd7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T21:20:13.644 INFO:teuthology.misc.health.smithi132.stdout: 2024-04-13T21:20:13.644 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":5,"fsid":"86f15865-1caf-41ce-80a5-8ccfb5344934","created":"2024-04-13T21:20:11.855036+0000","modified":"2024-04-13T21:20:12.870062+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T21:20:12.870062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3464daec-2e5b-4e82-ab01-75a5490342a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"65ff9e68-7cc9-494d-9d2d-00af934b2a19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32228de9-0d3a-42a2-88da-bc584c2f57d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:13.656 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-13T21:20:13.713 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:13.720+0000 7f2631dd7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T21:20:14.099 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:14.105+0000 7f2631dd7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T21:20:14.420 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:14.426+0000 7f2631dd7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T21:20:14.578 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:14.584+0000 7f2631dd7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T21:20:14.659 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:14.665+0000 7f2631dd7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T21:20:15.357 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:15.363+0000 7f2631dd7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T21:20:15.441 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:15.447+0000 7f2631dd7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T21:20:15.573 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:15.579+0000 7f2631dd7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T21:20:16.145 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:16.152+0000 7f2631dd7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T21:20:16.254 INFO:tasks.ceph.c1.osd.2.smithi132.stderr:2024-04-13T21:20:16.260+0000 7fabf6a0a7c0 -1 osd.2 0 log_to_monitors true 2024-04-13T21:20:16.257 INFO:tasks.ceph.c1.osd.1.smithi132.stderr:2024-04-13T21:20:16.263+0000 7f218ecb47c0 -1 osd.1 0 log_to_monitors true 2024-04-13T21:20:16.266 INFO:tasks.ceph.c1.osd.0.smithi132.stderr:2024-04-13T21:20:16.273+0000 7f07032cd7c0 -1 osd.0 0 log_to_monitors true 2024-04-13T21:20:16.331 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:16.337+0000 7f2631dd7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T21:20:16.496 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:16.502+0000 7f2631dd7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T21:20:17.305 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:17.311+0000 7f2631dd7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T21:20:17.588 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:17.595+0000 7f2631dd7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T21:20:17.666 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:17.672+0000 7f2631dd7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T21:20:18.098 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:18.104+0000 7f2631dd7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T21:20:18.554 INFO:tasks.ceph.c1.osd.0.smithi132.stderr:2024-04-13T21:20:18.560+0000 7f06f7685700 -1 osd.0 0 waiting for initial osdmap 2024-04-13T21:20:18.556 INFO:tasks.ceph.c1.osd.2.smithi132.stderr:2024-04-13T21:20:18.562+0000 7fabeadc2700 -1 osd.2 0 waiting for initial osdmap 2024-04-13T21:20:18.557 INFO:tasks.ceph.c1.osd.1.smithi132.stderr:2024-04-13T21:20:18.564+0000 7f218306c700 -1 osd.1 0 waiting for initial osdmap 2024-04-13T21:20:18.566 INFO:tasks.ceph.c1.osd.2.smithi132.stderr:2024-04-13T21:20:18.572+0000 7fabe299c700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T21:20:18.566 INFO:tasks.ceph.c1.osd.0.smithi132.stderr:2024-04-13T21:20:18.572+0000 7f06ef25f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T21:20:18.575 INFO:tasks.ceph.c1.osd.1.smithi132.stderr:2024-04-13T21:20:18.582+0000 7f217ac46700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T21:20:19.166 INFO:tasks.ceph.c1.mgr.x.smithi132.stderr:2024-04-13T21:20:19.172+0000 7f25ed051700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-13T21:20:19.958 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-13T21:20:20.291 INFO:teuthology.misc.health.smithi132.stdout: 2024-04-13T21:20:20.291 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":8,"fsid":"86f15865-1caf-41ce-80a5-8ccfb5344934","created":"2024-04-13T21:20:11.855036+0000","modified":"2024-04-13T21:20:19.567781+0000","last_up_change":"2024-04-13T21:20:19.567781+0000","last_in_change":"2024-04-13T21:20:12.870062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3464daec-2e5b-4e82-ab01-75a5490342a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":40788},{"type":"v1","addr":"172.21.15.132:6802","nonce":40788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":40788},{"type":"v1","addr":"172.21.15.132:6806","nonce":40788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":40788},{"type":"v1","addr":"172.21.15.132:6814","nonce":40788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":40788},{"type":"v1","addr":"172.21.15.132:6810","nonce":40788}]},"public_addr":"172.21.15.132:6802/40788","cluster_addr":"172.21.15.132:6806/40788","heartbeat_back_addr":"172.21.15.132:6814/40788","heartbeat_front_addr":"172.21.15.132:6810/40788","state":["exists","up"]},{"osd":1,"uuid":"65ff9e68-7cc9-494d-9d2d-00af934b2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":40814},{"type":"v1","addr":"172.21.15.132:6817","nonce":40814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":40814},{"type":"v1","addr":"172.21.15.132:6819","nonce":40814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":40814},{"type":"v1","addr":"172.21.15.132:6823","nonce":40814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":40814},{"type":"v1","addr":"172.21.15.132:6821","nonce":40814}]},"public_addr":"172.21.15.132:6817/40814","cluster_addr":"172.21.15.132:6819/40814","heartbeat_back_addr":"172.21.15.132:6823/40814","heartbeat_front_addr":"172.21.15.132:6821/40814","state":["exists","up"]},{"osd":2,"uuid":"32228de9-0d3a-42a2-88da-bc584c2f57d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":40789},{"type":"v1","addr":"172.21.15.132:6803","nonce":40789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":40789},{"type":"v1","addr":"172.21.15.132:6807","nonce":40789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":40789},{"type":"v1","addr":"172.21.15.132:6815","nonce":40789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":40789},{"type":"v1","addr":"172.21.15.132:6811","nonce":40789}]},"public_addr":"172.21.15.132:6803/40789","cluster_addr":"172.21.15.132:6807/40789","heartbeat_back_addr":"172.21.15.132:6815/40789","heartbeat_front_addr":"172.21.15.132:6811/40789","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:20.304 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-13T21:20:20.304 INFO:tasks.ceph:Creating RBD pool 2024-04-13T21:20:20.304 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 osd pool create rbd 8 2024-04-13T21:20:21.175 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'rbd' created 2024-04-13T21:20:21.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-13T21:20:22.177 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-13T21:20:22.191 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-04-13T21:20:22.192 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-04-13T21:20:22.192 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-04-13T21:20:22.192 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-04-13T21:20:22.192 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-13T21:20:22.622 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:22.636 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":40409},{"type":"v1","addr":"172.21.15.132:6825","nonce":40409}]},"active_addr":"172.21.15.132:6825/40409","active_change":"2024-04-13T21:20:18.118387+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":3710334085}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":137018298}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":4244954985}]}]} 2024-04-13T21:20:22.638 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-04-13T21:20:22.638 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-04-13T21:20:22.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-13T21:20:22.964 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:22.965 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":10,"fsid":"86f15865-1caf-41ce-80a5-8ccfb5344934","created":"2024-04-13T21:20:11.855036+0000","modified":"2024-04-13T21:20:22.174004+0000","last_up_change":"2024-04-13T21:20:19.567781+0000","last_in_change":"2024-04-13T21:20:12.870062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T21:20:20.212564+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T21:20:20.656762+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3464daec-2e5b-4e82-ab01-75a5490342a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":40788},{"type":"v1","addr":"172.21.15.132:6802","nonce":40788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":40788},{"type":"v1","addr":"172.21.15.132:6806","nonce":40788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":40788},{"type":"v1","addr":"172.21.15.132:6814","nonce":40788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":40788},{"type":"v1","addr":"172.21.15.132:6810","nonce":40788}]},"public_addr":"172.21.15.132:6802/40788","cluster_addr":"172.21.15.132:6806/40788","heartbeat_back_addr":"172.21.15.132:6814/40788","heartbeat_front_addr":"172.21.15.132:6810/40788","state":["exists","up"]},{"osd":1,"uuid":"65ff9e68-7cc9-494d-9d2d-00af934b2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":40814},{"type":"v1","addr":"172.21.15.132:6817","nonce":40814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":40814},{"type":"v1","addr":"172.21.15.132:6819","nonce":40814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":40814},{"type":"v1","addr":"172.21.15.132:6823","nonce":40814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":40814},{"type":"v1","addr":"172.21.15.132:6821","nonce":40814}]},"public_addr":"172.21.15.132:6817/40814","cluster_addr":"172.21.15.132:6819/40814","heartbeat_back_addr":"172.21.15.132:6823/40814","heartbeat_front_addr":"172.21.15.132:6821/40814","state":["exists","up"]},{"osd":2,"uuid":"32228de9-0d3a-42a2-88da-bc584c2f57d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":40789},{"type":"v1","addr":"172.21.15.132:6803","nonce":40789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":40789},{"type":"v1","addr":"172.21.15.132:6807","nonce":40789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":40789},{"type":"v1","addr":"172.21.15.132:6815","nonce":40789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":40789},{"type":"v1","addr":"172.21.15.132:6811","nonce":40789}]},"public_addr":"172.21.15.132:6803/40789","cluster_addr":"172.21.15.132:6807/40789","heartbeat_back_addr":"172.21.15.132:6815/40789","heartbeat_front_addr":"172.21.15.132:6811/40789","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:17.312581+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:17.277052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:17.272434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:22.978 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-04-13T21:20:22.979 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-13T21:20:23.308 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:23.308 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":11,"fsid":"86f15865-1caf-41ce-80a5-8ccfb5344934","created":"2024-04-13T21:20:11.855036+0000","modified":"2024-04-13T21:20:23.175031+0000","last_up_change":"2024-04-13T21:20:19.567781+0000","last_in_change":"2024-04-13T21:20:12.870062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T21:20:20.212564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T21:20:20.656762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3464daec-2e5b-4e82-ab01-75a5490342a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":40788},{"type":"v1","addr":"172.21.15.132:6802","nonce":40788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":40788},{"type":"v1","addr":"172.21.15.132:6806","nonce":40788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":40788},{"type":"v1","addr":"172.21.15.132:6814","nonce":40788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":40788},{"type":"v1","addr":"172.21.15.132:6810","nonce":40788}]},"public_addr":"172.21.15.132:6802/40788","cluster_addr":"172.21.15.132:6806/40788","heartbeat_back_addr":"172.21.15.132:6814/40788","heartbeat_front_addr":"172.21.15.132:6810/40788","state":["exists","up"]},{"osd":1,"uuid":"65ff9e68-7cc9-494d-9d2d-00af934b2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":40814},{"type":"v1","addr":"172.21.15.132:6817","nonce":40814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":40814},{"type":"v1","addr":"172.21.15.132:6819","nonce":40814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":40814},{"type":"v1","addr":"172.21.15.132:6823","nonce":40814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":40814},{"type":"v1","addr":"172.21.15.132:6821","nonce":40814}]},"public_addr":"172.21.15.132:6817/40814","cluster_addr":"172.21.15.132:6819/40814","heartbeat_back_addr":"172.21.15.132:6823/40814","heartbeat_front_addr":"172.21.15.132:6821/40814","state":["exists","up"]},{"osd":2,"uuid":"32228de9-0d3a-42a2-88da-bc584c2f57d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":40789},{"type":"v1","addr":"172.21.15.132:6803","nonce":40789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":40789},{"type":"v1","addr":"172.21.15.132:6807","nonce":40789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":40789},{"type":"v1","addr":"172.21.15.132:6815","nonce":40789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":40789},{"type":"v1","addr":"172.21.15.132:6811","nonce":40789}]},"public_addr":"172.21.15.132:6803/40789","cluster_addr":"172.21.15.132:6807/40789","heartbeat_back_addr":"172.21.15.132:6815/40789","heartbeat_front_addr":"172.21.15.132:6811/40789","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:17.312581+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:17.277052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:17.272434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:23.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-04-13T21:20:23.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-04-13T21:20:23.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-04-13T21:20:23.487 INFO:teuthology.orchestra.run.smithi132.stdout:34359738370 2024-04-13T21:20:23.487 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-13T21:20:23.496 INFO:teuthology.orchestra.run.smithi132.stdout:34359738370 2024-04-13T21:20:23.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-13T21:20:23.498 INFO:teuthology.orchestra.run.smithi132.stdout:34359738370 2024-04-13T21:20:23.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-13T21:20:23.837 INFO:teuthology.orchestra.run.smithi132.stdout:34359738369 2024-04-13T21:20:23.852 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2024-04-13T21:20:23.880 INFO:teuthology.orchestra.run.smithi132.stdout:34359738369 2024-04-13T21:20:23.886 INFO:teuthology.orchestra.run.smithi132.stdout:34359738369 2024-04-13T21:20:23.896 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2024-04-13T21:20:23.901 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2024-04-13T21:20:24.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-13T21:20:24.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-13T21:20:24.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-13T21:20:25.244 INFO:teuthology.orchestra.run.smithi132.stdout:34359738370 2024-04-13T21:20:25.251 INFO:teuthology.orchestra.run.smithi132.stdout:34359738370 2024-04-13T21:20:25.259 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2024-04-13T21:20:25.259 DEBUG:teuthology.parallel:result is None 2024-04-13T21:20:25.266 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2024-04-13T21:20:25.266 DEBUG:teuthology.parallel:result is None 2024-04-13T21:20:25.320 INFO:teuthology.orchestra.run.smithi132.stdout:34359738370 2024-04-13T21:20:25.333 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2024-04-13T21:20:25.334 DEBUG:teuthology.parallel:result is None 2024-04-13T21:20:25.334 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-13T21:20:25.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-13T21:20:25.654 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:25.656 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-04-13T21:20:25.668 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-13T21:20:24.171432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82976,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489824,"statfs":{"total":322122547200,"available":322037579776,"internally_reserved":0,"allocated":1454080,"data_stored":1247365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181510+0000","last_change":"2024-04-13T21:20:22.182479+0000","last_active":"2024-04-13T21:20:23.181510+0000","last_peered":"2024-04-13T21:20:23.181510+0000","last_clean":"2024-04-13T21:20:23.181510+0000","last_became_active":"2024-04-13T21:20:22.182196+0000","last_became_peered":"2024-04-13T21:20:22.182196+0000","last_unstale":"2024-04-13T21:20:23.181510+0000","last_undegraded":"2024-04-13T21:20:23.181510+0000","last_fullsized":"2024-04-13T21:20:23.181510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T07:38:17.217421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181459+0000","last_change":"2024-04-13T21:20:22.183546+0000","last_active":"2024-04-13T21:20:23.181459+0000","last_peered":"2024-04-13T21:20:23.181459+0000","last_clean":"2024-04-13T21:20:23.181459+0000","last_became_active":"2024-04-13T21:20:22.183315+0000","last_became_peered":"2024-04-13T21:20:22.183315+0000","last_unstale":"2024-04-13T21:20:23.181459+0000","last_undegraded":"2024-04-13T21:20:23.181459+0000","last_fullsized":"2024-04-13T21:20:23.181459+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T07:36:32.502761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181466+0000","last_change":"2024-04-13T21:20:22.183134+0000","last_active":"2024-04-13T21:20:23.181466+0000","last_peered":"2024-04-13T21:20:23.181466+0000","last_clean":"2024-04-13T21:20:23.181466+0000","last_became_active":"2024-04-13T21:20:22.182881+0000","last_became_peered":"2024-04-13T21:20:22.182881+0000","last_unstale":"2024-04-13T21:20:23.181466+0000","last_undegraded":"2024-04-13T21:20:23.181466+0000","last_fullsized":"2024-04-13T21:20:23.181466+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:55:58.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-13T21:20:22.187604+0000","last_change":"2024-04-13T21:20:22.181889+0000","last_active":"2024-04-13T21:20:22.187604+0000","last_peered":"2024-04-13T21:20:22.187604+0000","last_clean":"2024-04-13T21:20:22.187604+0000","last_became_active":"2024-04-13T21:20:22.181621+0000","last_became_peered":"2024-04-13T21:20:22.181621+0000","last_unstale":"2024-04-13T21:20:22.187604+0000","last_undegraded":"2024-04-13T21:20:22.187604+0000","last_fullsized":"2024-04-13T21:20:22.187604+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T01:48:51.389248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181430+0000","last_change":"2024-04-13T21:20:22.181588+0000","last_active":"2024-04-13T21:20:23.181430+0000","last_peered":"2024-04-13T21:20:23.181430+0000","last_clean":"2024-04-13T21:20:23.181430+0000","last_became_active":"2024-04-13T21:20:22.181344+0000","last_became_peered":"2024-04-13T21:20:22.181344+0000","last_unstale":"2024-04-13T21:20:23.181430+0000","last_undegraded":"2024-04-13T21:20:23.181430+0000","last_fullsized":"2024-04-13T21:20:23.181430+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:02:03.145279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181429+0000","last_change":"2024-04-13T21:20:22.181160+0000","last_active":"2024-04-13T21:20:23.181429+0000","last_peered":"2024-04-13T21:20:23.181429+0000","last_clean":"2024-04-13T21:20:23.181429+0000","last_became_active":"2024-04-13T21:20:22.180883+0000","last_became_peered":"2024-04-13T21:20:22.180883+0000","last_unstale":"2024-04-13T21:20:23.181429+0000","last_undegraded":"2024-04-13T21:20:23.181429+0000","last_fullsized":"2024-04-13T21:20:23.181429+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T07:40:45.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-13T21:20:22.187616+0000","last_change":"2024-04-13T21:20:22.181996+0000","last_active":"2024-04-13T21:20:22.187616+0000","last_peered":"2024-04-13T21:20:22.187616+0000","last_clean":"2024-04-13T21:20:22.187616+0000","last_became_active":"2024-04-13T21:20:22.181746+0000","last_became_peered":"2024-04-13T21:20:22.181746+0000","last_unstale":"2024-04-13T21:20:22.187616+0000","last_undegraded":"2024-04-13T21:20:22.187616+0000","last_fullsized":"2024-04-13T21:20:22.187616+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:00:01.685237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-13T21:20:22.187429+0000","last_change":"2024-04-13T21:20:22.181823+0000","last_active":"2024-04-13T21:20:22.187429+0000","last_peered":"2024-04-13T21:20:22.187429+0000","last_clean":"2024-04-13T21:20:22.187429+0000","last_became_active":"2024-04-13T21:20:22.181583+0000","last_became_peered":"2024-04-13T21:20:22.181583+0000","last_unstale":"2024-04-13T21:20:22.187429+0000","last_undegraded":"2024-04-13T21:20:22.187429+0000","last_fullsized":"2024-04-13T21:20:22.187429+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:55:12.179091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181488+0000","last_change":"2024-04-13T21:20:22.182549+0000","last_active":"2024-04-13T21:20:23.181488+0000","last_peered":"2024-04-13T21:20:23.181488+0000","last_clean":"2024-04-13T21:20:23.181488+0000","last_became_active":"2024-04-13T21:20:22.182255+0000","last_became_peered":"2024-04-13T21:20:22.182255+0000","last_unstale":"2024-04-13T21:20:23.181488+0000","last_undegraded":"2024-04-13T21:20:23.181488+0000","last_fullsized":"2024-04-13T21:20:23.181488+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T02:05:09.732021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27108,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28188,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829412,"statfs":{"total":107374182400,"available":107345317888,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T21:20:25.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-13T21:20:25.988 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:25.991 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-04-13T21:20:26.002 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-13T21:20:24.171432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82976,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489824,"statfs":{"total":322122547200,"available":322037579776,"internally_reserved":0,"allocated":1454080,"data_stored":1247365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181510+0000","last_change":"2024-04-13T21:20:22.182479+0000","last_active":"2024-04-13T21:20:23.181510+0000","last_peered":"2024-04-13T21:20:23.181510+0000","last_clean":"2024-04-13T21:20:23.181510+0000","last_became_active":"2024-04-13T21:20:22.182196+0000","last_became_peered":"2024-04-13T21:20:22.182196+0000","last_unstale":"2024-04-13T21:20:23.181510+0000","last_undegraded":"2024-04-13T21:20:23.181510+0000","last_fullsized":"2024-04-13T21:20:23.181510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T07:38:17.217421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181459+0000","last_change":"2024-04-13T21:20:22.183546+0000","last_active":"2024-04-13T21:20:23.181459+0000","last_peered":"2024-04-13T21:20:23.181459+0000","last_clean":"2024-04-13T21:20:23.181459+0000","last_became_active":"2024-04-13T21:20:22.183315+0000","last_became_peered":"2024-04-13T21:20:22.183315+0000","last_unstale":"2024-04-13T21:20:23.181459+0000","last_undegraded":"2024-04-13T21:20:23.181459+0000","last_fullsized":"2024-04-13T21:20:23.181459+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T07:36:32.502761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181466+0000","last_change":"2024-04-13T21:20:22.183134+0000","last_active":"2024-04-13T21:20:23.181466+0000","last_peered":"2024-04-13T21:20:23.181466+0000","last_clean":"2024-04-13T21:20:23.181466+0000","last_became_active":"2024-04-13T21:20:22.182881+0000","last_became_peered":"2024-04-13T21:20:22.182881+0000","last_unstale":"2024-04-13T21:20:23.181466+0000","last_undegraded":"2024-04-13T21:20:23.181466+0000","last_fullsized":"2024-04-13T21:20:23.181466+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:55:58.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-13T21:20:22.187604+0000","last_change":"2024-04-13T21:20:22.181889+0000","last_active":"2024-04-13T21:20:22.187604+0000","last_peered":"2024-04-13T21:20:22.187604+0000","last_clean":"2024-04-13T21:20:22.187604+0000","last_became_active":"2024-04-13T21:20:22.181621+0000","last_became_peered":"2024-04-13T21:20:22.181621+0000","last_unstale":"2024-04-13T21:20:22.187604+0000","last_undegraded":"2024-04-13T21:20:22.187604+0000","last_fullsized":"2024-04-13T21:20:22.187604+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T01:48:51.389248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181430+0000","last_change":"2024-04-13T21:20:22.181588+0000","last_active":"2024-04-13T21:20:23.181430+0000","last_peered":"2024-04-13T21:20:23.181430+0000","last_clean":"2024-04-13T21:20:23.181430+0000","last_became_active":"2024-04-13T21:20:22.181344+0000","last_became_peered":"2024-04-13T21:20:22.181344+0000","last_unstale":"2024-04-13T21:20:23.181430+0000","last_undegraded":"2024-04-13T21:20:23.181430+0000","last_fullsized":"2024-04-13T21:20:23.181430+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:02:03.145279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181429+0000","last_change":"2024-04-13T21:20:22.181160+0000","last_active":"2024-04-13T21:20:23.181429+0000","last_peered":"2024-04-13T21:20:23.181429+0000","last_clean":"2024-04-13T21:20:23.181429+0000","last_became_active":"2024-04-13T21:20:22.180883+0000","last_became_peered":"2024-04-13T21:20:22.180883+0000","last_unstale":"2024-04-13T21:20:23.181429+0000","last_undegraded":"2024-04-13T21:20:23.181429+0000","last_fullsized":"2024-04-13T21:20:23.181429+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T07:40:45.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-13T21:20:22.187616+0000","last_change":"2024-04-13T21:20:22.181996+0000","last_active":"2024-04-13T21:20:22.187616+0000","last_peered":"2024-04-13T21:20:22.187616+0000","last_clean":"2024-04-13T21:20:22.187616+0000","last_became_active":"2024-04-13T21:20:22.181746+0000","last_became_peered":"2024-04-13T21:20:22.181746+0000","last_unstale":"2024-04-13T21:20:22.187616+0000","last_undegraded":"2024-04-13T21:20:22.187616+0000","last_fullsized":"2024-04-13T21:20:22.187616+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:00:01.685237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-13T21:20:22.187429+0000","last_change":"2024-04-13T21:20:22.181823+0000","last_active":"2024-04-13T21:20:22.187429+0000","last_peered":"2024-04-13T21:20:22.187429+0000","last_clean":"2024-04-13T21:20:22.187429+0000","last_became_active":"2024-04-13T21:20:22.181583+0000","last_became_peered":"2024-04-13T21:20:22.181583+0000","last_unstale":"2024-04-13T21:20:22.187429+0000","last_undegraded":"2024-04-13T21:20:22.187429+0000","last_fullsized":"2024-04-13T21:20:22.187429+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:55:12.179091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:23.181488+0000","last_change":"2024-04-13T21:20:22.182549+0000","last_active":"2024-04-13T21:20:23.181488+0000","last_peered":"2024-04-13T21:20:23.181488+0000","last_clean":"2024-04-13T21:20:23.181488+0000","last_became_active":"2024-04-13T21:20:22.182255+0000","last_became_peered":"2024-04-13T21:20:22.182255+0000","last_unstale":"2024-04-13T21:20:23.181488+0000","last_undegraded":"2024-04-13T21:20:23.181488+0000","last_fullsized":"2024-04-13T21:20:23.181488+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:21.173244+0000","last_clean_scrub_stamp":"2024-04-13T21:20:21.173244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T02:05:09.732021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27108,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28188,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829412,"statfs":{"total":107374182400,"available":107345317888,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T21:20:26.003 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-13T21:20:26.003 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-04-13T21:20:26.003 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-04-13T21:20:26.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-04-13T21:20:26.373 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:26.374 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-13T21:20:26.387 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-04-13T21:20:26.388 INFO:teuthology.run_tasks:Running task ceph... 2024-04-13T21:20:26.397 INFO:tasks.ceph:Creating ceph cluster c2... 2024-04-13T21:20:26.397 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-13T21:20:26.397 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7654822', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 3741, 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'targets': {'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA/N/FKLBquRPdl60M0y1Az5c6woju1lkGZ2FTsAqgZWA3bDiyOusiaFVnYZpk0uy/nC9ouvTL4sTO6i7eJUTrk=', 'smithi175.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9iF1EUT4zEZ4kQHllVdkLfUpKC05i7vMlbtsRrcaTRcRWNj2GSkNeTg3+iKQ9jFBfrQGq3nEZemVeieYd/EBw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7d36322c373c73b184fd252cd881b122d8c51a3b', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-12_22:40:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-13T21:20:26.398 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-13T21:20:26.400 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-13T21:20:26.421 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T21:20:26.457 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T21:20:26.494 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:26.494 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T21:20:26.554 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T21:20:26.554 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 2024-04-13T21:20:26.613 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T21:20:26.613 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:26.613 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 179417 Links: 1 2024-04-13T21:20:26.613 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:26.613 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:26.613 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-13 21:19:23.131300039 +0000 2024-04-13T21:20:26.614 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-13 21:11:46.501406354 +0000 2024-04-13T21:20:26.614 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-13 21:11:46.501406354 +0000 2024-04-13T21:20:26.614 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2024-04-13T21:20:26.614 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T21:20:26.684 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-13T21:20:26.691 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-13T21:20:26.691 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000280149 s, 1.8 MB/s 2024-04-13T21:20:26.692 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T21:20:26.753 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 182600 Links: 1 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-13 21:19:23.131300039 +0000 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-13 21:11:46.853401028 +0000 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-13 21:11:46.853401028 +0000 2024-04-13T21:20:26.813 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2024-04-13T21:20:26.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T21:20:26.881 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-13T21:20:26.882 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-13T21:20:26.882 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000217136 s, 2.4 MB/s 2024-04-13T21:20:26.883 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T21:20:26.943 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 2024-04-13T21:20:27.000 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 179538 Links: 1 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-13 21:19:23.131300039 +0000 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-13 21:11:47.204395717 +0000 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-13 21:11:47.204395717 +0000 2024-04-13T21:20:27.001 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2024-04-13T21:20:27.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T21:20:27.067 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-13T21:20:27.067 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-13T21:20:27.067 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000188173 s, 2.7 MB/s 2024-04-13T21:20:27.068 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T21:20:27.127 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 2024-04-13T21:20:27.185 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T21:20:27.185 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T21:20:27.185 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 179592 Links: 1 2024-04-13T21:20:27.186 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T21:20:27.186 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T21:20:27.186 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-13 21:19:23.131300039 +0000 2024-04-13T21:20:27.186 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-13 21:11:47.552390451 +0000 2024-04-13T21:20:27.186 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-13 21:11:47.552390451 +0000 2024-04-13T21:20:27.186 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2024-04-13T21:20:27.186 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T21:20:27.251 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-13T21:20:27.251 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-13T21:20:27.251 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000214996 s, 2.4 MB/s 2024-04-13T21:20:27.253 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T21:20:27.311 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:27.311 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T21:20:27.311 INFO:tasks.ceph:Generating config... 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-13T21:20:27.312 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client] setuser = ceph 2024-04-13T21:20:27.313 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-13T21:20:27.313 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-13T21:20:27.313 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-13T21:20:27.313 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-13T21:20:27.313 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-13T21:20:27.313 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-13T21:20:27.313 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-13T21:20:27.313 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-13T21:20:27.314 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-13T21:20:27.314 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-13T21:20:27.314 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-13T21:20:27.314 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-13T21:20:27.314 INFO:tasks.ceph:Setting up c2.mon.a... 2024-04-13T21:20:27.314 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-04-13T21:20:27.405 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c2.keyring 2024-04-13T21:20:27.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-04-13T21:20:27.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-13T21:20:27.519 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.175')] 2024-04-13T21:20:27.519 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.175'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-13T21:20:27.520 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:27.520 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T21:20:27.575 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.175 --print /home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:27.656 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:20:27.657 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:20:27.657 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:27.657 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: generated fsid 627f123d-57cb-4597-b0f1-8de1173b7a8b 2024-04-13T21:20:27.657 INFO:teuthology.orchestra.run.smithi175.stdout:setting min_mon_release = pacific 2024-04-13T21:20:27.657 INFO:teuthology.orchestra.run.smithi175.stdout:epoch 0 2024-04-13T21:20:27.657 INFO:teuthology.orchestra.run.smithi175.stdout:fsid 627f123d-57cb-4597-b0f1-8de1173b7a8b 2024-04-13T21:20:27.658 INFO:teuthology.orchestra.run.smithi175.stdout:last_changed 2024-04-13T21:20:27.665236+0000 2024-04-13T21:20:27.658 INFO:teuthology.orchestra.run.smithi175.stdout:created 2024-04-13T21:20:27.665236+0000 2024-04-13T21:20:27.658 INFO:teuthology.orchestra.run.smithi175.stdout:min_mon_release 16 (pacific) 2024-04-13T21:20:27.658 INFO:teuthology.orchestra.run.smithi175.stdout:election_strategy: 1 2024-04-13T21:20:27.658 INFO:teuthology.orchestra.run.smithi175.stdout:0: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.a 2024-04-13T21:20:27.658 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-04-13T21:20:27.659 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T21:20:27.716 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 627f123d-57cb-4597-b0f1-8de1173b7a8b... 2024-04-13T21:20:27.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-13T21:20:27.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2024-04-13T21:20:27.773 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.774 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2024-04-13T21:20:27.775 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = 172.21.15.175 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = 627f123d-57cb-4597-b0f1-8de1173b7a8b 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2024-04-13T21:20:27.776 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.777 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: rocksdb delete range threshold = 0 2024-04-13T21:20:27.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-04-13T21:20:27.779 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T21:20:27.780 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: debug rgw = 20 2024-04-13T21:20:27.781 INFO:teuthology.orchestra.run.smithi132.stdout: debug rgw sync = 20 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw crypt require ssl = False 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw crypt s3 kms backend = testing 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw curl low speed time = 300 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw data log num shards = 4 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw md log max shards = 4 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync data inject err probability = 0.1 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync log trim interval = 0 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync meta inject err probability = 0.1 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: rgw sync obj etag verify = True 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: setgroup = ceph 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout: setuser = ceph 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2024-04-13T21:20:27.782 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout: rgw data notify interval msec = 0 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout: osd_max_omap_entries_per_request = 10 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout: osd_max_omap_entries_per_request = 1000 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2024-04-13T21:20:27.783 INFO:teuthology.orchestra.run.smithi132.stdout: osd_max_omap_entries_per_request = 10000 2024-04-13T21:20:27.802 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2024-04-13T21:20:27.802 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2024-04-13T21:20:27.802 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T21:20:27.802 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.803 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T21:20:27.804 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = 172.21.15.175 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 627f123d-57cb-4597-b0f1-8de1173b7a8b 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2024-04-13T21:20:27.805 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd mclock profile = high_recovery_ops 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2024-04-13T21:20:27.806 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 20 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: rocksdb delete range threshold = 0 2024-04-13T21:20:27.807 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2024-04-13T21:20:27.808 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.809 INFO:teuthology.orchestra.run.smithi175.stdout: # 1m isn't quite enough 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: mon_down_mkfs_grace = 2m 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_filestore_osds = false 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw = 20 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw sync = 20 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt require ssl = False 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms backend = testing 2024-04-13T21:20:27.810 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw curl low speed time = 300 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data log num shards = 4 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw md log max shards = 4 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync data inject err probability = 0.1 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync log trim interval = 0 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync meta inject err probability = 0.1 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync obj etag verify = True 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: setgroup = ceph 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: setuser = ceph 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data notify interval msec = 0 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2024-04-13T21:20:27.811 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10 2024-04-13T21:20:27.812 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2024-04-13T21:20:27.812 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 1000 2024-04-13T21:20:27.812 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2024-04-13T21:20:27.812 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10000 2024-04-13T21:20:27.814 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-04-13T21:20:27.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-04-13T21:20:27.911 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-13T21:20:27.911 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:27.911 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-04-13T21:20:27.929 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:27.929 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-04-13T21:20:27.985 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T21:20:27.985 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:27.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/c2.keyring 2024-04-13T21:20:27.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-13T21:20:28.042 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-13T21:20:28.042 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:28.102 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T21:20:28.102 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:28.102 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/c2.keyring 2024-04-13T21:20:28.102 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-13T21:20:28.139 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:28.139 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:28.195 INFO:tasks.ceph:Setting up mon nodes... 2024-04-13T21:20:28.195 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-13T21:20:28.195 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-04-13T21:20:28.289 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-04-13T21:20:28.292 INFO:tasks.ceph:Setting up mds nodes... 2024-04-13T21:20:28.292 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-13T21:20:28.292 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-04-13T21:20:28.336 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c2.client.0.keyring 2024-04-13T21:20:28.349 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-04-13T21:20:28.433 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c2.client.1.keyring 2024-04-13T21:20:28.446 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-13T21:20:28.446 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T21:20:28.446 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-04-13T21:20:28.512 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:28.512 INFO:tasks.ceph:role: c2.osd.0 2024-04-13T21:20:28.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T21:20:28.513 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-13T21:20:28.586 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T21:20:28.586 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T21:20:28.586 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T21:20:28.587 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T21:20:29.014 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-13T21:20:29.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-13T21:20:29.016 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-04-13T21:20:29.184 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-04-13T21:20:29.216 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-04-13T21:20:29.283 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:29.283 INFO:tasks.ceph:role: c2.osd.1 2024-04-13T21:20:29.284 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T21:20:29.284 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-13T21:20:29.357 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T21:20:29.357 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T21:20:29.357 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T21:20:29.357 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-13T21:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T21:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-13T21:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T21:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-13T21:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T21:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T21:20:29.793 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-13T21:20:29.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-13T21:20:29.794 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-04-13T21:20:29.898 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-04-13T21:20:29.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-04-13T21:20:29.998 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T21:20:29.998 INFO:tasks.ceph:role: c2.osd.2 2024-04-13T21:20:29.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T21:20:29.998 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-13T21:20:30.072 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T21:20:30.073 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T21:20:30.516 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-13T21:20:30.518 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-13T21:20:30.518 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-04-13T21:20:30.688 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-04-13T21:20:30.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:30.808 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:30.815+0000 7f9027be97c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-04-13T21:20:30.811 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:30.818+0000 7f9027be97c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-04-13T21:20:30.811 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:30.818+0000 7f9027be97c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-13T21:20:30.811 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:30.818+0000 7f9027be97c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-13T21:20:30.811 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:30.818+0000 7f9027be97c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-13T21:20:30.811 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:30.819+0000 7f9027be97c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-04-13T21:20:32.886 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-04-13T21:20:32.918 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:33.011 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:33.018+0000 7fa3323c57c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-04-13T21:20:33.012 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:33.018+0000 7fa3323c57c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-04-13T21:20:33.012 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:33.018+0000 7fa3323c57c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-13T21:20:33.012 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:33.018+0000 7fa3323c57c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-13T21:20:33.012 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:33.018+0000 7fa3323c57c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-13T21:20:33.012 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:33.019+0000 7fa3323c57c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-04-13T21:20:35.096 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-04-13T21:20:35.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:35.220 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:35.227+0000 7f6c485437c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-04-13T21:20:35.221 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:35.227+0000 7f6c485437c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-04-13T21:20:35.221 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:35.227+0000 7f6c485437c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-13T21:20:35.221 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:35.227+0000 7f6c485437c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-13T21:20:35.221 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:35.227+0000 7f6c485437c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-13T21:20:35.221 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:20:35.228+0000 7f6c485437c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-04-13T21:20:37.313 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-04-13T21:20:37.359 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-13T21:20:37.359 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:37.359 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-04-13T21:20:37.427 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:37.427 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-04-13T21:20:37.495 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:37.495 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-04-13T21:20:37.563 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:37.563 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-04-13T21:20:37.631 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:37.632 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-04-13T21:20:37.689 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:37.689 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-04-13T21:20:37.745 INFO:tasks.ceph:Adding keys to all mons... 2024-04-13T21:20:37.746 DEBUG:teuthology.orchestra.run.smithi175:> sudo tee -a /etc/ceph/c2.keyring 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.x] 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAc9xpm7XC+ERAAA0CfJrc5lWis92MwHUzmkA== 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAe9xpmoPCiMBAAZwsSW1yEI3pakQrNwhblUQ== 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAh9xpmzeIhARAAWzXDv9k2q3qWCdK+ajAgXw== 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAj9xpmnJGXDRAAUx2xs/IuXC2MzNAxBgLhfg== 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAc9xpm1JqBFBAA7E6dErPVCgO4cv0euGayNg== 2024-04-13T21:20:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:[client.1] 2024-04-13T21:20:37.812 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAc9xpm2jFLGhAA66mY+uPUrCMlIll0TUC4/A== 2024-04-13T21:20:37.812 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-13T21:20:37.906 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T21:20:37.959 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T21:20:38.013 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T21:20:38.067 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T21:20:38.123 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T21:20:38.180 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-13T21:20:38.180 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-04-13T21:20:38.207 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-04-13T21:20:38.305 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-04-13T21:20:38.332 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-04-13T21:20:38.388 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-04-13T21:20:38.389 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-04-13T21:20:38.389 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-04-13T21:20:38.431 INFO:tasks.ceph.c2.mon.a:Started 2024-04-13T21:20:38.431 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-04-13T21:20:38.431 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-04-13T21:20:38.431 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-04-13T21:20:38.433 INFO:tasks.ceph.c2.mgr.x:Started 2024-04-13T21:20:38.433 INFO:tasks.ceph:Setting crush tunables to default 2024-04-13T21:20:38.433 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 osd crush tunables default 2024-04-13T21:20:38.657 INFO:teuthology.orchestra.run.smithi175.stderr:adjusted tunables profile to default 2024-04-13T21:20:38.669 INFO:tasks.ceph:check_enable_crimson: False 2024-04-13T21:20:38.669 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-04-13T21:20:38.669 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:38.669 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-04-13T21:20:38.696 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:38.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-04-13T21:20:38.763 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-13T21:20:38.763 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-04-13T21:20:38.830 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 osd new 51793587-3769-437b-b4b4-fd0c23d314a1 0 2024-04-13T21:20:39.086 INFO:teuthology.orchestra.run.smithi175.stdout:0 2024-04-13T21:20:39.099 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 osd new b08d451c-14fd-45cb-afae-32b6b2e45b5c 1 2024-04-13T21:20:39.143 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.150+0000 7f3d9f3b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T21:20:39.227 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.234+0000 7f3d9f3b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T21:20:39.302 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.309+0000 7f3d9f3b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T21:20:39.315 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-13T21:20:39.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 osd new a02dce7f-eeea-4e11-a4a6-a679cbe46517 2 2024-04-13T21:20:39.377 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.385+0000 7f3d9f3b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T21:20:39.546 INFO:teuthology.orchestra.run.smithi175.stdout:2 2024-04-13T21:20:39.558 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-04-13T21:20:39.558 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-04-13T21:20:39.559 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.567+0000 7f3d9f3b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T21:20:39.560 INFO:tasks.ceph.c2.osd.0:Started 2024-04-13T21:20:39.560 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-04-13T21:20:39.560 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-04-13T21:20:39.562 INFO:tasks.ceph.c2.osd.1:Started 2024-04-13T21:20:39.563 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-04-13T21:20:39.563 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-04-13T21:20:39.564 INFO:tasks.ceph.c2.osd.2:Started 2024-04-13T21:20:39.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-13T21:20:39.753 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.760+0000 7f3d9f3b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T21:20:39.780 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:39.780 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":5,"fsid":"627f123d-57cb-4597-b0f1-8de1173b7a8b","created":"2024-04-13T21:20:38.570104+0000","modified":"2024-04-13T21:20:39.550815+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T21:20:39.550815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51793587-3769-437b-b4b4-fd0c23d314a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b08d451c-14fd-45cb-afae-32b6b2e45b5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a02dce7f-eeea-4e11-a4a6-a679cbe46517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:39.795 INFO:tasks.ceph.ceph_manager.c2:[] 2024-04-13T21:20:39.795 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-13T21:20:39.866 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.873+0000 7f3d9f3b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T21:20:39.960 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:39.968+0000 7f3d9f3b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T21:20:40.036 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:40.043+0000 7f3d9f3b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T21:20:40.098 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-13T21:20:40.175 INFO:tasks.ceph.c2.osd.2.smithi175.stderr:2024-04-13T21:20:40.182+0000 7f0daee147c0 -1 Falling back to public interface 2024-04-13T21:20:40.190 INFO:tasks.ceph.c2.osd.0.smithi175.stderr:2024-04-13T21:20:40.198+0000 7ff72007a7c0 -1 Falling back to public interface 2024-04-13T21:20:40.191 INFO:tasks.ceph.c2.osd.1.smithi175.stderr:2024-04-13T21:20:40.198+0000 7f7f3e1aa7c0 -1 Falling back to public interface 2024-04-13T21:20:40.277 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:40.285+0000 7f3d9f3b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T21:20:40.320 INFO:teuthology.misc.health.smithi175.stdout: 2024-04-13T21:20:40.320 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":5,"fsid":"627f123d-57cb-4597-b0f1-8de1173b7a8b","created":"2024-04-13T21:20:38.570104+0000","modified":"2024-04-13T21:20:39.550815+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T21:20:39.550815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51793587-3769-437b-b4b4-fd0c23d314a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b08d451c-14fd-45cb-afae-32b6b2e45b5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a02dce7f-eeea-4e11-a4a6-a679cbe46517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:40.332 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-13T21:20:40.351 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:40.358+0000 7f3d9f3b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T21:20:40.734 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:40.742+0000 7f3d9f3b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T21:20:41.057 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:41.065+0000 7f3d9f3b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T21:20:41.216 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:41.223+0000 7f3d9f3b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T21:20:41.295 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:41.303+0000 7f3d9f3b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T21:20:41.997 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:42.004+0000 7f3d9f3b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T21:20:42.079 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:42.086+0000 7f3d9f3b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T21:20:42.211 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:42.218+0000 7f3d9f3b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T21:20:42.769 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:42.776+0000 7f3d9f3b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T21:20:42.886 INFO:tasks.ceph.c2.osd.2.smithi175.stderr:2024-04-13T21:20:42.893+0000 7f0daee147c0 -1 osd.2 0 log_to_monitors true 2024-04-13T21:20:42.913 INFO:tasks.ceph.c2.osd.1.smithi175.stderr:2024-04-13T21:20:42.920+0000 7f7f3e1aa7c0 -1 osd.1 0 log_to_monitors true 2024-04-13T21:20:42.929 INFO:tasks.ceph.c2.osd.0.smithi175.stderr:2024-04-13T21:20:42.936+0000 7ff72007a7c0 -1 osd.0 0 log_to_monitors true 2024-04-13T21:20:42.947 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:42.955+0000 7f3d9f3b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T21:20:43.112 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:43.120+0000 7f3d9f3b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T21:20:43.924 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:43.932+0000 7f3d9f3b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T21:20:44.194 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:44.202+0000 7f3d9f3b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T21:20:44.294 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:44.301+0000 7f3d9f3b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T21:20:44.745 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:44.752+0000 7f3d9f3b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T21:20:45.101 INFO:tasks.ceph.c2.osd.2.smithi175.stderr:2024-04-13T21:20:45.109+0000 7f0da31cc700 -1 osd.2 0 waiting for initial osdmap 2024-04-13T21:20:45.111 INFO:tasks.ceph.c2.osd.2.smithi175.stderr:2024-04-13T21:20:45.119+0000 7f0d9ada6700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T21:20:45.182 INFO:tasks.ceph.c2.osd.0.smithi175.stderr:2024-04-13T21:20:45.189+0000 7ff714c33700 -1 osd.0 0 waiting for initial osdmap 2024-04-13T21:20:45.187 INFO:tasks.ceph.c2.osd.1.smithi175.stderr:2024-04-13T21:20:45.194+0000 7f7f32562700 -1 osd.1 0 waiting for initial osdmap 2024-04-13T21:20:45.191 INFO:tasks.ceph.c2.osd.0.smithi175.stderr:2024-04-13T21:20:45.199+0000 7ff70c00c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T21:20:45.196 INFO:tasks.ceph.c2.osd.1.smithi175.stderr:2024-04-13T21:20:45.204+0000 7f7f2a13c700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T21:20:45.851 INFO:tasks.ceph.c2.mgr.x.smithi175.stderr:2024-04-13T21:20:45.859+0000 7f3d5a633700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-13T21:20:46.635 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-13T21:20:46.973 INFO:teuthology.misc.health.smithi175.stdout: 2024-04-13T21:20:46.973 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":8,"fsid":"627f123d-57cb-4597-b0f1-8de1173b7a8b","created":"2024-04-13T21:20:38.570104+0000","modified":"2024-04-13T21:20:46.110406+0000","last_up_change":"2024-04-13T21:20:46.110406+0000","last_in_change":"2024-04-13T21:20:39.550815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51793587-3769-437b-b4b4-fd0c23d314a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":40882},{"type":"v1","addr":"172.21.15.175:6810","nonce":40882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":40882},{"type":"v1","addr":"172.21.15.175:6814","nonce":40882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":40882},{"type":"v1","addr":"172.21.15.175:6822","nonce":40882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":40882},{"type":"v1","addr":"172.21.15.175:6818","nonce":40882}]},"public_addr":"172.21.15.175:6810/40882","cluster_addr":"172.21.15.175:6814/40882","heartbeat_back_addr":"172.21.15.175:6822/40882","heartbeat_front_addr":"172.21.15.175:6818/40882","state":["exists","up"]},{"osd":1,"uuid":"b08d451c-14fd-45cb-afae-32b6b2e45b5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6809","nonce":40883},{"type":"v1","addr":"172.21.15.175:6811","nonce":40883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6813","nonce":40883},{"type":"v1","addr":"172.21.15.175:6815","nonce":40883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6821","nonce":40883},{"type":"v1","addr":"172.21.15.175:6823","nonce":40883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6817","nonce":40883},{"type":"v1","addr":"172.21.15.175:6819","nonce":40883}]},"public_addr":"172.21.15.175:6811/40883","cluster_addr":"172.21.15.175:6815/40883","heartbeat_back_addr":"172.21.15.175:6823/40883","heartbeat_front_addr":"172.21.15.175:6819/40883","state":["exists","up"]},{"osd":2,"uuid":"a02dce7f-eeea-4e11-a4a6-a679cbe46517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":40881},{"type":"v1","addr":"172.21.15.175:6801","nonce":40881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":40881},{"type":"v1","addr":"172.21.15.175:6803","nonce":40881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":40881},{"type":"v1","addr":"172.21.15.175:6807","nonce":40881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":40881},{"type":"v1","addr":"172.21.15.175:6805","nonce":40881}]},"public_addr":"172.21.15.175:6801/40881","cluster_addr":"172.21.15.175:6803/40881","heartbeat_back_addr":"172.21.15.175:6807/40881","heartbeat_front_addr":"172.21.15.175:6805/40881","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:46.987 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-13T21:20:46.987 INFO:tasks.ceph:Creating RBD pool 2024-04-13T21:20:46.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 osd pool create rbd 8 2024-04-13T21:20:48.119 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'rbd' created 2024-04-13T21:20:48.133 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-13T21:20:49.125 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-13T21:20:49.137 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-04-13T21:20:49.138 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-04-13T21:20:49.138 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-04-13T21:20:49.142 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-04-13T21:20:49.142 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-13T21:20:49.528 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:49.540 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":40501},{"type":"v1","addr":"172.21.15.175:6825","nonce":40501}]},"active_addr":"172.21.15.175:6825/40501","active_change":"2024-04-13T21:20:44.766280+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":1094961259}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":72779259}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":3566105925}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":887757270}]}]} 2024-04-13T21:20:49.545 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-04-13T21:20:49.545 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-04-13T21:20:49.545 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-13T21:20:49.874 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:49.874 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":11,"fsid":"627f123d-57cb-4597-b0f1-8de1173b7a8b","created":"2024-04-13T21:20:38.570104+0000","modified":"2024-04-13T21:20:49.122360+0000","last_up_change":"2024-04-13T21:20:46.110406+0000","last_in_change":"2024-04-13T21:20:39.550815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T21:20:46.896302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T21:20:47.339919+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51793587-3769-437b-b4b4-fd0c23d314a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":40882},{"type":"v1","addr":"172.21.15.175:6810","nonce":40882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":40882},{"type":"v1","addr":"172.21.15.175:6814","nonce":40882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":40882},{"type":"v1","addr":"172.21.15.175:6822","nonce":40882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":40882},{"type":"v1","addr":"172.21.15.175:6818","nonce":40882}]},"public_addr":"172.21.15.175:6810/40882","cluster_addr":"172.21.15.175:6814/40882","heartbeat_back_addr":"172.21.15.175:6822/40882","heartbeat_front_addr":"172.21.15.175:6818/40882","state":["exists","up"]},{"osd":1,"uuid":"b08d451c-14fd-45cb-afae-32b6b2e45b5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6809","nonce":40883},{"type":"v1","addr":"172.21.15.175:6811","nonce":40883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6813","nonce":40883},{"type":"v1","addr":"172.21.15.175:6815","nonce":40883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6821","nonce":40883},{"type":"v1","addr":"172.21.15.175:6823","nonce":40883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6817","nonce":40883},{"type":"v1","addr":"172.21.15.175:6819","nonce":40883}]},"public_addr":"172.21.15.175:6811/40883","cluster_addr":"172.21.15.175:6815/40883","heartbeat_back_addr":"172.21.15.175:6823/40883","heartbeat_front_addr":"172.21.15.175:6819/40883","state":["exists","up"]},{"osd":2,"uuid":"a02dce7f-eeea-4e11-a4a6-a679cbe46517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":40881},{"type":"v1","addr":"172.21.15.175:6801","nonce":40881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":40881},{"type":"v1","addr":"172.21.15.175:6803","nonce":40881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":40881},{"type":"v1","addr":"172.21.15.175:6807","nonce":40881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":40881},{"type":"v1","addr":"172.21.15.175:6805","nonce":40881}]},"public_addr":"172.21.15.175:6801/40881","cluster_addr":"172.21.15.175:6803/40881","heartbeat_back_addr":"172.21.15.175:6807/40881","heartbeat_front_addr":"172.21.15.175:6805/40881","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:43.906160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:43.884516+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:43.908685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:49.888 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-04-13T21:20:49.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-13T21:20:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:50.215 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":12,"fsid":"627f123d-57cb-4597-b0f1-8de1173b7a8b","created":"2024-04-13T21:20:38.570104+0000","modified":"2024-04-13T21:20:50.126879+0000","last_up_change":"2024-04-13T21:20:46.110406+0000","last_in_change":"2024-04-13T21:20:39.550815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T21:20:46.896302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T21:20:47.339919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51793587-3769-437b-b4b4-fd0c23d314a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":40882},{"type":"v1","addr":"172.21.15.175:6810","nonce":40882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":40882},{"type":"v1","addr":"172.21.15.175:6814","nonce":40882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":40882},{"type":"v1","addr":"172.21.15.175:6822","nonce":40882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":40882},{"type":"v1","addr":"172.21.15.175:6818","nonce":40882}]},"public_addr":"172.21.15.175:6810/40882","cluster_addr":"172.21.15.175:6814/40882","heartbeat_back_addr":"172.21.15.175:6822/40882","heartbeat_front_addr":"172.21.15.175:6818/40882","state":["exists","up"]},{"osd":1,"uuid":"b08d451c-14fd-45cb-afae-32b6b2e45b5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6809","nonce":40883},{"type":"v1","addr":"172.21.15.175:6811","nonce":40883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6813","nonce":40883},{"type":"v1","addr":"172.21.15.175:6815","nonce":40883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6821","nonce":40883},{"type":"v1","addr":"172.21.15.175:6823","nonce":40883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6817","nonce":40883},{"type":"v1","addr":"172.21.15.175:6819","nonce":40883}]},"public_addr":"172.21.15.175:6811/40883","cluster_addr":"172.21.15.175:6815/40883","heartbeat_back_addr":"172.21.15.175:6823/40883","heartbeat_front_addr":"172.21.15.175:6819/40883","state":["exists","up"]},{"osd":2,"uuid":"a02dce7f-eeea-4e11-a4a6-a679cbe46517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":40881},{"type":"v1","addr":"172.21.15.175:6801","nonce":40881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":40881},{"type":"v1","addr":"172.21.15.175:6803","nonce":40881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":40881},{"type":"v1","addr":"172.21.15.175:6807","nonce":40881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":40881},{"type":"v1","addr":"172.21.15.175:6805","nonce":40881}]},"public_addr":"172.21.15.175:6801/40881","cluster_addr":"172.21.15.175:6803/40881","heartbeat_back_addr":"172.21.15.175:6807/40881","heartbeat_front_addr":"172.21.15.175:6805/40881","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:43.906160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:43.884516+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T21:20:43.908685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T21:20:50.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-04-13T21:20:50.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-04-13T21:20:50.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-04-13T21:20:50.397 INFO:teuthology.orchestra.run.smithi175.stdout:34359738370 2024-04-13T21:20:50.397 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-13T21:20:50.406 INFO:teuthology.orchestra.run.smithi175.stdout:34359738370 2024-04-13T21:20:50.406 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-13T21:20:50.407 INFO:teuthology.orchestra.run.smithi175.stdout:34359738370 2024-04-13T21:20:50.407 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-13T21:20:50.773 INFO:teuthology.orchestra.run.smithi175.stdout:34359738369 2024-04-13T21:20:50.788 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2024-04-13T21:20:50.790 INFO:teuthology.orchestra.run.smithi175.stdout:34359738369 2024-04-13T21:20:50.806 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-04-13T21:20:50.817 INFO:teuthology.orchestra.run.smithi175.stdout:34359738369 2024-04-13T21:20:50.833 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2024-04-13T21:20:51.789 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-13T21:20:51.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-13T21:20:51.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-13T21:20:52.155 INFO:teuthology.orchestra.run.smithi175.stdout:34359738370 2024-04-13T21:20:52.169 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2024-04-13T21:20:52.170 DEBUG:teuthology.parallel:result is None 2024-04-13T21:20:52.188 INFO:teuthology.orchestra.run.smithi175.stdout:34359738370 2024-04-13T21:20:52.193 INFO:teuthology.orchestra.run.smithi175.stdout:34359738370 2024-04-13T21:20:52.204 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2024-04-13T21:20:52.204 DEBUG:teuthology.parallel:result is None 2024-04-13T21:20:52.209 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2024-04-13T21:20:52.209 DEBUG:teuthology.parallel:result is None 2024-04-13T21:20:52.209 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-13T21:20:52.209 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-13T21:20:52.572 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:52.574 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2024-04-13T21:20:52.585 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-13T21:20:50.856567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82764,"kb_used_data":1444,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490036,"statfs":{"total":322122547200,"available":322037796864,"internally_reserved":0,"allocated":1478656,"data_stored":1257202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:49.144056+0000","last_change":"2024-04-13T21:20:49.131818+0000","last_active":"2024-04-13T21:20:49.144056+0000","last_peered":"2024-04-13T21:20:49.144056+0000","last_clean":"2024-04-13T21:20:49.144056+0000","last_became_active":"2024-04-13T21:20:49.131548+0000","last_became_peered":"2024-04-13T21:20:49.131548+0000","last_unstale":"2024-04-13T21:20:49.144056+0000","last_undegraded":"2024-04-13T21:20:49.144056+0000","last_fullsized":"2024-04-13T21:20:49.144056+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:32:02.859769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:49.144048+0000","last_change":"2024-04-13T21:20:49.132109+0000","last_active":"2024-04-13T21:20:49.144048+0000","last_peered":"2024-04-13T21:20:49.144048+0000","last_clean":"2024-04-13T21:20:49.144048+0000","last_became_active":"2024-04-13T21:20:49.131525+0000","last_became_peered":"2024-04-13T21:20:49.131525+0000","last_unstale":"2024-04-13T21:20:49.144048+0000","last_undegraded":"2024-04-13T21:20:49.144048+0000","last_fullsized":"2024-04-13T21:20:49.144048+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:06:13.224356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:49.144055+0000","last_change":"2024-04-13T21:20:49.131972+0000","last_active":"2024-04-13T21:20:49.144055+0000","last_peered":"2024-04-13T21:20:49.144055+0000","last_clean":"2024-04-13T21:20:49.144055+0000","last_became_active":"2024-04-13T21:20:49.131778+0000","last_became_peered":"2024-04-13T21:20:49.131778+0000","last_unstale":"2024-04-13T21:20:49.144055+0000","last_undegraded":"2024-04-13T21:20:49.144055+0000","last_fullsized":"2024-04-13T21:20:49.144055+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T02:49:15.583370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.129187+0000","last_change":"2024-04-13T21:20:49.129995+0000","last_active":"2024-04-13T21:20:50.129187+0000","last_peered":"2024-04-13T21:20:50.129187+0000","last_clean":"2024-04-13T21:20:50.129187+0000","last_became_active":"2024-04-13T21:20:49.129791+0000","last_became_peered":"2024-04-13T21:20:49.129791+0000","last_unstale":"2024-04-13T21:20:50.129187+0000","last_undegraded":"2024-04-13T21:20:50.129187+0000","last_fullsized":"2024-04-13T21:20:50.129187+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T08:44:54.866448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:49.125269+0000","last_change":"2024-04-13T21:20:48.129186+0000","last_active":"2024-04-13T21:20:49.125269+0000","last_peered":"2024-04-13T21:20:49.125269+0000","last_clean":"2024-04-13T21:20:49.125269+0000","last_became_active":"2024-04-13T21:20:48.128370+0000","last_became_peered":"2024-04-13T21:20:48.128370+0000","last_unstale":"2024-04-13T21:20:49.125269+0000","last_undegraded":"2024-04-13T21:20:49.125269+0000","last_fullsized":"2024-04-13T21:20:49.125269+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:47.114412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:47.114412+0000","last_clean_scrub_stamp":"2024-04-13T21:20:47.114412+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:36:37.784935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:49.130305+0000","last_change":"2024-04-13T21:20:49.129230+0000","last_active":"2024-04-13T21:20:49.130305+0000","last_peered":"2024-04-13T21:20:49.130305+0000","last_clean":"2024-04-13T21:20:49.130305+0000","last_became_active":"2024-04-13T21:20:49.128975+0000","last_became_peered":"2024-04-13T21:20:49.128975+0000","last_unstale":"2024-04-13T21:20:49.130305+0000","last_undegraded":"2024-04-13T21:20:49.130305+0000","last_fullsized":"2024-04-13T21:20:49.130305+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T01:10:05.432761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.129172+0000","last_change":"2024-04-13T21:20:49.130960+0000","last_active":"2024-04-13T21:20:50.129172+0000","last_peered":"2024-04-13T21:20:50.129172+0000","last_clean":"2024-04-13T21:20:50.129172+0000","last_became_active":"2024-04-13T21:20:49.130696+0000","last_became_peered":"2024-04-13T21:20:49.130696+0000","last_unstale":"2024-04-13T21:20:50.129172+0000","last_undegraded":"2024-04-13T21:20:50.129172+0000","last_fullsized":"2024-04-13T21:20:50.129172+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T06:51:23.877297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.397949+0000","last_change":"2024-04-13T21:20:49.130815+0000","last_active":"2024-04-13T21:20:50.397949+0000","last_peered":"2024-04-13T21:20:50.397949+0000","last_clean":"2024-04-13T21:20:50.397949+0000","last_became_active":"2024-04-13T21:20:49.130580+0000","last_became_peered":"2024-04-13T21:20:49.130580+0000","last_unstale":"2024-04-13T21:20:50.397949+0000","last_undegraded":"2024-04-13T21:20:50.397949+0000","last_fullsized":"2024-04-13T21:20:50.397949+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:36:38.787815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:49.144047+0000","last_change":"2024-04-13T21:20:49.131831+0000","last_active":"2024-04-13T21:20:49.144047+0000","last_peered":"2024-04-13T21:20:49.144047+0000","last_clean":"2024-04-13T21:20:49.144047+0000","last_became_active":"2024-04-13T21:20:49.131565+0000","last_became_peered":"2024-04-13T21:20:49.131565+0000","last_unstale":"2024-04-13T21:20:49.144047+0000","last_undegraded":"2024-04-13T21:20:49.144047+0000","last_fullsized":"2024-04-13T21:20:49.144047+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T05:42:47.141386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":98304,"data_stored":25745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27820,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":688128,"data_stored":615344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27824,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829776,"statfs":{"total":107374182400,"available":107345690624,"internally_reserved":0,"allocated":692224,"data_stored":616113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T21:20:52.586 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-13T21:20:52.909 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:52.911 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2024-04-13T21:20:52.923 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-13T21:20:52.856940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82800,"kb_used_data":1468,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490000,"statfs":{"total":322122547200,"available":322037760000,"internally_reserved":0,"allocated":1503232,"data_stored":1267636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000373"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.390518+0000","last_change":"2024-04-13T21:20:49.131818+0000","last_active":"2024-04-13T21:20:50.390518+0000","last_peered":"2024-04-13T21:20:50.390518+0000","last_clean":"2024-04-13T21:20:50.390518+0000","last_became_active":"2024-04-13T21:20:49.131548+0000","last_became_peered":"2024-04-13T21:20:49.131548+0000","last_unstale":"2024-04-13T21:20:50.390518+0000","last_undegraded":"2024-04-13T21:20:50.390518+0000","last_fullsized":"2024-04-13T21:20:50.390518+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:32:02.859769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.390501+0000","last_change":"2024-04-13T21:20:49.132109+0000","last_active":"2024-04-13T21:20:50.390501+0000","last_peered":"2024-04-13T21:20:50.390501+0000","last_clean":"2024-04-13T21:20:50.390501+0000","last_became_active":"2024-04-13T21:20:49.131525+0000","last_became_peered":"2024-04-13T21:20:49.131525+0000","last_unstale":"2024-04-13T21:20:50.390501+0000","last_undegraded":"2024-04-13T21:20:50.390501+0000","last_fullsized":"2024-04-13T21:20:50.390501+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:06:13.224356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.390483+0000","last_change":"2024-04-13T21:20:49.131972+0000","last_active":"2024-04-13T21:20:50.390483+0000","last_peered":"2024-04-13T21:20:50.390483+0000","last_clean":"2024-04-13T21:20:50.390483+0000","last_became_active":"2024-04-13T21:20:49.131778+0000","last_became_peered":"2024-04-13T21:20:49.131778+0000","last_unstale":"2024-04-13T21:20:50.390483+0000","last_undegraded":"2024-04-13T21:20:50.390483+0000","last_fullsized":"2024-04-13T21:20:50.390483+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T02:49:15.583370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.129187+0000","last_change":"2024-04-13T21:20:49.129995+0000","last_active":"2024-04-13T21:20:50.129187+0000","last_peered":"2024-04-13T21:20:50.129187+0000","last_clean":"2024-04-13T21:20:50.129187+0000","last_became_active":"2024-04-13T21:20:49.129791+0000","last_became_peered":"2024-04-13T21:20:49.129791+0000","last_unstale":"2024-04-13T21:20:50.129187+0000","last_undegraded":"2024-04-13T21:20:50.129187+0000","last_fullsized":"2024-04-13T21:20:50.129187+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T08:44:54.866448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.390417+0000","last_change":"2024-04-13T21:20:48.129186+0000","last_active":"2024-04-13T21:20:50.390417+0000","last_peered":"2024-04-13T21:20:50.390417+0000","last_clean":"2024-04-13T21:20:50.390417+0000","last_became_active":"2024-04-13T21:20:48.128370+0000","last_became_peered":"2024-04-13T21:20:48.128370+0000","last_unstale":"2024-04-13T21:20:50.390417+0000","last_undegraded":"2024-04-13T21:20:50.390417+0000","last_fullsized":"2024-04-13T21:20:50.390417+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:47.114412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:47.114412+0000","last_clean_scrub_stamp":"2024-04-13T21:20:47.114412+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:36:37.784935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.390443+0000","last_change":"2024-04-13T21:20:49.129230+0000","last_active":"2024-04-13T21:20:50.390443+0000","last_peered":"2024-04-13T21:20:50.390443+0000","last_clean":"2024-04-13T21:20:50.390443+0000","last_became_active":"2024-04-13T21:20:49.128975+0000","last_became_peered":"2024-04-13T21:20:49.128975+0000","last_unstale":"2024-04-13T21:20:50.390443+0000","last_undegraded":"2024-04-13T21:20:50.390443+0000","last_fullsized":"2024-04-13T21:20:50.390443+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T01:10:05.432761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.129172+0000","last_change":"2024-04-13T21:20:49.130960+0000","last_active":"2024-04-13T21:20:50.129172+0000","last_peered":"2024-04-13T21:20:50.129172+0000","last_clean":"2024-04-13T21:20:50.129172+0000","last_became_active":"2024-04-13T21:20:49.130696+0000","last_became_peered":"2024-04-13T21:20:49.130696+0000","last_unstale":"2024-04-13T21:20:50.129172+0000","last_undegraded":"2024-04-13T21:20:50.129172+0000","last_fullsized":"2024-04-13T21:20:50.129172+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T06:51:23.877297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.397949+0000","last_change":"2024-04-13T21:20:49.130815+0000","last_active":"2024-04-13T21:20:50.397949+0000","last_peered":"2024-04-13T21:20:50.397949+0000","last_clean":"2024-04-13T21:20:50.397949+0000","last_became_active":"2024-04-13T21:20:49.130580+0000","last_became_peered":"2024-04-13T21:20:49.130580+0000","last_unstale":"2024-04-13T21:20:50.397949+0000","last_undegraded":"2024-04-13T21:20:50.397949+0000","last_fullsized":"2024-04-13T21:20:50.397949+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:36:38.787815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-13T21:20:50.390465+0000","last_change":"2024-04-13T21:20:49.131831+0000","last_active":"2024-04-13T21:20:50.390465+0000","last_peered":"2024-04-13T21:20:50.390465+0000","last_clean":"2024-04-13T21:20:50.390465+0000","last_became_active":"2024-04-13T21:20:49.131565+0000","last_became_peered":"2024-04-13T21:20:49.131565+0000","last_unstale":"2024-04-13T21:20:50.390465+0000","last_undegraded":"2024-04-13T21:20:50.390465+0000","last_fullsized":"2024-04-13T21:20:50.390465+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T21:20:48.117292+0000","last_clean_scrub_stamp":"2024-04-13T21:20:48.117292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T05:42:47.141386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":106496,"data_stored":29223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27832,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829768,"statfs":{"total":107374182400,"available":107345682432,"internally_reserved":0,"allocated":696320,"data_stored":618822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27836,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":700416,"data_stored":619591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T21:20:52.924 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-04-13T21:20:52.924 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-04-13T21:20:52.924 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-04-13T21:20:52.924 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-04-13T21:20:53.292 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-13T21:20:53.292 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-13T21:20:53.306 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-04-13T21:20:53.307 INFO:teuthology.run_tasks:Running task rgw... 2024-04-13T21:20:53.322 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-04-13T21:20:53.322 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-04-13T21:20:53.322 INFO:tasks.rgw:Creating data pools 2024-04-13T21:20:53.322 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-04-13T21:20:53.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-13T21:20:54.211 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'default.rgw.buckets.data' created 2024-04-13T21:20:54.220 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-13T21:20:55.219 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-13T21:20:55.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-13T21:20:56.212 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'default.rgw.buckets.index' created 2024-04-13T21:20:56.229 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-13T21:20:57.225 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-13T21:20:57.244 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-04-13T21:20:57.244 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-13T21:20:57.573 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-13T21:20:57.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-13T21:20:58.219 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-13T21:20:58.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-13T21:20:58.560 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-13T21:20:58.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-13T21:20:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-13T21:20:59.233 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-04-13T21:20:59.233 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-13T21:21:00.145 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.data' created 2024-04-13T21:21:00.162 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-13T21:21:01.151 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-13T21:21:01.172 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-13T21:21:02.156 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.index' created 2024-04-13T21:21:02.172 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-13T21:21:03.166 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-13T21:21:03.189 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-04-13T21:21:03.190 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-13T21:21:03.519 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-13T21:21:03.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-13T21:21:04.156 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-13T21:21:04.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-13T21:21:04.498 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-13T21:21:04.509 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-13T21:21:05.159 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-13T21:21:05.171 DEBUG:tasks.rgw:Pools created 2024-04-13T21:21:05.171 INFO:tasks.rgw:Configuring compression type = random 2024-04-13T21:21:05.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-04-13T21:21:05.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-04-13T21:21:05.171 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-04-13T21:21:05.250 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:05.250 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:05.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:05.266+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.255+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.255+0000 7fba2af3f780 20 realm 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.255+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.256+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.256+0000 7fba2af3f780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.256+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.256+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.256+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.262+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.262+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.262+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.263+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.263+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.263+0000 7fba2af3f780 10 Creating default zonegroup 2024-04-13T21:21:07.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.264+0000 7fba2af3f780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-13T21:21:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.273+0000 7fba2af3f780 10 WARNING: driver->list_zones() returned r=-2 2024-04-13T21:21:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.273+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.274+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.278+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.279+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.279+0000 7fba2af3f780 10 could not read realm id: (2) No such file or directory 2024-04-13T21:21:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.279+0000 7fba2af3f780 10 WARNING: failed to set zone as default, r=-22 2024-04-13T21:21:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.279+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.279+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.281+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.281+0000 7fba2af3f780 20 rados_obj.operate() r=0 bl.length=874 2024-04-13T21:21:07.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.281+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.282+0000 7fba2af3f780 20 rados_obj.operate() r=0 bl.length=420 2024-04-13T21:21:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.282+0000 7fba2af3f780 10 Using default name default 2024-04-13T21:21:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.282+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 rados_obj.operate() r=0 bl.length=46 2024-04-13T21:21:07.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 rados_obj.operate() r=0 bl.length=874 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 zone default found 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 4 Realm: () 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 4 ZoneGroup: default (e65077be-b818-426a-a0ba-3a94bde5020e) 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 4 Zone: default (5cf9713e-fde4-4eda-9275-73a5d840a748) 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 zonegroup default 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:07.283+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:09.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:09.269+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:09.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:09.269+0000 7fba2af3f780 20 rados->read ofs=0 len=0 2024-04-13T21:21:09.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:09.271+0000 7fba2af3f780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:09.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:09.271+0000 7fba2af3f780 20 started sync module instance, tier type = 2024-04-13T21:21:09.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:09.271+0000 7fba2af3f780 20 started zone id=5cf9713e-fde4-4eda-9275-73a5d840a748 (name=default) with tier type = 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=0 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=1 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=2 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=3 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=4 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=5 2024-04-13T21:21:11.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.310+0000 7fba2af3f780 20 add_watcher() i=6 2024-04-13T21:21:11.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.312+0000 7fba2af3f780 20 add_watcher() i=7 2024-04-13T21:21:11.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.312+0000 7fba2af3f780 2 all 8 watchers are set, enabling cache 2024-04-13T21:21:11.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.312+0000 7fba2af3f780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-13T21:21:11.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.312+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-13T21:21:11.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.312+0000 7fba2af3f780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:11.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-13T21:21:11.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-13T21:21:11.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:11.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-13T21:21:11.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.313+0000 7fba2af3f780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-13T21:21:11.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.315+0000 7fba2af3f780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:11.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.320+0000 7fba0feba700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:11.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.320+0000 7fba2af3f780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:11.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:11.320+0000 7fba2af3f780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:14.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.276+0000 7fba2af3f780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-13T21:21:14.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.276+0000 7fba2af3f780 5 note: GC not initialized 2024-04-13T21:21:14.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.278+0000 7fba2af3f780 20 init_complete bucket index max shards: 11 2024-04-13T21:21:14.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.278+0000 7fba2af3f780 10 rgw notify: Started notification manager with: 1 workers 2024-04-13T21:21:14.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.278+0000 7fba096ad700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 13 21:21:44 2024 2024-04-13T21:21:14.272 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-04-13T21:21:14.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.278+0000 7fba08e8b700 20 reqs_thread_entry: start 2024-04-13T21:21:14.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.278+0000 7fba0b6b1700 20 reqs_thread_entry: start 2024-04-13T21:21:14.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.300+0000 7fba0feba700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:14.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.302+0000 7fba2af3f780 20 remove_watcher() i=0 2024-04-13T21:21:14.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.302+0000 7fba2af3f780 2 removed watcher, disabling cache 2024-04-13T21:21:14.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.304+0000 7fba2af3f780 20 remove_watcher() i=1 2024-04-13T21:21:14.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.307+0000 7fba2af3f780 20 remove_watcher() i=2 2024-04-13T21:21:14.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.310+0000 7fba2af3f780 20 remove_watcher() i=3 2024-04-13T21:21:14.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.313+0000 7fba2af3f780 20 remove_watcher() i=4 2024-04-13T21:21:14.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.314+0000 7fba2af3f780 20 remove_watcher() i=5 2024-04-13T21:21:14.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.315+0000 7fba2af3f780 20 remove_watcher() i=6 2024-04-13T21:21:14.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.316+0000 7fba2af3f780 20 remove_watcher() i=7 2024-04-13T21:21:14.311 INFO:teuthology.orchestra.run.smithi132.stdout:[] 2024-04-13T21:21:14.313 DEBUG:tasks.util.rgw: json result: [] 2024-04-13T21:21:14.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:14.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:14.314 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-13T21:21:14.366 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:14.366 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:14.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.383+0000 7f338c8aa780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:14.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.383+0000 7f338c8aa780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:14.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.387+0000 7f336e01e700 20 reqs_thread_entry: start 2024-04-13T21:21:14.386 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:14.386 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5cf9713e-fde4-4eda-9275-73a5d840a748", 2024-04-13T21:21:14.386 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "default.rgw.meta:root", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "default.rgw.control", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "default.rgw.log", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-13T21:21:14.387 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "default.rgw.otp", 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "", 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "" 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-13T21:21:14.388 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "", 2024-04-13T21:21:14.389 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-13T21:21:14.390 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:14.391 DEBUG:tasks.util.rgw: json result: {'id': '5cf9713e-fde4-4eda-9275-73a5d840a748', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-13T21:21:14.391 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-04-13T21:21:14.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-04-13T21:21:14.391 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-04-13T21:21:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:14.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.449+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 realm 2024-04-13T21:21:14.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados_obj.operate() r=0 bl.length=46 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 rados_obj.operate() r=0 bl.length=890 2024-04-13T21:21:14.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.451+0000 7fea5867a780 20 searching for the correct realm 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 RGWRados::pool_iterate: got zone_info.5cf9713e-fde4-4eda-9275-73a5d840a748 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 RGWRados::pool_iterate: got zonegroup_info.e65077be-b818-426a-a0ba-3a94bde5020e 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 rados_obj.operate() r=0 bl.length=46 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.459+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados_obj.operate() r=0 bl.length=420 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 zone default found 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 4 Realm: () 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 4 ZoneGroup: default (e65077be-b818-426a-a0ba-3a94bde5020e) 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 4 Zone: default (5cf9713e-fde4-4eda-9275-73a5d840a748) 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 zonegroup default 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados->read ofs=0 len=0 2024-04-13T21:21:14.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:14.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 started sync module instance, tier type = 2024-04-13T21:21:14.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.460+0000 7fea5867a780 20 started zone id=5cf9713e-fde4-4eda-9275-73a5d840a748 (name=default) with tier type = 2024-04-13T21:21:14.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=0 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=1 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=2 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=3 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=4 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=5 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.472+0000 7fea5867a780 20 add_watcher() i=6 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.473+0000 7fea5867a780 20 add_watcher() i=7 2024-04-13T21:21:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.473+0000 7fea5867a780 2 all 8 watchers are set, enabling cache 2024-04-13T21:21:14.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.474+0000 7fea3d5f5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:14.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.474+0000 7fea5867a780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:14.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.474+0000 7fea5867a780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:14.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.478+0000 7fea5867a780 5 note: GC not initialized 2024-04-13T21:21:14.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.479+0000 7fea38dec700 20 reqs_thread_entry: start 2024-04-13T21:21:14.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.479+0000 7fea5867a780 20 init_complete bucket index max shards: 11 2024-04-13T21:21:14.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.479+0000 7fea5867a780 10 rgw notify: Started notification manager with: 1 workers 2024-04-13T21:21:14.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.479+0000 7fea36de8700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 13 21:21:44 2024 2024-04-13T21:21:14.472 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-04-13T21:21:14.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.479+0000 7fea365c6700 20 reqs_thread_entry: start 2024-04-13T21:21:14.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.486+0000 7fea3d5f5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:14.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.488+0000 7fea5867a780 20 remove_watcher() i=0 2024-04-13T21:21:14.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.488+0000 7fea5867a780 2 removed watcher, disabling cache 2024-04-13T21:21:14.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.488+0000 7fea5867a780 20 remove_watcher() i=1 2024-04-13T21:21:14.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.489+0000 7fea5867a780 20 remove_watcher() i=2 2024-04-13T21:21:14.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.490+0000 7fea5867a780 20 remove_watcher() i=3 2024-04-13T21:21:14.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.491+0000 7fea5867a780 20 remove_watcher() i=4 2024-04-13T21:21:14.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.492+0000 7fea5867a780 20 remove_watcher() i=5 2024-04-13T21:21:14.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.493+0000 7fea5867a780 20 remove_watcher() i=6 2024-04-13T21:21:14.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.494+0000 7fea5867a780 20 remove_watcher() i=7 2024-04-13T21:21:14.488 INFO:teuthology.orchestra.run.smithi132.stdout:[] 2024-04-13T21:21:14.490 DEBUG:tasks.util.rgw: json result: [] 2024-04-13T21:21:14.490 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:14.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:14.491 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-13T21:21:14.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:14.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:14.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.595+0000 7f353dc9a780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:14.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.595+0000 7f353dc9a780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:14.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:14.598+0000 7f351f40e700 20 reqs_thread_entry: start 2024-04-13T21:21:14.597 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:14.597 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5cf9713e-fde4-4eda-9275-73a5d840a748", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "default.rgw.meta:root", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "default.rgw.control", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "default.rgw.log", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-13T21:21:14.598 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "default.rgw.otp", 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "", 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "" 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-13T21:21:14.599 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "", 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-13T21:21:14.600 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:14.601 DEBUG:tasks.util.rgw: json result: {'id': '5cf9713e-fde4-4eda-9275-73a5d840a748', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-13T21:21:14.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-04-13T21:21:14.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-04-13T21:21:14.602 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-04-13T21:21:14.671 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:14.671 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:14.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:14.688+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.193+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.193+0000 7f61db459780 20 realm 2024-04-13T21:21:16.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.193+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.195+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.195+0000 7f61db459780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-13T21:21:16.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.195+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.189 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.195+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.189 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.195+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.196+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.196+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.196+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.196+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.196+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.196+0000 7f61db459780 10 Creating default zonegroup 2024-04-13T21:21:16.190 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.197+0000 7f61db459780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-13T21:21:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.205+0000 7f61db459780 10 WARNING: driver->list_zones() returned r=-2 2024-04-13T21:21:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.205+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.205+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.209+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.209+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.209+0000 7f61db459780 10 could not read realm id: (2) No such file or directory 2024-04-13T21:21:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.209+0000 7f61db459780 10 WARNING: failed to set zone as default, r=-22 2024-04-13T21:21:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.209+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.209+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.212+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.212+0000 7f61db459780 20 rados_obj.operate() r=0 bl.length=874 2024-04-13T21:21:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.212+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados_obj.operate() r=0 bl.length=420 2024-04-13T21:21:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 10 Using default name default 2024-04-13T21:21:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados_obj.operate() r=0 bl.length=46 2024-04-13T21:21:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados_obj.operate() r=0 bl.length=874 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 zone default found 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 4 Realm: () 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 4 ZoneGroup: default (b82a02b6-ada4-4aff-9536-cf2b4550fc24) 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 4 Zone: default (57276bbe-b8c4-4c89-b257-291bacdcb286) 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 zonegroup default 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:16.213+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:18.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:18.203+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:18.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:18.203+0000 7f61db459780 20 rados->read ofs=0 len=0 2024-04-13T21:21:18.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:18.209+0000 7f61db459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:18.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:18.209+0000 7f61db459780 20 started sync module instance, tier type = 2024-04-13T21:21:18.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:18.209+0000 7f61db459780 20 started zone id=57276bbe-b8c4-4c89-b257-291bacdcb286 (name=default) with tier type = 2024-04-13T21:21:20.242 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=0 2024-04-13T21:21:20.243 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=1 2024-04-13T21:21:20.243 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=2 2024-04-13T21:21:20.243 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=3 2024-04-13T21:21:20.243 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=4 2024-04-13T21:21:20.243 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=5 2024-04-13T21:21:20.243 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.249+0000 7f61db459780 20 add_watcher() i=6 2024-04-13T21:21:20.245 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.251+0000 7f61db459780 20 add_watcher() i=7 2024-04-13T21:21:20.245 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.251+0000 7f61db459780 2 all 8 watchers are set, enabling cache 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-13T21:21:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-13T21:21:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-13T21:21:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-13T21:21:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.252+0000 7f61db459780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-13T21:21:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.253+0000 7f61db459780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-13T21:21:20.254 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.259+0000 7f61c03d4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:20.254 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.259+0000 7f61db459780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:20.254 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:20.259+0000 7f61db459780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:23.199 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.205+0000 7f61db459780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-13T21:21:23.199 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.205+0000 7f61db459780 5 note: GC not initialized 2024-04-13T21:21:23.200 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.207+0000 7f61bbbcb700 20 reqs_thread_entry: start 2024-04-13T21:21:23.200 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.207+0000 7f61db459780 20 init_complete bucket index max shards: 11 2024-04-13T21:21:23.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.207+0000 7f61db459780 10 rgw notify: Started notification manager with: 1 workers 2024-04-13T21:21:23.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.207+0000 7f61b9bc7700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 13 21:21:53 2024 2024-04-13T21:21:23.201 INFO:teuthology.orchestra.run.smithi175.stderr: 2024-04-13T21:21:23.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.207+0000 7f61b93a5700 20 reqs_thread_entry: start 2024-04-13T21:21:23.222 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.228+0000 7f61c03d4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:23.225 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.231+0000 7f61db459780 20 remove_watcher() i=0 2024-04-13T21:21:23.225 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.231+0000 7f61db459780 2 removed watcher, disabling cache 2024-04-13T21:21:23.226 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.232+0000 7f61db459780 20 remove_watcher() i=1 2024-04-13T21:21:23.227 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.233+0000 7f61db459780 20 remove_watcher() i=2 2024-04-13T21:21:23.227 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.234+0000 7f61db459780 20 remove_watcher() i=3 2024-04-13T21:21:23.228 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.235+0000 7f61db459780 20 remove_watcher() i=4 2024-04-13T21:21:23.229 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.236+0000 7f61db459780 20 remove_watcher() i=5 2024-04-13T21:21:23.230 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.237+0000 7f61db459780 20 remove_watcher() i=6 2024-04-13T21:21:23.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.238+0000 7f61db459780 20 remove_watcher() i=7 2024-04-13T21:21:23.234 INFO:teuthology.orchestra.run.smithi175.stdout:[] 2024-04-13T21:21:23.236 DEBUG:tasks.util.rgw: json result: [] 2024-04-13T21:21:23.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:23.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:23.237 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-13T21:21:23.288 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:23.288 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:23.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.303+0000 7fa9f570f780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:23.297 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.303+0000 7fa9f570f780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.307+0000 7fa9d6e83700 20 reqs_thread_entry: start 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "57276bbe-b8c4-4c89-b257-291bacdcb286", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "default.rgw.meta:root", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "default.rgw.control", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "default.rgw.log", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-13T21:21:23.307 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "default.rgw.otp", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-13T21:21:23.308 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-13T21:21:23.309 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:21:23.310 DEBUG:tasks.util.rgw: json result: {'id': '57276bbe-b8c4-4c89-b257-291bacdcb286', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-13T21:21:23.310 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-04-13T21:21:23.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-04-13T21:21:23.311 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-04-13T21:21:23.353 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:23.353 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:23.362 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.363 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.363 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 20 realm 2024-04-13T21:21:23.363 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.369+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.370+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.370+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.370+0000 7f4d78690780 20 rados_obj.operate() r=0 bl.length=46 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.370+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.364 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.371+0000 7f4d78690780 20 rados_obj.operate() r=0 bl.length=890 2024-04-13T21:21:23.365 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.371+0000 7f4d78690780 20 searching for the correct realm 2024-04-13T21:21:23.371 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.377+0000 7f4d78690780 20 RGWRados::pool_iterate: got zone_info.57276bbe-b8c4-4c89-b257-291bacdcb286 2024-04-13T21:21:23.371 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.377+0000 7f4d78690780 20 RGWRados::pool_iterate: got zonegroup_info.b82a02b6-ada4-4aff-9536-cf2b4550fc24 2024-04-13T21:21:23.371 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.377+0000 7f4d78690780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-13T21:21:23.371 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.377+0000 7f4d78690780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-13T21:21:23.371 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.377+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.372 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.372 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.372 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados_obj.operate() r=0 bl.length=46 2024-04-13T21:21:23.372 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados_obj.operate() r=0 bl.length=420 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 zone default found 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 4 Realm: () 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 4 ZoneGroup: default (b82a02b6-ada4-4aff-9536-cf2b4550fc24) 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 4 Zone: default (57276bbe-b8c4-4c89-b257-291bacdcb286) 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 zonegroup default 2024-04-13T21:21:23.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.378+0000 7f4d78690780 20 rados->read ofs=0 len=0 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.379+0000 7f4d78690780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.379+0000 7f4d78690780 20 started sync module instance, tier type = 2024-04-13T21:21:23.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.379+0000 7f4d78690780 20 started zone id=57276bbe-b8c4-4c89-b257-291bacdcb286 (name=default) with tier type = 2024-04-13T21:21:23.388 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=0 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=1 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=2 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=3 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=4 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=5 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.395+0000 7f4d78690780 20 add_watcher() i=6 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.396+0000 7f4d78690780 20 add_watcher() i=7 2024-04-13T21:21:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.396+0000 7f4d78690780 2 all 8 watchers are set, enabling cache 2024-04-13T21:21:23.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.399+0000 7f4d5d60b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:23.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.399+0000 7f4d78690780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:23.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.399+0000 7f4d78690780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:23.396 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.403+0000 7f4d78690780 5 note: GC not initialized 2024-04-13T21:21:23.397 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.403+0000 7f4d58e02700 20 reqs_thread_entry: start 2024-04-13T21:21:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.404+0000 7f4d78690780 20 init_complete bucket index max shards: 11 2024-04-13T21:21:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.404+0000 7f4d78690780 10 rgw notify: Started notification manager with: 1 workers 2024-04-13T21:21:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.404+0000 7f4d56dfe700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 13 21:21:53 2024 2024-04-13T21:21:23.398 INFO:teuthology.orchestra.run.smithi175.stderr: 2024-04-13T21:21:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.404+0000 7f4d565dc700 20 reqs_thread_entry: start 2024-04-13T21:21:23.404 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.411+0000 7f4d5d60b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-13T21:21:23.409 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.415+0000 7f4d78690780 20 remove_watcher() i=0 2024-04-13T21:21:23.409 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.415+0000 7f4d78690780 2 removed watcher, disabling cache 2024-04-13T21:21:23.411 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.418+0000 7f4d78690780 20 remove_watcher() i=1 2024-04-13T21:21:23.414 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.421+0000 7f4d78690780 20 remove_watcher() i=2 2024-04-13T21:21:23.417 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.424+0000 7f4d78690780 20 remove_watcher() i=3 2024-04-13T21:21:23.420 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.426+0000 7f4d78690780 20 remove_watcher() i=4 2024-04-13T21:21:23.423 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.429+0000 7f4d78690780 20 remove_watcher() i=5 2024-04-13T21:21:23.425 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.432+0000 7f4d78690780 20 remove_watcher() i=6 2024-04-13T21:21:23.426 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.433+0000 7f4d78690780 20 remove_watcher() i=7 2024-04-13T21:21:23.428 INFO:teuthology.orchestra.run.smithi175.stdout:[] 2024-04-13T21:21:23.430 DEBUG:tasks.util.rgw: json result: [] 2024-04-13T21:21:23.430 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:23.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-13T21:21:23.431 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-13T21:21:23.519 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:23.519 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:23.527 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.533+0000 7f5b30d1f780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-13T21:21:23.527 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.533+0000 7f5b30d1f780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-13T21:21:23.530 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:23.537+0000 7f5b12493700 20 reqs_thread_entry: start 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "57276bbe-b8c4-4c89-b257-291bacdcb286", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "default.rgw.meta:root", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "default.rgw.control", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "default.rgw.log", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-13T21:21:23.536 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "default.rgw.otp", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:21:23.537 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:21:23.539 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2024-04-13T21:21:23.539 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-13T21:21:23.539 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:21:23.540 DEBUG:tasks.util.rgw: json result: {'id': '57276bbe-b8c4-4c89-b257-291bacdcb286', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-13T21:21:23.540 INFO:tasks.rgw:Starting rgw... 2024-04-13T21:21:23.540 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-04-13T21:21:23.540 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-13T21:21:23.540 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-13T21:21:23.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-13T21:21:23.543 INFO:tasks.rgw.c1.client.0:Started 2024-04-13T21:21:23.543 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-04-13T21:21:23.543 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-13T21:21:23.543 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T21:21:23.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T21:21:23.545 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T21:21:23.546 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-04-13T21:21:23.546 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-13T21:21:23.546 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-13T21:21:23.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-13T21:21:23.548 INFO:tasks.rgw.c2.client.0:Started 2024-04-13T21:21:23.548 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-04-13T21:21:23.548 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-13T21:21:23.549 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-13T21:21:23.549 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-13T21:21:23.551 INFO:tasks.rgw.c2.client.1:Started 2024-04-13T21:21:23.551 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi132.front.sepia.ceph.com:8000/ 2024-04-13T21:21:23.551 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8000/ 2024-04-13T21:21:23.589 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:23.589 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T21:21:23.590 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8000: Connection refused 2024-04-13T21:21:24.591 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8000/ 2024-04-13T21:21:24.612 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:24.612 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:24.613 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-13T21:21:24.614 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T21:21:24.614 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi132.front.sepia.ceph.com:8001/ 2024-04-13T21:21:24.614 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001/ 2024-04-13T21:21:24.676 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:24.676 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:24.676 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T21:21:24.677 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T21:21:24.677 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi175.front.sepia.ceph.com:8000/ 2024-04-13T21:21:24.677 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000/ 2024-04-13T21:21:24.715 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:24.716 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:24.716 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T21:21:24.716 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2024-04-13T21:21:24.717 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi175.front.sepia.ceph.com:8001/ 2024-04-13T21:21:24.717 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001/ 2024-04-13T21:21:24.779 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:24.779 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:24.780 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-13T21:21:24.780 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2024-04-13T21:21:24.781 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-04-13T21:21:25.287 INFO:tasks.rgw_multisite:creating realm.. 2024-04-13T21:21:25.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:25.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:25.288 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:25.334 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:25.334 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:25.364 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:25.364 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:25.364 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "test-realm", 2024-04-13T21:21:25.364 INFO:teuthology.orchestra.run.smithi132.stdout: "current_period": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:21:25.364 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 1 2024-04-13T21:21:25.364 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:25.365 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-04-13T21:21:25.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:25.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:25.366 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:25.412 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:25.412 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:25.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:25.435+0000 7f6204fe5780 -1 old format 2024-04-13T21:21:25.435 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:25.435 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:21:25.435 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:21:25.435 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:21:25.435 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:21:25.435 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "", 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [], 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [] 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:25.436 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:25.437 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:25.438 INFO:tasks.rgw_multisite:creating master zone.. 2024-04-13T21:21:25.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi132.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:25.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi132.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:25.439 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --endpoints http://smithi132.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:25.532 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:25.532 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:25.546 INFO:teuthology.orchestra.run.smithi132.stderr:NOTICE: set zone's realm_id=645d07fe-13a2-4d07-bad9-a662f2f6c3cf 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a1.rgw.control", 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-13T21:21:25.562 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a1.rgw.log", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a1.rgw.otp", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:21:25.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a1.rgw.buckets.data" 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:21:25.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:25.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:25.565 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:25.565 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:25.565 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-13T21:21:25.565 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:25.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2024-04-13T21:21:26.300 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'a1.rgw.buckets.data' created 2024-04-13T21:21:26.319 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2024-04-13T21:21:27.335 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2024-04-13T21:21:27.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:27.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:27.350 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:27.402 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:27.402 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:27.419 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:27.419 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a1.rgw.control", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a1.rgw.log", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-13T21:21:27.420 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a1.rgw.otp", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:21:27.421 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a1.rgw.buckets.data", 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:27.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:27.423 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:27.423 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:27.423 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-13T21:21:27.423 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:27.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:27.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:27.424 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:27.471 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:27.471 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:27.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:27.489+0000 7f67608e8780 0 period (8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4 does not have zone f6f208d6-f791-479b-bc3d-112512eec2d4 configured 2024-04-13T21:21:34.389 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 1, 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:21:34.390 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:34.391 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": false, 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:34.392 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:21:34.393 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.394 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:21:34.395 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:21:34.396 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:21:34.397 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:21:34.398 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:34.399 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-13T21:21:34.399 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T21:21:34.400 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T21:21:34.406+0000 7fd9f31fd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 46248) UID: 0 2024-04-13T21:21:34.400 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T21:21:34.406+0000 7fd9f83d27c0 -1 shutting down 2024-04-13T21:21:40.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-13T21:21:40.501 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-13T21:21:40.501 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-13T21:21:40.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-13T21:21:40.504 INFO:tasks.rgw.c1.client.0:Started 2024-04-13T21:21:40.504 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8000 2024-04-13T21:21:40.528 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:40.528 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T21:21:40.530 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8000: Connection refused 2024-04-13T21:21:41.531 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8000 2024-04-13T21:21:41.553 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:41.553 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:41.554 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T21:21:41.555 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T21:21:41.555 INFO:tasks.rgw_multisite:creating admin user.. 2024-04-13T21:21:41.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:41.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:41.556 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:41.652 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:41.652 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:41.685 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:41.685 INFO:teuthology.orchestra.run.smithi132.stdout: "user_id": "realm-admin", 2024-04-13T21:21:41.685 INFO:teuthology.orchestra.run.smithi132.stdout: "display_name": "Realm Admin", 2024-04-13T21:21:41.685 INFO:teuthology.orchestra.run.smithi132.stdout: "email": "", 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "suspended": 0, 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "max_buckets": 1000, 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "subusers": [], 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "keys": [ 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "realm-admin", 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "swift_keys": [], 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": [], 2024-04-13T21:21:41.686 INFO:teuthology.orchestra.run.smithi132.stdout: "op_mask": "read, write, delete", 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "system": true, 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "", 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "default_storage_class": "", 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_tags": [], 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:21:41.687 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "temp_url_keys": [], 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "rgw", 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: "mfa_ids": [] 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:41.688 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:21:41.762 INFO:tasks.rgw_multisite:creating zone.. 2024-04-13T21:21:41.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi132.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:41.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi132.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:41.763 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --endpoints http://smithi132.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:41.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:41.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:41.828 INFO:teuthology.orchestra.run.smithi132.stderr:NOTICE: set zone's realm_id=645d07fe-13a2-4d07-bad9-a662f2f6c3cf 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a2.rgw.control", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a2.rgw.log", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-13T21:21:41.842 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a2.rgw.otp", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:21:41.843 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a2.rgw.buckets.data" 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:41.844 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:41.845 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:41.845 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-13T21:21:41.845 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:41.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2024-04-13T21:21:42.338 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'a2.rgw.buckets.data' created 2024-04-13T21:21:42.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2024-04-13T21:21:43.355 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2024-04-13T21:21:43.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:43.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:43.375 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:43.426 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:43.426 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:43.445 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:43.445 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:21:43.445 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:21:43.445 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-13T21:21:43.445 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a2.rgw.control", 2024-04-13T21:21:43.445 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a2.rgw.log", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a2.rgw.otp", 2024-04-13T21:21:43.446 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:21:43.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-13T21:21:43.448 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:43.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:43.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:43.449 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:43.496 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:43.496 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:43.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:21:43.514+0000 7f7dd01b1780 0 period (5039ce81-80d5-4d03-a26f-34b9088aaab4 does not have zone 5da80fbc-5edc-4ba7-826d-471359a9fc92 configured 2024-04-13T21:21:45.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-13T21:21:50.417 INFO:teuthology.orchestra.run.smithi132.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:21:50.452 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:21:50.452 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:21:50.452 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2, 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:21:50.453 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:21:50.454 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:50.455 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:21:50.456 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:21:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:21:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:21:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:21:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:21:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:21:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:21:50.464 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.465 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:21:50.466 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:21:50.466 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:21:50.466 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:21:50.466 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:21:50.467 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:21:50.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:21:50.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:21:50.468 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:50.468 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:21:50.468 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:21:50.468 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:21:50.469 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T21:21:50.469 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T21:21:50.470 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T21:21:50.477+0000 7fdb62c75700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 46251) UID: 0 2024-04-13T21:21:50.470 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T21:21:50.477+0000 7fdb67e4a7c0 -1 shutting down 2024-04-13T21:21:51.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T21:21:56.571 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T21:21:56.571 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T21:21:56.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T21:21:56.575 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T21:21:56.575 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T21:21:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T21:21:56.600 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:56.600 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:56.600 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T21:21:57.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T21:21:57.601 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T21:21:57.623 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:21:57.623 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:21:57.624 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T21:21:57.625 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T21:21:57.625 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-13T21:21:57.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi132.front.sepia.ceph.com:8000', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:57.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi132.front.sepia.ceph.com:8000', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:57.626 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi132.front.sepia.ceph.com:8000 --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:57.685 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:57.685 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:57.728 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:21:57.728 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:57.728 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-realm", 2024-04-13T21:21:57.728 INFO:teuthology.orchestra.run.smithi175.stdout: "current_period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:21:57.728 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 2 2024-04-13T21:21:57.728 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:21:57.729 INFO:tasks.rgw_multisite:creating zonegroup.. 2024-04-13T21:21:57.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:57.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:57.730 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:57.777 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:57.777 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:57.792 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:21:57.798+0000 7fd2bd9f9780 -1 old format 2024-04-13T21:21:57.805 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:21:57.805 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "b", 2024-04-13T21:21:57.805 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b", 2024-04-13T21:21:57.805 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": false, 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "", 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [], 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2024-04-13T21:21:57.806 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [] 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:21:57.807 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:21:57.808 INFO:tasks.rgw_multisite:creating zone.. 2024-04-13T21:21:57.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:57.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:21:57.809 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --endpoints http://smithi175.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:21:57.902 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:21:57.903 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:21:57.917 INFO:teuthology.orchestra.run.smithi175.stderr:NOTICE: set zone's realm_id=645d07fe-13a2-4d07-bad9-a662f2f6c3cf 2024-04-13T21:21:57.935 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:21:57.935 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:21:57.935 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b1", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "b1.rgw.control", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "b1.rgw.log", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-13T21:21:57.936 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "b1.rgw.otp", 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-13T21:21:57.937 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "b1.rgw.buckets.data" 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:21:57.938 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-13T21:21:57.939 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:21:57.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2024-04-13T21:21:59.254 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'b1.rgw.buckets.data' created 2024-04-13T21:21:59.275 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2024-04-13T21:22:00.277 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2024-04-13T21:22:00.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:00.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:00.292 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:00.343 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:00.343 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:00.361 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:22:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b1", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "b1.rgw.control", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "b1.rgw.log", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-13T21:22:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "b1.rgw.otp", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:22:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "b1.rgw.buckets.data", 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-13T21:22:00.365 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:22:00.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:00.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:00.366 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:00.413 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:00.413 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:00.425 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:22:00.431+0000 7ff133c3a780 0 period (5039ce81-80d5-4d03-a26f-34b9088aaab4 does not have zone ba6de64d-3aa0-42be-83ea-cebdba376cd8 configured 2024-04-13T21:22:07.323 INFO:teuthology.orchestra.run.smithi175.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 3, 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2024-04-13T21:22:07.358 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "a", 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "a", 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:07.359 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "a2", 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:07.360 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "a1", 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:07.361 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.362 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:07.363 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "b", 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b", 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": false, 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2024-04-13T21:22:07.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b1", 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": false, 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:07.365 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:07.369 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:07.369 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2024-04-13T21:22:07.370 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 782658364 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2041901071 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2070976819 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "a", 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2024-04-13T21:22:07.372 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2024-04-13T21:22:07.373 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2024-04-13T21:22:07.374 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:22:07.375 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:22:07.377 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-13T21:22:07.377 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T21:22:07.378 INFO:tasks.rgw.c2.client.0.smithi175.stdout:2024-04-13T21:22:07.384+0000 7fa623e04700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 46256) UID: 0 2024-04-13T21:22:07.378 INFO:tasks.rgw.c2.client.0.smithi175.stdout:2024-04-13T21:22:07.384+0000 7fa628fd97c0 -1 shutting down 2024-04-13T21:22:08.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-13T21:22:13.478 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-13T21:22:13.478 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-13T21:22:13.479 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-13T21:22:13.481 INFO:tasks.rgw.c2.client.0:Started 2024-04-13T21:22:13.482 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000 2024-04-13T21:22:13.504 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:22:13.505 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:22:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T21:22:13.506 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8000: Connection refused 2024-04-13T21:22:13.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-13T21:22:14.507 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000 2024-04-13T21:22:14.529 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:22:14.529 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:22:14.530 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T21:22:14.531 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2024-04-13T21:22:14.531 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-13T21:22:14.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi132.front.sepia.ceph.com:8000', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--url', 'http://smithi132.front.sepia.ceph.com:8000', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:14.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi132.front.sepia.ceph.com:8000', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--url', 'http://smithi132.front.sepia.ceph.com:8000', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:14.532 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi132.front.sepia.ceph.com:8000 --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --url http://smithi132.front.sepia.ceph.com:8000 --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:14.627 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:14.628 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:14.664 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:22:14.664 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:14.664 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-realm", 2024-04-13T21:22:14.664 INFO:teuthology.orchestra.run.smithi175.stdout: "current_period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:14.664 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 2 2024-04-13T21:22:14.664 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:22:14.665 INFO:tasks.rgw_multisite:creating zone.. 2024-04-13T21:22:14.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:14.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:14.666 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --endpoints http://smithi175.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:14.717 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:14.717 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:14.731 INFO:teuthology.orchestra.run.smithi175.stderr:NOTICE: set zone's realm_id=645d07fe-13a2-4d07-bad9-a662f2f6c3cf 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b2", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "b2.rgw.control", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "b2.rgw.log", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-13T21:22:14.745 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "b2.rgw.otp", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:14.746 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "b2.rgw.buckets.data" 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:14.747 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:14.748 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:14.748 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-13T21:22:14.748 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:22:14.749 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2024-04-13T21:22:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'b2.rgw.buckets.data' created 2024-04-13T21:22:15.314 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2024-04-13T21:22:16.321 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2024-04-13T21:22:16.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:16.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:16.339 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:16.389 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:16.389 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:16.408 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:22:16.408 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:22:16.408 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b2", 2024-04-13T21:22:16.408 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-13T21:22:16.408 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "b2.rgw.control", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "b2.rgw.log", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "b2.rgw.otp", 2024-04-13T21:22:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "b2.rgw.buckets.data", 2024-04-13T21:22:16.410 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-13T21:22:16.411 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:22:16.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:16.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:16.413 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:16.459 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:16.459 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:16.472 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:22:16.478+0000 7f838b4e2780 0 period (5039ce81-80d5-4d03-a26f-34b9088aaab4 does not have zone 4123dd49-f32f-49d9-98c0-83e8adaa8c84 configured 2024-04-13T21:22:23.372 INFO:teuthology.orchestra.run.smithi175.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 4, 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "a", 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "a", 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2024-04-13T21:22:23.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "a2", 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:23.401 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "a1", 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:23.402 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2024-04-13T21:22:23.403 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "b", 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b", 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": false, 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:22:23.405 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b2", 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:23.406 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "b1", 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2024-04-13T21:22:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2024-04-13T21:22:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 555598186 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 782658364 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:22:23.415 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2041901071 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2070976819 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "a", 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2024-04-13T21:22:23.416 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2024-04-13T21:22:23.417 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.418 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:22:23.419 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:22:23.421 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-13T21:22:23.421 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T21:22:23.421 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T21:22:23.428+0000 7ff0331f0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 46257) UID: 0 2024-04-13T21:22:23.421 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T21:22:23.428+0000 7ff0383c57c0 -1 shutting down 2024-04-13T21:22:25.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-13T21:22:29.522 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-13T21:22:29.522 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-13T21:22:29.522 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-13T21:22:29.525 INFO:tasks.rgw.c2.client.1:Started 2024-04-13T21:22:29.525 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2024-04-13T21:22:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T21:22:29.551 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:22:29.551 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:22:29.551 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T21:22:30.552 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2024-04-13T21:22:30.573 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T21:22:30.573 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2024-04-13T21:22:30.574 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T21:22:30.574 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2024-04-13T21:22:30.575 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-04-13T21:22:30.575 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-04-13T21:22:30.593 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi 2024-04-13T21:22:30.653 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-04-13T21:22:30.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:30.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:22:30.653 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:22:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:30.752 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "user_id": "rgw-multisite-test-user", 2024-04-13T21:22:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "display_name": "Multisite Test User", 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "email": "", 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "suspended": 0, 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "max_buckets": 1000, 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "subusers": [], 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "keys": [ 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "rgw-multisite-test-user", 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "C81CVW06BOFYI3GBQ4YX", 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "uSsFoUs8LsZr47IClCWYmUA17sRArJ9MRi3GDTOL" 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "swift_keys": [], 2024-04-13T21:22:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": [ 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "roles", 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "perm": "*" 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "op_mask": "read, write, delete", 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "", 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "default_storage_class": "", 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_tags": [], 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:22:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "temp_url_keys": [], 2024-04-13T21:22:30.756 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "rgw", 2024-04-13T21:22:30.756 INFO:teuthology.orchestra.run.smithi132.stdout: "mfa_ids": [] 2024-04-13T21:22:30.756 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:30.756 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T21:22:30.766 INFO:rgw_multi.tests:meta checkpoint 2024-04-13T21:22:30.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:30.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:30.766 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:30.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:30.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:30.844 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:22:30.856 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:22:30.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:22:30.856 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:22:30.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.857 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:22:30.858 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:22:30.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:30.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:30.859 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:30.906 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:30.906 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:30.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:30.942+0000 7fa07bdb3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:30.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:30.943+0000 7fa07bdb3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:30.946 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:30.946 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.947 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.948 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:30.949 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:30.950 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:22:30Z" 2024-04-13T21:22:30.951 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:30.952 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:22:30.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-13T21:22:35.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:35.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:35.954 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:36.013 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:36.013 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:36.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:36.051+0000 7fbbdec4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:36.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:36.052+0000 7fbbdec4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:36.056 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:22:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:22:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:22:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:22:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:22:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:22:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:22:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:22:36Z" 2024-04-13T21:22:36.061 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:36.062 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:22:41.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:41.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:41.064 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:41.120 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:41.120 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:41.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:41.157+0000 7fa31bbd3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:41.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:41.158+0000 7fa31bbd3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:41.162 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:41.162 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:22:41.162 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:41.163 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:41.164 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:41.165 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:41.166 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:41.167 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:22:41Z" 2024-04-13T21:22:41.168 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:41.168 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:22:46.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:46.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:46.170 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:46.226 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:46.227 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:46.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:46.262+0000 7fe2cb3cd780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:46.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:46.263+0000 7fe2cb3cd780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:46.268 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:46.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:46.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:46.271 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:46.272 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:22:46.273 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:22:46.273 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:22:46.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:46.273 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:22:46Z" 2024-04-13T21:22:46.273 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:46.274 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:22:51.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:51.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:51.275 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:51.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:51.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:51.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:51.368+0000 7f6a2b5ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:51.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:51.369+0000 7f6a2b5ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:51.375 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:51.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:22:51.377 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:51.378 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:22:51.379 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:22:51.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:51.380 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:22:51Z" 2024-04-13T21:22:51.380 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:51.381 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:22:56.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:56.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:22:56.382 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:22:56.437 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:22:56.437 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:22:56.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:56.474+0000 7f7e69665780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:56.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:22:56.475+0000 7f7e69665780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:22:56.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:56.483 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:22:56.484 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:22:56.485 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:22:56.486 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:22:56.487 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:22:56.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:22:56.487 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:22:56Z" 2024-04-13T21:22:56.487 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:22:56.488 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:01.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:01.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:01.489 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:01.543 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:01.543 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:01.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:01.580+0000 7f13b8474780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:01.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:01.581+0000 7f13b8474780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:01Z" 2024-04-13T21:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:01.592 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:06.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:06.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:06.594 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:06.654 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:06.654 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:06.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:06.691+0000 7f90e88c5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:06.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:06.692+0000 7f90e88c5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:06.696 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:06.697 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:06.698 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:06.699 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:06.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:06Z" 2024-04-13T21:23:06.701 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:06.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:11.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:11.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:11.704 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:11.760 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:11.760 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:11.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:11.798+0000 7f8eb275b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:11.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:11.799+0000 7f8eb275b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:11.803 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:11.803 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:11.803 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:11.803 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:11.804 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:11.805 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:11.806 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:11.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:11.808 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:11.808 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:11.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:11.808 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:11.808 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:11.808 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:11.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:11.809 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:11Z" 2024-04-13T21:23:11.809 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:11.810 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:16.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:16.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:16.812 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:16.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:16.905+0000 7fbe74381780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:16.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:16.906+0000 7fbe74381780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:16.911 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:16.911 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:16.911 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:16.911 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:16.912 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:16.914 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:16Z" 2024-04-13T21:23:16.916 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:16.917 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:21.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:21.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:21.918 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:21.973 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:21.974 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:22.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:22.011+0000 7feb44e68780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:22.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:22.012+0000 7feb44e68780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:22.018 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:22.019 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:22.020 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:22.021 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:22.022 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:22Z" 2024-04-13T21:23:22.023 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:22.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:27.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:27.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:27.029 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:27.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:27.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:27.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:27.121+0000 7ff6a9567780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:27.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:27.122+0000 7ff6a9567780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:27.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:27.130 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:27.131 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:27.132 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:27Z" 2024-04-13T21:23:27.133 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:27.134 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:32.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:32.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:32.135 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:32.191 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:32.191 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:32.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:32.228+0000 7fcb39976780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:32.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:32.229+0000 7fcb39976780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:32.234 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:32.235 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:32.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:32.237 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:32.238 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:32.239 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:32Z" 2024-04-13T21:23:32.240 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:32.242 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:37.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:37.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:37.243 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:37.298 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:37.298 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:37.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:37.336+0000 7ffb270af780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:37.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:37.337+0000 7ffb270af780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:37.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:37.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:37.344 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:37.345 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:37.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:37.347 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:37Z" 2024-04-13T21:23:37.347 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:37.348 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:42.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:42.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:42.349 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:42.404 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:42.405 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:42.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:42.447+0000 7fe15c110780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:42.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:42.448+0000 7fe15c110780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:42.453 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:42Z" 2024-04-13T21:23:42.458 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:42.459 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:47.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:47.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:47.462 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:47.518 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:47.518 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:47.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:47.557+0000 7fba25e49780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:47.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:47.558+0000 7fba25e49780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:47.562 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:47.562 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:47.564 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:47.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:47.566 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:47.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:47.568 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:47Z" 2024-04-13T21:23:47.568 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:47.569 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:52.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:52.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:52.571 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:52.626 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:52.627 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:52.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:52.665+0000 7f063e70d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:52.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:52.666+0000 7f063e70d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:52.670 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:52.670 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:52.671 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:52.672 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:52.673 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:52.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:52Z" 2024-04-13T21:23:52.675 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:52.677 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:23:57.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:57.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:23:57.678 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:23:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:23:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:23:57.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:57.771+0000 7f2635e62780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:57.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:23:57.772+0000 7f2635e62780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:23:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:57.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:57.779 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:23:57.780 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:23:57.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:23:57.782 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:23:57Z" 2024-04-13T21:23:57.782 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:23:57.783 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:24:02.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:02.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:02.784 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:02.839 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:02.839 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:02.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:02.878+0000 7f326ae11780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:02.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:02.879+0000 7f326ae11780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:02.884 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:02.884 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:02.884 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:02.884 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:02.884 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:02.884 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:02.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:02.886 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:02.887 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:02.889 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:02.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:02.889 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:02Z" 2024-04-13T21:24:02.889 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:02.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:24:07.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:07.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:07.891 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:07.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:07.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:07.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:07.984+0000 7f5e6a855780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:07.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:07.985+0000 7f5e6a855780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:07Z" 2024-04-13T21:24:07.996 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:07.998 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:24:12.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:12.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:13.000 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:13.056 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:13.056 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:13.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:13.092+0000 7f60c7ca1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:13.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:13.093+0000 7f60c7ca1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:13.099 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:13.099 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:13.099 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:13.099 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:13.099 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:13.099 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:13.100 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:13.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:13.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:13.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:13.104 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:13.104 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:13.104 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:13.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:13.104 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:13Z" 2024-04-13T21:24:13.104 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:13.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:24:18.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:18.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:18.106 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:18.162 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:18.162 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:18.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:18.201+0000 7ff4eeda4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:18.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:18.202+0000 7ff4eeda4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:18.207 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:18.207 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:18.208 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:18.208 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:18.208 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:18.208 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:18.208 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:18.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:18.209 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:18.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:18.209 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:18.209 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:18.209 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:18.210 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:18.210 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:18.210 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:18.210 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:18.210 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:18.211 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:18.211 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:18.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:18.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:18.211 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:18.212 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:18.212 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:18.212 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:18.212 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:18.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:18.214 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:18.214 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:18.214 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:18.214 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:18.214 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:18.214 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:18.216 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:18.216 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:18.216 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:18.216 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:18.216 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:18.216 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:18.217 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:18.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:18.218 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:18Z" 2024-04-13T21:24:18.218 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:18.220 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:24:23.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:23.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:23.222 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:23.279 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:23.280 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:23.318+0000 7fea79c98780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:23.319+0000 7fea79c98780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:23.324 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:23.324 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:23.324 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:23.324 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:23.324 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:23.326 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:23.327 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:23.328 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:23.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:23.330 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:23Z" 2024-04-13T21:24:23.330 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:23.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713043350.757914_77.1 target= 2024-04-13T21:24:28.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.333 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.388 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.389 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:28.427+0000 7ff893ae5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:28.428+0000 7ff893ae5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.432 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:28.432 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:28.432 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:28.432 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:28.432 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:28.432 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.433 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.435 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:28.436 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:28.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.437 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:28Z" 2024-04-13T21:24:28.437 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:28.438 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:28.438 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-13T21:24:28.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.438 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.495 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.495 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:28.526 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:28.532+0000 7f9666baf780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.527 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:28.533+0000 7f9666baf780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.538 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.539 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.540 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.541 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.542 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2024-04-13T21:24:28Z" 2024-04-13T21:24:28.543 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:24:28.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-13T21:24:28.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-13T21:24:28.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.545 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.595 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.595 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:28.625 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:28.631+0000 7fa6d15ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.626 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:28.632+0000 7fa6d15ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.637 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2024-04-13T21:24:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2024-04-13T21:24:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2024-04-13T21:24:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2024-04-13T21:24:28Z" 2024-04-13T21:24:28.641 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:24:28.642 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-13T21:24:28.642 INFO:rgw_multi.tests:meta checkpoint 2024-04-13T21:24:28.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.642 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.692 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.692 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:28.724 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:28.735 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:28.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:28.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.737 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:28.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:28.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.738 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.785 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.786 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:28.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:28.819+0000 7f895c4b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:28.820+0000 7f895c4b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:28.824 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.825 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.828 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:28Z" 2024-04-13T21:24:28.830 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:28.831 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:28.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-13T21:24:28.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.831 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:28.907 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:28.913+0000 7fcdf6ef8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.907 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:28.914+0000 7fcdf6ef8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:28.918 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:24:28.918 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2024-04-13T21:24:28.918 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.919 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.920 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:28.922 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2024-04-13T21:24:28Z" 2024-04-13T21:24:28.923 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:24:28.924 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-13T21:24:28.924 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-13T21:24:28.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:28.925 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:28.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:28.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:29.000 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:29.006+0000 7f02cfb91780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:29.000 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-13T21:24:29.007+0000 7f02cfb91780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:29.011 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-13T21:24:29.011 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2024-04-13T21:24:29.011 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2024-04-13T21:24:29.011 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2024-04-13T21:24:29.011 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2024-04-13T21:24:29.011 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:29.012 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:29.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-04-13T21:24:29.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:29.016 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2024-04-13T21:24:29.016 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2024-04-13T21:24:29.016 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2024-04-13T21:24:29.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-13T21:24:29.016 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2024-04-13T21:24:29Z" 2024-04-13T21:24:29.016 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-13T21:24:29.017 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-13T21:24:29.017 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-04-13T21:24:29.219 DEBUG:boto:Using access key provided by client. 2024-04-13T21:24:29.220 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:24:29.220 DEBUG:boto:Using access key provided by client. 2024-04-13T21:24:29.220 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:24:29.221 DEBUG:boto:Using access key provided by client. 2024-04-13T21:24:29.221 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:24:29.221 DEBUG:boto:Using access key provided by client. 2024-04-13T21:24:29.221 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:24:29.221 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-1 2024-04-13T21:24:29.222 DEBUG:boto:path=/vwnfns-1/ 2024-04-13T21:24:29.222 DEBUG:boto:auth_path=/vwnfns-1/ 2024-04-13T21:24:29.222 DEBUG:boto:Method: PUT 2024-04-13T21:24:29.222 DEBUG:boto:Path: /vwnfns-1/ 2024-04-13T21:24:29.222 DEBUG:boto:Data: 2024-04-13T21:24:29.222 DEBUG:boto:Headers: {} 2024-04-13T21:24:29.222 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:29.222 DEBUG:boto:Port: 8000 2024-04-13T21:24:29.222 DEBUG:boto:Params: {} 2024-04-13T21:24:29.222 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:24:29.223 DEBUG:boto:Token: None 2024-04-13T21:24:29.223 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:29 GMT /vwnfns-1/ 2024-04-13T21:24:29.223 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NFWpc88ae5Fb1NBmYzLsm7rnhpU= 2024-04-13T21:24:29.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NFWpc88ae5Fb1NBmYzLsm7rnhpU=', 'Content-Length': '0'} 2024-04-13T21:24:31.544 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048638798db298c6d-00661af80d-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:31.544 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-2 2024-04-13T21:24:31.544 DEBUG:boto:path=/vwnfns-2/ 2024-04-13T21:24:31.544 DEBUG:boto:auth_path=/vwnfns-2/ 2024-04-13T21:24:31.544 DEBUG:boto:Method: PUT 2024-04-13T21:24:31.545 DEBUG:boto:Path: /vwnfns-2/ 2024-04-13T21:24:31.545 DEBUG:boto:Data: 2024-04-13T21:24:31.545 DEBUG:boto:Headers: {} 2024-04-13T21:24:31.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:31.545 DEBUG:boto:Port: 8001 2024-04-13T21:24:31.545 DEBUG:boto:Params: {} 2024-04-13T21:24:31.545 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:24:31.546 DEBUG:boto:Token: None 2024-04-13T21:24:31.546 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:31 GMT /vwnfns-2/ 2024-04-13T21:24:31.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QcgZ4wd+/qjKD/t1OXdDgOqQRlg= 2024-04-13T21:24:31.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QcgZ4wd+/qjKD/t1OXdDgOqQRlg=', 'Content-Length': '0'} 2024-04-13T21:24:34.562 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1f4e158401c30cf-00661af80f-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:34.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:34.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:34.563 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:34.613 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:34.613 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:34.645 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:34.656 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043471.570939_37.1", 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:24:31.570939Z" 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:34.658 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:34.659 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043350.757914_77.1', 1: '1_1713043471.570939_37.1', 2: '', 3: ''} 2024-04-13T21:24:34.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:34.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:34.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:34.660 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:34.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:34.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:34.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:34.743+0000 7f83ce1a4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:34.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:34.744+0000 7f83ce1a4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:34.748 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:34.748 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:34.748 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:34.748 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:34.749 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:34.750 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043471.526008_30.1", 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:31.526008Z", 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.751 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:34.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:34.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:34Z" 2024-04-13T21:24:34.754 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:34.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043471.526008_30.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:31.526008Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:34Z" } 2024-04-13T21:24:34.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043471.526008_30.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:31.526008Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:34.756 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043471.570939_37.1', 2: '', 3: ''} 2024-04-13T21:24:34.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043471.526008_30.1', 2: '', 3: ''} 2024-04-13T21:24:34.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1713043471.570939_37.1 target=1_1713043471.526008_30.1 2024-04-13T21:24:39.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:39.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:39.758 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:39.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:39.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:39.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:39.853+0000 7f033fe1d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:39.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:39.854+0000 7f033fe1d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:39.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:39.861 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:39.862 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043471.570939_37.1", 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:31.570939Z", 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:39.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:39.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:39.865 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:39.865 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:39.865 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:39.865 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:39.866 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:39.867 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:39.867 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:39.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:39.867 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:39Z" 2024-04-13T21:24:39.867 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:39.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043471.570939_37.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:31.570939Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:39Z" } 2024-04-13T21:24:39.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043471.570939_37.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:31.570939Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:39.868 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043471.570939_37.1', 2: '', 3: ''} 2024-04-13T21:24:39.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043471.570939_37.1', 2: '', 3: ''} 2024-04-13T21:24:39.869 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:39.869 DEBUG:boto:path=/vwnfns-1/ 2024-04-13T21:24:39.869 DEBUG:boto:auth_path=/vwnfns-1/ 2024-04-13T21:24:39.869 DEBUG:boto:Method: HEAD 2024-04-13T21:24:39.869 DEBUG:boto:Path: /vwnfns-1/ 2024-04-13T21:24:39.869 DEBUG:boto:Data: 2024-04-13T21:24:39.869 DEBUG:boto:Headers: {} 2024-04-13T21:24:39.869 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:39.870 DEBUG:boto:Port: 8000 2024-04-13T21:24:39.870 DEBUG:boto:Params: {} 2024-04-13T21:24:39.870 DEBUG:boto:Token: None 2024-04-13T21:24:39.870 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:39 GMT /vwnfns-1/ 2024-04-13T21:24:39.870 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qJPKodSjUenRARDuOs/Q7KeDtuA= 2024-04-13T21:24:39.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qJPKodSjUenRARDuOs/Q7KeDtuA=', 'Content-Length': '0'} 2024-04-13T21:24:39.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e77e32067e1788f7-00661af817-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:39.873 DEBUG:boto:path=/vwnfns-2/ 2024-04-13T21:24:39.873 DEBUG:boto:auth_path=/vwnfns-2/ 2024-04-13T21:24:39.873 DEBUG:boto:Method: HEAD 2024-04-13T21:24:39.873 DEBUG:boto:Path: /vwnfns-2/ 2024-04-13T21:24:39.873 DEBUG:boto:Data: 2024-04-13T21:24:39.873 DEBUG:boto:Headers: {} 2024-04-13T21:24:39.873 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:39.873 DEBUG:boto:Port: 8000 2024-04-13T21:24:39.874 DEBUG:boto:Params: {} 2024-04-13T21:24:39.874 DEBUG:boto:Token: None 2024-04-13T21:24:39.874 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:39 GMT /vwnfns-2/ 2024-04-13T21:24:39.874 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cPr3iCy03JAlnobUQ5mZ4QAxRyM= 2024-04-13T21:24:39.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cPr3iCy03JAlnobUQ5mZ4QAxRyM=', 'Content-Length': '0'} 2024-04-13T21:24:39.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c59a4ebd15c457-00661af817-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:39.876 DEBUG:boto:path=/vwnfns-1/ 2024-04-13T21:24:39.876 DEBUG:boto:auth_path=/vwnfns-1/ 2024-04-13T21:24:39.876 DEBUG:boto:Method: HEAD 2024-04-13T21:24:39.876 DEBUG:boto:Path: /vwnfns-1/ 2024-04-13T21:24:39.876 DEBUG:boto:Data: 2024-04-13T21:24:39.877 DEBUG:boto:Headers: {} 2024-04-13T21:24:39.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:39.877 DEBUG:boto:Port: 8001 2024-04-13T21:24:39.877 DEBUG:boto:Params: {} 2024-04-13T21:24:39.877 DEBUG:boto:Token: None 2024-04-13T21:24:39.877 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:39 GMT /vwnfns-1/ 2024-04-13T21:24:39.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qJPKodSjUenRARDuOs/Q7KeDtuA= 2024-04-13T21:24:39.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qJPKodSjUenRARDuOs/Q7KeDtuA=', 'Content-Length': '0'} 2024-04-13T21:24:39.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005970ef7aa5328ce5-00661af817-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:39.879 DEBUG:boto:path=/vwnfns-2/ 2024-04-13T21:24:39.879 DEBUG:boto:auth_path=/vwnfns-2/ 2024-04-13T21:24:39.880 DEBUG:boto:Method: HEAD 2024-04-13T21:24:39.880 DEBUG:boto:Path: /vwnfns-2/ 2024-04-13T21:24:39.880 DEBUG:boto:Data: 2024-04-13T21:24:39.880 DEBUG:boto:Headers: {} 2024-04-13T21:24:39.880 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:39.880 DEBUG:boto:Port: 8001 2024-04-13T21:24:39.880 DEBUG:boto:Params: {} 2024-04-13T21:24:39.881 DEBUG:boto:Token: None 2024-04-13T21:24:39.882 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:39 GMT /vwnfns-2/ 2024-04-13T21:24:39.882 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cPr3iCy03JAlnobUQ5mZ4QAxRyM= 2024-04-13T21:24:39.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cPr3iCy03JAlnobUQ5mZ4QAxRyM=', 'Content-Length': '0'} 2024-04-13T21:24:39.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e07b8b989c291bfa-00661af817-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:39.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-04-13T21:24:39.885 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-3 2024-04-13T21:24:39.885 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:39.885 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:39.885 DEBUG:boto:Method: PUT 2024-04-13T21:24:39.885 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:39.885 DEBUG:boto:Data: 2024-04-13T21:24:39.885 DEBUG:boto:Headers: {} 2024-04-13T21:24:39.885 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:39.886 DEBUG:boto:Port: 8000 2024-04-13T21:24:39.886 DEBUG:boto:Params: {} 2024-04-13T21:24:39.887 DEBUG:boto:Token: None 2024-04-13T21:24:39.887 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:39 GMT /vwnfns-3/ 2024-04-13T21:24:39.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KA00BOwRUwSZKH6akSLW1MQIObs= 2024-04-13T21:24:39.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KA00BOwRUwSZKH6akSLW1MQIObs=', 'Content-Length': '0'} 2024-04-13T21:24:39.901 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e49537cfeeb54eba-00661af817-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:39.901 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-4 2024-04-13T21:24:39.901 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:39.902 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:39.902 DEBUG:boto:Method: PUT 2024-04-13T21:24:39.902 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:39.902 DEBUG:boto:Data: 2024-04-13T21:24:39.902 DEBUG:boto:Headers: {} 2024-04-13T21:24:39.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:39.902 DEBUG:boto:Port: 8001 2024-04-13T21:24:39.902 DEBUG:boto:Params: {} 2024-04-13T21:24:39.902 DEBUG:boto:Token: None 2024-04-13T21:24:39.903 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:39 GMT /vwnfns-4/ 2024-04-13T21:24:39.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qMHhbUYL33U30yn5emmvQZR0Wds= 2024-04-13T21:24:39.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qMHhbUYL33U30yn5emmvQZR0Wds=', 'Content-Length': '0'} 2024-04-13T21:24:39.929 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000890e15df5345c795-00661af817-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:39.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:39.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:39.930 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:39.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:39.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:40.006 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:40.017 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:40.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.017 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:40.017 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:40.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043479.923131_45.1", 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:24:39.923131Z" 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.019 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:40.020 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:40.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.021 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:40.069 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:40.070 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:40.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:40.115+0000 7f6255242780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:40.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:40.116+0000 7f6255242780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:40.120 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:40.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:40.120 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.121 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.122 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043479.923131_45.1", 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:39.923131Z", 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:40.123 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:40.125 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:40Z" 2024-04-13T21:24:40.126 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:40.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043479.923131_45.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:39.923131Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:40Z" } 2024-04-13T21:24:40.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043479.923131_45.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:39.923131Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:40.128 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.128 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:40.128 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.129 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.129 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.129 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.129 DEBUG:boto:Data: 2024-04-13T21:24:40.129 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.129 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.129 DEBUG:boto:Params: {} 2024-04-13T21:24:40.129 DEBUG:boto:Token: None 2024-04-13T21:24:40.129 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.130 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU= 2024-04-13T21:24:40.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU=', 'Content-Length': '0'} 2024-04-13T21:24:40.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc8d4112c9245716-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.132 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.132 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.133 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.133 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.133 DEBUG:boto:Data: 2024-04-13T21:24:40.134 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.134 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.134 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.134 DEBUG:boto:Params: {} 2024-04-13T21:24:40.134 DEBUG:boto:Token: None 2024-04-13T21:24:40.135 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s= 2024-04-13T21:24:40.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s=', 'Content-Length': '0'} 2024-04-13T21:24:40.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1d227f7d135e737-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.137 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.137 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.138 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.138 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.138 DEBUG:boto:Data: 2024-04-13T21:24:40.138 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.138 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.138 DEBUG:boto:Params: {} 2024-04-13T21:24:40.138 DEBUG:boto:Token: None 2024-04-13T21:24:40.138 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.138 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU= 2024-04-13T21:24:40.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU=', 'Content-Length': '0'} 2024-04-13T21:24:40.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000052370981edc9adb-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.141 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.141 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.141 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.141 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.141 DEBUG:boto:Data: 2024-04-13T21:24:40.141 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.141 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.141 DEBUG:boto:Params: {} 2024-04-13T21:24:40.142 DEBUG:boto:Token: None 2024-04-13T21:24:40.142 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s= 2024-04-13T21:24:40.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s=', 'Content-Length': '0'} 2024-04-13T21:24:40.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f45f00db44c6af-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.144 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.144 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.144 DEBUG:boto:Method: PUT 2024-04-13T21:24:40.144 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.144 DEBUG:boto:Data: 2024-04-13T21:24:40.144 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.144 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.145 DEBUG:boto:Params: {} 2024-04-13T21:24:40.145 DEBUG:boto:Token: None 2024-04-13T21:24:40.145 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lkl4p8RDrg0AUdCMXJoOH6KV/Dw= 2024-04-13T21:24:40.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lkl4p8RDrg0AUdCMXJoOH6KV/Dw=', 'Content-Length': '0'} 2024-04-13T21:24:40.147 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef8f59cf9a68ec57-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.147 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.147 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.148 DEBUG:boto:Method: PUT 2024-04-13T21:24:40.148 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.148 DEBUG:boto:Data: 2024-04-13T21:24:40.148 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.148 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.148 DEBUG:boto:Params: {} 2024-04-13T21:24:40.148 DEBUG:boto:Token: None 2024-04-13T21:24:40.148 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z6MYU9b2mEVj/qBO7PonWBvzG58= 2024-04-13T21:24:40.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z6MYU9b2mEVj/qBO7PonWBvzG58=', 'Content-Length': '0'} 2024-04-13T21:24:40.151 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b39a071dbc31487-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.151 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.151 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.151 DEBUG:boto:Method: PUT 2024-04-13T21:24:40.151 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.151 DEBUG:boto:Data: 2024-04-13T21:24:40.151 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.151 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.152 DEBUG:boto:Params: {} 2024-04-13T21:24:40.152 DEBUG:boto:Token: None 2024-04-13T21:24:40.152 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lkl4p8RDrg0AUdCMXJoOH6KV/Dw= 2024-04-13T21:24:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lkl4p8RDrg0AUdCMXJoOH6KV/Dw=', 'Content-Length': '0'} 2024-04-13T21:24:40.156 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003788747b610d9a75-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.156 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.156 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.157 DEBUG:boto:Method: PUT 2024-04-13T21:24:40.157 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.157 DEBUG:boto:Data: 2024-04-13T21:24:40.157 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.157 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.157 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.157 DEBUG:boto:Params: {} 2024-04-13T21:24:40.157 DEBUG:boto:Token: None 2024-04-13T21:24:40.158 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.158 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z6MYU9b2mEVj/qBO7PonWBvzG58= 2024-04-13T21:24:40.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z6MYU9b2mEVj/qBO7PonWBvzG58=', 'Content-Length': '0'} 2024-04-13T21:24:40.204 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000946877ccdb82ec07-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.205 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.205 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.205 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.205 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.205 DEBUG:boto:Data: 2024-04-13T21:24:40.205 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.205 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.205 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.205 DEBUG:boto:Params: {} 2024-04-13T21:24:40.206 DEBUG:boto:Token: None 2024-04-13T21:24:40.206 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.206 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU= 2024-04-13T21:24:40.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU=', 'Content-Length': '0'} 2024-04-13T21:24:40.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035cddf06e6a3c782-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.208 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.208 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.208 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.208 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.208 DEBUG:boto:Data: 2024-04-13T21:24:40.209 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.209 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.209 DEBUG:boto:Params: {} 2024-04-13T21:24:40.209 DEBUG:boto:Token: None 2024-04-13T21:24:40.209 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s= 2024-04-13T21:24:40.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s=', 'Content-Length': '0'} 2024-04-13T21:24:40.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003423afc555445d5-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.211 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.211 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.211 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.211 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.212 DEBUG:boto:Data: 2024-04-13T21:24:40.212 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.212 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.212 DEBUG:boto:Params: {} 2024-04-13T21:24:40.212 DEBUG:boto:Token: None 2024-04-13T21:24:40.213 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU= 2024-04-13T21:24:40.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU=', 'Content-Length': '0'} 2024-04-13T21:24:40.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e91cf221325ff7c-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.215 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.215 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.215 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.215 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.215 DEBUG:boto:Data: 2024-04-13T21:24:40.215 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.215 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.216 DEBUG:boto:Params: {} 2024-04-13T21:24:40.216 DEBUG:boto:Token: None 2024-04-13T21:24:40.216 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s= 2024-04-13T21:24:40.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s=', 'Content-Length': '0'} 2024-04-13T21:24:40.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005930441b7acf2487-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.218 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:40.266 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:40.266 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:40.297 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:40.308 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043479.923131_45.1", 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:24:39.923131Z" 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.310 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.310 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:40.310 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.310 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:40.311 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:40.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.311 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:40.358 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:40.359 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:40.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:40.394+0000 7f40384cb780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:40.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:40.395+0000 7f40384cb780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:40.400 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:40.400 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:40.400 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:40.400 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:40.400 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043479.923131_45.1", 2024-04-13T21:24:40.402 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:39.923131Z", 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:40.404 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:40Z" 2024-04-13T21:24:40.406 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:40.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043479.923131_45.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:39.923131Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:40Z" } 2024-04-13T21:24:40.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043479.923131_45.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:39.923131Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:40.408 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.408 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:40.408 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.408 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.409 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.409 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.409 DEBUG:boto:Data: 2024-04-13T21:24:40.409 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.409 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.409 DEBUG:boto:Params: {} 2024-04-13T21:24:40.409 DEBUG:boto:Token: None 2024-04-13T21:24:40.409 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU= 2024-04-13T21:24:40.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU=', 'Content-Length': '0'} 2024-04-13T21:24:40.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035d5776ccbd20706-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.412 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.412 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.412 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.412 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.412 DEBUG:boto:Data: 2024-04-13T21:24:40.412 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.413 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.413 DEBUG:boto:Params: {} 2024-04-13T21:24:40.413 DEBUG:boto:Token: None 2024-04-13T21:24:40.413 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s= 2024-04-13T21:24:40.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s=', 'Content-Length': '0'} 2024-04-13T21:24:40.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f5010226567a96c-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.415 DEBUG:boto:path=/vwnfns-3/ 2024-04-13T21:24:40.415 DEBUG:boto:auth_path=/vwnfns-3/ 2024-04-13T21:24:40.415 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.415 DEBUG:boto:Path: /vwnfns-3/ 2024-04-13T21:24:40.415 DEBUG:boto:Data: 2024-04-13T21:24:40.416 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.416 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.416 DEBUG:boto:Params: {} 2024-04-13T21:24:40.416 DEBUG:boto:Token: None 2024-04-13T21:24:40.416 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-3/ 2024-04-13T21:24:40.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU= 2024-04-13T21:24:40.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVg8jVzAxc9Rn/68Jl2oh2mX5mU=', 'Content-Length': '0'} 2024-04-13T21:24:40.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031f4495ef22988ff-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.418 DEBUG:boto:path=/vwnfns-4/ 2024-04-13T21:24:40.418 DEBUG:boto:auth_path=/vwnfns-4/ 2024-04-13T21:24:40.418 DEBUG:boto:Method: HEAD 2024-04-13T21:24:40.418 DEBUG:boto:Path: /vwnfns-4/ 2024-04-13T21:24:40.419 DEBUG:boto:Data: 2024-04-13T21:24:40.419 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.419 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.419 DEBUG:boto:Params: {} 2024-04-13T21:24:40.419 DEBUG:boto:Token: None 2024-04-13T21:24:40.419 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-4/ 2024-04-13T21:24:40.419 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s= 2024-04-13T21:24:40.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CAwZPgOWgnVDPZ7u5rweoFqs9s=', 'Content-Length': '0'} 2024-04-13T21:24:40.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094c0329ad97655ed-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-04-13T21:24:40.422 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-5 2024-04-13T21:24:40.422 DEBUG:boto:path=/vwnfns-5/ 2024-04-13T21:24:40.422 DEBUG:boto:auth_path=/vwnfns-5/ 2024-04-13T21:24:40.422 DEBUG:boto:Method: PUT 2024-04-13T21:24:40.422 DEBUG:boto:Path: /vwnfns-5/ 2024-04-13T21:24:40.423 DEBUG:boto:Data: 2024-04-13T21:24:40.423 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:40.423 DEBUG:boto:Port: 8000 2024-04-13T21:24:40.423 DEBUG:boto:Params: {} 2024-04-13T21:24:40.423 DEBUG:boto:Token: None 2024-04-13T21:24:40.423 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-5/ 2024-04-13T21:24:40.423 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qEj73FFRFuH/ZlD6WnvGl3nKig= 2024-04-13T21:24:40.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qEj73FFRFuH/ZlD6WnvGl3nKig=', 'Content-Length': '0'} 2024-04-13T21:24:40.436 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000058c0fbe7e8b1eab2-00661af818-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.436 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-6 2024-04-13T21:24:40.436 DEBUG:boto:path=/vwnfns-6/ 2024-04-13T21:24:40.436 DEBUG:boto:auth_path=/vwnfns-6/ 2024-04-13T21:24:40.436 DEBUG:boto:Method: PUT 2024-04-13T21:24:40.436 DEBUG:boto:Path: /vwnfns-6/ 2024-04-13T21:24:40.436 DEBUG:boto:Data: 2024-04-13T21:24:40.437 DEBUG:boto:Headers: {} 2024-04-13T21:24:40.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:40.437 DEBUG:boto:Port: 8001 2024-04-13T21:24:40.437 DEBUG:boto:Params: {} 2024-04-13T21:24:40.437 DEBUG:boto:Token: None 2024-04-13T21:24:40.437 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:40 GMT /vwnfns-6/ 2024-04-13T21:24:40.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GdijbpV/XlcqWARq+NJR+Y8250U= 2024-04-13T21:24:40.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GdijbpV/XlcqWARq+NJR+Y8250U=', 'Content-Length': '0'} 2024-04-13T21:24:40.458 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008101b8c3ac0dc2e1-00661af818-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:40.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.459 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:40.506 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:40.506 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:40.534 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:40.548 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:40.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.548 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:40.548 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:40.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.549 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043480.457926_53.1", 2024-04-13T21:24:40.549 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:24:40.457926Z" 2024-04-13T21:24:40.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.550 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.550 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:40.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.551 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.551 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:40.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.551 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:40.553 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043350.757914_77.1', 1: '1_1713043480.457926_53.1', 2: '', 3: ''} 2024-04-13T21:24:40.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:40.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:40.555 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:40.603 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:40.603 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:40.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:40.637+0000 7f2f57dfe780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:40.632 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:40.638+0000 7f2f57dfe780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:40.653 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:40.653 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:40.653 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:40.653 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:40.653 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:40.653 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.654 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:40.655 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043479.923131_45.1", 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:39.923131Z", 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:40.656 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:40.657 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:40.658 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:40.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:40.660 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:40Z" 2024-04-13T21:24:40.660 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:40.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043479.923131_45.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:39.923131Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:40Z" } 2024-04-13T21:24:40.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043479.923131_45.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:39.923131Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:40.661 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043480.457926_53.1', 2: '', 3: ''} 2024-04-13T21:24:40.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043479.923131_45.1', 2: '', 3: ''} 2024-04-13T21:24:40.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1713043480.457926_53.1 target=1_1713043479.923131_45.1 2024-04-13T21:24:45.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:45.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:45.663 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:45.718 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:45.719 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:45.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:45.755+0000 7f106820a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:45.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:45.756+0000 7f106820a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:45.762 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:45.762 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:45.762 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:45.762 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:45.763 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:45.764 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043480.457926_53.1", 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:40.457926Z", 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.765 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:45.766 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:45.767 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:45Z" 2024-04-13T21:24:45.768 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:45.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043480.457926_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:40.457926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:45Z" } 2024-04-13T21:24:45.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043480.457926_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:40.457926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:45.770 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043480.457926_53.1', 2: '', 3: ''} 2024-04-13T21:24:45.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043480.457926_53.1', 2: '', 3: ''} 2024-04-13T21:24:45.770 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:45.770 DEBUG:boto:path=/vwnfns-5/ 2024-04-13T21:24:45.770 DEBUG:boto:auth_path=/vwnfns-5/ 2024-04-13T21:24:45.771 DEBUG:boto:Method: HEAD 2024-04-13T21:24:45.771 DEBUG:boto:Path: /vwnfns-5/ 2024-04-13T21:24:45.771 DEBUG:boto:Data: 2024-04-13T21:24:45.771 DEBUG:boto:Headers: {} 2024-04-13T21:24:45.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:45.772 DEBUG:boto:Port: 8000 2024-04-13T21:24:45.772 DEBUG:boto:Params: {} 2024-04-13T21:24:45.772 DEBUG:boto:Token: None 2024-04-13T21:24:45.772 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:45 GMT /vwnfns-5/ 2024-04-13T21:24:45.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMqsJgNmuY7SQ2mlpuExZD1o5fw= 2024-04-13T21:24:45.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMqsJgNmuY7SQ2mlpuExZD1o5fw=', 'Content-Length': '0'} 2024-04-13T21:24:45.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003540a85ffada85ab-00661af81d-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:45.775 DEBUG:boto:path=/vwnfns-6/ 2024-04-13T21:24:45.775 DEBUG:boto:auth_path=/vwnfns-6/ 2024-04-13T21:24:45.775 DEBUG:boto:Method: HEAD 2024-04-13T21:24:45.775 DEBUG:boto:Path: /vwnfns-6/ 2024-04-13T21:24:45.775 DEBUG:boto:Data: 2024-04-13T21:24:45.775 DEBUG:boto:Headers: {} 2024-04-13T21:24:45.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:45.775 DEBUG:boto:Port: 8000 2024-04-13T21:24:45.775 DEBUG:boto:Params: {} 2024-04-13T21:24:45.775 DEBUG:boto:Token: None 2024-04-13T21:24:45.776 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:45 GMT /vwnfns-6/ 2024-04-13T21:24:45.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sBwAmOwEvcRtilZCMx16la9PwQk= 2024-04-13T21:24:45.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sBwAmOwEvcRtilZCMx16la9PwQk=', 'Content-Length': '0'} 2024-04-13T21:24:45.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076bf1f9d206e3c6d-00661af81d-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:45.778 DEBUG:boto:path=/vwnfns-5/ 2024-04-13T21:24:45.778 DEBUG:boto:auth_path=/vwnfns-5/ 2024-04-13T21:24:45.778 DEBUG:boto:Method: HEAD 2024-04-13T21:24:45.778 DEBUG:boto:Path: /vwnfns-5/ 2024-04-13T21:24:45.778 DEBUG:boto:Data: 2024-04-13T21:24:45.778 DEBUG:boto:Headers: {} 2024-04-13T21:24:45.778 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:45.778 DEBUG:boto:Port: 8001 2024-04-13T21:24:45.778 DEBUG:boto:Params: {} 2024-04-13T21:24:45.778 DEBUG:boto:Token: None 2024-04-13T21:24:45.779 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:45 GMT /vwnfns-5/ 2024-04-13T21:24:45.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMqsJgNmuY7SQ2mlpuExZD1o5fw= 2024-04-13T21:24:45.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMqsJgNmuY7SQ2mlpuExZD1o5fw=', 'Content-Length': '0'} 2024-04-13T21:24:45.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9d8032f8215aa2b-00661af81d-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:45.781 DEBUG:boto:path=/vwnfns-6/ 2024-04-13T21:24:45.781 DEBUG:boto:auth_path=/vwnfns-6/ 2024-04-13T21:24:45.782 DEBUG:boto:Method: HEAD 2024-04-13T21:24:45.782 DEBUG:boto:Path: /vwnfns-6/ 2024-04-13T21:24:45.782 DEBUG:boto:Data: 2024-04-13T21:24:45.782 DEBUG:boto:Headers: {} 2024-04-13T21:24:45.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:45.782 DEBUG:boto:Port: 8001 2024-04-13T21:24:45.782 DEBUG:boto:Params: {} 2024-04-13T21:24:45.782 DEBUG:boto:Token: None 2024-04-13T21:24:45.782 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:45 GMT /vwnfns-6/ 2024-04-13T21:24:45.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sBwAmOwEvcRtilZCMx16la9PwQk= 2024-04-13T21:24:45.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sBwAmOwEvcRtilZCMx16la9PwQk=', 'Content-Length': '0'} 2024-04-13T21:24:45.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000738dfc180f0ce5b5-00661af81d-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:45.785 DEBUG:boto:path=/vwnfns-5/ 2024-04-13T21:24:45.785 DEBUG:boto:auth_path=/vwnfns-5/ 2024-04-13T21:24:45.785 DEBUG:boto:Method: DELETE 2024-04-13T21:24:45.785 DEBUG:boto:Path: /vwnfns-5/ 2024-04-13T21:24:45.785 DEBUG:boto:Data: 2024-04-13T21:24:45.785 DEBUG:boto:Headers: {} 2024-04-13T21:24:45.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:45.786 DEBUG:boto:Port: 8000 2024-04-13T21:24:45.786 DEBUG:boto:Params: {} 2024-04-13T21:24:45.786 DEBUG:boto:Token: None 2024-04-13T21:24:45.786 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:24:45 GMT /vwnfns-5/ 2024-04-13T21:24:45.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l0A+5/wQYa4oGkD+lkPGzz8FqgY= 2024-04-13T21:24:45.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l0A+5/wQYa4oGkD+lkPGzz8FqgY=', 'Content-Length': '0'} 2024-04-13T21:24:45.812 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7cff306eb7b8e58-00661af81d-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:45.813 DEBUG:boto:path=/vwnfns-6/ 2024-04-13T21:24:45.813 DEBUG:boto:auth_path=/vwnfns-6/ 2024-04-13T21:24:45.813 DEBUG:boto:Method: DELETE 2024-04-13T21:24:45.813 DEBUG:boto:Path: /vwnfns-6/ 2024-04-13T21:24:45.813 DEBUG:boto:Data: 2024-04-13T21:24:45.814 DEBUG:boto:Headers: {} 2024-04-13T21:24:45.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:45.814 DEBUG:boto:Port: 8001 2024-04-13T21:24:45.814 DEBUG:boto:Params: {} 2024-04-13T21:24:45.814 DEBUG:boto:Token: None 2024-04-13T21:24:45.814 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:24:45 GMT /vwnfns-6/ 2024-04-13T21:24:45.814 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M2Zyk720p3QiQspa96l6Kuf22+E= 2024-04-13T21:24:45.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M2Zyk720p3QiQspa96l6Kuf22+E=', 'Content-Length': '0'} 2024-04-13T21:24:45.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038063876e0b1f429-00661af81d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:45.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:45.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:45.872 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:45.948 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:45.959 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043485.857118_61.1", 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:24:45.857118Z" 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:45.960 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:45.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:45.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:45.961 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:45.961 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:45.961 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:45.961 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:45.962 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043350.757914_77.1', 1: '1_1713043485.857118_61.1', 2: '', 3: ''} 2024-04-13T21:24:45.962 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:45.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:45.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:45.963 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:46.010 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:46.011 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:46.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:46.047+0000 7fb65243b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:46.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:46.047+0000 7fb65243b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:46.052 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:46.052 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:46.052 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:46.052 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:46.052 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:46.053 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:46.054 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043485.838896_58.1", 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:45.838896Z", 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:46.055 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:46.056 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:46Z" 2024-04-13T21:24:46.058 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:46.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043485.838896_58.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:45.838896Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:46Z" } 2024-04-13T21:24:46.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043485.838896_58.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:45.838896Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:46.060 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043485.857118_61.1', 2: '', 3: ''} 2024-04-13T21:24:46.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043485.838896_58.1', 2: '', 3: ''} 2024-04-13T21:24:46.060 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1713043485.857118_61.1 target=1_1713043485.838896_58.1 2024-04-13T21:24:48.617 INFO:tasks.rgw.c2.client.0.smithi175.stdout:2024-04-13T21:24:48.625+0000 7fa9b5224700 -1 rgw async rados processor: could not delete bucket=vwnfns-5 2024-04-13T21:24:48.655 INFO:tasks.rgw.c2.client.0.smithi175.stdout:2024-04-13T21:24:48.662+0000 7fa9b7a29700 -1 rgw async rados processor: could not delete bucket=vwnfns-6 2024-04-13T21:24:51.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:51.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:51.061 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:51.118 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:51.118 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:51.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:51.155+0000 7f9a35f0f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:51.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:51.156+0000 7f9a35f0f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:51.161 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:51.161 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:51.162 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.163 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043485.857118_61.1", 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:45.857118Z", 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:51.165 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.166 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:51.166 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.166 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.167 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:51.167 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.167 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:51.167 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.167 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:51.167 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.168 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.168 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.168 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:51.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.168 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:51.169 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:51.169 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:51.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.169 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:51Z" 2024-04-13T21:24:51.169 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:51.170 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043485.857118_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:45.857118Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:51Z" } 2024-04-13T21:24:51.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043485.857118_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:45.857118Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:51.171 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043485.857118_61.1', 2: '', 3: ''} 2024-04-13T21:24:51.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043485.857118_61.1', 2: '', 3: ''} 2024-04-13T21:24:51.171 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:51.172 DEBUG:boto:path=/vwnfns-5/ 2024-04-13T21:24:51.172 DEBUG:boto:auth_path=/vwnfns-5/ 2024-04-13T21:24:51.172 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.172 DEBUG:boto:Path: /vwnfns-5/ 2024-04-13T21:24:51.172 DEBUG:boto:Data: 2024-04-13T21:24:51.172 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.172 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.173 DEBUG:boto:Params: {} 2024-04-13T21:24:51.173 DEBUG:boto:Token: None 2024-04-13T21:24:51.173 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-5/ 2024-04-13T21:24:51.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pr/wVo+FlkQGkM28QZcbKWov4Ck= 2024-04-13T21:24:51.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pr/wVo+FlkQGkM28QZcbKWov4Ck=', 'Content-Length': '0'} 2024-04-13T21:24:51.175 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000414e9c1b1a9c3a63-00661af823-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.175 DEBUG:boto:path=/vwnfns-6/ 2024-04-13T21:24:51.175 DEBUG:boto:auth_path=/vwnfns-6/ 2024-04-13T21:24:51.175 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.175 DEBUG:boto:Path: /vwnfns-6/ 2024-04-13T21:24:51.175 DEBUG:boto:Data: 2024-04-13T21:24:51.175 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.176 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.176 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.176 DEBUG:boto:Params: {} 2024-04-13T21:24:51.176 DEBUG:boto:Token: None 2024-04-13T21:24:51.176 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-6/ 2024-04-13T21:24:51.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bZyaOEdyBGntwK24GCmYwz7S2B8= 2024-04-13T21:24:51.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bZyaOEdyBGntwK24GCmYwz7S2B8=', 'Content-Length': '0'} 2024-04-13T21:24:51.178 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000dbba51bd5bb30b26-00661af823-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.178 DEBUG:boto:path=/vwnfns-5/ 2024-04-13T21:24:51.178 DEBUG:boto:auth_path=/vwnfns-5/ 2024-04-13T21:24:51.178 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.178 DEBUG:boto:Path: /vwnfns-5/ 2024-04-13T21:24:51.178 DEBUG:boto:Data: 2024-04-13T21:24:51.178 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.179 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.179 DEBUG:boto:Params: {} 2024-04-13T21:24:51.179 DEBUG:boto:Token: None 2024-04-13T21:24:51.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-5/ 2024-04-13T21:24:51.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pr/wVo+FlkQGkM28QZcbKWov4Ck= 2024-04-13T21:24:51.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pr/wVo+FlkQGkM28QZcbKWov4Ck=', 'Content-Length': '0'} 2024-04-13T21:24:51.184 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000086d5390c95d638f5-00661af823-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.184 DEBUG:boto:path=/vwnfns-6/ 2024-04-13T21:24:51.184 DEBUG:boto:auth_path=/vwnfns-6/ 2024-04-13T21:24:51.185 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.185 DEBUG:boto:Path: /vwnfns-6/ 2024-04-13T21:24:51.185 DEBUG:boto:Data: 2024-04-13T21:24:51.185 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.185 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.185 DEBUG:boto:Params: {} 2024-04-13T21:24:51.185 DEBUG:boto:Token: None 2024-04-13T21:24:51.185 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-6/ 2024-04-13T21:24:51.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bZyaOEdyBGntwK24GCmYwz7S2B8= 2024-04-13T21:24:51.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bZyaOEdyBGntwK24GCmYwz7S2B8=', 'Content-Length': '0'} 2024-04-13T21:24:51.187 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000009216923cc75dde06-00661af823-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-04-13T21:24:51.188 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-7 2024-04-13T21:24:51.188 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:24:51.188 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:24:51.189 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.189 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:24:51.189 DEBUG:boto:Data: 2024-04-13T21:24:51.189 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.189 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.189 DEBUG:boto:Params: {} 2024-04-13T21:24:51.189 DEBUG:boto:Token: None 2024-04-13T21:24:51.189 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/ 2024-04-13T21:24:51.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mvInA0VcWPwx4ey+vZC+3cM4EiU= 2024-04-13T21:24:51.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mvInA0VcWPwx4ey+vZC+3cM4EiU=', 'Content-Length': '0'} 2024-04-13T21:24:51.208 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006c5dcd3b532583b-00661af823-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.208 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-8 2024-04-13T21:24:51.209 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:24:51.209 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:24:51.209 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.209 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:24:51.209 DEBUG:boto:Data: 2024-04-13T21:24:51.209 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.209 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.210 DEBUG:boto:Params: {} 2024-04-13T21:24:51.210 DEBUG:boto:Token: None 2024-04-13T21:24:51.210 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/ 2024-04-13T21:24:51.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CuYmBtyst0cy1uIX4EtMPX6yQ6I= 2024-04-13T21:24:51.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CuYmBtyst0cy1uIX4EtMPX6yQ6I=', 'Content-Length': '0'} 2024-04-13T21:24:51.251 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d48b4c393848dfa0-00661af823-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.251 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:24:51.251 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:24:51.252 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.252 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:24:51.252 DEBUG:boto:Data: 2024-04-13T21:24:51.252 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.253 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.253 DEBUG:boto:Params: {} 2024-04-13T21:24:51.253 DEBUG:boto:Token: None 2024-04-13T21:24:51.254 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/ 2024-04-13T21:24:51.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U= 2024-04-13T21:24:51.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U=', 'Content-Length': '0'} 2024-04-13T21:24:51.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f820a8986fdc6fd-00661af823-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.258 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:24:51.258 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:24:51.258 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.259 DEBUG:boto:Path: /vwnfns-7/myobj 2024-04-13T21:24:51.259 DEBUG:boto:Data: 2024-04-13T21:24:51.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.259 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.260 DEBUG:boto:Params: {} 2024-04-13T21:24:51.260 DEBUG:boto:Token: None 2024-04-13T21:24:51.260 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/myobj 2024-04-13T21:24:51.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xZGAEJM1WUF33iq9v07tclTAMTU= 2024-04-13T21:24:51.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xZGAEJM1WUF33iq9v07tclTAMTU='} 2024-04-13T21:24:51.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004eaf9570183ad9ae-00661af823-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.278 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:24:51.278 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:24:51.278 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.279 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:24:51.279 DEBUG:boto:Data: 2024-04-13T21:24:51.279 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.279 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.279 DEBUG:boto:Params: {} 2024-04-13T21:24:51.280 DEBUG:boto:Token: None 2024-04-13T21:24:51.280 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/ 2024-04-13T21:24:51.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U= 2024-04-13T21:24:51.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U=', 'Content-Length': '0'} 2024-04-13T21:24:51.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbc2bb018bff346d-00661af823-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.283 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:24:51.283 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:24:51.283 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.283 DEBUG:boto:Path: /vwnfns-7/_myobj 2024-04-13T21:24:51.284 DEBUG:boto:Data: 2024-04-13T21:24:51.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.284 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.284 DEBUG:boto:Params: {} 2024-04-13T21:24:51.285 DEBUG:boto:Token: None 2024-04-13T21:24:51.285 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/_myobj 2024-04-13T21:24:51.285 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rpzJQqVTn+CRvhmbtZ6424eFuxE= 2024-04-13T21:24:51.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rpzJQqVTn+CRvhmbtZ6424eFuxE='} 2024-04-13T21:24:51.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000696d36a6dc8af5a8-00661af823-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.335 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:24:51.335 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:24:51.335 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.335 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:24:51.335 DEBUG:boto:Data: 2024-04-13T21:24:51.335 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.335 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.335 DEBUG:boto:Params: {} 2024-04-13T21:24:51.336 DEBUG:boto:Token: None 2024-04-13T21:24:51.336 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/ 2024-04-13T21:24:51.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U= 2024-04-13T21:24:51.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U=', 'Content-Length': '0'} 2024-04-13T21:24:51.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000896062ac085d23d-00661af823-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.338 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:24:51.338 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:24:51.338 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.339 DEBUG:boto:Path: /vwnfns-7/%3A 2024-04-13T21:24:51.339 DEBUG:boto:Data: 2024-04-13T21:24:51.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.340 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.340 DEBUG:boto:Params: {} 2024-04-13T21:24:51.340 DEBUG:boto:Token: None 2024-04-13T21:24:51.340 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/%3A 2024-04-13T21:24:51.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PkIjoyvSAJq061hKdsYes33F+wM= 2024-04-13T21:24:51.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PkIjoyvSAJq061hKdsYes33F+wM='} 2024-04-13T21:24:51.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e56d7453d42e181a-00661af823-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.349 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:24:51.350 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:24:51.350 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.350 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:24:51.350 DEBUG:boto:Data: 2024-04-13T21:24:51.350 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.350 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.350 DEBUG:boto:Params: {} 2024-04-13T21:24:51.351 DEBUG:boto:Token: None 2024-04-13T21:24:51.351 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/ 2024-04-13T21:24:51.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U= 2024-04-13T21:24:51.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fGNYqI1ZpDAYtz3IvSTj47miT0U=', 'Content-Length': '0'} 2024-04-13T21:24:51.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a7fab435d8c5f06-00661af823-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.353 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:24:51.353 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:24:51.353 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.354 DEBUG:boto:Path: /vwnfns-7/%26 2024-04-13T21:24:51.354 DEBUG:boto:Data: 2024-04-13T21:24:51.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:24:51.354 DEBUG:boto:Port: 8000 2024-04-13T21:24:51.354 DEBUG:boto:Params: {} 2024-04-13T21:24:51.354 DEBUG:boto:Token: None 2024-04-13T21:24:51.354 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-7/%26 2024-04-13T21:24:51.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MFBreBuQi+5cJp8KDIx+i0Gt8Jk= 2024-04-13T21:24:51.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MFBreBuQi+5cJp8KDIx+i0Gt8Jk='} 2024-04-13T21:24:51.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000108ea84dc95da82-00661af823-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.363 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:24:51.364 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:24:51.364 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.364 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:24:51.364 DEBUG:boto:Data: 2024-04-13T21:24:51.364 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.364 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.364 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.364 DEBUG:boto:Params: {} 2024-04-13T21:24:51.364 DEBUG:boto:Token: None 2024-04-13T21:24:51.365 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/ 2024-04-13T21:24:51.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y= 2024-04-13T21:24:51.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y=', 'Content-Length': '0'} 2024-04-13T21:24:51.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c2a2c56f8e31dc0-00661af823-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.367 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:24:51.368 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:24:51.368 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.368 DEBUG:boto:Path: /vwnfns-8/myobj 2024-04-13T21:24:51.368 DEBUG:boto:Data: 2024-04-13T21:24:51.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.368 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.368 DEBUG:boto:Params: {} 2024-04-13T21:24:51.368 DEBUG:boto:Token: None 2024-04-13T21:24:51.368 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/myobj 2024-04-13T21:24:51.369 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llHVUHhoJ4oHPEDZYku3wjasO18= 2024-04-13T21:24:51.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llHVUHhoJ4oHPEDZYku3wjasO18='} 2024-04-13T21:24:51.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c05cf75dad77af54-00661af823-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.377 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:24:51.378 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:24:51.378 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.378 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:24:51.378 DEBUG:boto:Data: 2024-04-13T21:24:51.378 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.378 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.378 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.378 DEBUG:boto:Params: {} 2024-04-13T21:24:51.378 DEBUG:boto:Token: None 2024-04-13T21:24:51.379 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/ 2024-04-13T21:24:51.379 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y= 2024-04-13T21:24:51.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y=', 'Content-Length': '0'} 2024-04-13T21:24:51.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a951f467e1c498ee-00661af823-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.381 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:24:51.381 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:24:51.382 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.382 DEBUG:boto:Path: /vwnfns-8/_myobj 2024-04-13T21:24:51.382 DEBUG:boto:Data: 2024-04-13T21:24:51.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.382 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.382 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.382 DEBUG:boto:Params: {} 2024-04-13T21:24:51.383 DEBUG:boto:Token: None 2024-04-13T21:24:51.383 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/_myobj 2024-04-13T21:24:51.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6LpYEwoLycPK1Oz5kS/9hsQY1ZQ= 2024-04-13T21:24:51.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6LpYEwoLycPK1Oz5kS/9hsQY1ZQ='} 2024-04-13T21:24:51.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fbfd887d9449eb7-00661af823-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.427 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:24:51.427 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:24:51.428 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.428 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:24:51.428 DEBUG:boto:Data: 2024-04-13T21:24:51.428 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.429 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.429 DEBUG:boto:Params: {} 2024-04-13T21:24:51.429 DEBUG:boto:Token: None 2024-04-13T21:24:51.429 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/ 2024-04-13T21:24:51.430 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y= 2024-04-13T21:24:51.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y=', 'Content-Length': '0'} 2024-04-13T21:24:51.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e128b048e81da12-00661af823-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.432 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:24:51.433 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:24:51.433 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.433 DEBUG:boto:Path: /vwnfns-8/%3A 2024-04-13T21:24:51.433 DEBUG:boto:Data: 2024-04-13T21:24:51.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.433 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.433 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.433 DEBUG:boto:Params: {} 2024-04-13T21:24:51.433 DEBUG:boto:Token: None 2024-04-13T21:24:51.434 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/%3A 2024-04-13T21:24:51.434 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tIjrgo1ZIr3TY/hvAgUCezMyTTs= 2024-04-13T21:24:51.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tIjrgo1ZIr3TY/hvAgUCezMyTTs='} 2024-04-13T21:24:51.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5bad08420b7a058-00661af823-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.442 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:24:51.442 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:24:51.442 DEBUG:boto:Method: HEAD 2024-04-13T21:24:51.443 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:24:51.443 DEBUG:boto:Data: 2024-04-13T21:24:51.443 DEBUG:boto:Headers: {} 2024-04-13T21:24:51.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.443 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.443 DEBUG:boto:Params: {} 2024-04-13T21:24:51.444 DEBUG:boto:Token: None 2024-04-13T21:24:51.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/ 2024-04-13T21:24:51.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y= 2024-04-13T21:24:51.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/jaRCadak9IyC5cqtm2rGwrrY5Y=', 'Content-Length': '0'} 2024-04-13T21:24:51.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007257302c2adc2e3e-00661af823-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.447 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:24:51.447 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:24:51.447 DEBUG:boto:Method: PUT 2024-04-13T21:24:51.447 DEBUG:boto:Path: /vwnfns-8/%26 2024-04-13T21:24:51.448 DEBUG:boto:Data: 2024-04-13T21:24:51.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:24:51.448 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:24:51.448 DEBUG:boto:Port: 8001 2024-04-13T21:24:51.448 DEBUG:boto:Params: {} 2024-04-13T21:24:51.449 DEBUG:boto:Token: None 2024-04-13T21:24:51.449 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:24:51 GMT /vwnfns-8/%26 2024-04-13T21:24:51.449 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5xBVlAxAF3TZTF7xF9qnwaIcNJo= 2024-04-13T21:24:51.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:24:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5xBVlAxAF3TZTF7xF9qnwaIcNJo='} 2024-04-13T21:24:51.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000536bd31dca34bcd6-00661af823-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:24:51.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:51.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:51.455 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:51.503 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:51.553 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:51.554 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:22:30.757914Z" 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043491.238779_69.1", 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:24:51.238779Z" 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.556 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:24:51.557 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043350.757914_77.1', 1: '1_1713043491.238779_69.1', 2: '', 3: ''} 2024-04-13T21:24:51.557 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:24:51.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:51.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:24:51.557 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:24:51.605 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:51.605 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:51.632 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:51.638+0000 7f795b975780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:51.632 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:51.639+0000 7f795b975780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:24:51.644 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:24:51.644 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:24:51.644 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:24:51.644 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:24:51.644 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:24:51.644 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043350.757914_77.1", 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:24:51.645 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:22:30.757914Z", 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043491.238779_69.1", 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.646 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:24:51.238779Z", 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:51.647 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:24:51.648 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:24:51.649 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:24:51.649 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:24:51.650 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:24:51.651 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:24:51.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:24:51.651 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:24:51Z" 2024-04-13T21:24:51.651 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:24:51.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043350.757914_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:22:30.757914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043491.238779_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:24:51.238779Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:24:51Z" } 2024-04-13T21:24:51.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043350.757914_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:22:30.757914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043491.238779_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:24:51.238779Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:24:51.652 DEBUG:rgw_multi.tests:log_status={0: '1_1713043350.757914_77.1', 1: '1_1713043491.238779_69.1', 2: '', 3: ''} 2024-04-13T21:24:51.652 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043350.757914_77.1', 1: '1_1713043491.238779_69.1', 2: '', 3: ''} 2024-04-13T21:24:51.652 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:24:51.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:24:51.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:24:51.653 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:24:51.707 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:24:51.707 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:24:51.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:51.750+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:24:56.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:24:56.751+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:01.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:01.752+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:06.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:06.753+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:11.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:11.754+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:16.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:16.754+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:21.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:21.754+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:26.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:26.755+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:31.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:31.755+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:36.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:36.756+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:41.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:41.758+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:46.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:46.758+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:51.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:51.759+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:25:56.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:25:56.760+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:01.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:01.760+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:06.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:06.760+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:11.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:11.761+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:16.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:16.762+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:21.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:21.763+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:26.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:26.764+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:31.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:31.764+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:36.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:36.764+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:41.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:41.765+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:46.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:46.766+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:51.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:51.766+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:26:56.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:26:56.767+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:27:01.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:01.768+0000 7fc4adc9f780 1 waiting to reach incremental sync.. 2024-04-13T21:27:06.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:06.770+0000 7fc4adc9f780 1 bucket sync caught up with source: 2024-04-13T21:27:06.763 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.3.6, , 00000000001.5.6, , , , 00000000001.5.6, 00000000001.4.6, , ] 2024-04-13T21:27:06.763 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.3.6, , 00000000001.5.6, , , , 00000000001.5.6, 00000000001.4.6, , ] 2024-04-13T21:27:06.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:06.770+0000 7fc4adc9f780 0 bucket checkpoint complete 2024-04-13T21:27:06.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-7 zones={a2, a1} 2024-04-13T21:27:06.779 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.779 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.779 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.779 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:27:06.780 DEBUG:boto:Data: 2024-04-13T21:27:06.780 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.780 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.780 DEBUG:boto:Params: {} 2024-04-13T21:27:06.780 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:27:06.781 DEBUG:boto:Token: None 2024-04-13T21:27:06.781 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.781 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xEHSWAzawX63hS9Eq9gHR1w6s5c= 2024-04-13T21:27:06.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xEHSWAzawX63hS9Eq9gHR1w6s5c=', 'Content-Length': '0'} 2024-04-13T21:27:06.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7cdc56bed852589-00661af8aa-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.786 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.786 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.787 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.787 DEBUG:boto:Path: /vwnfns-7/ 2024-04-13T21:27:06.787 DEBUG:boto:Data: 2024-04-13T21:27:06.787 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.787 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.787 DEBUG:boto:Params: {} 2024-04-13T21:27:06.787 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:27:06.787 DEBUG:boto:Token: None 2024-04-13T21:27:06.787 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xEHSWAzawX63hS9Eq9gHR1w6s5c= 2024-04-13T21:27:06.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xEHSWAzawX63hS9Eq9gHR1w6s5c=', 'Content-Length': '0'} 2024-04-13T21:27:06.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf34bd3762157d92-00661af8aa-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.791 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:27:06.792 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.792 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.792 DEBUG:boto:path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.792 DEBUG:boto:auth_path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.792 DEBUG:boto:Method: GET 2024-04-13T21:27:06.792 DEBUG:boto:Path: /vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.792 DEBUG:boto:Data: 2024-04-13T21:27:06.792 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.792 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.792 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.792 DEBUG:boto:Params: {} 2024-04-13T21:27:06.793 DEBUG:boto:Token: None 2024-04-13T21:27:06.793 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/?versions 2024-04-13T21:27:06.793 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY= 2024-04-13T21:27:06.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY=', 'Content-Length': '0'} 2024-04-13T21:27:06.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad5c82ebf3bf5b3b-00661af8aa-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.795 DEBUG:boto:b'vwnfns-7999false&nulltrue2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.798 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-13T21:27:06.798 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-13T21:27:06.798 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-13T21:27:06.798 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:27:06.798 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:27:06.798 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.798 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.799 DEBUG:boto:path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.799 DEBUG:boto:auth_path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.799 DEBUG:boto:Method: GET 2024-04-13T21:27:06.799 DEBUG:boto:Path: /vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.799 DEBUG:boto:Data: 2024-04-13T21:27:06.799 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.799 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.799 DEBUG:boto:Params: {} 2024-04-13T21:27:06.799 DEBUG:boto:Token: None 2024-04-13T21:27:06.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/?versions 2024-04-13T21:27:06.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY= 2024-04-13T21:27:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY=', 'Content-Length': '0'} 2024-04-13T21:27:06.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040997e8b3820fe09-00661af8aa-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.802 DEBUG:boto:b'vwnfns-7999false&nulltrue2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.802 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-13T21:27:06.803 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-13T21:27:06.803 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-13T21:27:06.803 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:27:06.803 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.803 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.803 DEBUG:boto:path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.803 DEBUG:boto:auth_path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.803 DEBUG:boto:Method: GET 2024-04-13T21:27:06.803 DEBUG:boto:Path: /vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.803 DEBUG:boto:Data: 2024-04-13T21:27:06.804 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.804 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.804 DEBUG:boto:Params: {} 2024-04-13T21:27:06.804 DEBUG:boto:Token: None 2024-04-13T21:27:06.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/?versions 2024-04-13T21:27:06.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY= 2024-04-13T21:27:06.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY=', 'Content-Length': '0'} 2024-04-13T21:27:06.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052bcd2808a4df61b-00661af8aa-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.806 DEBUG:boto:b'vwnfns-7999false&nulltrue2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.807 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.807 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.807 DEBUG:boto:path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.807 DEBUG:boto:auth_path=/vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.807 DEBUG:boto:Method: GET 2024-04-13T21:27:06.808 DEBUG:boto:Path: /vwnfns-7/?versions&max-keys=999 2024-04-13T21:27:06.808 DEBUG:boto:Data: 2024-04-13T21:27:06.808 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.808 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.808 DEBUG:boto:Params: {} 2024-04-13T21:27:06.808 DEBUG:boto:Token: None 2024-04-13T21:27:06.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/?versions 2024-04-13T21:27:06.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY= 2024-04-13T21:27:06.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GRb5FcPQapPrceYZBWrNENxJsUY=', 'Content-Length': '0'} 2024-04-13T21:27:06.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6d83c24e16117c-00661af8aa-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.811 DEBUG:boto:b'vwnfns-7999false&nulltrue2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-13T21:27:06.811 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.811 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.812 DEBUG:boto:path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.812 DEBUG:boto:auth_path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.812 DEBUG:boto:Method: GET 2024-04-13T21:27:06.812 DEBUG:boto:Path: /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.812 DEBUG:boto:Data: 2024-04-13T21:27:06.812 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.812 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.812 DEBUG:boto:Params: {} 2024-04-13T21:27:06.812 DEBUG:boto:Token: None 2024-04-13T21:27:06.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98= 2024-04-13T21:27:06.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98=', 'Content-Length': '0'} 2024-04-13T21:27:06.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000765fdefcc6d3adcc-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.815 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.815 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.815 DEBUG:boto:path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.815 DEBUG:boto:auth_path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.815 DEBUG:boto:Method: GET 2024-04-13T21:27:06.815 DEBUG:boto:Path: /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.815 DEBUG:boto:Data: 2024-04-13T21:27:06.816 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.816 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.816 DEBUG:boto:Params: {} 2024-04-13T21:27:06.816 DEBUG:boto:Token: None 2024-04-13T21:27:06.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98= 2024-04-13T21:27:06.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98=', 'Content-Length': '0'} 2024-04-13T21:27:06.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068ae4e2f74669bd9-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.818 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.818 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.818 DEBUG:boto:path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.818 DEBUG:boto:auth_path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.818 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.819 DEBUG:boto:Path: /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.819 DEBUG:boto:Data: 2024-04-13T21:27:06.819 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.819 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.819 DEBUG:boto:Params: {} 2024-04-13T21:27:06.819 DEBUG:boto:Token: None 2024-04-13T21:27:06.819 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vT2tYym+2Rne/HMavQdn886Nc0A= 2024-04-13T21:27:06.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vT2tYym+2Rne/HMavQdn886Nc0A=', 'Content-Length': '0'} 2024-04-13T21:27:06.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a63b20b04ed43a92-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.821 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.821 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.821 DEBUG:boto:path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.821 DEBUG:boto:auth_path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.822 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.822 DEBUG:boto:Path: /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.822 DEBUG:boto:Data: 2024-04-13T21:27:06.822 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.822 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.822 DEBUG:boto:Params: {} 2024-04-13T21:27:06.822 DEBUG:boto:Token: None 2024-04-13T21:27:06.822 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.822 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vT2tYym+2Rne/HMavQdn886Nc0A= 2024-04-13T21:27:06.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vT2tYym+2Rne/HMavQdn886Nc0A=', 'Content-Length': '0'} 2024-04-13T21:27:06.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc186d2f7efa332-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-13T21:27:06.824 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.824 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.825 DEBUG:boto:path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.825 DEBUG:boto:auth_path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.825 DEBUG:boto:Method: GET 2024-04-13T21:27:06.825 DEBUG:boto:Path: /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.825 DEBUG:boto:Data: 2024-04-13T21:27:06.825 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.825 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.825 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.825 DEBUG:boto:Params: {} 2024-04-13T21:27:06.825 DEBUG:boto:Token: None 2024-04-13T21:27:06.825 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.826 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98= 2024-04-13T21:27:06.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98=', 'Content-Length': '0'} 2024-04-13T21:27:06.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030c142a57cf8a9a5-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.870 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.870 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.870 DEBUG:boto:path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.871 DEBUG:boto:auth_path=/vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.871 DEBUG:boto:Method: GET 2024-04-13T21:27:06.871 DEBUG:boto:Path: /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.871 DEBUG:boto:Data: 2024-04-13T21:27:06.871 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.871 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.871 DEBUG:boto:Params: {} 2024-04-13T21:27:06.871 DEBUG:boto:Token: None 2024-04-13T21:27:06.871 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26?versionId=null 2024-04-13T21:27:06.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98= 2024-04-13T21:27:06.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZxhwmOHV8UT/aY2QaLqR/ajd98=', 'Content-Length': '0'} 2024-04-13T21:27:06.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c6c3b14f9852176-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.873 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.873 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.873 DEBUG:boto:path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.874 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.874 DEBUG:boto:Method: GET 2024-04-13T21:27:06.874 DEBUG:boto:Path: /vwnfns-7/?prefix=%26 2024-04-13T21:27:06.874 DEBUG:boto:Data: 2024-04-13T21:27:06.874 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.874 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.874 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.874 DEBUG:boto:Params: {} 2024-04-13T21:27:06.874 DEBUG:boto:Token: None 2024-04-13T21:27:06.874 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78= 2024-04-13T21:27:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78=', 'Content-Length': '0'} 2024-04-13T21:27:06.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eba85f7547ace66-00661af8aa-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.877 DEBUG:boto:b'vwnfns-7&1000false&2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.878 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.878 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.878 DEBUG:boto:path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.878 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.878 DEBUG:boto:Method: GET 2024-04-13T21:27:06.878 DEBUG:boto:Path: /vwnfns-7/?prefix=%26 2024-04-13T21:27:06.878 DEBUG:boto:Data: 2024-04-13T21:27:06.878 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.878 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.878 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.878 DEBUG:boto:Params: {} 2024-04-13T21:27:06.879 DEBUG:boto:Token: None 2024-04-13T21:27:06.879 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.879 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78= 2024-04-13T21:27:06.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78=', 'Content-Length': '0'} 2024-04-13T21:27:06.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d7620bb56edece-00661af8aa-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.881 DEBUG:boto:b'vwnfns-7&1000false&2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.882 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.882 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.882 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.882 DEBUG:boto:Path: /vwnfns-7/%26 2024-04-13T21:27:06.882 DEBUG:boto:Data: 2024-04-13T21:27:06.882 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.882 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.882 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.882 DEBUG:boto:Params: {} 2024-04-13T21:27:06.882 DEBUG:boto:Token: None 2024-04-13T21:27:06.883 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26 2024-04-13T21:27:06.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kax59vOU+sWjHuEUwI+tCYHnOUI= 2024-04-13T21:27:06.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kax59vOU+sWjHuEUwI+tCYHnOUI=', 'Content-Length': '0'} 2024-04-13T21:27:06.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006062892dc8af9ac2-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.885 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.885 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.885 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.885 DEBUG:boto:Path: /vwnfns-7/%26 2024-04-13T21:27:06.885 DEBUG:boto:Data: 2024-04-13T21:27:06.885 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.885 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.885 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.885 DEBUG:boto:Params: {} 2024-04-13T21:27:06.885 DEBUG:boto:Token: None 2024-04-13T21:27:06.885 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26 2024-04-13T21:27:06.886 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kax59vOU+sWjHuEUwI+tCYHnOUI= 2024-04-13T21:27:06.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kax59vOU+sWjHuEUwI+tCYHnOUI=', 'Content-Length': '0'} 2024-04-13T21:27:06.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e391377b9761f7c2-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-13T21:27:06.888 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.888 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.888 DEBUG:boto:Method: GET 2024-04-13T21:27:06.888 DEBUG:boto:Path: /vwnfns-7/%26 2024-04-13T21:27:06.888 DEBUG:boto:Data: 2024-04-13T21:27:06.888 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.889 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.889 DEBUG:boto:Params: {} 2024-04-13T21:27:06.889 DEBUG:boto:Token: None 2024-04-13T21:27:06.889 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26 2024-04-13T21:27:06.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cQoxeM1AdNds2S5jz8eY3Zv1tUg= 2024-04-13T21:27:06.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cQoxeM1AdNds2S5jz8eY3Zv1tUg=', 'Content-Length': '0'} 2024-04-13T21:27:06.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c9361f2adbc1564-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.891 DEBUG:boto:path=/vwnfns-7/%26 2024-04-13T21:27:06.891 DEBUG:boto:auth_path=/vwnfns-7/%26 2024-04-13T21:27:06.891 DEBUG:boto:Method: GET 2024-04-13T21:27:06.891 DEBUG:boto:Path: /vwnfns-7/%26 2024-04-13T21:27:06.891 DEBUG:boto:Data: 2024-04-13T21:27:06.891 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.892 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.892 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.892 DEBUG:boto:Params: {} 2024-04-13T21:27:06.892 DEBUG:boto:Token: None 2024-04-13T21:27:06.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%26 2024-04-13T21:27:06.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cQoxeM1AdNds2S5jz8eY3Zv1tUg= 2024-04-13T21:27:06.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cQoxeM1AdNds2S5jz8eY3Zv1tUg=', 'Content-Length': '0'} 2024-04-13T21:27:06.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c79807d47df6993-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.934 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.935 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.935 DEBUG:boto:path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.935 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.935 DEBUG:boto:Method: GET 2024-04-13T21:27:06.935 DEBUG:boto:Path: /vwnfns-7/?prefix=%26 2024-04-13T21:27:06.935 DEBUG:boto:Data: 2024-04-13T21:27:06.935 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.935 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.935 DEBUG:boto:Params: {} 2024-04-13T21:27:06.936 DEBUG:boto:Token: None 2024-04-13T21:27:06.936 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.936 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78= 2024-04-13T21:27:06.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78=', 'Content-Length': '0'} 2024-04-13T21:27:06.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f85d4c6289124cdd-00661af8aa-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.938 DEBUG:boto:b'vwnfns-7&1000false&2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.938 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.939 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.939 DEBUG:boto:path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.939 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%26 2024-04-13T21:27:06.939 DEBUG:boto:Method: GET 2024-04-13T21:27:06.939 DEBUG:boto:Path: /vwnfns-7/?prefix=%26 2024-04-13T21:27:06.939 DEBUG:boto:Data: 2024-04-13T21:27:06.939 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.939 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.939 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.939 DEBUG:boto:Params: {} 2024-04-13T21:27:06.940 DEBUG:boto:Token: None 2024-04-13T21:27:06.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78= 2024-04-13T21:27:06.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78=', 'Content-Length': '0'} 2024-04-13T21:27:06.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e688d9d8ab0c80af-00661af8aa-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.942 DEBUG:boto:b'vwnfns-7&1000false&2024-04-13T21:24:51.364Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-13T21:27:06.943 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.943 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.943 DEBUG:boto:path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.943 DEBUG:boto:auth_path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.943 DEBUG:boto:Method: GET 2024-04-13T21:27:06.943 DEBUG:boto:Path: /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.943 DEBUG:boto:Data: 2024-04-13T21:27:06.943 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.943 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.943 DEBUG:boto:Params: {} 2024-04-13T21:27:06.944 DEBUG:boto:Token: None 2024-04-13T21:27:06.944 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.944 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8= 2024-04-13T21:27:06.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8=', 'Content-Length': '0'} 2024-04-13T21:27:06.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d86357db62c56f3-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.946 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.946 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.946 DEBUG:boto:path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.946 DEBUG:boto:auth_path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.946 DEBUG:boto:Method: GET 2024-04-13T21:27:06.946 DEBUG:boto:Path: /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.946 DEBUG:boto:Data: 2024-04-13T21:27:06.946 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.946 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.947 DEBUG:boto:Params: {} 2024-04-13T21:27:06.947 DEBUG:boto:Token: None 2024-04-13T21:27:06.947 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8= 2024-04-13T21:27:06.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8=', 'Content-Length': '0'} 2024-04-13T21:27:06.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000623f61b0d5513b89-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.949 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.949 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.949 DEBUG:boto:path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.949 DEBUG:boto:auth_path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.949 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.949 DEBUG:boto:Path: /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.949 DEBUG:boto:Data: 2024-04-13T21:27:06.949 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.950 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.950 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.950 DEBUG:boto:Params: {} 2024-04-13T21:27:06.950 DEBUG:boto:Token: None 2024-04-13T21:27:06.950 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.950 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:moxh9R6gMfwnMCGYEq/kht1iHYk= 2024-04-13T21:27:06.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:moxh9R6gMfwnMCGYEq/kht1iHYk=', 'Content-Length': '0'} 2024-04-13T21:27:06.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc80999aebcf3b0-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.952 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.952 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.952 DEBUG:boto:path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.952 DEBUG:boto:auth_path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.952 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.952 DEBUG:boto:Path: /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.952 DEBUG:boto:Data: 2024-04-13T21:27:06.953 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.953 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.953 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.953 DEBUG:boto:Params: {} 2024-04-13T21:27:06.953 DEBUG:boto:Token: None 2024-04-13T21:27:06.953 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.953 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:moxh9R6gMfwnMCGYEq/kht1iHYk= 2024-04-13T21:27:06.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:moxh9R6gMfwnMCGYEq/kht1iHYk=', 'Content-Length': '0'} 2024-04-13T21:27:06.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eba94b9bc0a3ae0-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-13T21:27:06.955 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.955 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.955 DEBUG:boto:path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.955 DEBUG:boto:auth_path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.956 DEBUG:boto:Method: GET 2024-04-13T21:27:06.956 DEBUG:boto:Path: /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.956 DEBUG:boto:Data: 2024-04-13T21:27:06.956 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.956 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.956 DEBUG:boto:Params: {} 2024-04-13T21:27:06.956 DEBUG:boto:Token: None 2024-04-13T21:27:06.956 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8= 2024-04-13T21:27:06.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8=', 'Content-Length': '0'} 2024-04-13T21:27:06.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002709dd1fd501b4f1-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.958 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.959 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.959 DEBUG:boto:path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.959 DEBUG:boto:auth_path=/vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.959 DEBUG:boto:Method: GET 2024-04-13T21:27:06.959 DEBUG:boto:Path: /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.959 DEBUG:boto:Data: 2024-04-13T21:27:06.959 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.959 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.959 DEBUG:boto:Params: {} 2024-04-13T21:27:06.959 DEBUG:boto:Token: None 2024-04-13T21:27:06.960 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A?versionId=null 2024-04-13T21:27:06.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8= 2024-04-13T21:27:06.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WiJED3UVttgU0nkobO1/uLhjkZ8=', 'Content-Length': '0'} 2024-04-13T21:27:06.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee0310df5e890561-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.962 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.962 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.962 DEBUG:boto:path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:06.962 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:06.962 DEBUG:boto:Method: GET 2024-04-13T21:27:06.962 DEBUG:boto:Path: /vwnfns-7/?prefix=%3A 2024-04-13T21:27:06.962 DEBUG:boto:Data: 2024-04-13T21:27:06.962 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.962 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.963 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.963 DEBUG:boto:Params: {} 2024-04-13T21:27:06.963 DEBUG:boto:Token: None 2024-04-13T21:27:06.963 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.963 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78= 2024-04-13T21:27:06.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78=', 'Content-Length': '0'} 2024-04-13T21:27:06.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005270355b3fbe9a1b-00661af8aa-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.965 DEBUG:boto:b'vwnfns-7:1000false:2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.966 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:06.966 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:06.966 DEBUG:boto:path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:06.966 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:06.966 DEBUG:boto:Method: GET 2024-04-13T21:27:06.966 DEBUG:boto:Path: /vwnfns-7/?prefix=%3A 2024-04-13T21:27:06.966 DEBUG:boto:Data: 2024-04-13T21:27:06.966 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.966 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.966 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.967 DEBUG:boto:Params: {} 2024-04-13T21:27:06.967 DEBUG:boto:Token: None 2024-04-13T21:27:06.967 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/ 2024-04-13T21:27:06.967 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78= 2024-04-13T21:27:06.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fxSELqtUvA2SS241R8U4bXSXH78=', 'Content-Length': '0'} 2024-04-13T21:27:06.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d804ce4e66f93c42-00661af8aa-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.969 DEBUG:boto:b'vwnfns-7:1000false:2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:06.970 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.970 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.970 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.970 DEBUG:boto:Path: /vwnfns-7/%3A 2024-04-13T21:27:06.970 DEBUG:boto:Data: 2024-04-13T21:27:06.970 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.970 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.970 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.970 DEBUG:boto:Params: {} 2024-04-13T21:27:06.970 DEBUG:boto:Token: None 2024-04-13T21:27:06.971 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A 2024-04-13T21:27:06.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NLyueRw46LIA5geEheUSj/OmrDM= 2024-04-13T21:27:06.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NLyueRw46LIA5geEheUSj/OmrDM=', 'Content-Length': '0'} 2024-04-13T21:27:06.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000148421c30b78e20f-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.972 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.972 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.973 DEBUG:boto:Method: HEAD 2024-04-13T21:27:06.973 DEBUG:boto:Path: /vwnfns-7/%3A 2024-04-13T21:27:06.973 DEBUG:boto:Data: 2024-04-13T21:27:06.973 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.973 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.973 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.973 DEBUG:boto:Params: {} 2024-04-13T21:27:06.973 DEBUG:boto:Token: None 2024-04-13T21:27:06.973 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A 2024-04-13T21:27:06.973 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NLyueRw46LIA5geEheUSj/OmrDM= 2024-04-13T21:27:06.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NLyueRw46LIA5geEheUSj/OmrDM=', 'Content-Length': '0'} 2024-04-13T21:27:06.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1868f2f4c2dc794-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-13T21:27:06.975 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.975 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.975 DEBUG:boto:Method: GET 2024-04-13T21:27:06.975 DEBUG:boto:Path: /vwnfns-7/%3A 2024-04-13T21:27:06.976 DEBUG:boto:Data: 2024-04-13T21:27:06.976 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:06.976 DEBUG:boto:Port: 8001 2024-04-13T21:27:06.976 DEBUG:boto:Params: {} 2024-04-13T21:27:06.976 DEBUG:boto:Token: None 2024-04-13T21:27:06.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A 2024-04-13T21:27:06.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E5PBYtFHlkQ5xYh6GDVrC44wQ4E= 2024-04-13T21:27:06.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E5PBYtFHlkQ5xYh6GDVrC44wQ4E=', 'Content-Length': '0'} 2024-04-13T21:27:06.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2215e789100c2d4-00661af8aa-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:06.978 DEBUG:boto:path=/vwnfns-7/%3A 2024-04-13T21:27:06.978 DEBUG:boto:auth_path=/vwnfns-7/%3A 2024-04-13T21:27:06.978 DEBUG:boto:Method: GET 2024-04-13T21:27:06.978 DEBUG:boto:Path: /vwnfns-7/%3A 2024-04-13T21:27:06.979 DEBUG:boto:Data: 2024-04-13T21:27:06.979 DEBUG:boto:Headers: {} 2024-04-13T21:27:06.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:06.979 DEBUG:boto:Port: 8000 2024-04-13T21:27:06.979 DEBUG:boto:Params: {} 2024-04-13T21:27:06.979 DEBUG:boto:Token: None 2024-04-13T21:27:06.979 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:06 GMT /vwnfns-7/%3A 2024-04-13T21:27:06.979 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E5PBYtFHlkQ5xYh6GDVrC44wQ4E= 2024-04-13T21:27:06.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E5PBYtFHlkQ5xYh6GDVrC44wQ4E=', 'Content-Length': '0'} 2024-04-13T21:27:06.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000094db319731c1d10-00661af8aa-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.022 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.023 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.023 DEBUG:boto:path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:07.023 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:07.023 DEBUG:boto:Method: GET 2024-04-13T21:27:07.023 DEBUG:boto:Path: /vwnfns-7/?prefix=%3A 2024-04-13T21:27:07.023 DEBUG:boto:Data: 2024-04-13T21:27:07.023 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.023 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.024 DEBUG:boto:Params: {} 2024-04-13T21:27:07.024 DEBUG:boto:Token: None 2024-04-13T21:27:07.024 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae728ce88b3a74d5-00661af8ab-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.026 DEBUG:boto:b'vwnfns-7:1000false:2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.027 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.027 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.027 DEBUG:boto:path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:07.027 DEBUG:boto:auth_path=/vwnfns-7/?prefix=%3A 2024-04-13T21:27:07.027 DEBUG:boto:Method: GET 2024-04-13T21:27:07.027 DEBUG:boto:Path: /vwnfns-7/?prefix=%3A 2024-04-13T21:27:07.027 DEBUG:boto:Data: 2024-04-13T21:27:07.028 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.028 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.028 DEBUG:boto:Params: {} 2024-04-13T21:27:07.028 DEBUG:boto:Token: None 2024-04-13T21:27:07.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004015f6704375d4b4-00661af8ab-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.031 DEBUG:boto:b'vwnfns-7:1000false:2024-04-13T21:24:51.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-13T21:27:07.031 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.032 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.032 DEBUG:boto:path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.032 DEBUG:boto:auth_path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.032 DEBUG:boto:Method: GET 2024-04-13T21:27:07.032 DEBUG:boto:Path: /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.032 DEBUG:boto:Data: 2024-04-13T21:27:07.032 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.032 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.032 DEBUG:boto:Params: {} 2024-04-13T21:27:07.033 DEBUG:boto:Token: None 2024-04-13T21:27:07.033 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.033 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74= 2024-04-13T21:27:07.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74=', 'Content-Length': '0'} 2024-04-13T21:27:07.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f9408c0c14767f5-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.078 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.078 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.079 DEBUG:boto:path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.079 DEBUG:boto:auth_path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.079 DEBUG:boto:Method: GET 2024-04-13T21:27:07.079 DEBUG:boto:Path: /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.079 DEBUG:boto:Data: 2024-04-13T21:27:07.079 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.079 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.079 DEBUG:boto:Params: {} 2024-04-13T21:27:07.079 DEBUG:boto:Token: None 2024-04-13T21:27:07.080 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74= 2024-04-13T21:27:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74=', 'Content-Length': '0'} 2024-04-13T21:27:07.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059a3ae0dab62dc97-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.082 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.082 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.082 DEBUG:boto:path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.082 DEBUG:boto:auth_path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.082 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.082 DEBUG:boto:Path: /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.082 DEBUG:boto:Data: 2024-04-13T21:27:07.082 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.083 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.083 DEBUG:boto:Params: {} 2024-04-13T21:27:07.083 DEBUG:boto:Token: None 2024-04-13T21:27:07.083 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I4JnxoDx91Uc/BncqJIIQBBjTAo= 2024-04-13T21:27:07.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I4JnxoDx91Uc/BncqJIIQBBjTAo=', 'Content-Length': '0'} 2024-04-13T21:27:07.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aa9d1ee3b852e72-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.085 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.085 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.085 DEBUG:boto:path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.085 DEBUG:boto:auth_path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.085 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.085 DEBUG:boto:Path: /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.085 DEBUG:boto:Data: 2024-04-13T21:27:07.085 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.085 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.086 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.086 DEBUG:boto:Params: {} 2024-04-13T21:27:07.086 DEBUG:boto:Token: None 2024-04-13T21:27:07.086 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.086 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I4JnxoDx91Uc/BncqJIIQBBjTAo= 2024-04-13T21:27:07.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I4JnxoDx91Uc/BncqJIIQBBjTAo=', 'Content-Length': '0'} 2024-04-13T21:27:07.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c3a281e2b58fbfc-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-13T21:27:07.088 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.088 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.088 DEBUG:boto:path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.088 DEBUG:boto:auth_path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.088 DEBUG:boto:Method: GET 2024-04-13T21:27:07.088 DEBUG:boto:Path: /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.088 DEBUG:boto:Data: 2024-04-13T21:27:07.088 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.089 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.089 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.089 DEBUG:boto:Params: {} 2024-04-13T21:27:07.089 DEBUG:boto:Token: None 2024-04-13T21:27:07.089 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74= 2024-04-13T21:27:07.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74=', 'Content-Length': '0'} 2024-04-13T21:27:07.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009485883bc352793b-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.091 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.091 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.091 DEBUG:boto:path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.091 DEBUG:boto:auth_path=/vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.091 DEBUG:boto:Method: GET 2024-04-13T21:27:07.091 DEBUG:boto:Path: /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.091 DEBUG:boto:Data: 2024-04-13T21:27:07.092 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.092 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.092 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.092 DEBUG:boto:Params: {} 2024-04-13T21:27:07.092 DEBUG:boto:Token: None 2024-04-13T21:27:07.092 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj?versionId=null 2024-04-13T21:27:07.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74= 2024-04-13T21:27:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5KG7V7FKTh3lnAF6l3/GjDXmB74=', 'Content-Length': '0'} 2024-04-13T21:27:07.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b17b3221584dd53-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.094 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.094 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.094 DEBUG:boto:path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.095 DEBUG:boto:auth_path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.095 DEBUG:boto:Method: GET 2024-04-13T21:27:07.095 DEBUG:boto:Path: /vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.095 DEBUG:boto:Data: 2024-04-13T21:27:07.095 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.095 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.095 DEBUG:boto:Params: {} 2024-04-13T21:27:07.095 DEBUG:boto:Token: None 2024-04-13T21:27:07.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c5746861a68ca4-00661af8ab-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.098 DEBUG:boto:b'vwnfns-7_myobj1000false_myobj2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.098 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.098 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.098 DEBUG:boto:path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.098 DEBUG:boto:auth_path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.098 DEBUG:boto:Method: GET 2024-04-13T21:27:07.098 DEBUG:boto:Path: /vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.099 DEBUG:boto:Data: 2024-04-13T21:27:07.099 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.099 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.099 DEBUG:boto:Params: {} 2024-04-13T21:27:07.099 DEBUG:boto:Token: None 2024-04-13T21:27:07.099 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a478c093558ba72-00661af8ab-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.102 DEBUG:boto:b'vwnfns-7_myobj1000false_myobj2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.102 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.102 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.102 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.102 DEBUG:boto:Path: /vwnfns-7/_myobj 2024-04-13T21:27:07.102 DEBUG:boto:Data: 2024-04-13T21:27:07.102 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.103 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.103 DEBUG:boto:Params: {} 2024-04-13T21:27:07.103 DEBUG:boto:Token: None 2024-04-13T21:27:07.103 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj 2024-04-13T21:27:07.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ov38lYX02jn1S4Gu4CbrSMMKcvo= 2024-04-13T21:27:07.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ov38lYX02jn1S4Gu4CbrSMMKcvo=', 'Content-Length': '0'} 2024-04-13T21:27:07.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087011f091936e99f-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.105 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.105 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.105 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.105 DEBUG:boto:Path: /vwnfns-7/_myobj 2024-04-13T21:27:07.105 DEBUG:boto:Data: 2024-04-13T21:27:07.105 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.105 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.105 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.105 DEBUG:boto:Params: {} 2024-04-13T21:27:07.105 DEBUG:boto:Token: None 2024-04-13T21:27:07.106 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj 2024-04-13T21:27:07.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ov38lYX02jn1S4Gu4CbrSMMKcvo= 2024-04-13T21:27:07.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ov38lYX02jn1S4Gu4CbrSMMKcvo=', 'Content-Length': '0'} 2024-04-13T21:27:07.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fce83de1f2dbe90-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-13T21:27:07.108 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.108 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.108 DEBUG:boto:Method: GET 2024-04-13T21:27:07.108 DEBUG:boto:Path: /vwnfns-7/_myobj 2024-04-13T21:27:07.108 DEBUG:boto:Data: 2024-04-13T21:27:07.108 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.108 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.108 DEBUG:boto:Params: {} 2024-04-13T21:27:07.109 DEBUG:boto:Token: None 2024-04-13T21:27:07.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj 2024-04-13T21:27:07.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NKtBREYkKg979yI5tEwaLaBuSrg= 2024-04-13T21:27:07.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NKtBREYkKg979yI5tEwaLaBuSrg=', 'Content-Length': '0'} 2024-04-13T21:27:07.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5cbabb045bec0c7-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.111 DEBUG:boto:path=/vwnfns-7/_myobj 2024-04-13T21:27:07.111 DEBUG:boto:auth_path=/vwnfns-7/_myobj 2024-04-13T21:27:07.111 DEBUG:boto:Method: GET 2024-04-13T21:27:07.111 DEBUG:boto:Path: /vwnfns-7/_myobj 2024-04-13T21:27:07.111 DEBUG:boto:Data: 2024-04-13T21:27:07.111 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.111 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.111 DEBUG:boto:Params: {} 2024-04-13T21:27:07.112 DEBUG:boto:Token: None 2024-04-13T21:27:07.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/_myobj 2024-04-13T21:27:07.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NKtBREYkKg979yI5tEwaLaBuSrg= 2024-04-13T21:27:07.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NKtBREYkKg979yI5tEwaLaBuSrg=', 'Content-Length': '0'} 2024-04-13T21:27:07.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000690422e87910c4ab-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.113 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.114 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.114 DEBUG:boto:path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.114 DEBUG:boto:auth_path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.114 DEBUG:boto:Method: GET 2024-04-13T21:27:07.114 DEBUG:boto:Path: /vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.114 DEBUG:boto:Data: 2024-04-13T21:27:07.114 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.114 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.114 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.114 DEBUG:boto:Params: {} 2024-04-13T21:27:07.115 DEBUG:boto:Token: None 2024-04-13T21:27:07.115 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.115 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4610070a7476e69-00661af8ab-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.162 DEBUG:boto:b'vwnfns-7_myobj1000false_myobj2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.163 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.163 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.163 DEBUG:boto:path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.163 DEBUG:boto:auth_path=/vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.163 DEBUG:boto:Method: GET 2024-04-13T21:27:07.163 DEBUG:boto:Path: /vwnfns-7/?prefix=_myobj 2024-04-13T21:27:07.163 DEBUG:boto:Data: 2024-04-13T21:27:07.163 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.163 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.164 DEBUG:boto:Params: {} 2024-04-13T21:27:07.164 DEBUG:boto:Token: None 2024-04-13T21:27:07.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c28cee5fdabb1e26-00661af8ab-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.166 DEBUG:boto:b'vwnfns-7_myobj1000false_myobj2024-04-13T21:24:51.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:27:07.167 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.167 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.167 DEBUG:boto:path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.167 DEBUG:boto:auth_path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.167 DEBUG:boto:Method: GET 2024-04-13T21:27:07.167 DEBUG:boto:Path: /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.167 DEBUG:boto:Data: 2024-04-13T21:27:07.167 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.167 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.168 DEBUG:boto:Params: {} 2024-04-13T21:27:07.168 DEBUG:boto:Token: None 2024-04-13T21:27:07.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM= 2024-04-13T21:27:07.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM=', 'Content-Length': '0'} 2024-04-13T21:27:07.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000381c7bf8cbacfdd4-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.170 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.170 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.170 DEBUG:boto:path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.170 DEBUG:boto:auth_path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.170 DEBUG:boto:Method: GET 2024-04-13T21:27:07.171 DEBUG:boto:Path: /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.171 DEBUG:boto:Data: 2024-04-13T21:27:07.171 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.171 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.171 DEBUG:boto:Params: {} 2024-04-13T21:27:07.171 DEBUG:boto:Token: None 2024-04-13T21:27:07.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM= 2024-04-13T21:27:07.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM=', 'Content-Length': '0'} 2024-04-13T21:27:07.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7d0ad054ade4bc4-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.174 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.174 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.174 DEBUG:boto:path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.174 DEBUG:boto:auth_path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.174 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.174 DEBUG:boto:Path: /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.174 DEBUG:boto:Data: 2024-04-13T21:27:07.174 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.175 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.175 DEBUG:boto:Params: {} 2024-04-13T21:27:07.175 DEBUG:boto:Token: None 2024-04-13T21:27:07.175 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6fA5QaU2hF+FyEKPvwHqv7ATtKc= 2024-04-13T21:27:07.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6fA5QaU2hF+FyEKPvwHqv7ATtKc=', 'Content-Length': '0'} 2024-04-13T21:27:07.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016d45e144588087e-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.177 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.177 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.177 DEBUG:boto:path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.177 DEBUG:boto:auth_path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.177 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.177 DEBUG:boto:Path: /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.178 DEBUG:boto:Data: 2024-04-13T21:27:07.178 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.178 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.178 DEBUG:boto:Params: {} 2024-04-13T21:27:07.178 DEBUG:boto:Token: None 2024-04-13T21:27:07.178 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6fA5QaU2hF+FyEKPvwHqv7ATtKc= 2024-04-13T21:27:07.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6fA5QaU2hF+FyEKPvwHqv7ATtKc=', 'Content-Length': '0'} 2024-04-13T21:27:07.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026675ef3ff78f938-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:27:07.180 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.180 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.180 DEBUG:boto:path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.180 DEBUG:boto:auth_path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.181 DEBUG:boto:Method: GET 2024-04-13T21:27:07.181 DEBUG:boto:Path: /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.181 DEBUG:boto:Data: 2024-04-13T21:27:07.181 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.181 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.181 DEBUG:boto:Params: {} 2024-04-13T21:27:07.181 DEBUG:boto:Token: None 2024-04-13T21:27:07.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM= 2024-04-13T21:27:07.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM=', 'Content-Length': '0'} 2024-04-13T21:27:07.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f8c56c36ef78105-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.184 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.184 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.184 DEBUG:boto:path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.184 DEBUG:boto:auth_path=/vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.184 DEBUG:boto:Method: GET 2024-04-13T21:27:07.184 DEBUG:boto:Path: /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.184 DEBUG:boto:Data: 2024-04-13T21:27:07.184 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.184 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.185 DEBUG:boto:Params: {} 2024-04-13T21:27:07.185 DEBUG:boto:Token: None 2024-04-13T21:27:07.185 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj?versionId=null 2024-04-13T21:27:07.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM= 2024-04-13T21:27:07.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hd0EcnkvCGGMEtpGhDw+nTgszfM=', 'Content-Length': '0'} 2024-04-13T21:27:07.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000034eea055690797c-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.187 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.187 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.187 DEBUG:boto:path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.187 DEBUG:boto:auth_path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.187 DEBUG:boto:Method: GET 2024-04-13T21:27:07.187 DEBUG:boto:Path: /vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.187 DEBUG:boto:Data: 2024-04-13T21:27:07.188 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.188 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.188 DEBUG:boto:Params: {} 2024-04-13T21:27:07.188 DEBUG:boto:Token: None 2024-04-13T21:27:07.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007616aec7be03715-00661af8ab-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.190 DEBUG:boto:b'vwnfns-7myobj1000falsemyobj2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.191 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.191 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.191 DEBUG:boto:path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.191 DEBUG:boto:auth_path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.191 DEBUG:boto:Method: GET 2024-04-13T21:27:07.191 DEBUG:boto:Path: /vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.191 DEBUG:boto:Data: 2024-04-13T21:27:07.191 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.192 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.192 DEBUG:boto:Params: {} 2024-04-13T21:27:07.192 DEBUG:boto:Token: None 2024-04-13T21:27:07.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039813a3e40298fd9-00661af8ab-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.194 DEBUG:boto:b'vwnfns-7myobj1000falsemyobj2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.194 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.195 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.195 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.195 DEBUG:boto:Path: /vwnfns-7/myobj 2024-04-13T21:27:07.195 DEBUG:boto:Data: 2024-04-13T21:27:07.195 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.195 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.195 DEBUG:boto:Params: {} 2024-04-13T21:27:07.195 DEBUG:boto:Token: None 2024-04-13T21:27:07.196 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj 2024-04-13T21:27:07.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mRaWwLlUHoPtvHTRkvDuT/+vabM= 2024-04-13T21:27:07.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mRaWwLlUHoPtvHTRkvDuT/+vabM=', 'Content-Length': '0'} 2024-04-13T21:27:07.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6fe4e8d47b88a6b-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.198 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.198 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.198 DEBUG:boto:Method: HEAD 2024-04-13T21:27:07.198 DEBUG:boto:Path: /vwnfns-7/myobj 2024-04-13T21:27:07.198 DEBUG:boto:Data: 2024-04-13T21:27:07.198 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.198 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.198 DEBUG:boto:Params: {} 2024-04-13T21:27:07.199 DEBUG:boto:Token: None 2024-04-13T21:27:07.199 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj 2024-04-13T21:27:07.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mRaWwLlUHoPtvHTRkvDuT/+vabM= 2024-04-13T21:27:07.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mRaWwLlUHoPtvHTRkvDuT/+vabM=', 'Content-Length': '0'} 2024-04-13T21:27:07.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b5397394c8c9136-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:27:07.200 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.201 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.201 DEBUG:boto:Method: GET 2024-04-13T21:27:07.201 DEBUG:boto:Path: /vwnfns-7/myobj 2024-04-13T21:27:07.201 DEBUG:boto:Data: 2024-04-13T21:27:07.201 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.201 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.201 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.201 DEBUG:boto:Params: {} 2024-04-13T21:27:07.201 DEBUG:boto:Token: None 2024-04-13T21:27:07.201 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj 2024-04-13T21:27:07.202 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bt0ViWlVU8LVBLIhxaIJUxxbD3I= 2024-04-13T21:27:07.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bt0ViWlVU8LVBLIhxaIJUxxbD3I=', 'Content-Length': '0'} 2024-04-13T21:27:07.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6c53dfda3bf5c07-00661af8ab-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.250 DEBUG:boto:path=/vwnfns-7/myobj 2024-04-13T21:27:07.251 DEBUG:boto:auth_path=/vwnfns-7/myobj 2024-04-13T21:27:07.251 DEBUG:boto:Method: GET 2024-04-13T21:27:07.251 DEBUG:boto:Path: /vwnfns-7/myobj 2024-04-13T21:27:07.251 DEBUG:boto:Data: 2024-04-13T21:27:07.251 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.251 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.251 DEBUG:boto:Params: {} 2024-04-13T21:27:07.251 DEBUG:boto:Token: None 2024-04-13T21:27:07.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/myobj 2024-04-13T21:27:07.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bt0ViWlVU8LVBLIhxaIJUxxbD3I= 2024-04-13T21:27:07.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bt0ViWlVU8LVBLIhxaIJUxxbD3I=', 'Content-Length': '0'} 2024-04-13T21:27:07.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c7236da8b16ffab-00661af8ab-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.254 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.254 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.254 DEBUG:boto:path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.254 DEBUG:boto:auth_path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.254 DEBUG:boto:Method: GET 2024-04-13T21:27:07.254 DEBUG:boto:Path: /vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.254 DEBUG:boto:Data: 2024-04-13T21:27:07.254 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:27:07.255 DEBUG:boto:Port: 8001 2024-04-13T21:27:07.255 DEBUG:boto:Params: {} 2024-04-13T21:27:07.255 DEBUG:boto:Token: None 2024-04-13T21:27:07.255 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003990e3d3c87815a0-00661af8ab-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.258 DEBUG:boto:b'vwnfns-7myobj1000falsemyobj2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.258 DEBUG:boto:path=/vwnfns-7/ 2024-04-13T21:27:07.258 DEBUG:boto:auth_path=/vwnfns-7/ 2024-04-13T21:27:07.258 DEBUG:boto:path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.258 DEBUG:boto:auth_path=/vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.259 DEBUG:boto:Method: GET 2024-04-13T21:27:07.259 DEBUG:boto:Path: /vwnfns-7/?prefix=myobj 2024-04-13T21:27:07.259 DEBUG:boto:Data: 2024-04-13T21:27:07.259 DEBUG:boto:Headers: {} 2024-04-13T21:27:07.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:27:07.259 DEBUG:boto:Port: 8000 2024-04-13T21:27:07.259 DEBUG:boto:Params: {} 2024-04-13T21:27:07.259 DEBUG:boto:Token: None 2024-04-13T21:27:07.259 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:27:07 GMT /vwnfns-7/ 2024-04-13T21:27:07.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ= 2024-04-13T21:27:07.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:27:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruamWJ8W8n+Y64uSBdFnh4wYaVQ=', 'Content-Length': '0'} 2024-04-13T21:27:07.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4f9e99632c9566c-00661af8ab-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:27:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:27:07.263 DEBUG:boto:b'vwnfns-7myobj1000falsemyobj2024-04-13T21:24:51.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:27:07.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-7 zones={a2, a1} 2024-04-13T21:27:07.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:27:07.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:27:07.264 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:27:07.321 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:27:07.321 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:27:07.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:07.361+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:12.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:12.362+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:17.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:17.362+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:22.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:22.363+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:27.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:27.363+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:32.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:32.363+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:37.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:37.365+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:42.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:42.366+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:47.366+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:52.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:52.367+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:27:57.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:27:57.366+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:28:02.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:02.367+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:28:07.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:07.367+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:28:12.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:12.368+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:28:17.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:17.370+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:28:22.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:22.371+0000 7feb529e2780 1 waiting to reach incremental sync.. 2024-04-13T21:28:27.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:27.372+0000 7feb529e2780 1 bucket sync caught up with source: 2024-04-13T21:28:27.365 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.8.6, , 00000000001.10.6, , , , 00000000001.6.6, 00000000001.17.6, , ] 2024-04-13T21:28:27.365 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.8.6, , 00000000001.10.6, , , , 00000000001.6.6, 00000000001.17.6, , ] 2024-04-13T21:28:27.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:27.372+0000 7feb529e2780 0 bucket checkpoint complete 2024-04-13T21:28:27.380 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-8 zones={a1, a2} 2024-04-13T21:28:27.381 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.381 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.381 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.381 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:28:27.381 DEBUG:boto:Data: 2024-04-13T21:28:27.381 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.381 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.382 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.382 DEBUG:boto:Params: {} 2024-04-13T21:28:27.382 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:28:27.382 DEBUG:boto:Token: None 2024-04-13T21:28:27.383 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6eHH+61k41FpKhAoXMlPMokoBjs= 2024-04-13T21:28:27.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6eHH+61k41FpKhAoXMlPMokoBjs=', 'Content-Length': '0'} 2024-04-13T21:28:27.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015a88d0f02e7cb33-00661af8fb-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.389 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.389 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.389 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.389 DEBUG:boto:Path: /vwnfns-8/ 2024-04-13T21:28:27.389 DEBUG:boto:Data: 2024-04-13T21:28:27.389 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.389 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.389 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.389 DEBUG:boto:Params: {} 2024-04-13T21:28:27.390 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:28:27.390 DEBUG:boto:Token: None 2024-04-13T21:28:27.390 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.390 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6eHH+61k41FpKhAoXMlPMokoBjs= 2024-04-13T21:28:27.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6eHH+61k41FpKhAoXMlPMokoBjs=', 'Content-Length': '0'} 2024-04-13T21:28:27.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000450a79f851758ae-00661af8fb-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.394 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:28:27.394 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.394 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.394 DEBUG:boto:path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.394 DEBUG:boto:auth_path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.395 DEBUG:boto:Method: GET 2024-04-13T21:28:27.395 DEBUG:boto:Path: /vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.395 DEBUG:boto:Data: 2024-04-13T21:28:27.395 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.395 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.395 DEBUG:boto:Params: {} 2024-04-13T21:28:27.395 DEBUG:boto:Token: None 2024-04-13T21:28:27.395 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/?versions 2024-04-13T21:28:27.395 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU= 2024-04-13T21:28:27.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU=', 'Content-Length': '0'} 2024-04-13T21:28:27.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000db710f8249a1f67-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.398 DEBUG:boto:b'vwnfns-8999false&nulltrue2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.399 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-13T21:28:27.399 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-13T21:28:27.399 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-13T21:28:27.399 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:28:27.400 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:28:27.400 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.400 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.400 DEBUG:boto:path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.400 DEBUG:boto:auth_path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.400 DEBUG:boto:Method: GET 2024-04-13T21:28:27.400 DEBUG:boto:Path: /vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.400 DEBUG:boto:Data: 2024-04-13T21:28:27.400 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.401 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.401 DEBUG:boto:Params: {} 2024-04-13T21:28:27.401 DEBUG:boto:Token: None 2024-04-13T21:28:27.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/?versions 2024-04-13T21:28:27.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU= 2024-04-13T21:28:27.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU=', 'Content-Length': '0'} 2024-04-13T21:28:27.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb4adbe97267879-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.404 DEBUG:boto:b'vwnfns-8999false&nulltrue2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.404 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-13T21:28:27.404 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-13T21:28:27.404 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-13T21:28:27.404 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:28:27.404 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.404 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.405 DEBUG:boto:path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.405 DEBUG:boto:auth_path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.405 DEBUG:boto:Method: GET 2024-04-13T21:28:27.405 DEBUG:boto:Path: /vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.405 DEBUG:boto:Data: 2024-04-13T21:28:27.405 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.405 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.405 DEBUG:boto:Params: {} 2024-04-13T21:28:27.405 DEBUG:boto:Token: None 2024-04-13T21:28:27.405 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/?versions 2024-04-13T21:28:27.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU= 2024-04-13T21:28:27.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU=', 'Content-Length': '0'} 2024-04-13T21:28:27.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4b5991b09da8de-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.408 DEBUG:boto:b'vwnfns-8999false&nulltrue2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.408 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.408 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.409 DEBUG:boto:path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.409 DEBUG:boto:auth_path=/vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.409 DEBUG:boto:Method: GET 2024-04-13T21:28:27.409 DEBUG:boto:Path: /vwnfns-8/?versions&max-keys=999 2024-04-13T21:28:27.409 DEBUG:boto:Data: 2024-04-13T21:28:27.409 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.409 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.409 DEBUG:boto:Params: {} 2024-04-13T21:28:27.409 DEBUG:boto:Token: None 2024-04-13T21:28:27.410 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/?versions 2024-04-13T21:28:27.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU= 2024-04-13T21:28:27.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WToCCKSJx84lqDOnaUQBTJF8qtU=', 'Content-Length': '0'} 2024-04-13T21:28:27.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006edf5433d4a6ad00-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.413 DEBUG:boto:b'vwnfns-8999false&nulltrue2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-13T21:28:27.413 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.413 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.414 DEBUG:boto:path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.414 DEBUG:boto:auth_path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.414 DEBUG:boto:Method: GET 2024-04-13T21:28:27.414 DEBUG:boto:Path: /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.414 DEBUG:boto:Data: 2024-04-13T21:28:27.414 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.414 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.414 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.414 DEBUG:boto:Params: {} 2024-04-13T21:28:27.414 DEBUG:boto:Token: None 2024-04-13T21:28:27.415 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.415 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw= 2024-04-13T21:28:27.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw=', 'Content-Length': '0'} 2024-04-13T21:28:27.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068f8674074a58e0f-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.417 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.417 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.417 DEBUG:boto:path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.417 DEBUG:boto:auth_path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.417 DEBUG:boto:Method: GET 2024-04-13T21:28:27.417 DEBUG:boto:Path: /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.417 DEBUG:boto:Data: 2024-04-13T21:28:27.417 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.418 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.418 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.418 DEBUG:boto:Params: {} 2024-04-13T21:28:27.418 DEBUG:boto:Token: None 2024-04-13T21:28:27.418 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.418 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw= 2024-04-13T21:28:27.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw=', 'Content-Length': '0'} 2024-04-13T21:28:27.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000763de755db47a263-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.420 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.420 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.420 DEBUG:boto:path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.420 DEBUG:boto:auth_path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.421 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.421 DEBUG:boto:Path: /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.421 DEBUG:boto:Data: 2024-04-13T21:28:27.421 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.421 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.421 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.421 DEBUG:boto:Params: {} 2024-04-13T21:28:27.421 DEBUG:boto:Token: None 2024-04-13T21:28:27.421 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.421 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u7FQ3h18xLw9nztxpca9tvReVpM= 2024-04-13T21:28:27.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u7FQ3h18xLw9nztxpca9tvReVpM=', 'Content-Length': '0'} 2024-04-13T21:28:27.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001330b0477647e080-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.423 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.423 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.423 DEBUG:boto:path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.424 DEBUG:boto:auth_path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.424 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.424 DEBUG:boto:Path: /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.424 DEBUG:boto:Data: 2024-04-13T21:28:27.424 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.424 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.424 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.424 DEBUG:boto:Params: {} 2024-04-13T21:28:27.424 DEBUG:boto:Token: None 2024-04-13T21:28:27.424 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.425 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u7FQ3h18xLw9nztxpca9tvReVpM= 2024-04-13T21:28:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u7FQ3h18xLw9nztxpca9tvReVpM=', 'Content-Length': '0'} 2024-04-13T21:28:27.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a973a7b32a58e21-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-13T21:28:27.427 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.427 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.427 DEBUG:boto:path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.427 DEBUG:boto:auth_path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.427 DEBUG:boto:Method: GET 2024-04-13T21:28:27.428 DEBUG:boto:Path: /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.428 DEBUG:boto:Data: 2024-04-13T21:28:27.428 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.428 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.428 DEBUG:boto:Params: {} 2024-04-13T21:28:27.428 DEBUG:boto:Token: None 2024-04-13T21:28:27.428 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.429 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw= 2024-04-13T21:28:27.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw=', 'Content-Length': '0'} 2024-04-13T21:28:27.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029009b37a99af1e5-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.431 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.431 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.431 DEBUG:boto:path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.431 DEBUG:boto:auth_path=/vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.431 DEBUG:boto:Method: GET 2024-04-13T21:28:27.431 DEBUG:boto:Path: /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.431 DEBUG:boto:Data: 2024-04-13T21:28:27.431 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.431 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.432 DEBUG:boto:Params: {} 2024-04-13T21:28:27.432 DEBUG:boto:Token: None 2024-04-13T21:28:27.432 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26?versionId=null 2024-04-13T21:28:27.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw= 2024-04-13T21:28:27.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vInvY3bGUzAomTHPWH+eve8HNjw=', 'Content-Length': '0'} 2024-04-13T21:28:27.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047a5eccf9e25f291-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.435 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.435 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.435 DEBUG:boto:path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.435 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.435 DEBUG:boto:Method: GET 2024-04-13T21:28:27.435 DEBUG:boto:Path: /vwnfns-8/?prefix=%26 2024-04-13T21:28:27.436 DEBUG:boto:Data: 2024-04-13T21:28:27.436 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.436 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.436 DEBUG:boto:Params: {} 2024-04-13T21:28:27.437 DEBUG:boto:Token: None 2024-04-13T21:28:27.437 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a08858d7ace2a6a5-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.482 DEBUG:boto:b'vwnfns-8&1000false&2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.483 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.483 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.483 DEBUG:boto:path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.484 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.484 DEBUG:boto:Method: GET 2024-04-13T21:28:27.484 DEBUG:boto:Path: /vwnfns-8/?prefix=%26 2024-04-13T21:28:27.484 DEBUG:boto:Data: 2024-04-13T21:28:27.484 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.484 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.484 DEBUG:boto:Params: {} 2024-04-13T21:28:27.484 DEBUG:boto:Token: None 2024-04-13T21:28:27.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005939370491b296ea-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.487 DEBUG:boto:b'vwnfns-8&1000false&2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.488 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.488 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.488 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.488 DEBUG:boto:Path: /vwnfns-8/%26 2024-04-13T21:28:27.488 DEBUG:boto:Data: 2024-04-13T21:28:27.488 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.489 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.489 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.489 DEBUG:boto:Params: {} 2024-04-13T21:28:27.489 DEBUG:boto:Token: None 2024-04-13T21:28:27.489 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26 2024-04-13T21:28:27.489 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t2MBDmJWEiFXVEaK2R21B2ESVg4= 2024-04-13T21:28:27.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t2MBDmJWEiFXVEaK2R21B2ESVg4=', 'Content-Length': '0'} 2024-04-13T21:28:27.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076a1060430a1ccba-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.492 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.492 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.492 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.492 DEBUG:boto:Path: /vwnfns-8/%26 2024-04-13T21:28:27.492 DEBUG:boto:Data: 2024-04-13T21:28:27.492 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.493 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.493 DEBUG:boto:Params: {} 2024-04-13T21:28:27.493 DEBUG:boto:Token: None 2024-04-13T21:28:27.493 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26 2024-04-13T21:28:27.493 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t2MBDmJWEiFXVEaK2R21B2ESVg4= 2024-04-13T21:28:27.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t2MBDmJWEiFXVEaK2R21B2ESVg4=', 'Content-Length': '0'} 2024-04-13T21:28:27.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000444db3678148748-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-13T21:28:27.498 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.498 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.499 DEBUG:boto:Method: GET 2024-04-13T21:28:27.499 DEBUG:boto:Path: /vwnfns-8/%26 2024-04-13T21:28:27.499 DEBUG:boto:Data: 2024-04-13T21:28:27.499 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.499 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.499 DEBUG:boto:Params: {} 2024-04-13T21:28:27.499 DEBUG:boto:Token: None 2024-04-13T21:28:27.499 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26 2024-04-13T21:28:27.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PO19lSl+OlBqS3LR0eWK1PtmtxI= 2024-04-13T21:28:27.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PO19lSl+OlBqS3LR0eWK1PtmtxI=', 'Content-Length': '0'} 2024-04-13T21:28:27.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf8f7d0e4c7b635f-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.502 DEBUG:boto:path=/vwnfns-8/%26 2024-04-13T21:28:27.502 DEBUG:boto:auth_path=/vwnfns-8/%26 2024-04-13T21:28:27.502 DEBUG:boto:Method: GET 2024-04-13T21:28:27.502 DEBUG:boto:Path: /vwnfns-8/%26 2024-04-13T21:28:27.502 DEBUG:boto:Data: 2024-04-13T21:28:27.502 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.502 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.503 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.503 DEBUG:boto:Params: {} 2024-04-13T21:28:27.503 DEBUG:boto:Token: None 2024-04-13T21:28:27.503 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%26 2024-04-13T21:28:27.503 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PO19lSl+OlBqS3LR0eWK1PtmtxI= 2024-04-13T21:28:27.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PO19lSl+OlBqS3LR0eWK1PtmtxI=', 'Content-Length': '0'} 2024-04-13T21:28:27.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bcf25b12a2c2efd-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.546 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.547 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.547 DEBUG:boto:path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.547 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.547 DEBUG:boto:Method: GET 2024-04-13T21:28:27.547 DEBUG:boto:Path: /vwnfns-8/?prefix=%26 2024-04-13T21:28:27.547 DEBUG:boto:Data: 2024-04-13T21:28:27.547 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.547 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.548 DEBUG:boto:Params: {} 2024-04-13T21:28:27.548 DEBUG:boto:Token: None 2024-04-13T21:28:27.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a7d95df2c241c2-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.551 DEBUG:boto:b'vwnfns-8&1000false&2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.551 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.551 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.551 DEBUG:boto:path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.551 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%26 2024-04-13T21:28:27.552 DEBUG:boto:Method: GET 2024-04-13T21:28:27.552 DEBUG:boto:Path: /vwnfns-8/?prefix=%26 2024-04-13T21:28:27.552 DEBUG:boto:Data: 2024-04-13T21:28:27.552 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.552 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.552 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.552 DEBUG:boto:Params: {} 2024-04-13T21:28:27.552 DEBUG:boto:Token: None 2024-04-13T21:28:27.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.553 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52da96dbce90ad4-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.555 DEBUG:boto:b'vwnfns-8&1000false&2024-04-13T21:24:51.459Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-13T21:28:27.555 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.556 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.556 DEBUG:boto:path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.556 DEBUG:boto:auth_path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.556 DEBUG:boto:Method: GET 2024-04-13T21:28:27.556 DEBUG:boto:Path: /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.556 DEBUG:boto:Data: 2024-04-13T21:28:27.556 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.556 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.556 DEBUG:boto:Params: {} 2024-04-13T21:28:27.557 DEBUG:boto:Token: None 2024-04-13T21:28:27.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw= 2024-04-13T21:28:27.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw=', 'Content-Length': '0'} 2024-04-13T21:28:27.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006097e0d223f41e06-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.559 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.559 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.559 DEBUG:boto:path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.559 DEBUG:boto:auth_path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.559 DEBUG:boto:Method: GET 2024-04-13T21:28:27.560 DEBUG:boto:Path: /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.560 DEBUG:boto:Data: 2024-04-13T21:28:27.560 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.560 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.560 DEBUG:boto:Params: {} 2024-04-13T21:28:27.560 DEBUG:boto:Token: None 2024-04-13T21:28:27.560 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw= 2024-04-13T21:28:27.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw=', 'Content-Length': '0'} 2024-04-13T21:28:27.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ba3c529eb3d32cb-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.562 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.563 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.563 DEBUG:boto:path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.563 DEBUG:boto:auth_path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.563 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.563 DEBUG:boto:Path: /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.563 DEBUG:boto:Data: 2024-04-13T21:28:27.563 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.563 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.563 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.563 DEBUG:boto:Params: {} 2024-04-13T21:28:27.563 DEBUG:boto:Token: None 2024-04-13T21:28:27.564 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VDMSwPeR8TGc+mtjP29dIHmguOc= 2024-04-13T21:28:27.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VDMSwPeR8TGc+mtjP29dIHmguOc=', 'Content-Length': '0'} 2024-04-13T21:28:27.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004013e26d329e82d4-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.566 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.566 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.566 DEBUG:boto:path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.566 DEBUG:boto:auth_path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.567 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.567 DEBUG:boto:Path: /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.567 DEBUG:boto:Data: 2024-04-13T21:28:27.567 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.567 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.567 DEBUG:boto:Params: {} 2024-04-13T21:28:27.567 DEBUG:boto:Token: None 2024-04-13T21:28:27.567 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VDMSwPeR8TGc+mtjP29dIHmguOc= 2024-04-13T21:28:27.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VDMSwPeR8TGc+mtjP29dIHmguOc=', 'Content-Length': '0'} 2024-04-13T21:28:27.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a6ba4e5388184c7-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-13T21:28:27.570 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.570 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.570 DEBUG:boto:path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.570 DEBUG:boto:auth_path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.570 DEBUG:boto:Method: GET 2024-04-13T21:28:27.570 DEBUG:boto:Path: /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.570 DEBUG:boto:Data: 2024-04-13T21:28:27.570 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.570 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.570 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.571 DEBUG:boto:Params: {} 2024-04-13T21:28:27.571 DEBUG:boto:Token: None 2024-04-13T21:28:27.571 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw= 2024-04-13T21:28:27.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw=', 'Content-Length': '0'} 2024-04-13T21:28:27.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c86acc87158b9216-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.618 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.619 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.619 DEBUG:boto:path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.619 DEBUG:boto:auth_path=/vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.619 DEBUG:boto:Method: GET 2024-04-13T21:28:27.619 DEBUG:boto:Path: /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.619 DEBUG:boto:Data: 2024-04-13T21:28:27.619 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.620 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.620 DEBUG:boto:Params: {} 2024-04-13T21:28:27.620 DEBUG:boto:Token: None 2024-04-13T21:28:27.620 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A?versionId=null 2024-04-13T21:28:27.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw= 2024-04-13T21:28:27.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpyDLJaarj8FWv3Gl5sFl1kvYyw=', 'Content-Length': '0'} 2024-04-13T21:28:27.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000892da58682e804a1-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.622 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.623 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.623 DEBUG:boto:path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.623 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.623 DEBUG:boto:Method: GET 2024-04-13T21:28:27.623 DEBUG:boto:Path: /vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.623 DEBUG:boto:Data: 2024-04-13T21:28:27.623 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.624 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.624 DEBUG:boto:Params: {} 2024-04-13T21:28:27.624 DEBUG:boto:Token: None 2024-04-13T21:28:27.624 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.624 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000381c95eed8fd60f-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.626 DEBUG:boto:b'vwnfns-8:1000false:2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.627 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.627 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.627 DEBUG:boto:path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.627 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.627 DEBUG:boto:Method: GET 2024-04-13T21:28:27.628 DEBUG:boto:Path: /vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.628 DEBUG:boto:Data: 2024-04-13T21:28:27.628 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.628 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.628 DEBUG:boto:Params: {} 2024-04-13T21:28:27.628 DEBUG:boto:Token: None 2024-04-13T21:28:27.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8b3aec3b6624fd6-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.631 DEBUG:boto:b'vwnfns-8:1000false:2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.631 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.631 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.632 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.632 DEBUG:boto:Path: /vwnfns-8/%3A 2024-04-13T21:28:27.632 DEBUG:boto:Data: 2024-04-13T21:28:27.632 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.632 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.633 DEBUG:boto:Params: {} 2024-04-13T21:28:27.633 DEBUG:boto:Token: None 2024-04-13T21:28:27.633 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A 2024-04-13T21:28:27.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EW3+gL3Z7XfBehMTqwjWqxmtc5o= 2024-04-13T21:28:27.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EW3+gL3Z7XfBehMTqwjWqxmtc5o=', 'Content-Length': '0'} 2024-04-13T21:28:27.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000301161e7ca20b9aa-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.635 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.635 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.635 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.635 DEBUG:boto:Path: /vwnfns-8/%3A 2024-04-13T21:28:27.635 DEBUG:boto:Data: 2024-04-13T21:28:27.636 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.636 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.636 DEBUG:boto:Params: {} 2024-04-13T21:28:27.636 DEBUG:boto:Token: None 2024-04-13T21:28:27.637 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A 2024-04-13T21:28:27.637 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EW3+gL3Z7XfBehMTqwjWqxmtc5o= 2024-04-13T21:28:27.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EW3+gL3Z7XfBehMTqwjWqxmtc5o=', 'Content-Length': '0'} 2024-04-13T21:28:27.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bc4237464d841e6-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-13T21:28:27.640 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.640 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.640 DEBUG:boto:Method: GET 2024-04-13T21:28:27.640 DEBUG:boto:Path: /vwnfns-8/%3A 2024-04-13T21:28:27.640 DEBUG:boto:Data: 2024-04-13T21:28:27.641 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.641 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.641 DEBUG:boto:Params: {} 2024-04-13T21:28:27.641 DEBUG:boto:Token: None 2024-04-13T21:28:27.642 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A 2024-04-13T21:28:27.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q20W1wIBxG1xA6WlDsUgdOoBOJ8= 2024-04-13T21:28:27.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q20W1wIBxG1xA6WlDsUgdOoBOJ8=', 'Content-Length': '0'} 2024-04-13T21:28:27.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2d22dc11e3ac50-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.686 DEBUG:boto:path=/vwnfns-8/%3A 2024-04-13T21:28:27.687 DEBUG:boto:auth_path=/vwnfns-8/%3A 2024-04-13T21:28:27.687 DEBUG:boto:Method: GET 2024-04-13T21:28:27.687 DEBUG:boto:Path: /vwnfns-8/%3A 2024-04-13T21:28:27.687 DEBUG:boto:Data: 2024-04-13T21:28:27.687 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.687 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.687 DEBUG:boto:Params: {} 2024-04-13T21:28:27.687 DEBUG:boto:Token: None 2024-04-13T21:28:27.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/%3A 2024-04-13T21:28:27.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q20W1wIBxG1xA6WlDsUgdOoBOJ8= 2024-04-13T21:28:27.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q20W1wIBxG1xA6WlDsUgdOoBOJ8=', 'Content-Length': '0'} 2024-04-13T21:28:27.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e69265e9107b6efe-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.690 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.690 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.690 DEBUG:boto:path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.690 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.690 DEBUG:boto:Method: GET 2024-04-13T21:28:27.690 DEBUG:boto:Path: /vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.690 DEBUG:boto:Data: 2024-04-13T21:28:27.691 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.691 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.691 DEBUG:boto:Params: {} 2024-04-13T21:28:27.691 DEBUG:boto:Token: None 2024-04-13T21:28:27.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000668a33ec5c8677a3-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.694 DEBUG:boto:b'vwnfns-8:1000false:2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.695 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.695 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.695 DEBUG:boto:path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.695 DEBUG:boto:auth_path=/vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.695 DEBUG:boto:Method: GET 2024-04-13T21:28:27.695 DEBUG:boto:Path: /vwnfns-8/?prefix=%3A 2024-04-13T21:28:27.695 DEBUG:boto:Data: 2024-04-13T21:28:27.696 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.696 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.696 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.696 DEBUG:boto:Params: {} 2024-04-13T21:28:27.696 DEBUG:boto:Token: None 2024-04-13T21:28:27.697 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.697 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c011a5658cc01557-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.702 DEBUG:boto:b'vwnfns-8:1000false:2024-04-13T21:24:51.444Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-13T21:28:27.703 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.703 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.703 DEBUG:boto:path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.703 DEBUG:boto:auth_path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.703 DEBUG:boto:Method: GET 2024-04-13T21:28:27.703 DEBUG:boto:Path: /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.704 DEBUG:boto:Data: 2024-04-13T21:28:27.704 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.704 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.704 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.704 DEBUG:boto:Params: {} 2024-04-13T21:28:27.704 DEBUG:boto:Token: None 2024-04-13T21:28:27.704 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU= 2024-04-13T21:28:27.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU=', 'Content-Length': '0'} 2024-04-13T21:28:27.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f78f0aa441cf111e-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.707 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.707 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.707 DEBUG:boto:path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.707 DEBUG:boto:auth_path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.707 DEBUG:boto:Method: GET 2024-04-13T21:28:27.707 DEBUG:boto:Path: /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.708 DEBUG:boto:Data: 2024-04-13T21:28:27.708 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.708 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.708 DEBUG:boto:Params: {} 2024-04-13T21:28:27.709 DEBUG:boto:Token: None 2024-04-13T21:28:27.709 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.709 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU= 2024-04-13T21:28:27.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU=', 'Content-Length': '0'} 2024-04-13T21:28:27.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015725cf1b89f0b9a-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.712 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.712 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.712 DEBUG:boto:path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.713 DEBUG:boto:auth_path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.713 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.713 DEBUG:boto:Path: /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.713 DEBUG:boto:Data: 2024-04-13T21:28:27.713 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.714 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.714 DEBUG:boto:Params: {} 2024-04-13T21:28:27.714 DEBUG:boto:Token: None 2024-04-13T21:28:27.715 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TEx8YYKcGUVztxlkyk/EXmICNcE= 2024-04-13T21:28:27.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TEx8YYKcGUVztxlkyk/EXmICNcE=', 'Content-Length': '0'} 2024-04-13T21:28:27.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec33214df0d64ef6-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.718 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.718 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.718 DEBUG:boto:path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.718 DEBUG:boto:auth_path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.718 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.719 DEBUG:boto:Path: /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.719 DEBUG:boto:Data: 2024-04-13T21:28:27.719 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.719 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.719 DEBUG:boto:Params: {} 2024-04-13T21:28:27.720 DEBUG:boto:Token: None 2024-04-13T21:28:27.720 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TEx8YYKcGUVztxlkyk/EXmICNcE= 2024-04-13T21:28:27.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TEx8YYKcGUVztxlkyk/EXmICNcE=', 'Content-Length': '0'} 2024-04-13T21:28:27.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e63dd50bb1e83620-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-13T21:28:27.723 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.723 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.723 DEBUG:boto:path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.724 DEBUG:boto:auth_path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.724 DEBUG:boto:Method: GET 2024-04-13T21:28:27.724 DEBUG:boto:Path: /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.724 DEBUG:boto:Data: 2024-04-13T21:28:27.724 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.725 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.725 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.725 DEBUG:boto:Params: {} 2024-04-13T21:28:27.725 DEBUG:boto:Token: None 2024-04-13T21:28:27.726 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU= 2024-04-13T21:28:27.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU=', 'Content-Length': '0'} 2024-04-13T21:28:27.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002368e149679a21b7-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.728 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.728 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.729 DEBUG:boto:path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.729 DEBUG:boto:auth_path=/vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.729 DEBUG:boto:Method: GET 2024-04-13T21:28:27.729 DEBUG:boto:Path: /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.729 DEBUG:boto:Data: 2024-04-13T21:28:27.729 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.730 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.730 DEBUG:boto:Params: {} 2024-04-13T21:28:27.730 DEBUG:boto:Token: None 2024-04-13T21:28:27.730 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj?versionId=null 2024-04-13T21:28:27.730 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU= 2024-04-13T21:28:27.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+MeOKpqEjNCnG9R+n6Y8LvIMFvU=', 'Content-Length': '0'} 2024-04-13T21:28:27.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff326fc013c7a70c-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.779 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.779 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.779 DEBUG:boto:path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.779 DEBUG:boto:auth_path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.780 DEBUG:boto:Method: GET 2024-04-13T21:28:27.780 DEBUG:boto:Path: /vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.780 DEBUG:boto:Data: 2024-04-13T21:28:27.780 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.781 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.781 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.781 DEBUG:boto:Params: {} 2024-04-13T21:28:27.781 DEBUG:boto:Token: None 2024-04-13T21:28:27.782 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.782 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009081cedcb2630101-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.785 DEBUG:boto:b'vwnfns-8_myobj1000false_myobj2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.786 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.786 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.786 DEBUG:boto:path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.787 DEBUG:boto:auth_path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.787 DEBUG:boto:Method: GET 2024-04-13T21:28:27.787 DEBUG:boto:Path: /vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.787 DEBUG:boto:Data: 2024-04-13T21:28:27.787 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.788 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.788 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.788 DEBUG:boto:Params: {} 2024-04-13T21:28:27.788 DEBUG:boto:Token: None 2024-04-13T21:28:27.788 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034bd51fc1fae603c-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.792 DEBUG:boto:b'vwnfns-8_myobj1000false_myobj2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.793 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.793 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.793 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.793 DEBUG:boto:Path: /vwnfns-8/_myobj 2024-04-13T21:28:27.793 DEBUG:boto:Data: 2024-04-13T21:28:27.794 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.794 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.794 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.794 DEBUG:boto:Params: {} 2024-04-13T21:28:27.794 DEBUG:boto:Token: None 2024-04-13T21:28:27.794 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj 2024-04-13T21:28:27.794 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pZHtjFVYd427jErFzdtrJ3/VMBk= 2024-04-13T21:28:27.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pZHtjFVYd427jErFzdtrJ3/VMBk=', 'Content-Length': '0'} 2024-04-13T21:28:27.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5d9ea14b3c982db-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.796 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.797 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.797 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.797 DEBUG:boto:Path: /vwnfns-8/_myobj 2024-04-13T21:28:27.797 DEBUG:boto:Data: 2024-04-13T21:28:27.797 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.797 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.797 DEBUG:boto:Params: {} 2024-04-13T21:28:27.797 DEBUG:boto:Token: None 2024-04-13T21:28:27.797 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj 2024-04-13T21:28:27.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pZHtjFVYd427jErFzdtrJ3/VMBk= 2024-04-13T21:28:27.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pZHtjFVYd427jErFzdtrJ3/VMBk=', 'Content-Length': '0'} 2024-04-13T21:28:27.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d294964e67e106ae-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-13T21:28:27.799 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.800 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.800 DEBUG:boto:Method: GET 2024-04-13T21:28:27.800 DEBUG:boto:Path: /vwnfns-8/_myobj 2024-04-13T21:28:27.800 DEBUG:boto:Data: 2024-04-13T21:28:27.800 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.800 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.800 DEBUG:boto:Params: {} 2024-04-13T21:28:27.801 DEBUG:boto:Token: None 2024-04-13T21:28:27.801 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj 2024-04-13T21:28:27.801 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x7/RK8TMi/6AzymK8ChIB4GIiG8= 2024-04-13T21:28:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x7/RK8TMi/6AzymK8ChIB4GIiG8=', 'Content-Length': '0'} 2024-04-13T21:28:27.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074784f833b790639-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.803 DEBUG:boto:path=/vwnfns-8/_myobj 2024-04-13T21:28:27.803 DEBUG:boto:auth_path=/vwnfns-8/_myobj 2024-04-13T21:28:27.804 DEBUG:boto:Method: GET 2024-04-13T21:28:27.804 DEBUG:boto:Path: /vwnfns-8/_myobj 2024-04-13T21:28:27.804 DEBUG:boto:Data: 2024-04-13T21:28:27.804 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.804 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.804 DEBUG:boto:Params: {} 2024-04-13T21:28:27.804 DEBUG:boto:Token: None 2024-04-13T21:28:27.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/_myobj 2024-04-13T21:28:27.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x7/RK8TMi/6AzymK8ChIB4GIiG8= 2024-04-13T21:28:27.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x7/RK8TMi/6AzymK8ChIB4GIiG8=', 'Content-Length': '0'} 2024-04-13T21:28:27.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997fc6b0419e2983-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.850 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.851 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.851 DEBUG:boto:path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.851 DEBUG:boto:auth_path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.851 DEBUG:boto:Method: GET 2024-04-13T21:28:27.851 DEBUG:boto:Path: /vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.851 DEBUG:boto:Data: 2024-04-13T21:28:27.851 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.851 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.851 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.851 DEBUG:boto:Params: {} 2024-04-13T21:28:27.852 DEBUG:boto:Token: None 2024-04-13T21:28:27.852 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d57bd1a41245aa-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.855 DEBUG:boto:b'vwnfns-8_myobj1000false_myobj2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.855 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.855 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.855 DEBUG:boto:path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.855 DEBUG:boto:auth_path=/vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.856 DEBUG:boto:Method: GET 2024-04-13T21:28:27.856 DEBUG:boto:Path: /vwnfns-8/?prefix=_myobj 2024-04-13T21:28:27.856 DEBUG:boto:Data: 2024-04-13T21:28:27.856 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.856 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.856 DEBUG:boto:Params: {} 2024-04-13T21:28:27.856 DEBUG:boto:Token: None 2024-04-13T21:28:27.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006346d1466f90fe4e-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.859 DEBUG:boto:b'vwnfns-8_myobj1000false_myobj2024-04-13T21:24:51.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:27.860 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.860 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.860 DEBUG:boto:path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.860 DEBUG:boto:auth_path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.860 DEBUG:boto:Method: GET 2024-04-13T21:28:27.860 DEBUG:boto:Path: /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.860 DEBUG:boto:Data: 2024-04-13T21:28:27.860 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.860 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.861 DEBUG:boto:Params: {} 2024-04-13T21:28:27.861 DEBUG:boto:Token: None 2024-04-13T21:28:27.861 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.861 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU= 2024-04-13T21:28:27.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU=', 'Content-Length': '0'} 2024-04-13T21:28:27.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037ac0a3720f5d54c-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.906 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.907 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.907 DEBUG:boto:path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.907 DEBUG:boto:auth_path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.907 DEBUG:boto:Method: GET 2024-04-13T21:28:27.907 DEBUG:boto:Path: /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.907 DEBUG:boto:Data: 2024-04-13T21:28:27.907 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.908 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.908 DEBUG:boto:Params: {} 2024-04-13T21:28:27.908 DEBUG:boto:Token: None 2024-04-13T21:28:27.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU= 2024-04-13T21:28:27.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU=', 'Content-Length': '0'} 2024-04-13T21:28:27.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cca293fb58bbdb97-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.910 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.911 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.911 DEBUG:boto:path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.911 DEBUG:boto:auth_path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.911 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.911 DEBUG:boto:Path: /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.911 DEBUG:boto:Data: 2024-04-13T21:28:27.911 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.911 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.911 DEBUG:boto:Params: {} 2024-04-13T21:28:27.912 DEBUG:boto:Token: None 2024-04-13T21:28:27.912 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vwwqqpB6erCxGBFEGlgJXMXeZCA= 2024-04-13T21:28:27.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vwwqqpB6erCxGBFEGlgJXMXeZCA=', 'Content-Length': '0'} 2024-04-13T21:28:27.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000590bd7dd4ce42076-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.914 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.914 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.914 DEBUG:boto:path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.914 DEBUG:boto:auth_path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.914 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.914 DEBUG:boto:Path: /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.914 DEBUG:boto:Data: 2024-04-13T21:28:27.914 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.915 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.915 DEBUG:boto:Params: {} 2024-04-13T21:28:27.915 DEBUG:boto:Token: None 2024-04-13T21:28:27.915 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.915 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vwwqqpB6erCxGBFEGlgJXMXeZCA= 2024-04-13T21:28:27.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vwwqqpB6erCxGBFEGlgJXMXeZCA=', 'Content-Length': '0'} 2024-04-13T21:28:27.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f22a9b6acddc7877-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:27.917 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.917 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.917 DEBUG:boto:path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.917 DEBUG:boto:auth_path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.918 DEBUG:boto:Method: GET 2024-04-13T21:28:27.918 DEBUG:boto:Path: /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.918 DEBUG:boto:Data: 2024-04-13T21:28:27.918 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.918 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.918 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.918 DEBUG:boto:Params: {} 2024-04-13T21:28:27.918 DEBUG:boto:Token: None 2024-04-13T21:28:27.918 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.918 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU= 2024-04-13T21:28:27.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU=', 'Content-Length': '0'} 2024-04-13T21:28:27.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da15ad44a1f1d522-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.921 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.921 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.921 DEBUG:boto:path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.921 DEBUG:boto:auth_path=/vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.921 DEBUG:boto:Method: GET 2024-04-13T21:28:27.921 DEBUG:boto:Path: /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.921 DEBUG:boto:Data: 2024-04-13T21:28:27.921 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.922 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.922 DEBUG:boto:Params: {} 2024-04-13T21:28:27.922 DEBUG:boto:Token: None 2024-04-13T21:28:27.922 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj?versionId=null 2024-04-13T21:28:27.922 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU= 2024-04-13T21:28:27.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX://iCDte5ojpDgfNhmIUU95CqSBU=', 'Content-Length': '0'} 2024-04-13T21:28:27.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000889422514863b60c-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.924 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.924 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.924 DEBUG:boto:path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.924 DEBUG:boto:auth_path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.925 DEBUG:boto:Method: GET 2024-04-13T21:28:27.925 DEBUG:boto:Path: /vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.925 DEBUG:boto:Data: 2024-04-13T21:28:27.925 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.925 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.925 DEBUG:boto:Params: {} 2024-04-13T21:28:27.925 DEBUG:boto:Token: None 2024-04-13T21:28:27.925 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4c3f5868780a83-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.928 DEBUG:boto:b'vwnfns-8myobj1000falsemyobj2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.928 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.928 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.928 DEBUG:boto:path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.928 DEBUG:boto:auth_path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.929 DEBUG:boto:Method: GET 2024-04-13T21:28:27.929 DEBUG:boto:Path: /vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.929 DEBUG:boto:Data: 2024-04-13T21:28:27.929 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.929 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.929 DEBUG:boto:Params: {} 2024-04-13T21:28:27.929 DEBUG:boto:Token: None 2024-04-13T21:28:27.929 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aae8f9589ef5ebb-00661af8fb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.932 DEBUG:boto:b'vwnfns-8myobj1000falsemyobj2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.932 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.932 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.933 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.933 DEBUG:boto:Path: /vwnfns-8/myobj 2024-04-13T21:28:27.933 DEBUG:boto:Data: 2024-04-13T21:28:27.933 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.933 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.933 DEBUG:boto:Params: {} 2024-04-13T21:28:27.933 DEBUG:boto:Token: None 2024-04-13T21:28:27.933 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj 2024-04-13T21:28:27.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hfOP2yU/V41WCUbQnS6nBCh2oMY= 2024-04-13T21:28:27.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hfOP2yU/V41WCUbQnS6nBCh2oMY=', 'Content-Length': '0'} 2024-04-13T21:28:27.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b03549ac3438c3a-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.935 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.935 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.936 DEBUG:boto:Method: HEAD 2024-04-13T21:28:27.936 DEBUG:boto:Path: /vwnfns-8/myobj 2024-04-13T21:28:27.936 DEBUG:boto:Data: 2024-04-13T21:28:27.936 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.936 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.936 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.936 DEBUG:boto:Params: {} 2024-04-13T21:28:27.936 DEBUG:boto:Token: None 2024-04-13T21:28:27.936 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj 2024-04-13T21:28:27.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hfOP2yU/V41WCUbQnS6nBCh2oMY= 2024-04-13T21:28:27.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hfOP2yU/V41WCUbQnS6nBCh2oMY=', 'Content-Length': '0'} 2024-04-13T21:28:27.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb0a23ea415e3bc-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:27.938 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.939 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.939 DEBUG:boto:Method: GET 2024-04-13T21:28:27.939 DEBUG:boto:Path: /vwnfns-8/myobj 2024-04-13T21:28:27.939 DEBUG:boto:Data: 2024-04-13T21:28:27.939 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.939 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.939 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.939 DEBUG:boto:Params: {} 2024-04-13T21:28:27.939 DEBUG:boto:Token: None 2024-04-13T21:28:27.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj 2024-04-13T21:28:27.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wOKRBCVfQ32tIm+Fd9uNHaWXHkk= 2024-04-13T21:28:27.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wOKRBCVfQ32tIm+Fd9uNHaWXHkk=', 'Content-Length': '0'} 2024-04-13T21:28:27.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f505904eac80eaf-00661af8fb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.982 DEBUG:boto:path=/vwnfns-8/myobj 2024-04-13T21:28:27.983 DEBUG:boto:auth_path=/vwnfns-8/myobj 2024-04-13T21:28:27.983 DEBUG:boto:Method: GET 2024-04-13T21:28:27.983 DEBUG:boto:Path: /vwnfns-8/myobj 2024-04-13T21:28:27.983 DEBUG:boto:Data: 2024-04-13T21:28:27.983 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.983 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.983 DEBUG:boto:Params: {} 2024-04-13T21:28:27.983 DEBUG:boto:Token: None 2024-04-13T21:28:27.984 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/myobj 2024-04-13T21:28:27.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wOKRBCVfQ32tIm+Fd9uNHaWXHkk= 2024-04-13T21:28:27.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wOKRBCVfQ32tIm+Fd9uNHaWXHkk=', 'Content-Length': '0'} 2024-04-13T21:28:27.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:24:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ba68b1fd7564752-00661af8fb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.986 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.986 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.986 DEBUG:boto:path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.986 DEBUG:boto:auth_path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.986 DEBUG:boto:Method: GET 2024-04-13T21:28:27.986 DEBUG:boto:Path: /vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.987 DEBUG:boto:Data: 2024-04-13T21:28:27.987 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.987 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.987 DEBUG:boto:Params: {} 2024-04-13T21:28:27.987 DEBUG:boto:Token: None 2024-04-13T21:28:27.987 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f248dfb953a14f9-00661af8fb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.990 DEBUG:boto:b'vwnfns-8myobj1000falsemyobj2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.990 DEBUG:boto:path=/vwnfns-8/ 2024-04-13T21:28:27.990 DEBUG:boto:auth_path=/vwnfns-8/ 2024-04-13T21:28:27.990 DEBUG:boto:path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.990 DEBUG:boto:auth_path=/vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.991 DEBUG:boto:Method: GET 2024-04-13T21:28:27.991 DEBUG:boto:Path: /vwnfns-8/?prefix=myobj 2024-04-13T21:28:27.991 DEBUG:boto:Data: 2024-04-13T21:28:27.991 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:27.991 DEBUG:boto:Port: 8001 2024-04-13T21:28:27.991 DEBUG:boto:Params: {} 2024-04-13T21:28:27.991 DEBUG:boto:Token: None 2024-04-13T21:28:27.991 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-8/ 2024-04-13T21:28:27.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8= 2024-04-13T21:28:27.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ghWjJOONtEUD1Rj0HxTnsoNAT8=', 'Content-Length': '0'} 2024-04-13T21:28:27.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000288c53d50776e775-00661af8fc-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:27.994 DEBUG:boto:b'vwnfns-8myobj1000falsemyobj2024-04-13T21:24:51.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:27.994 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-8 zones={a1, a2} 2024-04-13T21:28:27.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-04-13T21:28:27.996 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-9 2024-04-13T21:28:27.996 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:27.996 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:27.997 DEBUG:boto:Method: PUT 2024-04-13T21:28:27.997 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:28:27.997 DEBUG:boto:Data: 2024-04-13T21:28:27.997 DEBUG:boto:Headers: {} 2024-04-13T21:28:27.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:27.997 DEBUG:boto:Port: 8000 2024-04-13T21:28:27.997 DEBUG:boto:Params: {} 2024-04-13T21:28:27.997 DEBUG:boto:Token: None 2024-04-13T21:28:27.997 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:28:27 GMT /vwnfns-9/ 2024-04-13T21:28:27.998 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qf49fI90agvuAQuz0Pr4KfdEBS8= 2024-04-13T21:28:27.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qf49fI90agvuAQuz0Pr4KfdEBS8=', 'Content-Length': '0'} 2024-04-13T21:28:28.012 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd88ffa9647179b1-00661af8fc-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:28.012 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-10 2024-04-13T21:28:28.012 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:28.012 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:28.012 DEBUG:boto:Method: PUT 2024-04-13T21:28:28.013 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:28:28.013 DEBUG:boto:Data: 2024-04-13T21:28:28.013 DEBUG:boto:Headers: {} 2024-04-13T21:28:28.013 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:28.013 DEBUG:boto:Port: 8001 2024-04-13T21:28:28.013 DEBUG:boto:Params: {} 2024-04-13T21:28:28.013 DEBUG:boto:Token: None 2024-04-13T21:28:28.013 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:28:28 GMT /vwnfns-10/ 2024-04-13T21:28:28.013 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1wryLtM/Ozvg0/orkwa3/IYAwY0= 2024-04-13T21:28:28.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1wryLtM/Ozvg0/orkwa3/IYAwY0=', 'Content-Length': '0'} 2024-04-13T21:28:28.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d69e09e0e8c1714-00661af8fc-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:28.041 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:28.041 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:28.041 DEBUG:boto:Method: HEAD 2024-04-13T21:28:28.041 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:28:28.041 DEBUG:boto:Data: 2024-04-13T21:28:28.042 DEBUG:boto:Headers: {} 2024-04-13T21:28:28.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:28.042 DEBUG:boto:Port: 8000 2024-04-13T21:28:28.042 DEBUG:boto:Params: {} 2024-04-13T21:28:28.042 DEBUG:boto:Token: None 2024-04-13T21:28:28.042 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:28 GMT /vwnfns-9/ 2024-04-13T21:28:28.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NIm6pHhAzV6altYF0aVq4FhL0WE= 2024-04-13T21:28:28.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NIm6pHhAzV6altYF0aVq4FhL0WE=', 'Content-Length': '0'} 2024-04-13T21:28:28.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2027a83b96cc088-00661af8fc-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:28.045 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:28.045 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:28.045 DEBUG:boto:Method: PUT 2024-04-13T21:28:28.045 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:28.045 DEBUG:boto:Data: 2024-04-13T21:28:28.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:28:28.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:28.045 DEBUG:boto:Port: 8000 2024-04-13T21:28:28.045 DEBUG:boto:Params: {} 2024-04-13T21:28:28.046 DEBUG:boto:Token: None 2024-04-13T21:28:28.046 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:28:28 GMT /vwnfns-9/myobj 2024-04-13T21:28:28.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D1xBRXq4+/UzDBfSWDIWuOBgTcM= 2024-04-13T21:28:28.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:28:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D1xBRXq4+/UzDBfSWDIWuOBgTcM='} 2024-04-13T21:28:28.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000577cdc442ed019a3-00661af8fc-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:28.054 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:28.054 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:28.055 DEBUG:boto:Method: HEAD 2024-04-13T21:28:28.055 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:28:28.055 DEBUG:boto:Data: 2024-04-13T21:28:28.055 DEBUG:boto:Headers: {} 2024-04-13T21:28:28.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:28.055 DEBUG:boto:Port: 8001 2024-04-13T21:28:28.055 DEBUG:boto:Params: {} 2024-04-13T21:28:28.055 DEBUG:boto:Token: None 2024-04-13T21:28:28.055 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:28 GMT /vwnfns-10/ 2024-04-13T21:28:28.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3wWKtoOXKA+nf0tS19HZ6XC/sfE= 2024-04-13T21:28:28.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3wWKtoOXKA+nf0tS19HZ6XC/sfE=', 'Content-Length': '0'} 2024-04-13T21:28:28.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035b5b530023eaa43-00661af8fc-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:28.058 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:28.058 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:28.058 DEBUG:boto:Method: PUT 2024-04-13T21:28:28.058 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:28:28.058 DEBUG:boto:Data: 2024-04-13T21:28:28.058 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:28:28.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:28.058 DEBUG:boto:Port: 8001 2024-04-13T21:28:28.059 DEBUG:boto:Params: {} 2024-04-13T21:28:28.059 DEBUG:boto:Token: None 2024-04-13T21:28:28.059 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:28:28 GMT /vwnfns-10/myobj 2024-04-13T21:28:28.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Qb3qOF8e5CC84FWNntJtz/HNZo= 2024-04-13T21:28:28.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:28:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Qb3qOF8e5CC84FWNntJtz/HNZo='} 2024-04-13T21:28:28.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ef6c2ca39024f72-00661af8fc-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:28.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:28:28.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:28:28.071 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:28:28.119 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:28:28.120 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:28:28.153 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:28:28.167 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:28:28.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.037043_82.1", 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.037043Z" 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:28:28.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.169 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:28:28.169 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:28:28.169 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:28:28.169 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:28:28.170 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043708.037043_82.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:28:28.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:28:28.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:28:28.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:28:28.171 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:28:28.221 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:28:28.221 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:28:28.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:28.258+0000 7f6f3c4f3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:28:28.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:28.259+0000 7f6f3c4f3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:28:28.262 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:28:28.262 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:28:28.262 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:28:28.263 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.037043_82.1", 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.037043Z", 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:28:28.264 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:28:28.265 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:28:28.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:28:28Z" 2024-04-13T21:28:28.268 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:28:28.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043708.037043_82.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:28:28.037043Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:28:28Z" } 2024-04-13T21:28:28.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043708.037043_82.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.037043Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:28:28.270 DEBUG:rgw_multi.tests:log_status={0: '1_1713043708.037043_82.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:28:28.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043708.037043_82.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:28:28.270 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:28:28.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:28:28.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:28:28.270 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:28:28.325 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:28:28.325 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:28:28.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:28.365+0000 7ff02780c780 1 waiting to reach incremental sync.. 2024-04-13T21:28:33.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:33.365+0000 7ff02780c780 1 waiting to reach incremental sync.. 2024-04-13T21:28:38.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:38.366+0000 7ff02780c780 1 waiting to reach incremental sync.. 2024-04-13T21:28:43.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:43.366+0000 7ff02780c780 1 waiting to reach incremental sync.. 2024-04-13T21:28:48.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:48.368+0000 7ff02780c780 1 bucket sync caught up with source: 2024-04-13T21:28:48.361 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:48.362 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:48.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:48.368+0000 7ff02780c780 0 bucket checkpoint complete 2024-04-13T21:28:48.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-9 zones={a2, a1} 2024-04-13T21:28:48.378 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.381 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.382 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.382 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:28:48.382 DEBUG:boto:Data: 2024-04-13T21:28:48.382 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.382 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.382 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.382 DEBUG:boto:Params: {} 2024-04-13T21:28:48.383 DEBUG:boto:Token: None 2024-04-13T21:28:48.383 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RU5EKzUlWnfw40JoBvjuujdbJbQ= 2024-04-13T21:28:48.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RU5EKzUlWnfw40JoBvjuujdbJbQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1515dd8448e61b0-00661af910-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.386 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.386 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.386 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.386 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:28:48.387 DEBUG:boto:Data: 2024-04-13T21:28:48.387 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.387 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.387 DEBUG:boto:Params: {} 2024-04-13T21:28:48.387 DEBUG:boto:Token: None 2024-04-13T21:28:48.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RU5EKzUlWnfw40JoBvjuujdbJbQ= 2024-04-13T21:28:48.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RU5EKzUlWnfw40JoBvjuujdbJbQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d456c72c4904c2a8-00661af910-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:28:48.390 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.390 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.390 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.390 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.391 DEBUG:boto:Method: GET 2024-04-13T21:28:48.391 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.391 DEBUG:boto:Data: 2024-04-13T21:28:48.391 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.391 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.391 DEBUG:boto:Params: {} 2024-04-13T21:28:48.391 DEBUG:boto:Token: None 2024-04-13T21:28:48.391 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/?versions 2024-04-13T21:28:48.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc= 2024-04-13T21:28:48.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc=', 'Content-Length': '0'} 2024-04-13T21:28:48.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a18c31d9701d1a0-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.394 DEBUG:boto:b'vwnfns-9999falsemyobjnulltrue2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.395 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:28:48.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:28:48.395 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.395 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.395 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.395 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.396 DEBUG:boto:Method: GET 2024-04-13T21:28:48.396 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.396 DEBUG:boto:Data: 2024-04-13T21:28:48.396 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.396 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.396 DEBUG:boto:Params: {} 2024-04-13T21:28:48.396 DEBUG:boto:Token: None 2024-04-13T21:28:48.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/?versions 2024-04-13T21:28:48.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc= 2024-04-13T21:28:48.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc=', 'Content-Length': '0'} 2024-04-13T21:28:48.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006138b6c31fec4955-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.400 DEBUG:boto:b'vwnfns-9999falsemyobjnulltrue2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.400 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:28:48.400 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.400 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.400 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.400 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.400 DEBUG:boto:Method: GET 2024-04-13T21:28:48.401 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.401 DEBUG:boto:Data: 2024-04-13T21:28:48.401 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.401 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.401 DEBUG:boto:Params: {} 2024-04-13T21:28:48.401 DEBUG:boto:Token: None 2024-04-13T21:28:48.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/?versions 2024-04-13T21:28:48.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc= 2024-04-13T21:28:48.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc=', 'Content-Length': '0'} 2024-04-13T21:28:48.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000197022923c1b5197-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.404 DEBUG:boto:b'vwnfns-9999falsemyobjnulltrue2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.404 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.404 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.404 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.404 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.405 DEBUG:boto:Method: GET 2024-04-13T21:28:48.405 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:28:48.405 DEBUG:boto:Data: 2024-04-13T21:28:48.405 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.405 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.405 DEBUG:boto:Params: {} 2024-04-13T21:28:48.405 DEBUG:boto:Token: None 2024-04-13T21:28:48.405 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/?versions 2024-04-13T21:28:48.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc= 2024-04-13T21:28:48.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khao+Tmw8yVgvnSEPu6yg96dBxc=', 'Content-Length': '0'} 2024-04-13T21:28:48.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084529b188540b1da-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.454 DEBUG:boto:b'vwnfns-9999falsemyobjnulltrue2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:48.455 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.455 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.455 DEBUG:boto:path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.455 DEBUG:boto:auth_path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.455 DEBUG:boto:Method: GET 2024-04-13T21:28:48.456 DEBUG:boto:Path: /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.456 DEBUG:boto:Data: 2024-04-13T21:28:48.456 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.456 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.456 DEBUG:boto:Params: {} 2024-04-13T21:28:48.456 DEBUG:boto:Token: None 2024-04-13T21:28:48.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g= 2024-04-13T21:28:48.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g=', 'Content-Length': '0'} 2024-04-13T21:28:48.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097a93131eb17358f-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.459 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.460 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.460 DEBUG:boto:path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.460 DEBUG:boto:auth_path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.460 DEBUG:boto:Method: GET 2024-04-13T21:28:48.460 DEBUG:boto:Path: /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.460 DEBUG:boto:Data: 2024-04-13T21:28:48.460 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.460 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.460 DEBUG:boto:Params: {} 2024-04-13T21:28:48.461 DEBUG:boto:Token: None 2024-04-13T21:28:48.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g= 2024-04-13T21:28:48.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g=', 'Content-Length': '0'} 2024-04-13T21:28:48.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061fd02234b3c33de-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.463 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.463 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.463 DEBUG:boto:path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.463 DEBUG:boto:auth_path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.463 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.463 DEBUG:boto:Path: /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.463 DEBUG:boto:Data: 2024-04-13T21:28:48.463 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.464 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.464 DEBUG:boto:Params: {} 2024-04-13T21:28:48.464 DEBUG:boto:Token: None 2024-04-13T21:28:48.464 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eGBlPgfX/yyDVTsbcIWVivEeILY= 2024-04-13T21:28:48.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eGBlPgfX/yyDVTsbcIWVivEeILY=', 'Content-Length': '0'} 2024-04-13T21:28:48.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000be84c03500bcfba-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.466 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.466 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.466 DEBUG:boto:path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.466 DEBUG:boto:auth_path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.466 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.466 DEBUG:boto:Path: /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.467 DEBUG:boto:Data: 2024-04-13T21:28:48.467 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.467 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.467 DEBUG:boto:Params: {} 2024-04-13T21:28:48.467 DEBUG:boto:Token: None 2024-04-13T21:28:48.467 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eGBlPgfX/yyDVTsbcIWVivEeILY= 2024-04-13T21:28:48.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eGBlPgfX/yyDVTsbcIWVivEeILY=', 'Content-Length': '0'} 2024-04-13T21:28:48.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002741256c2efbc0c7-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:48.469 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.469 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.469 DEBUG:boto:path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.469 DEBUG:boto:auth_path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.470 DEBUG:boto:Method: GET 2024-04-13T21:28:48.470 DEBUG:boto:Path: /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.470 DEBUG:boto:Data: 2024-04-13T21:28:48.470 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.470 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.470 DEBUG:boto:Params: {} 2024-04-13T21:28:48.470 DEBUG:boto:Token: None 2024-04-13T21:28:48.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.471 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g= 2024-04-13T21:28:48.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g=', 'Content-Length': '0'} 2024-04-13T21:28:48.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2050d314b398cb9-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.472 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.472 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.473 DEBUG:boto:path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.473 DEBUG:boto:auth_path=/vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.473 DEBUG:boto:Method: GET 2024-04-13T21:28:48.473 DEBUG:boto:Path: /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.473 DEBUG:boto:Data: 2024-04-13T21:28:48.473 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.473 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.473 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.473 DEBUG:boto:Params: {} 2024-04-13T21:28:48.473 DEBUG:boto:Token: None 2024-04-13T21:28:48.474 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj?versionId=null 2024-04-13T21:28:48.474 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g= 2024-04-13T21:28:48.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+Bgqs8KlCBpqesN9Betk94BBj4g=', 'Content-Length': '0'} 2024-04-13T21:28:48.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000378c32804c5f16eb-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.476 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.476 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.476 DEBUG:boto:path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.476 DEBUG:boto:auth_path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.476 DEBUG:boto:Method: GET 2024-04-13T21:28:48.476 DEBUG:boto:Path: /vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.476 DEBUG:boto:Data: 2024-04-13T21:28:48.476 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.477 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.477 DEBUG:boto:Params: {} 2024-04-13T21:28:48.477 DEBUG:boto:Token: None 2024-04-13T21:28:48.477 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo= 2024-04-13T21:28:48.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo=', 'Content-Length': '0'} 2024-04-13T21:28:48.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d0104f147edfcb-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.479 DEBUG:boto:b'vwnfns-9myobj1000falsemyobj2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.480 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.480 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.480 DEBUG:boto:path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.480 DEBUG:boto:auth_path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.480 DEBUG:boto:Method: GET 2024-04-13T21:28:48.480 DEBUG:boto:Path: /vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.480 DEBUG:boto:Data: 2024-04-13T21:28:48.480 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.481 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.481 DEBUG:boto:Params: {} 2024-04-13T21:28:48.481 DEBUG:boto:Token: None 2024-04-13T21:28:48.481 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.481 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo= 2024-04-13T21:28:48.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo=', 'Content-Length': '0'} 2024-04-13T21:28:48.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ad391b084d0357-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.483 DEBUG:boto:b'vwnfns-9myobj1000falsemyobj2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.484 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.484 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.484 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.484 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:48.484 DEBUG:boto:Data: 2024-04-13T21:28:48.484 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.484 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.484 DEBUG:boto:Params: {} 2024-04-13T21:28:48.485 DEBUG:boto:Token: None 2024-04-13T21:28:48.485 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj 2024-04-13T21:28:48.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YMSHPWZII9RPF+FlSULJJyRe3iA= 2024-04-13T21:28:48.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YMSHPWZII9RPF+FlSULJJyRe3iA=', 'Content-Length': '0'} 2024-04-13T21:28:48.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2b1e307ce9bcedc-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.487 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.487 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.487 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.487 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:48.487 DEBUG:boto:Data: 2024-04-13T21:28:48.487 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.487 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.488 DEBUG:boto:Params: {} 2024-04-13T21:28:48.488 DEBUG:boto:Token: None 2024-04-13T21:28:48.488 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj 2024-04-13T21:28:48.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YMSHPWZII9RPF+FlSULJJyRe3iA= 2024-04-13T21:28:48.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YMSHPWZII9RPF+FlSULJJyRe3iA=', 'Content-Length': '0'} 2024-04-13T21:28:48.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34ba13d67618751-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:48.490 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.490 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.491 DEBUG:boto:Method: GET 2024-04-13T21:28:48.491 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:48.491 DEBUG:boto:Data: 2024-04-13T21:28:48.491 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.491 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.491 DEBUG:boto:Params: {} 2024-04-13T21:28:48.491 DEBUG:boto:Token: None 2024-04-13T21:28:48.491 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj 2024-04-13T21:28:48.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lB3WN9wZ1eXK2wVol69K4SKqH/s= 2024-04-13T21:28:48.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lB3WN9wZ1eXK2wVol69K4SKqH/s=', 'Content-Length': '0'} 2024-04-13T21:28:48.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3c9475dd7b62eb7-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.534 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.535 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.535 DEBUG:boto:Method: GET 2024-04-13T21:28:48.535 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:48.535 DEBUG:boto:Data: 2024-04-13T21:28:48.535 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.535 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.535 DEBUG:boto:Params: {} 2024-04-13T21:28:48.535 DEBUG:boto:Token: None 2024-04-13T21:28:48.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj 2024-04-13T21:28:48.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lB3WN9wZ1eXK2wVol69K4SKqH/s= 2024-04-13T21:28:48.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lB3WN9wZ1eXK2wVol69K4SKqH/s=', 'Content-Length': '0'} 2024-04-13T21:28:48.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd31aafd9adf3d45-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.538 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.538 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.538 DEBUG:boto:path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.538 DEBUG:boto:auth_path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.538 DEBUG:boto:Method: GET 2024-04-13T21:28:48.538 DEBUG:boto:Path: /vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.538 DEBUG:boto:Data: 2024-04-13T21:28:48.538 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.539 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.539 DEBUG:boto:Params: {} 2024-04-13T21:28:48.539 DEBUG:boto:Token: None 2024-04-13T21:28:48.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo= 2024-04-13T21:28:48.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo=', 'Content-Length': '0'} 2024-04-13T21:28:48.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af82c39ca60e144-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.541 DEBUG:boto:b'vwnfns-9myobj1000falsemyobj2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.542 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.542 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.542 DEBUG:boto:path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.543 DEBUG:boto:auth_path=/vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.543 DEBUG:boto:Method: GET 2024-04-13T21:28:48.543 DEBUG:boto:Path: /vwnfns-9/?prefix=myobj 2024-04-13T21:28:48.543 DEBUG:boto:Data: 2024-04-13T21:28:48.543 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.544 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.544 DEBUG:boto:Params: {} 2024-04-13T21:28:48.544 DEBUG:boto:Token: None 2024-04-13T21:28:48.544 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo= 2024-04-13T21:28:48.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0bzRrHsFGcaHaPJ2RSo4/l1LFjo=', 'Content-Length': '0'} 2024-04-13T21:28:48.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051024ab916e79e2a-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.546 DEBUG:boto:b'vwnfns-9myobj1000falsemyobj2024-04-13T21:28:28.056Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-9 zones={a2, a1} 2024-04-13T21:28:48.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:28:48.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:28:48.547 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:28:48.605 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:28:48.605 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:28:48.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:48.647+0000 7fa41c6e1780 1 bucket sync caught up with source: 2024-04-13T21:28:48.640 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:48.640 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:48.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:48.647+0000 7fa41c6e1780 0 bucket checkpoint complete 2024-04-13T21:28:48.652 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-10 zones={a1, a2} 2024-04-13T21:28:48.652 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.652 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.652 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.652 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:28:48.653 DEBUG:boto:Data: 2024-04-13T21:28:48.653 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.653 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.653 DEBUG:boto:Params: {} 2024-04-13T21:28:48.653 DEBUG:boto:Token: None 2024-04-13T21:28:48.653 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/ 2024-04-13T21:28:48.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zFyd6gtVk+qEKUpcuJgMTw4SsmI= 2024-04-13T21:28:48.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zFyd6gtVk+qEKUpcuJgMTw4SsmI=', 'Content-Length': '0'} 2024-04-13T21:28:48.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004151288969351c31-00661af910-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.656 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.656 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.656 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.656 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:28:48.656 DEBUG:boto:Data: 2024-04-13T21:28:48.656 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.656 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.656 DEBUG:boto:Params: {} 2024-04-13T21:28:48.657 DEBUG:boto:Token: None 2024-04-13T21:28:48.657 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/ 2024-04-13T21:28:48.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zFyd6gtVk+qEKUpcuJgMTw4SsmI= 2024-04-13T21:28:48.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zFyd6gtVk+qEKUpcuJgMTw4SsmI=', 'Content-Length': '0'} 2024-04-13T21:28:48.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1efe183afc144cf-00661af910-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.659 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:28:48.659 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.659 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.659 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.659 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.659 DEBUG:boto:Method: GET 2024-04-13T21:28:48.659 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.659 DEBUG:boto:Data: 2024-04-13T21:28:48.660 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.660 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.660 DEBUG:boto:Params: {} 2024-04-13T21:28:48.660 DEBUG:boto:Token: None 2024-04-13T21:28:48.660 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/?versions 2024-04-13T21:28:48.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M= 2024-04-13T21:28:48.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M=', 'Content-Length': '0'} 2024-04-13T21:28:48.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a88dd3e1cc6e5dc3-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.662 DEBUG:boto:b'vwnfns-10999falsemyobjnulltrue2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.663 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:28:48.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:28:48.663 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.663 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.663 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.663 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.663 DEBUG:boto:Method: GET 2024-04-13T21:28:48.664 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.664 DEBUG:boto:Data: 2024-04-13T21:28:48.664 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.664 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.664 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.664 DEBUG:boto:Params: {} 2024-04-13T21:28:48.664 DEBUG:boto:Token: None 2024-04-13T21:28:48.664 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/?versions 2024-04-13T21:28:48.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M= 2024-04-13T21:28:48.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M=', 'Content-Length': '0'} 2024-04-13T21:28:48.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5d75fdf2677f4d-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.667 DEBUG:boto:b'vwnfns-10999falsemyobjnulltrue2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.667 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-13T21:28:48.667 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.667 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.667 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.667 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.667 DEBUG:boto:Method: GET 2024-04-13T21:28:48.668 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.668 DEBUG:boto:Data: 2024-04-13T21:28:48.668 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.668 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.668 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.668 DEBUG:boto:Params: {} 2024-04-13T21:28:48.668 DEBUG:boto:Token: None 2024-04-13T21:28:48.668 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/?versions 2024-04-13T21:28:48.668 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M= 2024-04-13T21:28:48.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M=', 'Content-Length': '0'} 2024-04-13T21:28:48.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6db84f090b86d8-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.671 DEBUG:boto:b'vwnfns-10999falsemyobjnulltrue2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.671 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.671 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.671 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.671 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.671 DEBUG:boto:Method: GET 2024-04-13T21:28:48.671 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:28:48.672 DEBUG:boto:Data: 2024-04-13T21:28:48.672 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.672 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.672 DEBUG:boto:Params: {} 2024-04-13T21:28:48.672 DEBUG:boto:Token: None 2024-04-13T21:28:48.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/?versions 2024-04-13T21:28:48.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M= 2024-04-13T21:28:48.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+5z4kDkz8fTm8SIx4n1vYFzEH9M=', 'Content-Length': '0'} 2024-04-13T21:28:48.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036ada19327dfea85-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.723 DEBUG:boto:b'vwnfns-10999falsemyobjnulltrue2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:48.723 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.723 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.724 DEBUG:boto:path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.724 DEBUG:boto:auth_path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.724 DEBUG:boto:Method: GET 2024-04-13T21:28:48.724 DEBUG:boto:Path: /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.724 DEBUG:boto:Data: 2024-04-13T21:28:48.724 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.724 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.724 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.724 DEBUG:boto:Params: {} 2024-04-13T21:28:48.724 DEBUG:boto:Token: None 2024-04-13T21:28:48.725 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.725 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ= 2024-04-13T21:28:48.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c42d09c041056b36-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.727 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.727 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.727 DEBUG:boto:path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.727 DEBUG:boto:auth_path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.728 DEBUG:boto:Method: GET 2024-04-13T21:28:48.728 DEBUG:boto:Path: /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.728 DEBUG:boto:Data: 2024-04-13T21:28:48.728 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.728 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.728 DEBUG:boto:Params: {} 2024-04-13T21:28:48.728 DEBUG:boto:Token: None 2024-04-13T21:28:48.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ= 2024-04-13T21:28:48.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035c08e699289e16d-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.730 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.731 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.731 DEBUG:boto:path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.731 DEBUG:boto:auth_path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.731 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.731 DEBUG:boto:Path: /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.731 DEBUG:boto:Data: 2024-04-13T21:28:48.731 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.731 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.731 DEBUG:boto:Params: {} 2024-04-13T21:28:48.731 DEBUG:boto:Token: None 2024-04-13T21:28:48.732 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UL7K17BkbF3MgTjiND6R5PSaKLw= 2024-04-13T21:28:48.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UL7K17BkbF3MgTjiND6R5PSaKLw=', 'Content-Length': '0'} 2024-04-13T21:28:48.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd2fa30a3036a1d8-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.733 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.733 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.734 DEBUG:boto:path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.734 DEBUG:boto:auth_path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.734 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.734 DEBUG:boto:Path: /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.734 DEBUG:boto:Data: 2024-04-13T21:28:48.734 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.734 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.734 DEBUG:boto:Params: {} 2024-04-13T21:28:48.734 DEBUG:boto:Token: None 2024-04-13T21:28:48.735 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UL7K17BkbF3MgTjiND6R5PSaKLw= 2024-04-13T21:28:48.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UL7K17BkbF3MgTjiND6R5PSaKLw=', 'Content-Length': '0'} 2024-04-13T21:28:48.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000623b20eba6f1b393-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:48.736 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.737 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.737 DEBUG:boto:path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.737 DEBUG:boto:auth_path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.737 DEBUG:boto:Method: GET 2024-04-13T21:28:48.737 DEBUG:boto:Path: /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.737 DEBUG:boto:Data: 2024-04-13T21:28:48.737 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.737 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.737 DEBUG:boto:Params: {} 2024-04-13T21:28:48.737 DEBUG:boto:Token: None 2024-04-13T21:28:48.738 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.738 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ= 2024-04-13T21:28:48.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c078eb2b835999d-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.786 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.786 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.787 DEBUG:boto:path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.787 DEBUG:boto:auth_path=/vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.787 DEBUG:boto:Method: GET 2024-04-13T21:28:48.787 DEBUG:boto:Path: /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.787 DEBUG:boto:Data: 2024-04-13T21:28:48.787 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.787 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.787 DEBUG:boto:Params: {} 2024-04-13T21:28:48.787 DEBUG:boto:Token: None 2024-04-13T21:28:48.788 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj?versionId=null 2024-04-13T21:28:48.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ= 2024-04-13T21:28:48.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m2syGoth6D0taniPi+2+es7WioQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015d768fe8afeed1e-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.790 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.790 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.790 DEBUG:boto:path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.790 DEBUG:boto:auth_path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.791 DEBUG:boto:Method: GET 2024-04-13T21:28:48.791 DEBUG:boto:Path: /vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.791 DEBUG:boto:Data: 2024-04-13T21:28:48.791 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.791 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.791 DEBUG:boto:Params: {} 2024-04-13T21:28:48.791 DEBUG:boto:Token: None 2024-04-13T21:28:48.791 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/ 2024-04-13T21:28:48.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o= 2024-04-13T21:28:48.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o=', 'Content-Length': '0'} 2024-04-13T21:28:48.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb05e4f0b401c19-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.796 DEBUG:boto:b'vwnfns-10myobj1000falsemyobj2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.796 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.796 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.796 DEBUG:boto:path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.796 DEBUG:boto:auth_path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.796 DEBUG:boto:Method: GET 2024-04-13T21:28:48.797 DEBUG:boto:Path: /vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.797 DEBUG:boto:Data: 2024-04-13T21:28:48.797 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.797 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.797 DEBUG:boto:Params: {} 2024-04-13T21:28:48.797 DEBUG:boto:Token: None 2024-04-13T21:28:48.797 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/ 2024-04-13T21:28:48.797 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o= 2024-04-13T21:28:48.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o=', 'Content-Length': '0'} 2024-04-13T21:28:48.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c2ae0691560aa3-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.800 DEBUG:boto:b'vwnfns-10myobj1000falsemyobj2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.800 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.800 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.800 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.800 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:28:48.800 DEBUG:boto:Data: 2024-04-13T21:28:48.800 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.801 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.801 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.801 DEBUG:boto:Params: {} 2024-04-13T21:28:48.801 DEBUG:boto:Token: None 2024-04-13T21:28:48.801 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj 2024-04-13T21:28:48.801 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OV6TOEQUx5JITiaCanGtHDlLjIo= 2024-04-13T21:28:48.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OV6TOEQUx5JITiaCanGtHDlLjIo=', 'Content-Length': '0'} 2024-04-13T21:28:48.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f37318f7450fa9e5-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.803 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.803 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.803 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.803 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:28:48.803 DEBUG:boto:Data: 2024-04-13T21:28:48.803 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.803 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.804 DEBUG:boto:Params: {} 2024-04-13T21:28:48.804 DEBUG:boto:Token: None 2024-04-13T21:28:48.804 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj 2024-04-13T21:28:48.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OV6TOEQUx5JITiaCanGtHDlLjIo= 2024-04-13T21:28:48.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OV6TOEQUx5JITiaCanGtHDlLjIo=', 'Content-Length': '0'} 2024-04-13T21:28:48.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f8e157c3412020a-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-13T21:28:48.806 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.806 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.806 DEBUG:boto:Method: GET 2024-04-13T21:28:48.806 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:28:48.806 DEBUG:boto:Data: 2024-04-13T21:28:48.806 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.806 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.807 DEBUG:boto:Params: {} 2024-04-13T21:28:48.807 DEBUG:boto:Token: None 2024-04-13T21:28:48.807 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj 2024-04-13T21:28:48.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J0BohtVv5cHr0NRR2weYxiRv01I= 2024-04-13T21:28:48.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J0BohtVv5cHr0NRR2weYxiRv01I=', 'Content-Length': '0'} 2024-04-13T21:28:48.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc198859702a0480-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.851 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:28:48.852 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:28:48.852 DEBUG:boto:Method: GET 2024-04-13T21:28:48.852 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:28:48.852 DEBUG:boto:Data: 2024-04-13T21:28:48.852 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.853 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.853 DEBUG:boto:Params: {} 2024-04-13T21:28:48.853 DEBUG:boto:Token: None 2024-04-13T21:28:48.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/myobj 2024-04-13T21:28:48.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J0BohtVv5cHr0NRR2weYxiRv01I= 2024-04-13T21:28:48.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J0BohtVv5cHr0NRR2weYxiRv01I=', 'Content-Length': '0'} 2024-04-13T21:28:48.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acdcde3dfd94b66c-00661af910-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.856 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.856 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.856 DEBUG:boto:path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.856 DEBUG:boto:auth_path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.856 DEBUG:boto:Method: GET 2024-04-13T21:28:48.857 DEBUG:boto:Path: /vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.857 DEBUG:boto:Data: 2024-04-13T21:28:48.857 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.857 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.857 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.857 DEBUG:boto:Params: {} 2024-04-13T21:28:48.857 DEBUG:boto:Token: None 2024-04-13T21:28:48.857 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/ 2024-04-13T21:28:48.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o= 2024-04-13T21:28:48.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o=', 'Content-Length': '0'} 2024-04-13T21:28:48.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638a69c61a249394-00661af910-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.859 DEBUG:boto:b'vwnfns-10myobj1000falsemyobj2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.860 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:28:48.860 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:28:48.860 DEBUG:boto:path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.860 DEBUG:boto:auth_path=/vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.860 DEBUG:boto:Method: GET 2024-04-13T21:28:48.860 DEBUG:boto:Path: /vwnfns-10/?prefix=myobj 2024-04-13T21:28:48.861 DEBUG:boto:Data: 2024-04-13T21:28:48.861 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.861 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:28:48.861 DEBUG:boto:Port: 8001 2024-04-13T21:28:48.861 DEBUG:boto:Params: {} 2024-04-13T21:28:48.861 DEBUG:boto:Token: None 2024-04-13T21:28:48.861 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-10/ 2024-04-13T21:28:48.861 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o= 2024-04-13T21:28:48.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRUGvZ/LGbYmce/Pr6plgxpHD/o=', 'Content-Length': '0'} 2024-04-13T21:28:48.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005de3bb34aca1a339-00661af910-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.863 DEBUG:boto:b'vwnfns-10myobj1000falsemyobj2024-04-13T21:28:28.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:28:48.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-10 zones={a1, a2} 2024-04-13T21:28:48.864 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:28:48.864 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:28:48.864 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.864 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:28:48.864 DEBUG:boto:Data: 2024-04-13T21:28:48.864 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.864 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.865 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.865 DEBUG:boto:Params: {} 2024-04-13T21:28:48.865 DEBUG:boto:Token: None 2024-04-13T21:28:48.865 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/ 2024-04-13T21:28:48.865 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RU5EKzUlWnfw40JoBvjuujdbJbQ= 2024-04-13T21:28:48.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RU5EKzUlWnfw40JoBvjuujdbJbQ=', 'Content-Length': '0'} 2024-04-13T21:28:48.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000676643beb83bc2de-00661af910-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.868 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.868 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.868 DEBUG:boto:Method: HEAD 2024-04-13T21:28:48.868 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:48.868 DEBUG:boto:Data: 2024-04-13T21:28:48.868 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.868 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.869 DEBUG:boto:Params: {} 2024-04-13T21:28:48.869 DEBUG:boto:Token: None 2024-04-13T21:28:48.869 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj 2024-04-13T21:28:48.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YMSHPWZII9RPF+FlSULJJyRe3iA= 2024-04-13T21:28:48.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YMSHPWZII9RPF+FlSULJJyRe3iA=', 'Content-Length': '0'} 2024-04-13T21:28:48.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ea459d0357971ea-00661af910-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.873 DEBUG:boto:path=/vwnfns-9/myobj 2024-04-13T21:28:48.873 DEBUG:boto:auth_path=/vwnfns-9/myobj 2024-04-13T21:28:48.873 DEBUG:boto:Method: DELETE 2024-04-13T21:28:48.873 DEBUG:boto:Path: /vwnfns-9/myobj 2024-04-13T21:28:48.873 DEBUG:boto:Data: 2024-04-13T21:28:48.873 DEBUG:boto:Headers: {} 2024-04-13T21:28:48.873 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:28:48.873 DEBUG:boto:Port: 8000 2024-04-13T21:28:48.873 DEBUG:boto:Params: {} 2024-04-13T21:28:48.873 DEBUG:boto:Token: None 2024-04-13T21:28:48.874 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:28:48 GMT /vwnfns-9/myobj 2024-04-13T21:28:48.874 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Mp++cIO6oXAuhqSNe/AdF19TWqM= 2024-04-13T21:28:48.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:28:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Mp++cIO6oXAuhqSNe/AdF19TWqM=', 'Content-Length': '0'} 2024-04-13T21:28:48.878 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011c9101e075e5690-00661af910-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:28:48.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:28:48.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:28:48.879 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:28:48.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:28:48.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:28:48.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:48.974+0000 7f4a0b18b780 1 waiting for incremental sync to catch up: 2024-04-13T21:28:48.967 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:48.967 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-13T21:28:53.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:53.976+0000 7f4a0b18b780 1 waiting for incremental sync to catch up: 2024-04-13T21:28:53.969 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:53.969 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-13T21:28:58.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:28:58.978+0000 7f4a0b18b780 1 waiting for incremental sync to catch up: 2024-04-13T21:28:58.970 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:28:58.970 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-13T21:29:03.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:03.979+0000 7f4a0b18b780 1 waiting for incremental sync to catch up: 2024-04-13T21:29:03.972 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:29:03.972 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-13T21:29:08.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:08.980+0000 7f4a0b18b780 1 bucket sync caught up with source: 2024-04-13T21:29:08.973 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.7.6, , , , , , , , , ] 2024-04-13T21:29:08.973 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-13T21:29:08.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:08.980+0000 7f4a0b18b780 0 bucket checkpoint complete 2024-04-13T21:29:08.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-9 zones={a2, a1} 2024-04-13T21:29:08.989 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:29:08.989 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:29:08.989 DEBUG:boto:Method: HEAD 2024-04-13T21:29:08.989 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:29:08.990 DEBUG:boto:Data: 2024-04-13T21:29:08.990 DEBUG:boto:Headers: {} 2024-04-13T21:29:08.990 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:08.990 DEBUG:boto:Port: 8001 2024-04-13T21:29:08.990 DEBUG:boto:Params: {} 2024-04-13T21:29:08.990 DEBUG:boto:Token: None 2024-04-13T21:29:08.990 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:08 GMT /vwnfns-9/ 2024-04-13T21:29:08.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wbvkEY1G+4S9M7RpDbavOdefcI0= 2024-04-13T21:29:08.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wbvkEY1G+4S9M7RpDbavOdefcI0=', 'Content-Length': '0'} 2024-04-13T21:29:08.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032970155e5561c0c-00661af925-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:08.993 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:29:08.994 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:29:08.994 DEBUG:boto:Method: HEAD 2024-04-13T21:29:08.994 DEBUG:boto:Path: /vwnfns-9/ 2024-04-13T21:29:08.994 DEBUG:boto:Data: 2024-04-13T21:29:08.994 DEBUG:boto:Headers: {} 2024-04-13T21:29:08.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:08.995 DEBUG:boto:Port: 8000 2024-04-13T21:29:08.995 DEBUG:boto:Params: {} 2024-04-13T21:29:08.995 DEBUG:boto:Token: None 2024-04-13T21:29:08.995 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:08 GMT /vwnfns-9/ 2024-04-13T21:29:08.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wbvkEY1G+4S9M7RpDbavOdefcI0= 2024-04-13T21:29:08.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wbvkEY1G+4S9M7RpDbavOdefcI0=', 'Content-Length': '0'} 2024-04-13T21:29:08.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e7e311cc39c1840-00661af925-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:08.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:29:08.998 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:29:08.998 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:29:08.998 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:08.998 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:08.999 DEBUG:boto:Method: GET 2024-04-13T21:29:08.999 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:08.999 DEBUG:boto:Data: 2024-04-13T21:29:08.999 DEBUG:boto:Headers: {} 2024-04-13T21:29:08.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:08.999 DEBUG:boto:Port: 8001 2024-04-13T21:29:08.999 DEBUG:boto:Params: {} 2024-04-13T21:29:08.999 DEBUG:boto:Token: None 2024-04-13T21:29:08.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:08 GMT /vwnfns-9/?versions 2024-04-13T21:29:09.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HrqW57cJWdAHki8+sOyBscKkz9I= 2024-04-13T21:29:09.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HrqW57cJWdAHki8+sOyBscKkz9I=', 'Content-Length': '0'} 2024-04-13T21:29:09.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000445d4890ff1a79a2-00661af925-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.002 DEBUG:boto:b'vwnfns-9999false' 2024-04-13T21:29:09.003 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:29:09.003 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:29:09.003 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:29:09.003 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.003 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.003 DEBUG:boto:Method: GET 2024-04-13T21:29:09.003 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.003 DEBUG:boto:Data: 2024-04-13T21:29:09.003 DEBUG:boto:Headers: {} 2024-04-13T21:29:09.004 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:09.004 DEBUG:boto:Port: 8000 2024-04-13T21:29:09.004 DEBUG:boto:Params: {} 2024-04-13T21:29:09.004 DEBUG:boto:Token: None 2024-04-13T21:29:09.004 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:09 GMT /vwnfns-9/?versions 2024-04-13T21:29:09.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1XxJiCAIQLa0htFuMgjfVo+46VU= 2024-04-13T21:29:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1XxJiCAIQLa0htFuMgjfVo+46VU=', 'Content-Length': '0'} 2024-04-13T21:29:09.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c31bb94795e8b2d-00661af925-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.007 DEBUG:boto:b'vwnfns-9999false' 2024-04-13T21:29:09.007 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:29:09.007 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:29:09.007 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.007 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.007 DEBUG:boto:Method: GET 2024-04-13T21:29:09.008 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.008 DEBUG:boto:Data: 2024-04-13T21:29:09.008 DEBUG:boto:Headers: {} 2024-04-13T21:29:09.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:09.008 DEBUG:boto:Port: 8001 2024-04-13T21:29:09.008 DEBUG:boto:Params: {} 2024-04-13T21:29:09.008 DEBUG:boto:Token: None 2024-04-13T21:29:09.008 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:09 GMT /vwnfns-9/?versions 2024-04-13T21:29:09.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1XxJiCAIQLa0htFuMgjfVo+46VU= 2024-04-13T21:29:09.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1XxJiCAIQLa0htFuMgjfVo+46VU=', 'Content-Length': '0'} 2024-04-13T21:29:09.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f6afed6b556c58f-00661af925-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.011 DEBUG:boto:b'vwnfns-9999false' 2024-04-13T21:29:09.011 DEBUG:boto:path=/vwnfns-9/ 2024-04-13T21:29:09.011 DEBUG:boto:auth_path=/vwnfns-9/ 2024-04-13T21:29:09.011 DEBUG:boto:path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.011 DEBUG:boto:auth_path=/vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.011 DEBUG:boto:Method: GET 2024-04-13T21:29:09.012 DEBUG:boto:Path: /vwnfns-9/?versions&max-keys=999 2024-04-13T21:29:09.012 DEBUG:boto:Data: 2024-04-13T21:29:09.012 DEBUG:boto:Headers: {} 2024-04-13T21:29:09.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:09.012 DEBUG:boto:Port: 8000 2024-04-13T21:29:09.012 DEBUG:boto:Params: {} 2024-04-13T21:29:09.012 DEBUG:boto:Token: None 2024-04-13T21:29:09.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:09 GMT /vwnfns-9/?versions 2024-04-13T21:29:09.013 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1XxJiCAIQLa0htFuMgjfVo+46VU= 2024-04-13T21:29:09.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1XxJiCAIQLa0htFuMgjfVo+46VU=', 'Content-Length': '0'} 2024-04-13T21:29:09.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000318754bf117cf24f-00661af925-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.015 DEBUG:boto:b'vwnfns-9999false' 2024-04-13T21:29:09.015 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-9 zones={a2, a1} 2024-04-13T21:29:09.015 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:09.015 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:09.015 DEBUG:boto:Method: HEAD 2024-04-13T21:29:09.015 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:29:09.016 DEBUG:boto:Data: 2024-04-13T21:29:09.016 DEBUG:boto:Headers: {} 2024-04-13T21:29:09.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:09.016 DEBUG:boto:Port: 8001 2024-04-13T21:29:09.016 DEBUG:boto:Params: {} 2024-04-13T21:29:09.016 DEBUG:boto:Token: None 2024-04-13T21:29:09.016 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:09 GMT /vwnfns-10/ 2024-04-13T21:29:09.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2mczZVhR9l4sOzJ19bCVN4yciu4= 2024-04-13T21:29:09.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2mczZVhR9l4sOzJ19bCVN4yciu4=', 'Content-Length': '0'} 2024-04-13T21:29:09.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dac2d875711a2358-00661af925-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.019 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:29:09.019 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:29:09.019 DEBUG:boto:Method: HEAD 2024-04-13T21:29:09.019 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:29:09.019 DEBUG:boto:Data: 2024-04-13T21:29:09.019 DEBUG:boto:Headers: {} 2024-04-13T21:29:09.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:09.019 DEBUG:boto:Port: 8001 2024-04-13T21:29:09.019 DEBUG:boto:Params: {} 2024-04-13T21:29:09.020 DEBUG:boto:Token: None 2024-04-13T21:29:09.020 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:09 GMT /vwnfns-10/myobj 2024-04-13T21:29:09.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k/r8xSTvcRJyk3z4mv6QzU1NtOI= 2024-04-13T21:29:09.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k/r8xSTvcRJyk3z4mv6QzU1NtOI=', 'Content-Length': '0'} 2024-04-13T21:29:09.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:28:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093cfad6fffcad588-00661af925-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.022 DEBUG:boto:path=/vwnfns-10/myobj 2024-04-13T21:29:09.022 DEBUG:boto:auth_path=/vwnfns-10/myobj 2024-04-13T21:29:09.022 DEBUG:boto:Method: DELETE 2024-04-13T21:29:09.023 DEBUG:boto:Path: /vwnfns-10/myobj 2024-04-13T21:29:09.023 DEBUG:boto:Data: 2024-04-13T21:29:09.023 DEBUG:boto:Headers: {} 2024-04-13T21:29:09.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:09.023 DEBUG:boto:Port: 8001 2024-04-13T21:29:09.023 DEBUG:boto:Params: {} 2024-04-13T21:29:09.023 DEBUG:boto:Token: None 2024-04-13T21:29:09.023 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:29:09 GMT /vwnfns-10/myobj 2024-04-13T21:29:09.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgBWIj3Tk6t+oKxKYsTziZptTeQ= 2024-04-13T21:29:09.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgBWIj3Tk6t+oKxKYsTziZptTeQ=', 'Content-Length': '0'} 2024-04-13T21:29:09.030 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000daf6930bf889168e-00661af925-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:09.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:29:09.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:29:09.031 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:29:09.089 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:29:09.089 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:29:10.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:10.137+0000 7f1be311b780 1 waiting for incremental sync to catch up: 2024-04-13T21:29:10.130 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:29:10.130 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-13T21:29:15.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:15.138+0000 7f1be311b780 1 waiting for incremental sync to catch up: 2024-04-13T21:29:15.132 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:29:15.132 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-13T21:29:20.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:20.139+0000 7f1be311b780 1 waiting for incremental sync to catch up: 2024-04-13T21:29:20.133 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-13T21:29:20.133 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-13T21:29:25.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:25.140+0000 7f1be311b780 1 bucket sync caught up with source: 2024-04-13T21:29:25.134 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.8.6, , , , , , , , , ] 2024-04-13T21:29:25.134 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2024-04-13T21:29:25.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:25.140+0000 7f1be311b780 0 bucket checkpoint complete 2024-04-13T21:29:25.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-10 zones={a1, a2} 2024-04-13T21:29:25.150 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:25.150 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:25.150 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.151 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:29:25.151 DEBUG:boto:Data: 2024-04-13T21:29:25.151 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.151 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.151 DEBUG:boto:Params: {} 2024-04-13T21:29:25.151 DEBUG:boto:Token: None 2024-04-13T21:29:25.151 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-10/ 2024-04-13T21:29:25.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+SYod+oK3cq5qAA1GXgXPX8DFTo= 2024-04-13T21:29:25.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+SYod+oK3cq5qAA1GXgXPX8DFTo=', 'Content-Length': '0'} 2024-04-13T21:29:25.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a14c2d459b32593a-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.154 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:25.154 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:25.155 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.155 DEBUG:boto:Path: /vwnfns-10/ 2024-04-13T21:29:25.155 DEBUG:boto:Data: 2024-04-13T21:29:25.155 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:25.155 DEBUG:boto:Port: 8001 2024-04-13T21:29:25.155 DEBUG:boto:Params: {} 2024-04-13T21:29:25.155 DEBUG:boto:Token: None 2024-04-13T21:29:25.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-10/ 2024-04-13T21:29:25.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+SYod+oK3cq5qAA1GXgXPX8DFTo= 2024-04-13T21:29:25.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+SYod+oK3cq5qAA1GXgXPX8DFTo=', 'Content-Length': '0'} 2024-04-13T21:29:25.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001d6bf9c6c207aa0-00661af935-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.158 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:29:25.158 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:25.158 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:25.159 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.159 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.159 DEBUG:boto:Method: GET 2024-04-13T21:29:25.159 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.159 DEBUG:boto:Data: 2024-04-13T21:29:25.159 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.159 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.159 DEBUG:boto:Params: {} 2024-04-13T21:29:25.159 DEBUG:boto:Token: None 2024-04-13T21:29:25.160 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-10/?versions 2024-04-13T21:29:25.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM= 2024-04-13T21:29:25.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM=', 'Content-Length': '0'} 2024-04-13T21:29:25.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000987601b875025df-00661af935-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.162 DEBUG:boto:b'vwnfns-10999false' 2024-04-13T21:29:25.163 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:29:25.163 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:25.163 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:25.163 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.163 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.163 DEBUG:boto:Method: GET 2024-04-13T21:29:25.163 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.163 DEBUG:boto:Data: 2024-04-13T21:29:25.163 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:25.164 DEBUG:boto:Port: 8001 2024-04-13T21:29:25.164 DEBUG:boto:Params: {} 2024-04-13T21:29:25.164 DEBUG:boto:Token: None 2024-04-13T21:29:25.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-10/?versions 2024-04-13T21:29:25.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM= 2024-04-13T21:29:25.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM=', 'Content-Length': '0'} 2024-04-13T21:29:25.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8e32b0310ecba9-00661af935-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.166 DEBUG:boto:b'vwnfns-10999false' 2024-04-13T21:29:25.167 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:25.167 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:25.167 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.167 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.167 DEBUG:boto:Method: GET 2024-04-13T21:29:25.167 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.167 DEBUG:boto:Data: 2024-04-13T21:29:25.167 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.167 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.168 DEBUG:boto:Params: {} 2024-04-13T21:29:25.168 DEBUG:boto:Token: None 2024-04-13T21:29:25.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-10/?versions 2024-04-13T21:29:25.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM= 2024-04-13T21:29:25.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM=', 'Content-Length': '0'} 2024-04-13T21:29:25.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd67eac0ed2e74da-00661af935-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.170 DEBUG:boto:b'vwnfns-10999false' 2024-04-13T21:29:25.171 DEBUG:boto:path=/vwnfns-10/ 2024-04-13T21:29:25.171 DEBUG:boto:auth_path=/vwnfns-10/ 2024-04-13T21:29:25.171 DEBUG:boto:path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.171 DEBUG:boto:auth_path=/vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.171 DEBUG:boto:Method: GET 2024-04-13T21:29:25.171 DEBUG:boto:Path: /vwnfns-10/?versions&max-keys=999 2024-04-13T21:29:25.171 DEBUG:boto:Data: 2024-04-13T21:29:25.171 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:25.171 DEBUG:boto:Port: 8001 2024-04-13T21:29:25.172 DEBUG:boto:Params: {} 2024-04-13T21:29:25.172 DEBUG:boto:Token: None 2024-04-13T21:29:25.172 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-10/?versions 2024-04-13T21:29:25.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM= 2024-04-13T21:29:25.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bbNq6TjaIl0GeZZCkv9nyR3boEM=', 'Content-Length': '0'} 2024-04-13T21:29:25.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f71dd773b3f417b-00661af935-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.214 DEBUG:boto:b'vwnfns-10999false' 2024-04-13T21:29:25.215 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-10 zones={a1, a2} 2024-04-13T21:29:25.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-04-13T21:29:25.217 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-11 2024-04-13T21:29:25.217 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.217 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.217 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.217 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.217 DEBUG:boto:Data: 2024-04-13T21:29:25.217 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.217 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.217 DEBUG:boto:Params: {} 2024-04-13T21:29:25.218 DEBUG:boto:Token: None 2024-04-13T21:29:25.218 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9cl4+lXbVTrhDB+4jA6u5jwCZ1E= 2024-04-13T21:29:25.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9cl4+lXbVTrhDB+4jA6u5jwCZ1E=', 'Content-Length': '0'} 2024-04-13T21:29:25.236 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001090ccbbe29fb4fc-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.237 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-12 2024-04-13T21:29:25.237 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:25.237 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:25.237 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.237 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:25.237 DEBUG:boto:Data: 2024-04-13T21:29:25.237 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.237 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:25.237 DEBUG:boto:Port: 8001 2024-04-13T21:29:25.237 DEBUG:boto:Params: {} 2024-04-13T21:29:25.238 DEBUG:boto:Token: None 2024-04-13T21:29:25.238 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-12/ 2024-04-13T21:29:25.238 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EYBRKpvCF5hNX9HoD+fZvaUJFU4= 2024-04-13T21:29:25.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EYBRKpvCF5hNX9HoD+fZvaUJFU4=', 'Content-Length': '0'} 2024-04-13T21:29:25.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002fec7f875f9fe81f-00661af935-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.273 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.273 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.273 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.273 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.274 DEBUG:boto:Data: 2024-04-13T21:29:25.274 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.274 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.274 DEBUG:boto:Params: {} 2024-04-13T21:29:25.274 DEBUG:boto:Token: None 2024-04-13T21:29:25.274 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017e98b3bebc24d85-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.277 DEBUG:boto:path=/vwnfns-11/obj1 2024-04-13T21:29:25.277 DEBUG:boto:auth_path=/vwnfns-11/obj1 2024-04-13T21:29:25.277 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.277 DEBUG:boto:Path: /vwnfns-11/obj1 2024-04-13T21:29:25.277 DEBUG:boto:Data: 2024-04-13T21:29:25.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.277 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.278 DEBUG:boto:Params: {} 2024-04-13T21:29:25.278 DEBUG:boto:Token: None 2024-04-13T21:29:25.278 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj1 2024-04-13T21:29:25.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rc6qA9vMNB9eEqI/1XmQ5zp3GZY= 2024-04-13T21:29:25.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rc6qA9vMNB9eEqI/1XmQ5zp3GZY='} 2024-04-13T21:29:25.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022a117557181e462-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.285 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.285 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.285 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.285 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.285 DEBUG:boto:Data: 2024-04-13T21:29:25.285 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.285 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.285 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.285 DEBUG:boto:Params: {} 2024-04-13T21:29:25.286 DEBUG:boto:Token: None 2024-04-13T21:29:25.286 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aace2fd0e2160bb-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.288 DEBUG:boto:path=/vwnfns-11/obj2 2024-04-13T21:29:25.288 DEBUG:boto:auth_path=/vwnfns-11/obj2 2024-04-13T21:29:25.288 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.288 DEBUG:boto:Path: /vwnfns-11/obj2 2024-04-13T21:29:25.288 DEBUG:boto:Data: 2024-04-13T21:29:25.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.289 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.289 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.289 DEBUG:boto:Params: {} 2024-04-13T21:29:25.289 DEBUG:boto:Token: None 2024-04-13T21:29:25.289 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj2 2024-04-13T21:29:25.289 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1t77kGyEUeZf/rj6vXK6hTBK/II= 2024-04-13T21:29:25.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1t77kGyEUeZf/rj6vXK6hTBK/II='} 2024-04-13T21:29:25.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed1e253b77c7fb0e-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.294 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.294 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.295 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.295 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.295 DEBUG:boto:Data: 2024-04-13T21:29:25.295 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.295 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.295 DEBUG:boto:Params: {} 2024-04-13T21:29:25.295 DEBUG:boto:Token: None 2024-04-13T21:29:25.295 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b9e0e841d1fc6d8-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.298 DEBUG:boto:path=/vwnfns-11/obj3 2024-04-13T21:29:25.298 DEBUG:boto:auth_path=/vwnfns-11/obj3 2024-04-13T21:29:25.299 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.299 DEBUG:boto:Path: /vwnfns-11/obj3 2024-04-13T21:29:25.299 DEBUG:boto:Data: 2024-04-13T21:29:25.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.299 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.299 DEBUG:boto:Params: {} 2024-04-13T21:29:25.299 DEBUG:boto:Token: None 2024-04-13T21:29:25.299 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj3 2024-04-13T21:29:25.299 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FuBC2lixTTuFtzNN4HgDQv9dPDM= 2024-04-13T21:29:25.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FuBC2lixTTuFtzNN4HgDQv9dPDM='} 2024-04-13T21:29:25.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c154711e1850f3f-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.347 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.347 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.347 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.347 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.347 DEBUG:boto:Data: 2024-04-13T21:29:25.347 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.347 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.348 DEBUG:boto:Params: {} 2024-04-13T21:29:25.348 DEBUG:boto:Token: None 2024-04-13T21:29:25.348 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032d118812ffbcfb5-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.350 DEBUG:boto:path=/vwnfns-11/obj4 2024-04-13T21:29:25.351 DEBUG:boto:auth_path=/vwnfns-11/obj4 2024-04-13T21:29:25.351 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.351 DEBUG:boto:Path: /vwnfns-11/obj4 2024-04-13T21:29:25.351 DEBUG:boto:Data: 2024-04-13T21:29:25.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.351 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.351 DEBUG:boto:Params: {} 2024-04-13T21:29:25.351 DEBUG:boto:Token: None 2024-04-13T21:29:25.352 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj4 2024-04-13T21:29:25.352 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aHfiJxg6eBYbW7Rx/tjlExE+BuE= 2024-04-13T21:29:25.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aHfiJxg6eBYbW7Rx/tjlExE+BuE='} 2024-04-13T21:29:25.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dfcd17d2a483799-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.356 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.357 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.357 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.357 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.357 DEBUG:boto:Data: 2024-04-13T21:29:25.357 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.357 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.357 DEBUG:boto:Params: {} 2024-04-13T21:29:25.357 DEBUG:boto:Token: None 2024-04-13T21:29:25.357 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055a1be978549fcb9-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.360 DEBUG:boto:path=/vwnfns-11/obj5 2024-04-13T21:29:25.360 DEBUG:boto:auth_path=/vwnfns-11/obj5 2024-04-13T21:29:25.360 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.360 DEBUG:boto:Path: /vwnfns-11/obj5 2024-04-13T21:29:25.360 DEBUG:boto:Data: 2024-04-13T21:29:25.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.361 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.361 DEBUG:boto:Params: {} 2024-04-13T21:29:25.361 DEBUG:boto:Token: None 2024-04-13T21:29:25.361 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj5 2024-04-13T21:29:25.361 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:09jlYKnbSt/f6PmNno5PRnCyYUE= 2024-04-13T21:29:25.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:09jlYKnbSt/f6PmNno5PRnCyYUE='} 2024-04-13T21:29:25.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052ff82dabdd34009-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.407 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.407 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.407 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.407 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.407 DEBUG:boto:Data: 2024-04-13T21:29:25.407 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.407 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.408 DEBUG:boto:Params: {} 2024-04-13T21:29:25.408 DEBUG:boto:Token: None 2024-04-13T21:29:25.408 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89a2f0222a256b5-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.423 DEBUG:boto:path=/vwnfns-11/obj6 2024-04-13T21:29:25.423 DEBUG:boto:auth_path=/vwnfns-11/obj6 2024-04-13T21:29:25.423 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.423 DEBUG:boto:Path: /vwnfns-11/obj6 2024-04-13T21:29:25.423 DEBUG:boto:Data: 2024-04-13T21:29:25.423 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.423 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.424 DEBUG:boto:Params: {} 2024-04-13T21:29:25.424 DEBUG:boto:Token: None 2024-04-13T21:29:25.424 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj6 2024-04-13T21:29:25.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F+AN7SZusydjT+QnBrWzDi9CMKA= 2024-04-13T21:29:25.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F+AN7SZusydjT+QnBrWzDi9CMKA='} 2024-04-13T21:29:25.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e098cead1dfbfdc-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.428 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.429 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.429 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.429 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.429 DEBUG:boto:Data: 2024-04-13T21:29:25.429 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.429 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.429 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.429 DEBUG:boto:Params: {} 2024-04-13T21:29:25.429 DEBUG:boto:Token: None 2024-04-13T21:29:25.429 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.430 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b603802667998684-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.432 DEBUG:boto:path=/vwnfns-11/obj7 2024-04-13T21:29:25.432 DEBUG:boto:auth_path=/vwnfns-11/obj7 2024-04-13T21:29:25.432 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.432 DEBUG:boto:Path: /vwnfns-11/obj7 2024-04-13T21:29:25.432 DEBUG:boto:Data: 2024-04-13T21:29:25.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.432 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.432 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.433 DEBUG:boto:Params: {} 2024-04-13T21:29:25.433 DEBUG:boto:Token: None 2024-04-13T21:29:25.433 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj7 2024-04-13T21:29:25.433 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aVDIwZQN8nloMsMRqAn5joULjfA= 2024-04-13T21:29:25.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aVDIwZQN8nloMsMRqAn5joULjfA='} 2024-04-13T21:29:25.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088454787d7a35afc-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.475 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.475 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.475 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.475 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.475 DEBUG:boto:Data: 2024-04-13T21:29:25.475 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.475 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.475 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.476 DEBUG:boto:Params: {} 2024-04-13T21:29:25.476 DEBUG:boto:Token: None 2024-04-13T21:29:25.476 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000900c08ac4bed26a9-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.479 DEBUG:boto:path=/vwnfns-11/obj8 2024-04-13T21:29:25.479 DEBUG:boto:auth_path=/vwnfns-11/obj8 2024-04-13T21:29:25.479 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.479 DEBUG:boto:Path: /vwnfns-11/obj8 2024-04-13T21:29:25.479 DEBUG:boto:Data: 2024-04-13T21:29:25.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.480 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.480 DEBUG:boto:Params: {} 2024-04-13T21:29:25.480 DEBUG:boto:Token: None 2024-04-13T21:29:25.480 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj8 2024-04-13T21:29:25.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ahARxELLQSetAdgauxagMo9X4lg= 2024-04-13T21:29:25.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ahARxELLQSetAdgauxagMo9X4lg='} 2024-04-13T21:29:25.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b29dc9a55fcebebe-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.485 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.485 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.485 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.485 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.485 DEBUG:boto:Data: 2024-04-13T21:29:25.485 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.485 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.485 DEBUG:boto:Params: {} 2024-04-13T21:29:25.485 DEBUG:boto:Token: None 2024-04-13T21:29:25.486 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b25c710f5c96611f-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.488 DEBUG:boto:path=/vwnfns-11/obj9 2024-04-13T21:29:25.488 DEBUG:boto:auth_path=/vwnfns-11/obj9 2024-04-13T21:29:25.488 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.489 DEBUG:boto:Path: /vwnfns-11/obj9 2024-04-13T21:29:25.489 DEBUG:boto:Data: 2024-04-13T21:29:25.489 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.489 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.489 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.489 DEBUG:boto:Params: {} 2024-04-13T21:29:25.489 DEBUG:boto:Token: None 2024-04-13T21:29:25.489 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj9 2024-04-13T21:29:25.490 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ei3BpzF6SpjG0nW5ez5Rwtf6zmc= 2024-04-13T21:29:25.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ei3BpzF6SpjG0nW5ez5Rwtf6zmc='} 2024-04-13T21:29:25.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e484eca9138aa063-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.535 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.535 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.535 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.535 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.536 DEBUG:boto:Data: 2024-04-13T21:29:25.536 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.536 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.536 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.536 DEBUG:boto:Params: {} 2024-04-13T21:29:25.536 DEBUG:boto:Token: None 2024-04-13T21:29:25.536 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054983d2adb500ccc-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.539 DEBUG:boto:path=/vwnfns-11/obj10 2024-04-13T21:29:25.539 DEBUG:boto:auth_path=/vwnfns-11/obj10 2024-04-13T21:29:25.539 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.539 DEBUG:boto:Path: /vwnfns-11/obj10 2024-04-13T21:29:25.539 DEBUG:boto:Data: 2024-04-13T21:29:25.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.540 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.540 DEBUG:boto:Params: {} 2024-04-13T21:29:25.540 DEBUG:boto:Token: None 2024-04-13T21:29:25.540 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj10 2024-04-13T21:29:25.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x639aYamntLZApWARiXxjhit7xM= 2024-04-13T21:29:25.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x639aYamntLZApWARiXxjhit7xM='} 2024-04-13T21:29:25.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f737870d602cc21e-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.544 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.544 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.544 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.544 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.544 DEBUG:boto:Data: 2024-04-13T21:29:25.544 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.545 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.545 DEBUG:boto:Params: {} 2024-04-13T21:29:25.545 DEBUG:boto:Token: None 2024-04-13T21:29:25.545 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000204f90e30e24af0e-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.547 DEBUG:boto:path=/vwnfns-11/obj11 2024-04-13T21:29:25.547 DEBUG:boto:auth_path=/vwnfns-11/obj11 2024-04-13T21:29:25.548 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.548 DEBUG:boto:Path: /vwnfns-11/obj11 2024-04-13T21:29:25.548 DEBUG:boto:Data: 2024-04-13T21:29:25.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.548 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.548 DEBUG:boto:Params: {} 2024-04-13T21:29:25.548 DEBUG:boto:Token: None 2024-04-13T21:29:25.548 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj11 2024-04-13T21:29:25.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2C9I8x59FteE4egJsXL0JOxFyU8= 2024-04-13T21:29:25.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2C9I8x59FteE4egJsXL0JOxFyU8='} 2024-04-13T21:29:25.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eda1e07886ac5237-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.591 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.591 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.591 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.591 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.591 DEBUG:boto:Data: 2024-04-13T21:29:25.591 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.592 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.592 DEBUG:boto:Params: {} 2024-04-13T21:29:25.592 DEBUG:boto:Token: None 2024-04-13T21:29:25.592 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d22a8772039de504-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.595 DEBUG:boto:path=/vwnfns-11/obj12 2024-04-13T21:29:25.595 DEBUG:boto:auth_path=/vwnfns-11/obj12 2024-04-13T21:29:25.595 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.595 DEBUG:boto:Path: /vwnfns-11/obj12 2024-04-13T21:29:25.595 DEBUG:boto:Data: 2024-04-13T21:29:25.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.595 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.595 DEBUG:boto:Params: {} 2024-04-13T21:29:25.596 DEBUG:boto:Token: None 2024-04-13T21:29:25.596 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj12 2024-04-13T21:29:25.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SeH26MTdBWgUo8qTgJdXzMoTEJY= 2024-04-13T21:29:25.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SeH26MTdBWgUo8qTgJdXzMoTEJY='} 2024-04-13T21:29:25.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bf1fde388e0853a-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.599 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.599 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.600 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.600 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.600 DEBUG:boto:Data: 2024-04-13T21:29:25.600 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.600 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.600 DEBUG:boto:Params: {} 2024-04-13T21:29:25.600 DEBUG:boto:Token: None 2024-04-13T21:29:25.600 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098a58e6257b9f817-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.602 DEBUG:boto:path=/vwnfns-11/obj13 2024-04-13T21:29:25.603 DEBUG:boto:auth_path=/vwnfns-11/obj13 2024-04-13T21:29:25.603 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.603 DEBUG:boto:Path: /vwnfns-11/obj13 2024-04-13T21:29:25.603 DEBUG:boto:Data: 2024-04-13T21:29:25.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.603 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.603 DEBUG:boto:Params: {} 2024-04-13T21:29:25.604 DEBUG:boto:Token: None 2024-04-13T21:29:25.604 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj13 2024-04-13T21:29:25.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oyBuaC4rgfuG3gm7OOFpRjPk1uY= 2024-04-13T21:29:25.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oyBuaC4rgfuG3gm7OOFpRjPk1uY='} 2024-04-13T21:29:25.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e95cf705074ab7f3-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.651 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.651 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.651 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.651 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.651 DEBUG:boto:Data: 2024-04-13T21:29:25.651 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.652 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.652 DEBUG:boto:Params: {} 2024-04-13T21:29:25.652 DEBUG:boto:Token: None 2024-04-13T21:29:25.652 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002460738c5f02b313-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.654 DEBUG:boto:path=/vwnfns-11/obj14 2024-04-13T21:29:25.654 DEBUG:boto:auth_path=/vwnfns-11/obj14 2024-04-13T21:29:25.654 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.654 DEBUG:boto:Path: /vwnfns-11/obj14 2024-04-13T21:29:25.655 DEBUG:boto:Data: 2024-04-13T21:29:25.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.655 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.655 DEBUG:boto:Params: {} 2024-04-13T21:29:25.655 DEBUG:boto:Token: None 2024-04-13T21:29:25.655 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj14 2024-04-13T21:29:25.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LIC9CxTDqk/qBuEMRvPdzKecxQU= 2024-04-13T21:29:25.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LIC9CxTDqk/qBuEMRvPdzKecxQU='} 2024-04-13T21:29:25.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b29059c7fbae1029-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.659 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.659 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.659 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.659 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.659 DEBUG:boto:Data: 2024-04-13T21:29:25.660 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.660 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.660 DEBUG:boto:Params: {} 2024-04-13T21:29:25.660 DEBUG:boto:Token: None 2024-04-13T21:29:25.660 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ebae84286e2d422-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.663 DEBUG:boto:path=/vwnfns-11/obj15 2024-04-13T21:29:25.663 DEBUG:boto:auth_path=/vwnfns-11/obj15 2024-04-13T21:29:25.663 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.663 DEBUG:boto:Path: /vwnfns-11/obj15 2024-04-13T21:29:25.663 DEBUG:boto:Data: 2024-04-13T21:29:25.663 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.663 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.663 DEBUG:boto:Params: {} 2024-04-13T21:29:25.664 DEBUG:boto:Token: None 2024-04-13T21:29:25.664 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj15 2024-04-13T21:29:25.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:70U3DnmGf9gnIa4O8uEu9iaaagg= 2024-04-13T21:29:25.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:70U3DnmGf9gnIa4O8uEu9iaaagg='} 2024-04-13T21:29:25.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003512bd956e6f0773-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.707 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.707 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.707 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.707 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.707 DEBUG:boto:Data: 2024-04-13T21:29:25.707 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.707 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.708 DEBUG:boto:Params: {} 2024-04-13T21:29:25.708 DEBUG:boto:Token: None 2024-04-13T21:29:25.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c91556200b422de-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.710 DEBUG:boto:path=/vwnfns-11/obj16 2024-04-13T21:29:25.710 DEBUG:boto:auth_path=/vwnfns-11/obj16 2024-04-13T21:29:25.710 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.710 DEBUG:boto:Path: /vwnfns-11/obj16 2024-04-13T21:29:25.711 DEBUG:boto:Data: 2024-04-13T21:29:25.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.711 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.711 DEBUG:boto:Params: {} 2024-04-13T21:29:25.711 DEBUG:boto:Token: None 2024-04-13T21:29:25.711 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj16 2024-04-13T21:29:25.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D1+nI3qrggK/ENU2TeIEnd30bMA= 2024-04-13T21:29:25.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D1+nI3qrggK/ENU2TeIEnd30bMA='} 2024-04-13T21:29:25.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002650b57860318748-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.717 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.717 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.718 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.718 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.718 DEBUG:boto:Data: 2024-04-13T21:29:25.718 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.718 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.719 DEBUG:boto:Params: {} 2024-04-13T21:29:25.719 DEBUG:boto:Token: None 2024-04-13T21:29:25.719 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007387a8ae5089ccb0-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.723 DEBUG:boto:path=/vwnfns-11/obj17 2024-04-13T21:29:25.723 DEBUG:boto:auth_path=/vwnfns-11/obj17 2024-04-13T21:29:25.723 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.723 DEBUG:boto:Path: /vwnfns-11/obj17 2024-04-13T21:29:25.723 DEBUG:boto:Data: 2024-04-13T21:29:25.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.723 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.723 DEBUG:boto:Params: {} 2024-04-13T21:29:25.723 DEBUG:boto:Token: None 2024-04-13T21:29:25.724 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj17 2024-04-13T21:29:25.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LV/navtrVsJqY3UQTc1LVHMAewg= 2024-04-13T21:29:25.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LV/navtrVsJqY3UQTc1LVHMAewg='} 2024-04-13T21:29:25.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b380e8dbda6ea8e8-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.767 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.767 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.767 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.767 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.767 DEBUG:boto:Data: 2024-04-13T21:29:25.767 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.767 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.768 DEBUG:boto:Params: {} 2024-04-13T21:29:25.768 DEBUG:boto:Token: None 2024-04-13T21:29:25.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c3bc0953cd28a53-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.770 DEBUG:boto:path=/vwnfns-11/obj18 2024-04-13T21:29:25.770 DEBUG:boto:auth_path=/vwnfns-11/obj18 2024-04-13T21:29:25.771 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.771 DEBUG:boto:Path: /vwnfns-11/obj18 2024-04-13T21:29:25.771 DEBUG:boto:Data: 2024-04-13T21:29:25.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.771 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.771 DEBUG:boto:Params: {} 2024-04-13T21:29:25.771 DEBUG:boto:Token: None 2024-04-13T21:29:25.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj18 2024-04-13T21:29:25.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/GFranvcbchxDu9EWq0WFk3A09I= 2024-04-13T21:29:25.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/GFranvcbchxDu9EWq0WFk3A09I='} 2024-04-13T21:29:25.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae658dad60dd7cba-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.775 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.775 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.775 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.776 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.776 DEBUG:boto:Data: 2024-04-13T21:29:25.776 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.776 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.776 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.776 DEBUG:boto:Params: {} 2024-04-13T21:29:25.776 DEBUG:boto:Token: None 2024-04-13T21:29:25.776 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cd26dd02305df65-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.778 DEBUG:boto:path=/vwnfns-11/obj19 2024-04-13T21:29:25.779 DEBUG:boto:auth_path=/vwnfns-11/obj19 2024-04-13T21:29:25.779 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.779 DEBUG:boto:Path: /vwnfns-11/obj19 2024-04-13T21:29:25.779 DEBUG:boto:Data: 2024-04-13T21:29:25.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.779 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.779 DEBUG:boto:Params: {} 2024-04-13T21:29:25.779 DEBUG:boto:Token: None 2024-04-13T21:29:25.779 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj19 2024-04-13T21:29:25.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Uqu6jNTGvrNsK07WyTvoAg1aUkY= 2024-04-13T21:29:25.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Uqu6jNTGvrNsK07WyTvoAg1aUkY='} 2024-04-13T21:29:25.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcecea5dd86ca826-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.783 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.783 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.783 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.783 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.783 DEBUG:boto:Data: 2024-04-13T21:29:25.784 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.784 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.784 DEBUG:boto:Params: {} 2024-04-13T21:29:25.784 DEBUG:boto:Token: None 2024-04-13T21:29:25.784 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098299bc5c6bfc0db-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.787 DEBUG:boto:path=/vwnfns-11/obj20 2024-04-13T21:29:25.787 DEBUG:boto:auth_path=/vwnfns-11/obj20 2024-04-13T21:29:25.787 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.787 DEBUG:boto:Path: /vwnfns-11/obj20 2024-04-13T21:29:25.787 DEBUG:boto:Data: 2024-04-13T21:29:25.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.788 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.788 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.788 DEBUG:boto:Params: {} 2024-04-13T21:29:25.788 DEBUG:boto:Token: None 2024-04-13T21:29:25.789 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj20 2024-04-13T21:29:25.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Tj0SC2+lQ6MrOAyp/1FooVzHRyE= 2024-04-13T21:29:25.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Tj0SC2+lQ6MrOAyp/1FooVzHRyE='} 2024-04-13T21:29:25.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f866785cbd3a32d0-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.835 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.835 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.835 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.835 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.835 DEBUG:boto:Data: 2024-04-13T21:29:25.835 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.836 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.836 DEBUG:boto:Params: {} 2024-04-13T21:29:25.836 DEBUG:boto:Token: None 2024-04-13T21:29:25.836 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b13ba222bf4355e-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.838 DEBUG:boto:path=/vwnfns-11/obj21 2024-04-13T21:29:25.838 DEBUG:boto:auth_path=/vwnfns-11/obj21 2024-04-13T21:29:25.839 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.839 DEBUG:boto:Path: /vwnfns-11/obj21 2024-04-13T21:29:25.839 DEBUG:boto:Data: 2024-04-13T21:29:25.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.839 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.839 DEBUG:boto:Params: {} 2024-04-13T21:29:25.839 DEBUG:boto:Token: None 2024-04-13T21:29:25.839 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj21 2024-04-13T21:29:25.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/Cen8nwkYwF/0t2B5nOjVxCPv90= 2024-04-13T21:29:25.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/Cen8nwkYwF/0t2B5nOjVxCPv90='} 2024-04-13T21:29:25.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000545e64f6b0f4f189-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.846 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.846 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.846 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.846 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.846 DEBUG:boto:Data: 2024-04-13T21:29:25.847 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.847 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.847 DEBUG:boto:Params: {} 2024-04-13T21:29:25.847 DEBUG:boto:Token: None 2024-04-13T21:29:25.847 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.847 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c46cf1fc8775908-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.849 DEBUG:boto:path=/vwnfns-11/obj22 2024-04-13T21:29:25.849 DEBUG:boto:auth_path=/vwnfns-11/obj22 2024-04-13T21:29:25.849 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.850 DEBUG:boto:Path: /vwnfns-11/obj22 2024-04-13T21:29:25.850 DEBUG:boto:Data: 2024-04-13T21:29:25.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.850 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.850 DEBUG:boto:Params: {} 2024-04-13T21:29:25.850 DEBUG:boto:Token: None 2024-04-13T21:29:25.850 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj22 2024-04-13T21:29:25.850 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sXULMo1Ul7z0CTGwaaq4EYjFblg= 2024-04-13T21:29:25.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sXULMo1Ul7z0CTGwaaq4EYjFblg='} 2024-04-13T21:29:25.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dfec6e36b6e4574-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.854 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.854 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.854 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.854 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.855 DEBUG:boto:Data: 2024-04-13T21:29:25.855 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.855 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.855 DEBUG:boto:Params: {} 2024-04-13T21:29:25.855 DEBUG:boto:Token: None 2024-04-13T21:29:25.855 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.855 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c2cda0a588b278e-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.859 DEBUG:boto:path=/vwnfns-11/obj23 2024-04-13T21:29:25.859 DEBUG:boto:auth_path=/vwnfns-11/obj23 2024-04-13T21:29:25.859 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.859 DEBUG:boto:Path: /vwnfns-11/obj23 2024-04-13T21:29:25.859 DEBUG:boto:Data: 2024-04-13T21:29:25.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.860 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.860 DEBUG:boto:Params: {} 2024-04-13T21:29:25.860 DEBUG:boto:Token: None 2024-04-13T21:29:25.860 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj23 2024-04-13T21:29:25.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WD2qGzQCj6tfWwSMcK7i7FPV4uY= 2024-04-13T21:29:25.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WD2qGzQCj6tfWwSMcK7i7FPV4uY='} 2024-04-13T21:29:25.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdb8bf0c612d6ee8-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.903 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.903 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.903 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.903 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.903 DEBUG:boto:Data: 2024-04-13T21:29:25.903 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.903 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.904 DEBUG:boto:Params: {} 2024-04-13T21:29:25.904 DEBUG:boto:Token: None 2024-04-13T21:29:25.904 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac511cc0d12bd0bb-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.908 DEBUG:boto:path=/vwnfns-11/obj24 2024-04-13T21:29:25.908 DEBUG:boto:auth_path=/vwnfns-11/obj24 2024-04-13T21:29:25.908 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.908 DEBUG:boto:Path: /vwnfns-11/obj24 2024-04-13T21:29:25.908 DEBUG:boto:Data: 2024-04-13T21:29:25.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.908 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.908 DEBUG:boto:Params: {} 2024-04-13T21:29:25.909 DEBUG:boto:Token: None 2024-04-13T21:29:25.909 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj24 2024-04-13T21:29:25.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BoGBjP5H76Kblei9MJygYkDdS+s= 2024-04-13T21:29:25.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BoGBjP5H76Kblei9MJygYkDdS+s='} 2024-04-13T21:29:25.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029af3207b33efab8-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.913 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.913 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.913 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.913 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.913 DEBUG:boto:Data: 2024-04-13T21:29:25.914 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.914 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.914 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.914 DEBUG:boto:Params: {} 2024-04-13T21:29:25.914 DEBUG:boto:Token: None 2024-04-13T21:29:25.914 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.914 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab806ececbd6704b-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.916 DEBUG:boto:path=/vwnfns-11/obj25 2024-04-13T21:29:25.917 DEBUG:boto:auth_path=/vwnfns-11/obj25 2024-04-13T21:29:25.917 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.917 DEBUG:boto:Path: /vwnfns-11/obj25 2024-04-13T21:29:25.917 DEBUG:boto:Data: 2024-04-13T21:29:25.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.917 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.917 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.917 DEBUG:boto:Params: {} 2024-04-13T21:29:25.917 DEBUG:boto:Token: None 2024-04-13T21:29:25.918 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj25 2024-04-13T21:29:25.918 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1T3WKuV1OE8CBT6fmD3Ptcbri7A= 2024-04-13T21:29:25.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1T3WKuV1OE8CBT6fmD3Ptcbri7A='} 2024-04-13T21:29:25.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daeb2b4e9f2762e1-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.963 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.963 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.963 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.963 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.963 DEBUG:boto:Data: 2024-04-13T21:29:25.963 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.964 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.964 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.964 DEBUG:boto:Params: {} 2024-04-13T21:29:25.964 DEBUG:boto:Token: None 2024-04-13T21:29:25.964 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f95d6c89638f8e2a-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.966 DEBUG:boto:path=/vwnfns-11/obj26 2024-04-13T21:29:25.967 DEBUG:boto:auth_path=/vwnfns-11/obj26 2024-04-13T21:29:25.967 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.967 DEBUG:boto:Path: /vwnfns-11/obj26 2024-04-13T21:29:25.967 DEBUG:boto:Data: 2024-04-13T21:29:25.967 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.967 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.967 DEBUG:boto:Params: {} 2024-04-13T21:29:25.967 DEBUG:boto:Token: None 2024-04-13T21:29:25.968 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj26 2024-04-13T21:29:25.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EvPxjcv5P0jqUrKO4f+ML2bxySQ= 2024-04-13T21:29:25.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EvPxjcv5P0jqUrKO4f+ML2bxySQ='} 2024-04-13T21:29:25.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce199e49af6ac1a3-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.972 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:25.972 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:25.972 DEBUG:boto:Method: HEAD 2024-04-13T21:29:25.972 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:25.972 DEBUG:boto:Data: 2024-04-13T21:29:25.972 DEBUG:boto:Headers: {} 2024-04-13T21:29:25.972 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.972 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.973 DEBUG:boto:Params: {} 2024-04-13T21:29:25.973 DEBUG:boto:Token: None 2024-04-13T21:29:25.973 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/ 2024-04-13T21:29:25.973 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T21:29:25.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T21:29:25.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f11e0efe94605a86-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:25.976 DEBUG:boto:path=/vwnfns-11/obj27 2024-04-13T21:29:25.976 DEBUG:boto:auth_path=/vwnfns-11/obj27 2024-04-13T21:29:25.976 DEBUG:boto:Method: PUT 2024-04-13T21:29:25.976 DEBUG:boto:Path: /vwnfns-11/obj27 2024-04-13T21:29:25.976 DEBUG:boto:Data: 2024-04-13T21:29:25.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:25.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:25.976 DEBUG:boto:Port: 8000 2024-04-13T21:29:25.976 DEBUG:boto:Params: {} 2024-04-13T21:29:25.977 DEBUG:boto:Token: None 2024-04-13T21:29:25.977 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:25 GMT /vwnfns-11/obj27 2024-04-13T21:29:25.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqcW4HCpfZSIvl4gW7rPh3wrxKo= 2024-04-13T21:29:25.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqcW4HCpfZSIvl4gW7rPh3wrxKo='} 2024-04-13T21:29:26.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc0813069c153526-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.019 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.019 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.019 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.019 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.019 DEBUG:boto:Data: 2024-04-13T21:29:26.019 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.019 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.019 DEBUG:boto:Params: {} 2024-04-13T21:29:26.020 DEBUG:boto:Token: None 2024-04-13T21:29:26.020 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044637087a7552a69-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.022 DEBUG:boto:path=/vwnfns-11/obj28 2024-04-13T21:29:26.022 DEBUG:boto:auth_path=/vwnfns-11/obj28 2024-04-13T21:29:26.022 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.022 DEBUG:boto:Path: /vwnfns-11/obj28 2024-04-13T21:29:26.023 DEBUG:boto:Data: 2024-04-13T21:29:26.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.023 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.023 DEBUG:boto:Params: {} 2024-04-13T21:29:26.023 DEBUG:boto:Token: None 2024-04-13T21:29:26.023 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj28 2024-04-13T21:29:26.023 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ll23Rmws7apGCkRJZlIVYyBNsLw= 2024-04-13T21:29:26.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ll23Rmws7apGCkRJZlIVYyBNsLw='} 2024-04-13T21:29:26.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2049c3a31498a24-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.027 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.027 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.027 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.027 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.027 DEBUG:boto:Data: 2024-04-13T21:29:26.027 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.028 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.028 DEBUG:boto:Params: {} 2024-04-13T21:29:26.028 DEBUG:boto:Token: None 2024-04-13T21:29:26.028 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096e03c88dbce00e7-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.030 DEBUG:boto:path=/vwnfns-11/obj29 2024-04-13T21:29:26.030 DEBUG:boto:auth_path=/vwnfns-11/obj29 2024-04-13T21:29:26.030 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.030 DEBUG:boto:Path: /vwnfns-11/obj29 2024-04-13T21:29:26.031 DEBUG:boto:Data: 2024-04-13T21:29:26.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.031 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.031 DEBUG:boto:Params: {} 2024-04-13T21:29:26.031 DEBUG:boto:Token: None 2024-04-13T21:29:26.031 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj29 2024-04-13T21:29:26.031 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h3WMdTswguhlrRBg5bVK9qcfxOg= 2024-04-13T21:29:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h3WMdTswguhlrRBg5bVK9qcfxOg='} 2024-04-13T21:29:26.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e15e90180915a83e-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.035 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.035 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.035 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.035 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.035 DEBUG:boto:Data: 2024-04-13T21:29:26.035 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.036 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.036 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.036 DEBUG:boto:Params: {} 2024-04-13T21:29:26.036 DEBUG:boto:Token: None 2024-04-13T21:29:26.036 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009031537febff0272-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.038 DEBUG:boto:path=/vwnfns-11/obj30 2024-04-13T21:29:26.038 DEBUG:boto:auth_path=/vwnfns-11/obj30 2024-04-13T21:29:26.038 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.038 DEBUG:boto:Path: /vwnfns-11/obj30 2024-04-13T21:29:26.038 DEBUG:boto:Data: 2024-04-13T21:29:26.039 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.039 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.039 DEBUG:boto:Params: {} 2024-04-13T21:29:26.039 DEBUG:boto:Token: None 2024-04-13T21:29:26.039 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj30 2024-04-13T21:29:26.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Oa9TOQ9zsvNcgcUe0fs4Nq43TK4= 2024-04-13T21:29:26.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Oa9TOQ9zsvNcgcUe0fs4Nq43TK4='} 2024-04-13T21:29:26.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027fa6f0ab53724d6-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.087 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.087 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.087 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.087 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.087 DEBUG:boto:Data: 2024-04-13T21:29:26.087 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.087 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.087 DEBUG:boto:Params: {} 2024-04-13T21:29:26.088 DEBUG:boto:Token: None 2024-04-13T21:29:26.088 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e8f994a50f3d008-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.090 DEBUG:boto:path=/vwnfns-11/obj31 2024-04-13T21:29:26.090 DEBUG:boto:auth_path=/vwnfns-11/obj31 2024-04-13T21:29:26.090 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.090 DEBUG:boto:Path: /vwnfns-11/obj31 2024-04-13T21:29:26.090 DEBUG:boto:Data: 2024-04-13T21:29:26.090 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.091 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.091 DEBUG:boto:Params: {} 2024-04-13T21:29:26.091 DEBUG:boto:Token: None 2024-04-13T21:29:26.091 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj31 2024-04-13T21:29:26.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DOEQV7nLEwq+wRP7cqe99wkWLOQ= 2024-04-13T21:29:26.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOEQV7nLEwq+wRP7cqe99wkWLOQ='} 2024-04-13T21:29:26.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ec0d62051514a8f-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.095 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.095 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.095 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.095 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.095 DEBUG:boto:Data: 2024-04-13T21:29:26.095 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.095 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.095 DEBUG:boto:Params: {} 2024-04-13T21:29:26.096 DEBUG:boto:Token: None 2024-04-13T21:29:26.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835e0ca899ac347f-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.098 DEBUG:boto:path=/vwnfns-11/obj32 2024-04-13T21:29:26.098 DEBUG:boto:auth_path=/vwnfns-11/obj32 2024-04-13T21:29:26.098 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.098 DEBUG:boto:Path: /vwnfns-11/obj32 2024-04-13T21:29:26.099 DEBUG:boto:Data: 2024-04-13T21:29:26.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.099 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.099 DEBUG:boto:Params: {} 2024-04-13T21:29:26.099 DEBUG:boto:Token: None 2024-04-13T21:29:26.099 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj32 2024-04-13T21:29:26.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6CotuKHQPw5DVKZHo+dr6CKFZwA= 2024-04-13T21:29:26.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6CotuKHQPw5DVKZHo+dr6CKFZwA='} 2024-04-13T21:29:26.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000707823857ce950b2-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.103 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.103 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.103 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.103 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.103 DEBUG:boto:Data: 2024-04-13T21:29:26.103 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.103 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.103 DEBUG:boto:Params: {} 2024-04-13T21:29:26.104 DEBUG:boto:Token: None 2024-04-13T21:29:26.104 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.104 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000230db2cca474fbdd-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.106 DEBUG:boto:path=/vwnfns-11/obj33 2024-04-13T21:29:26.106 DEBUG:boto:auth_path=/vwnfns-11/obj33 2024-04-13T21:29:26.106 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.106 DEBUG:boto:Path: /vwnfns-11/obj33 2024-04-13T21:29:26.106 DEBUG:boto:Data: 2024-04-13T21:29:26.106 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.107 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.107 DEBUG:boto:Params: {} 2024-04-13T21:29:26.107 DEBUG:boto:Token: None 2024-04-13T21:29:26.107 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj33 2024-04-13T21:29:26.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BhiAX6OvgxF76Px0EWLiDe97Z+E= 2024-04-13T21:29:26.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BhiAX6OvgxF76Px0EWLiDe97Z+E='} 2024-04-13T21:29:26.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f78877b562b84b01-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.155 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.155 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.155 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.155 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.155 DEBUG:boto:Data: 2024-04-13T21:29:26.155 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.155 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.155 DEBUG:boto:Params: {} 2024-04-13T21:29:26.156 DEBUG:boto:Token: None 2024-04-13T21:29:26.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3247070ce15cd24-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.158 DEBUG:boto:path=/vwnfns-11/obj34 2024-04-13T21:29:26.158 DEBUG:boto:auth_path=/vwnfns-11/obj34 2024-04-13T21:29:26.158 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.158 DEBUG:boto:Path: /vwnfns-11/obj34 2024-04-13T21:29:26.159 DEBUG:boto:Data: 2024-04-13T21:29:26.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.159 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.159 DEBUG:boto:Params: {} 2024-04-13T21:29:26.159 DEBUG:boto:Token: None 2024-04-13T21:29:26.159 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj34 2024-04-13T21:29:26.159 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CuxxhuO8c95wZgg2s1ItzI0cskQ= 2024-04-13T21:29:26.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CuxxhuO8c95wZgg2s1ItzI0cskQ='} 2024-04-13T21:29:26.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd575bc40f3935c5-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.163 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.163 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.163 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.163 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.163 DEBUG:boto:Data: 2024-04-13T21:29:26.163 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.163 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.163 DEBUG:boto:Params: {} 2024-04-13T21:29:26.163 DEBUG:boto:Token: None 2024-04-13T21:29:26.164 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1f9e27cb124e76a-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.166 DEBUG:boto:path=/vwnfns-11/obj35 2024-04-13T21:29:26.166 DEBUG:boto:auth_path=/vwnfns-11/obj35 2024-04-13T21:29:26.166 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.166 DEBUG:boto:Path: /vwnfns-11/obj35 2024-04-13T21:29:26.166 DEBUG:boto:Data: 2024-04-13T21:29:26.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.166 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.166 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.167 DEBUG:boto:Params: {} 2024-04-13T21:29:26.167 DEBUG:boto:Token: None 2024-04-13T21:29:26.167 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj35 2024-04-13T21:29:26.167 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QhQvvphQhyyl/dhRWFItSOj6RWs= 2024-04-13T21:29:26.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QhQvvphQhyyl/dhRWFItSOj6RWs='} 2024-04-13T21:29:26.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4fb2c89786896c9-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.170 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.170 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.170 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.171 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.171 DEBUG:boto:Data: 2024-04-13T21:29:26.171 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.171 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.171 DEBUG:boto:Params: {} 2024-04-13T21:29:26.171 DEBUG:boto:Token: None 2024-04-13T21:29:26.171 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3de327dd2142a2c-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.173 DEBUG:boto:path=/vwnfns-11/obj36 2024-04-13T21:29:26.173 DEBUG:boto:auth_path=/vwnfns-11/obj36 2024-04-13T21:29:26.173 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.173 DEBUG:boto:Path: /vwnfns-11/obj36 2024-04-13T21:29:26.174 DEBUG:boto:Data: 2024-04-13T21:29:26.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.174 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.174 DEBUG:boto:Params: {} 2024-04-13T21:29:26.174 DEBUG:boto:Token: None 2024-04-13T21:29:26.174 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj36 2024-04-13T21:29:26.174 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CiSI57Fgw3oSJ0Otl83gqbbF1Uc= 2024-04-13T21:29:26.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CiSI57Fgw3oSJ0Otl83gqbbF1Uc='} 2024-04-13T21:29:26.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000351493995f7af09d-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.178 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.178 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.178 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.178 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.178 DEBUG:boto:Data: 2024-04-13T21:29:26.178 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.178 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.179 DEBUG:boto:Params: {} 2024-04-13T21:29:26.179 DEBUG:boto:Token: None 2024-04-13T21:29:26.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a63c94b90da0602e-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.181 DEBUG:boto:path=/vwnfns-11/obj37 2024-04-13T21:29:26.181 DEBUG:boto:auth_path=/vwnfns-11/obj37 2024-04-13T21:29:26.181 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.181 DEBUG:boto:Path: /vwnfns-11/obj37 2024-04-13T21:29:26.181 DEBUG:boto:Data: 2024-04-13T21:29:26.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.182 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.182 DEBUG:boto:Params: {} 2024-04-13T21:29:26.182 DEBUG:boto:Token: None 2024-04-13T21:29:26.182 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj37 2024-04-13T21:29:26.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fEbcEadGWr7kvEbkLfjZXQXJZeQ= 2024-04-13T21:29:26.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fEbcEadGWr7kvEbkLfjZXQXJZeQ='} 2024-04-13T21:29:26.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071490c2d205a4873-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.186 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.186 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.186 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.187 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.187 DEBUG:boto:Data: 2024-04-13T21:29:26.187 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.187 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.187 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.187 DEBUG:boto:Params: {} 2024-04-13T21:29:26.187 DEBUG:boto:Token: None 2024-04-13T21:29:26.187 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b130922d92da4c97-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.190 DEBUG:boto:path=/vwnfns-11/obj38 2024-04-13T21:29:26.190 DEBUG:boto:auth_path=/vwnfns-11/obj38 2024-04-13T21:29:26.190 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.190 DEBUG:boto:Path: /vwnfns-11/obj38 2024-04-13T21:29:26.190 DEBUG:boto:Data: 2024-04-13T21:29:26.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.190 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.190 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.190 DEBUG:boto:Params: {} 2024-04-13T21:29:26.191 DEBUG:boto:Token: None 2024-04-13T21:29:26.191 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj38 2024-04-13T21:29:26.191 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sksHMe7eJHo/BgiGbAAIy+rdloI= 2024-04-13T21:29:26.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sksHMe7eJHo/BgiGbAAIy+rdloI='} 2024-04-13T21:29:26.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c6bd62845c3b48b-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.194 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.195 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.195 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.195 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.195 DEBUG:boto:Data: 2024-04-13T21:29:26.195 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.195 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.195 DEBUG:boto:Params: {} 2024-04-13T21:29:26.195 DEBUG:boto:Token: None 2024-04-13T21:29:26.195 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000101433ec7c695a01-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.198 DEBUG:boto:path=/vwnfns-11/obj39 2024-04-13T21:29:26.198 DEBUG:boto:auth_path=/vwnfns-11/obj39 2024-04-13T21:29:26.198 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.198 DEBUG:boto:Path: /vwnfns-11/obj39 2024-04-13T21:29:26.198 DEBUG:boto:Data: 2024-04-13T21:29:26.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.198 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.198 DEBUG:boto:Params: {} 2024-04-13T21:29:26.198 DEBUG:boto:Token: None 2024-04-13T21:29:26.199 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj39 2024-04-13T21:29:26.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/dWa3RndVA1lIZy0EGl8qvV9giY= 2024-04-13T21:29:26.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/dWa3RndVA1lIZy0EGl8qvV9giY='} 2024-04-13T21:29:26.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081ba79296be95175-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.247 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.247 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.247 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.247 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.247 DEBUG:boto:Data: 2024-04-13T21:29:26.247 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.248 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.248 DEBUG:boto:Params: {} 2024-04-13T21:29:26.248 DEBUG:boto:Token: None 2024-04-13T21:29:26.248 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.249 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006053e4e0c660b571-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.251 DEBUG:boto:path=/vwnfns-11/obj40 2024-04-13T21:29:26.251 DEBUG:boto:auth_path=/vwnfns-11/obj40 2024-04-13T21:29:26.251 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.251 DEBUG:boto:Path: /vwnfns-11/obj40 2024-04-13T21:29:26.251 DEBUG:boto:Data: 2024-04-13T21:29:26.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.252 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.252 DEBUG:boto:Params: {} 2024-04-13T21:29:26.252 DEBUG:boto:Token: None 2024-04-13T21:29:26.252 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj40 2024-04-13T21:29:26.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BmIQ0XLlfxbR9DXREzbfcAWGW6U= 2024-04-13T21:29:26.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BmIQ0XLlfxbR9DXREzbfcAWGW6U='} 2024-04-13T21:29:26.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be3e7cc8f7edbd6e-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.256 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.256 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.256 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.256 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.256 DEBUG:boto:Data: 2024-04-13T21:29:26.257 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.257 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.257 DEBUG:boto:Params: {} 2024-04-13T21:29:26.257 DEBUG:boto:Token: None 2024-04-13T21:29:26.257 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000807bc2af786822ea-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.259 DEBUG:boto:path=/vwnfns-11/obj41 2024-04-13T21:29:26.259 DEBUG:boto:auth_path=/vwnfns-11/obj41 2024-04-13T21:29:26.260 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.260 DEBUG:boto:Path: /vwnfns-11/obj41 2024-04-13T21:29:26.260 DEBUG:boto:Data: 2024-04-13T21:29:26.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.260 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.260 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.260 DEBUG:boto:Params: {} 2024-04-13T21:29:26.260 DEBUG:boto:Token: None 2024-04-13T21:29:26.260 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj41 2024-04-13T21:29:26.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lux2LmWiMK275PFnY7XiKdeUnDQ= 2024-04-13T21:29:26.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lux2LmWiMK275PFnY7XiKdeUnDQ='} 2024-04-13T21:29:26.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae9b41f6eba536b0-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.307 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.307 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.307 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.307 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.307 DEBUG:boto:Data: 2024-04-13T21:29:26.307 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.307 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.308 DEBUG:boto:Params: {} 2024-04-13T21:29:26.308 DEBUG:boto:Token: None 2024-04-13T21:29:26.308 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae0c7917b194c702-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.310 DEBUG:boto:path=/vwnfns-11/obj42 2024-04-13T21:29:26.310 DEBUG:boto:auth_path=/vwnfns-11/obj42 2024-04-13T21:29:26.310 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.311 DEBUG:boto:Path: /vwnfns-11/obj42 2024-04-13T21:29:26.311 DEBUG:boto:Data: 2024-04-13T21:29:26.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.311 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.311 DEBUG:boto:Params: {} 2024-04-13T21:29:26.311 DEBUG:boto:Token: None 2024-04-13T21:29:26.311 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj42 2024-04-13T21:29:26.311 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2eKIo72W9p3u1BaZgMhDynPyU8M= 2024-04-13T21:29:26.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2eKIo72W9p3u1BaZgMhDynPyU8M='} 2024-04-13T21:29:26.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f37365bdec6f6ec6-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.315 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.315 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.315 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.315 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.316 DEBUG:boto:Data: 2024-04-13T21:29:26.316 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.316 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.316 DEBUG:boto:Params: {} 2024-04-13T21:29:26.316 DEBUG:boto:Token: None 2024-04-13T21:29:26.316 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6ebeaa69948d70a-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.318 DEBUG:boto:path=/vwnfns-11/obj43 2024-04-13T21:29:26.319 DEBUG:boto:auth_path=/vwnfns-11/obj43 2024-04-13T21:29:26.319 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.319 DEBUG:boto:Path: /vwnfns-11/obj43 2024-04-13T21:29:26.319 DEBUG:boto:Data: 2024-04-13T21:29:26.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.319 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.319 DEBUG:boto:Params: {} 2024-04-13T21:29:26.319 DEBUG:boto:Token: None 2024-04-13T21:29:26.319 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj43 2024-04-13T21:29:26.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N29q8g64GaEl+Ph7VT9rM3xxUDE= 2024-04-13T21:29:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N29q8g64GaEl+Ph7VT9rM3xxUDE='} 2024-04-13T21:29:26.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e0e7fd67b9c0792-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.323 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.323 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.324 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.324 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.324 DEBUG:boto:Data: 2024-04-13T21:29:26.324 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.324 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.324 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.324 DEBUG:boto:Params: {} 2024-04-13T21:29:26.324 DEBUG:boto:Token: None 2024-04-13T21:29:26.324 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f503f25e582e3846-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.326 DEBUG:boto:path=/vwnfns-11/obj44 2024-04-13T21:29:26.326 DEBUG:boto:auth_path=/vwnfns-11/obj44 2024-04-13T21:29:26.327 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.327 DEBUG:boto:Path: /vwnfns-11/obj44 2024-04-13T21:29:26.327 DEBUG:boto:Data: 2024-04-13T21:29:26.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.327 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.327 DEBUG:boto:Params: {} 2024-04-13T21:29:26.327 DEBUG:boto:Token: None 2024-04-13T21:29:26.327 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj44 2024-04-13T21:29:26.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k1UlD9yNn9t6+PoeXT2w95JrFM8= 2024-04-13T21:29:26.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k1UlD9yNn9t6+PoeXT2w95JrFM8='} 2024-04-13T21:29:26.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6d8ff19c54788cb-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.375 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.375 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.375 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.375 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.375 DEBUG:boto:Data: 2024-04-13T21:29:26.375 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.376 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.376 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.376 DEBUG:boto:Params: {} 2024-04-13T21:29:26.376 DEBUG:boto:Token: None 2024-04-13T21:29:26.376 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a4aeab5efc3573a-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.379 DEBUG:boto:path=/vwnfns-11/obj45 2024-04-13T21:29:26.379 DEBUG:boto:auth_path=/vwnfns-11/obj45 2024-04-13T21:29:26.379 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.379 DEBUG:boto:Path: /vwnfns-11/obj45 2024-04-13T21:29:26.379 DEBUG:boto:Data: 2024-04-13T21:29:26.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.380 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.380 DEBUG:boto:Params: {} 2024-04-13T21:29:26.380 DEBUG:boto:Token: None 2024-04-13T21:29:26.380 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj45 2024-04-13T21:29:26.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScYoLAqEIeRIb/zSnamg6YpLJEc= 2024-04-13T21:29:26.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScYoLAqEIeRIb/zSnamg6YpLJEc='} 2024-04-13T21:29:26.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a127707ec3f9f28e-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.384 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.384 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.384 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.384 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.384 DEBUG:boto:Data: 2024-04-13T21:29:26.384 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.384 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.385 DEBUG:boto:Params: {} 2024-04-13T21:29:26.385 DEBUG:boto:Token: None 2024-04-13T21:29:26.385 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.385 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003448de2966c02d6c-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.387 DEBUG:boto:path=/vwnfns-11/obj46 2024-04-13T21:29:26.387 DEBUG:boto:auth_path=/vwnfns-11/obj46 2024-04-13T21:29:26.388 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.388 DEBUG:boto:Path: /vwnfns-11/obj46 2024-04-13T21:29:26.388 DEBUG:boto:Data: 2024-04-13T21:29:26.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.388 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.388 DEBUG:boto:Params: {} 2024-04-13T21:29:26.388 DEBUG:boto:Token: None 2024-04-13T21:29:26.388 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj46 2024-04-13T21:29:26.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnazETLzd+4q8L54+1dN7C6V8vc= 2024-04-13T21:29:26.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnazETLzd+4q8L54+1dN7C6V8vc='} 2024-04-13T21:29:26.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063e7c6587b804dcd-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.435 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.435 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.435 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.435 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.435 DEBUG:boto:Data: 2024-04-13T21:29:26.435 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.435 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.436 DEBUG:boto:Params: {} 2024-04-13T21:29:26.436 DEBUG:boto:Token: None 2024-04-13T21:29:26.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069da71a5614e3f29-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.438 DEBUG:boto:path=/vwnfns-11/obj47 2024-04-13T21:29:26.439 DEBUG:boto:auth_path=/vwnfns-11/obj47 2024-04-13T21:29:26.439 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.439 DEBUG:boto:Path: /vwnfns-11/obj47 2024-04-13T21:29:26.439 DEBUG:boto:Data: 2024-04-13T21:29:26.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.439 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.439 DEBUG:boto:Params: {} 2024-04-13T21:29:26.439 DEBUG:boto:Token: None 2024-04-13T21:29:26.440 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj47 2024-04-13T21:29:26.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H2/+dSTbTmUGtrElste4/U7Coz0= 2024-04-13T21:29:26.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H2/+dSTbTmUGtrElste4/U7Coz0='} 2024-04-13T21:29:26.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a704dc20440eb9c1-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.443 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.443 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.443 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.444 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.444 DEBUG:boto:Data: 2024-04-13T21:29:26.444 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.444 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.444 DEBUG:boto:Params: {} 2024-04-13T21:29:26.444 DEBUG:boto:Token: None 2024-04-13T21:29:26.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ac0d34a67dd96b8-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.446 DEBUG:boto:path=/vwnfns-11/obj48 2024-04-13T21:29:26.446 DEBUG:boto:auth_path=/vwnfns-11/obj48 2024-04-13T21:29:26.447 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.447 DEBUG:boto:Path: /vwnfns-11/obj48 2024-04-13T21:29:26.447 DEBUG:boto:Data: 2024-04-13T21:29:26.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.447 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.447 DEBUG:boto:Params: {} 2024-04-13T21:29:26.447 DEBUG:boto:Token: None 2024-04-13T21:29:26.448 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj48 2024-04-13T21:29:26.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0B3Lianij9u31zICLgRB8hLzvvo= 2024-04-13T21:29:26.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0B3Lianij9u31zICLgRB8hLzvvo='} 2024-04-13T21:29:26.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002717e6a487ef8472-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.495 DEBUG:boto:path=/vwnfns-11/ 2024-04-13T21:29:26.495 DEBUG:boto:auth_path=/vwnfns-11/ 2024-04-13T21:29:26.495 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.495 DEBUG:boto:Path: /vwnfns-11/ 2024-04-13T21:29:26.495 DEBUG:boto:Data: 2024-04-13T21:29:26.495 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.495 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.496 DEBUG:boto:Params: {} 2024-04-13T21:29:26.496 DEBUG:boto:Token: None 2024-04-13T21:29:26.496 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/ 2024-04-13T21:29:26.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T21:29:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T21:29:26.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b07ca1eacae5547-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.498 DEBUG:boto:path=/vwnfns-11/obj49 2024-04-13T21:29:26.498 DEBUG:boto:auth_path=/vwnfns-11/obj49 2024-04-13T21:29:26.499 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.499 DEBUG:boto:Path: /vwnfns-11/obj49 2024-04-13T21:29:26.499 DEBUG:boto:Data: 2024-04-13T21:29:26.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:29:26.499 DEBUG:boto:Port: 8000 2024-04-13T21:29:26.499 DEBUG:boto:Params: {} 2024-04-13T21:29:26.499 DEBUG:boto:Token: None 2024-04-13T21:29:26.499 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-11/obj49 2024-04-13T21:29:26.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1RXIULWZKXkAuzyO20n5dC9c6JY= 2024-04-13T21:29:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1RXIULWZKXkAuzyO20n5dC9c6JY='} 2024-04-13T21:29:26.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070548a503b38a696-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.503 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.503 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.503 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.503 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.504 DEBUG:boto:Data: 2024-04-13T21:29:26.504 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.504 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.504 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.504 DEBUG:boto:Params: {} 2024-04-13T21:29:26.504 DEBUG:boto:Token: None 2024-04-13T21:29:26.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080bea28c444af011-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.507 DEBUG:boto:path=/vwnfns-12/obj1 2024-04-13T21:29:26.507 DEBUG:boto:auth_path=/vwnfns-12/obj1 2024-04-13T21:29:26.507 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.507 DEBUG:boto:Path: /vwnfns-12/obj1 2024-04-13T21:29:26.507 DEBUG:boto:Data: 2024-04-13T21:29:26.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.508 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.508 DEBUG:boto:Params: {} 2024-04-13T21:29:26.509 DEBUG:boto:Token: None 2024-04-13T21:29:26.509 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj1 2024-04-13T21:29:26.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hHKkMXWnJ0FmqyeWJeWfIWgS97k= 2024-04-13T21:29:26.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHKkMXWnJ0FmqyeWJeWfIWgS97k='} 2024-04-13T21:29:26.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a80455b74920256d-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.515 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.515 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.515 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.515 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.516 DEBUG:boto:Data: 2024-04-13T21:29:26.516 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.516 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.516 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.516 DEBUG:boto:Params: {} 2024-04-13T21:29:26.516 DEBUG:boto:Token: None 2024-04-13T21:29:26.516 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d86df8ba5614c3e6-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.518 DEBUG:boto:path=/vwnfns-12/obj2 2024-04-13T21:29:26.519 DEBUG:boto:auth_path=/vwnfns-12/obj2 2024-04-13T21:29:26.519 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.519 DEBUG:boto:Path: /vwnfns-12/obj2 2024-04-13T21:29:26.519 DEBUG:boto:Data: 2024-04-13T21:29:26.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.519 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.519 DEBUG:boto:Params: {} 2024-04-13T21:29:26.520 DEBUG:boto:Token: None 2024-04-13T21:29:26.520 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj2 2024-04-13T21:29:26.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sNltT/1NcP+F5G8DBX0rqNs9SYg= 2024-04-13T21:29:26.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNltT/1NcP+F5G8DBX0rqNs9SYg='} 2024-04-13T21:29:26.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000385e0ad86abffea-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.563 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.563 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.563 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.563 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.563 DEBUG:boto:Data: 2024-04-13T21:29:26.563 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.564 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.564 DEBUG:boto:Params: {} 2024-04-13T21:29:26.564 DEBUG:boto:Token: None 2024-04-13T21:29:26.564 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb3fe7f7343035d6-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.567 DEBUG:boto:path=/vwnfns-12/obj3 2024-04-13T21:29:26.567 DEBUG:boto:auth_path=/vwnfns-12/obj3 2024-04-13T21:29:26.567 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.567 DEBUG:boto:Path: /vwnfns-12/obj3 2024-04-13T21:29:26.567 DEBUG:boto:Data: 2024-04-13T21:29:26.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.567 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.568 DEBUG:boto:Params: {} 2024-04-13T21:29:26.568 DEBUG:boto:Token: None 2024-04-13T21:29:26.568 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj3 2024-04-13T21:29:26.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wuBvlJlh+TaxYxvtngl2qmpdIl8= 2024-04-13T21:29:26.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wuBvlJlh+TaxYxvtngl2qmpdIl8='} 2024-04-13T21:29:26.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce2b3e01393dd340-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.573 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.573 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.573 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.573 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.573 DEBUG:boto:Data: 2024-04-13T21:29:26.573 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.573 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.573 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.573 DEBUG:boto:Params: {} 2024-04-13T21:29:26.574 DEBUG:boto:Token: None 2024-04-13T21:29:26.574 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.574 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a48ef6e8b9c76b4-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.576 DEBUG:boto:path=/vwnfns-12/obj4 2024-04-13T21:29:26.576 DEBUG:boto:auth_path=/vwnfns-12/obj4 2024-04-13T21:29:26.576 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.577 DEBUG:boto:Path: /vwnfns-12/obj4 2024-04-13T21:29:26.577 DEBUG:boto:Data: 2024-04-13T21:29:26.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.577 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.577 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.577 DEBUG:boto:Params: {} 2024-04-13T21:29:26.577 DEBUG:boto:Token: None 2024-04-13T21:29:26.577 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj4 2024-04-13T21:29:26.577 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCAtJUWpOLzg+DldD3pmX9OrrLM= 2024-04-13T21:29:26.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCAtJUWpOLzg+DldD3pmX9OrrLM='} 2024-04-13T21:29:26.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f2a8f43c99d9746-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.623 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.623 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.623 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.623 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.623 DEBUG:boto:Data: 2024-04-13T21:29:26.623 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.624 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.624 DEBUG:boto:Params: {} 2024-04-13T21:29:26.624 DEBUG:boto:Token: None 2024-04-13T21:29:26.624 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.624 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a407801ac0bb4c99-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.626 DEBUG:boto:path=/vwnfns-12/obj5 2024-04-13T21:29:26.626 DEBUG:boto:auth_path=/vwnfns-12/obj5 2024-04-13T21:29:26.627 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.627 DEBUG:boto:Path: /vwnfns-12/obj5 2024-04-13T21:29:26.627 DEBUG:boto:Data: 2024-04-13T21:29:26.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.627 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.627 DEBUG:boto:Params: {} 2024-04-13T21:29:26.627 DEBUG:boto:Token: None 2024-04-13T21:29:26.627 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj5 2024-04-13T21:29:26.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LcUgZ3jiq2HhkejFAe56phXxA2M= 2024-04-13T21:29:26.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LcUgZ3jiq2HhkejFAe56phXxA2M='} 2024-04-13T21:29:26.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017ef1690c3a9131d-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.632 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.632 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.632 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.632 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.632 DEBUG:boto:Data: 2024-04-13T21:29:26.632 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.633 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.633 DEBUG:boto:Params: {} 2024-04-13T21:29:26.633 DEBUG:boto:Token: None 2024-04-13T21:29:26.633 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c1307649a49931d-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.635 DEBUG:boto:path=/vwnfns-12/obj6 2024-04-13T21:29:26.635 DEBUG:boto:auth_path=/vwnfns-12/obj6 2024-04-13T21:29:26.635 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.636 DEBUG:boto:Path: /vwnfns-12/obj6 2024-04-13T21:29:26.636 DEBUG:boto:Data: 2024-04-13T21:29:26.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.636 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.636 DEBUG:boto:Params: {} 2024-04-13T21:29:26.636 DEBUG:boto:Token: None 2024-04-13T21:29:26.636 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj6 2024-04-13T21:29:26.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t0qVjcjptkynFkpvqF3X2I6V5Ck= 2024-04-13T21:29:26.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t0qVjcjptkynFkpvqF3X2I6V5Ck='} 2024-04-13T21:29:26.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a47e10d80f121c5a-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.679 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.679 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.680 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.680 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.680 DEBUG:boto:Data: 2024-04-13T21:29:26.680 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.680 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.680 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.680 DEBUG:boto:Params: {} 2024-04-13T21:29:26.681 DEBUG:boto:Token: None 2024-04-13T21:29:26.681 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.681 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d64b26c068216e-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.683 DEBUG:boto:path=/vwnfns-12/obj7 2024-04-13T21:29:26.683 DEBUG:boto:auth_path=/vwnfns-12/obj7 2024-04-13T21:29:26.683 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.683 DEBUG:boto:Path: /vwnfns-12/obj7 2024-04-13T21:29:26.683 DEBUG:boto:Data: 2024-04-13T21:29:26.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.684 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.684 DEBUG:boto:Params: {} 2024-04-13T21:29:26.684 DEBUG:boto:Token: None 2024-04-13T21:29:26.684 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj7 2024-04-13T21:29:26.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:So+xo8TnLOyuWMhBb763X7vOTFQ= 2024-04-13T21:29:26.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:So+xo8TnLOyuWMhBb763X7vOTFQ='} 2024-04-13T21:29:26.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075f9e86efc2fbe57-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.688 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.688 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.688 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.688 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.688 DEBUG:boto:Data: 2024-04-13T21:29:26.688 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.689 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.689 DEBUG:boto:Params: {} 2024-04-13T21:29:26.689 DEBUG:boto:Token: None 2024-04-13T21:29:26.689 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009248844ae4ab0045-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.691 DEBUG:boto:path=/vwnfns-12/obj8 2024-04-13T21:29:26.691 DEBUG:boto:auth_path=/vwnfns-12/obj8 2024-04-13T21:29:26.692 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.692 DEBUG:boto:Path: /vwnfns-12/obj8 2024-04-13T21:29:26.692 DEBUG:boto:Data: 2024-04-13T21:29:26.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.692 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.692 DEBUG:boto:Params: {} 2024-04-13T21:29:26.692 DEBUG:boto:Token: None 2024-04-13T21:29:26.692 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj8 2024-04-13T21:29:26.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OH+iJzPD0a6RlKrktl8WGbJKQ9I= 2024-04-13T21:29:26.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OH+iJzPD0a6RlKrktl8WGbJKQ9I='} 2024-04-13T21:29:26.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a9c200395836874-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.735 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.735 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.735 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.735 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.735 DEBUG:boto:Data: 2024-04-13T21:29:26.735 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.736 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.736 DEBUG:boto:Params: {} 2024-04-13T21:29:26.736 DEBUG:boto:Token: None 2024-04-13T21:29:26.736 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070371e9fa49c33f3-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.739 DEBUG:boto:path=/vwnfns-12/obj9 2024-04-13T21:29:26.739 DEBUG:boto:auth_path=/vwnfns-12/obj9 2024-04-13T21:29:26.739 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.739 DEBUG:boto:Path: /vwnfns-12/obj9 2024-04-13T21:29:26.739 DEBUG:boto:Data: 2024-04-13T21:29:26.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.740 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.740 DEBUG:boto:Params: {} 2024-04-13T21:29:26.740 DEBUG:boto:Token: None 2024-04-13T21:29:26.740 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj9 2024-04-13T21:29:26.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:taQ0jt7G73zBqLGr6ZDLqBvc0Ps= 2024-04-13T21:29:26.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:taQ0jt7G73zBqLGr6ZDLqBvc0Ps='} 2024-04-13T21:29:26.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000363dca96cc888f95-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.748 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.748 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.748 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.748 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.748 DEBUG:boto:Data: 2024-04-13T21:29:26.748 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.748 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.748 DEBUG:boto:Params: {} 2024-04-13T21:29:26.749 DEBUG:boto:Token: None 2024-04-13T21:29:26.749 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012e573bae08c7a16-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.752 DEBUG:boto:path=/vwnfns-12/obj10 2024-04-13T21:29:26.752 DEBUG:boto:auth_path=/vwnfns-12/obj10 2024-04-13T21:29:26.752 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.752 DEBUG:boto:Path: /vwnfns-12/obj10 2024-04-13T21:29:26.752 DEBUG:boto:Data: 2024-04-13T21:29:26.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.753 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.753 DEBUG:boto:Params: {} 2024-04-13T21:29:26.753 DEBUG:boto:Token: None 2024-04-13T21:29:26.753 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj10 2024-04-13T21:29:26.753 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3cMHQuQRF/kYGTU7cuS2+CzaSJI= 2024-04-13T21:29:26.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3cMHQuQRF/kYGTU7cuS2+CzaSJI='} 2024-04-13T21:29:26.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006514fbc574e88784-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.799 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.799 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.799 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.799 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.799 DEBUG:boto:Data: 2024-04-13T21:29:26.799 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.799 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.799 DEBUG:boto:Params: {} 2024-04-13T21:29:26.800 DEBUG:boto:Token: None 2024-04-13T21:29:26.800 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000312d263ad19df3cb-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.803 DEBUG:boto:path=/vwnfns-12/obj11 2024-04-13T21:29:26.804 DEBUG:boto:auth_path=/vwnfns-12/obj11 2024-04-13T21:29:26.804 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.804 DEBUG:boto:Path: /vwnfns-12/obj11 2024-04-13T21:29:26.804 DEBUG:boto:Data: 2024-04-13T21:29:26.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.804 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.804 DEBUG:boto:Params: {} 2024-04-13T21:29:26.804 DEBUG:boto:Token: None 2024-04-13T21:29:26.804 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj11 2024-04-13T21:29:26.805 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W1tpjg+XjmIRa95Rkz6LOAqo8gY= 2024-04-13T21:29:26.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W1tpjg+XjmIRa95Rkz6LOAqo8gY='} 2024-04-13T21:29:26.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3f04ef39be0c65b-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.810 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.810 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.810 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.810 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.810 DEBUG:boto:Data: 2024-04-13T21:29:26.810 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.810 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.810 DEBUG:boto:Params: {} 2024-04-13T21:29:26.810 DEBUG:boto:Token: None 2024-04-13T21:29:26.811 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dfc4355c161dfdc-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.813 DEBUG:boto:path=/vwnfns-12/obj12 2024-04-13T21:29:26.814 DEBUG:boto:auth_path=/vwnfns-12/obj12 2024-04-13T21:29:26.814 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.814 DEBUG:boto:Path: /vwnfns-12/obj12 2024-04-13T21:29:26.814 DEBUG:boto:Data: 2024-04-13T21:29:26.814 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.814 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.814 DEBUG:boto:Params: {} 2024-04-13T21:29:26.814 DEBUG:boto:Token: None 2024-04-13T21:29:26.815 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj12 2024-04-13T21:29:26.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CvJUUaW3eFpEvZWcgnaGvow3snc= 2024-04-13T21:29:26.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CvJUUaW3eFpEvZWcgnaGvow3snc='} 2024-04-13T21:29:26.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5c1ad0d858ca59d-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.863 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.863 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.863 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.863 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.863 DEBUG:boto:Data: 2024-04-13T21:29:26.863 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.864 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.864 DEBUG:boto:Params: {} 2024-04-13T21:29:26.864 DEBUG:boto:Token: None 2024-04-13T21:29:26.864 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000667b061613aeb5b8-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.866 DEBUG:boto:path=/vwnfns-12/obj13 2024-04-13T21:29:26.866 DEBUG:boto:auth_path=/vwnfns-12/obj13 2024-04-13T21:29:26.867 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.867 DEBUG:boto:Path: /vwnfns-12/obj13 2024-04-13T21:29:26.867 DEBUG:boto:Data: 2024-04-13T21:29:26.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.867 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.867 DEBUG:boto:Params: {} 2024-04-13T21:29:26.867 DEBUG:boto:Token: None 2024-04-13T21:29:26.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj13 2024-04-13T21:29:26.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZspWn1KHTWlKqTpOjHrf8R7rmcs= 2024-04-13T21:29:26.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZspWn1KHTWlKqTpOjHrf8R7rmcs='} 2024-04-13T21:29:26.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d227a2bb07be231b-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.871 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.871 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.871 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.871 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.872 DEBUG:boto:Data: 2024-04-13T21:29:26.872 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.872 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.872 DEBUG:boto:Params: {} 2024-04-13T21:29:26.872 DEBUG:boto:Token: None 2024-04-13T21:29:26.873 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.873 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e35de774c749ae-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.876 DEBUG:boto:path=/vwnfns-12/obj14 2024-04-13T21:29:26.876 DEBUG:boto:auth_path=/vwnfns-12/obj14 2024-04-13T21:29:26.876 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.876 DEBUG:boto:Path: /vwnfns-12/obj14 2024-04-13T21:29:26.876 DEBUG:boto:Data: 2024-04-13T21:29:26.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.876 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.876 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.876 DEBUG:boto:Params: {} 2024-04-13T21:29:26.876 DEBUG:boto:Token: None 2024-04-13T21:29:26.877 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj14 2024-04-13T21:29:26.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UUWU5RwoPB+5rn9+blsJS4Y4PWs= 2024-04-13T21:29:26.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UUWU5RwoPB+5rn9+blsJS4Y4PWs='} 2024-04-13T21:29:26.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edf66b3262199a52-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.923 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.923 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.923 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.923 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.923 DEBUG:boto:Data: 2024-04-13T21:29:26.923 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.924 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.924 DEBUG:boto:Params: {} 2024-04-13T21:29:26.924 DEBUG:boto:Token: None 2024-04-13T21:29:26.924 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0e4812a9028d86b-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.927 DEBUG:boto:path=/vwnfns-12/obj15 2024-04-13T21:29:26.927 DEBUG:boto:auth_path=/vwnfns-12/obj15 2024-04-13T21:29:26.927 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.927 DEBUG:boto:Path: /vwnfns-12/obj15 2024-04-13T21:29:26.927 DEBUG:boto:Data: 2024-04-13T21:29:26.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.927 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.927 DEBUG:boto:Params: {} 2024-04-13T21:29:26.928 DEBUG:boto:Token: None 2024-04-13T21:29:26.928 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj15 2024-04-13T21:29:26.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m/l+SiRr9QKzUei80plqtr2EMNg= 2024-04-13T21:29:26.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m/l+SiRr9QKzUei80plqtr2EMNg='} 2024-04-13T21:29:26.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000e3e8c161fc4d59-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.933 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.933 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.934 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.934 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.934 DEBUG:boto:Data: 2024-04-13T21:29:26.934 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.934 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.934 DEBUG:boto:Params: {} 2024-04-13T21:29:26.934 DEBUG:boto:Token: None 2024-04-13T21:29:26.934 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d905ef0ac528db4-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.937 DEBUG:boto:path=/vwnfns-12/obj16 2024-04-13T21:29:26.937 DEBUG:boto:auth_path=/vwnfns-12/obj16 2024-04-13T21:29:26.937 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.937 DEBUG:boto:Path: /vwnfns-12/obj16 2024-04-13T21:29:26.937 DEBUG:boto:Data: 2024-04-13T21:29:26.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.937 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.937 DEBUG:boto:Params: {} 2024-04-13T21:29:26.938 DEBUG:boto:Token: None 2024-04-13T21:29:26.938 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj16 2024-04-13T21:29:26.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e6lGATnbgaO2jXkx1r0OBDqajRg= 2024-04-13T21:29:26.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e6lGATnbgaO2jXkx1r0OBDqajRg='} 2024-04-13T21:29:26.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000262b5167332de738-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.983 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.983 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.983 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.983 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.983 DEBUG:boto:Data: 2024-04-13T21:29:26.983 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.983 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.984 DEBUG:boto:Params: {} 2024-04-13T21:29:26.984 DEBUG:boto:Token: None 2024-04-13T21:29:26.984 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000483ade3080b85c34-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.987 DEBUG:boto:path=/vwnfns-12/obj17 2024-04-13T21:29:26.987 DEBUG:boto:auth_path=/vwnfns-12/obj17 2024-04-13T21:29:26.987 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.987 DEBUG:boto:Path: /vwnfns-12/obj17 2024-04-13T21:29:26.987 DEBUG:boto:Data: 2024-04-13T21:29:26.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.987 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.987 DEBUG:boto:Params: {} 2024-04-13T21:29:26.987 DEBUG:boto:Token: None 2024-04-13T21:29:26.988 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj17 2024-04-13T21:29:26.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:paKIelpSFnOuhT2D3YbdvHY3uYg= 2024-04-13T21:29:26.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:paKIelpSFnOuhT2D3YbdvHY3uYg='} 2024-04-13T21:29:26.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007aba4b10ce477ae-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.991 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:26.991 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:26.991 DEBUG:boto:Method: HEAD 2024-04-13T21:29:26.991 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:26.992 DEBUG:boto:Data: 2024-04-13T21:29:26.992 DEBUG:boto:Headers: {} 2024-04-13T21:29:26.992 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.992 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.992 DEBUG:boto:Params: {} 2024-04-13T21:29:26.992 DEBUG:boto:Token: None 2024-04-13T21:29:26.992 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/ 2024-04-13T21:29:26.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T21:29:26.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T21:29:26.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc11674204e2114c-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:26.994 DEBUG:boto:path=/vwnfns-12/obj18 2024-04-13T21:29:26.994 DEBUG:boto:auth_path=/vwnfns-12/obj18 2024-04-13T21:29:26.995 DEBUG:boto:Method: PUT 2024-04-13T21:29:26.995 DEBUG:boto:Path: /vwnfns-12/obj18 2024-04-13T21:29:26.995 DEBUG:boto:Data: 2024-04-13T21:29:26.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:26.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:26.995 DEBUG:boto:Port: 8001 2024-04-13T21:29:26.995 DEBUG:boto:Params: {} 2024-04-13T21:29:26.995 DEBUG:boto:Token: None 2024-04-13T21:29:26.995 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:26 GMT /vwnfns-12/obj18 2024-04-13T21:29:26.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q2k5bch1rpRzHtHXKDpAaV/SiSs= 2024-04-13T21:29:26.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q2k5bch1rpRzHtHXKDpAaV/SiSs='} 2024-04-13T21:29:27.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008db130d9a139c840-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.039 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.039 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.039 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.039 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.039 DEBUG:boto:Data: 2024-04-13T21:29:27.039 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.039 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.040 DEBUG:boto:Params: {} 2024-04-13T21:29:27.040 DEBUG:boto:Token: None 2024-04-13T21:29:27.040 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef88069758b53620-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.042 DEBUG:boto:path=/vwnfns-12/obj19 2024-04-13T21:29:27.043 DEBUG:boto:auth_path=/vwnfns-12/obj19 2024-04-13T21:29:27.043 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.043 DEBUG:boto:Path: /vwnfns-12/obj19 2024-04-13T21:29:27.043 DEBUG:boto:Data: 2024-04-13T21:29:27.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.043 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.043 DEBUG:boto:Params: {} 2024-04-13T21:29:27.043 DEBUG:boto:Token: None 2024-04-13T21:29:27.043 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj19 2024-04-13T21:29:27.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hyn3LCr61MMAMnOlIWby+4MPz6U= 2024-04-13T21:29:27.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hyn3LCr61MMAMnOlIWby+4MPz6U='} 2024-04-13T21:29:27.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f491bf6e471aae71-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.047 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.047 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.047 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.048 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.048 DEBUG:boto:Data: 2024-04-13T21:29:27.048 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.048 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.048 DEBUG:boto:Params: {} 2024-04-13T21:29:27.048 DEBUG:boto:Token: None 2024-04-13T21:29:27.048 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4a40645c6a2bac7-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.050 DEBUG:boto:path=/vwnfns-12/obj20 2024-04-13T21:29:27.050 DEBUG:boto:auth_path=/vwnfns-12/obj20 2024-04-13T21:29:27.051 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.051 DEBUG:boto:Path: /vwnfns-12/obj20 2024-04-13T21:29:27.051 DEBUG:boto:Data: 2024-04-13T21:29:27.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.051 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.051 DEBUG:boto:Params: {} 2024-04-13T21:29:27.051 DEBUG:boto:Token: None 2024-04-13T21:29:27.051 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj20 2024-04-13T21:29:27.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lhYLGNuygmlVhiTARy3WFs0MZY0= 2024-04-13T21:29:27.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lhYLGNuygmlVhiTARy3WFs0MZY0='} 2024-04-13T21:29:27.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a643bad33830b1b9-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.095 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.095 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.095 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.095 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.095 DEBUG:boto:Data: 2024-04-13T21:29:27.096 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.096 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.096 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.096 DEBUG:boto:Params: {} 2024-04-13T21:29:27.096 DEBUG:boto:Token: None 2024-04-13T21:29:27.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e9f60cc489bf7db-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.099 DEBUG:boto:path=/vwnfns-12/obj21 2024-04-13T21:29:27.099 DEBUG:boto:auth_path=/vwnfns-12/obj21 2024-04-13T21:29:27.099 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.099 DEBUG:boto:Path: /vwnfns-12/obj21 2024-04-13T21:29:27.099 DEBUG:boto:Data: 2024-04-13T21:29:27.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.100 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.100 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.100 DEBUG:boto:Params: {} 2024-04-13T21:29:27.100 DEBUG:boto:Token: None 2024-04-13T21:29:27.100 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj21 2024-04-13T21:29:27.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6+YwKs+kWQujGOu9NZnB7K5v724= 2024-04-13T21:29:27.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6+YwKs+kWQujGOu9NZnB7K5v724='} 2024-04-13T21:29:27.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058efcb7b6a726605-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.106 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.106 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.106 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.106 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.106 DEBUG:boto:Data: 2024-04-13T21:29:27.106 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.106 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.106 DEBUG:boto:Params: {} 2024-04-13T21:29:27.107 DEBUG:boto:Token: None 2024-04-13T21:29:27.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0c9f5ab331aeb14-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.109 DEBUG:boto:path=/vwnfns-12/obj22 2024-04-13T21:29:27.109 DEBUG:boto:auth_path=/vwnfns-12/obj22 2024-04-13T21:29:27.109 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.109 DEBUG:boto:Path: /vwnfns-12/obj22 2024-04-13T21:29:27.109 DEBUG:boto:Data: 2024-04-13T21:29:27.109 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.110 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.110 DEBUG:boto:Params: {} 2024-04-13T21:29:27.110 DEBUG:boto:Token: None 2024-04-13T21:29:27.110 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj22 2024-04-13T21:29:27.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NENl1lsV0r8LzOGDom+SIn4w/oQ= 2024-04-13T21:29:27.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NENl1lsV0r8LzOGDom+SIn4w/oQ='} 2024-04-13T21:29:27.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088255cccf94c1173-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.155 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.155 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.155 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.156 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.156 DEBUG:boto:Data: 2024-04-13T21:29:27.156 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.156 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.156 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.156 DEBUG:boto:Params: {} 2024-04-13T21:29:27.156 DEBUG:boto:Token: None 2024-04-13T21:29:27.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e9c51ab56fabf1f-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.159 DEBUG:boto:path=/vwnfns-12/obj23 2024-04-13T21:29:27.159 DEBUG:boto:auth_path=/vwnfns-12/obj23 2024-04-13T21:29:27.159 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.159 DEBUG:boto:Path: /vwnfns-12/obj23 2024-04-13T21:29:27.159 DEBUG:boto:Data: 2024-04-13T21:29:27.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.160 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.160 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.160 DEBUG:boto:Params: {} 2024-04-13T21:29:27.160 DEBUG:boto:Token: None 2024-04-13T21:29:27.160 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj23 2024-04-13T21:29:27.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j9P9aSVrdur+ePOocjWihqIEVTU= 2024-04-13T21:29:27.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j9P9aSVrdur+ePOocjWihqIEVTU='} 2024-04-13T21:29:27.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000405e8a1bfc014711-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.164 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.164 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.164 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.164 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.164 DEBUG:boto:Data: 2024-04-13T21:29:27.164 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.164 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.164 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.164 DEBUG:boto:Params: {} 2024-04-13T21:29:27.165 DEBUG:boto:Token: None 2024-04-13T21:29:27.165 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e47e5dcf368bc758-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.167 DEBUG:boto:path=/vwnfns-12/obj24 2024-04-13T21:29:27.167 DEBUG:boto:auth_path=/vwnfns-12/obj24 2024-04-13T21:29:27.167 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.167 DEBUG:boto:Path: /vwnfns-12/obj24 2024-04-13T21:29:27.167 DEBUG:boto:Data: 2024-04-13T21:29:27.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.168 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.168 DEBUG:boto:Params: {} 2024-04-13T21:29:27.168 DEBUG:boto:Token: None 2024-04-13T21:29:27.168 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj24 2024-04-13T21:29:27.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UoRfdXDmGYPE6vOs3JYayL4CINI= 2024-04-13T21:29:27.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UoRfdXDmGYPE6vOs3JYayL4CINI='} 2024-04-13T21:29:27.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f817cd5d64fc0624-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.211 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.211 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.211 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.211 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.211 DEBUG:boto:Data: 2024-04-13T21:29:27.211 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.211 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.212 DEBUG:boto:Params: {} 2024-04-13T21:29:27.212 DEBUG:boto:Token: None 2024-04-13T21:29:27.212 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003843efd2075c85b6-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.214 DEBUG:boto:path=/vwnfns-12/obj25 2024-04-13T21:29:27.214 DEBUG:boto:auth_path=/vwnfns-12/obj25 2024-04-13T21:29:27.215 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.215 DEBUG:boto:Path: /vwnfns-12/obj25 2024-04-13T21:29:27.215 DEBUG:boto:Data: 2024-04-13T21:29:27.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.215 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.215 DEBUG:boto:Params: {} 2024-04-13T21:29:27.215 DEBUG:boto:Token: None 2024-04-13T21:29:27.215 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj25 2024-04-13T21:29:27.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YHnv/u711BQ3VQd4J87Zb8jRGPg= 2024-04-13T21:29:27.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YHnv/u711BQ3VQd4J87Zb8jRGPg='} 2024-04-13T21:29:27.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e223360be2d93bea-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.219 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.219 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.219 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.219 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.219 DEBUG:boto:Data: 2024-04-13T21:29:27.220 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.220 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.220 DEBUG:boto:Params: {} 2024-04-13T21:29:27.220 DEBUG:boto:Token: None 2024-04-13T21:29:27.220 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090d7d29dcf2c1bc9-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.222 DEBUG:boto:path=/vwnfns-12/obj26 2024-04-13T21:29:27.222 DEBUG:boto:auth_path=/vwnfns-12/obj26 2024-04-13T21:29:27.223 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.223 DEBUG:boto:Path: /vwnfns-12/obj26 2024-04-13T21:29:27.223 DEBUG:boto:Data: 2024-04-13T21:29:27.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.223 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.223 DEBUG:boto:Params: {} 2024-04-13T21:29:27.223 DEBUG:boto:Token: None 2024-04-13T21:29:27.223 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj26 2024-04-13T21:29:27.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VktOa4pih3WdiNy5SNOfe6dVkYg= 2024-04-13T21:29:27.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VktOa4pih3WdiNy5SNOfe6dVkYg='} 2024-04-13T21:29:27.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e7eb66f67fc8e21-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.227 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.227 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.228 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.228 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.228 DEBUG:boto:Data: 2024-04-13T21:29:27.228 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.228 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.228 DEBUG:boto:Params: {} 2024-04-13T21:29:27.228 DEBUG:boto:Token: None 2024-04-13T21:29:27.228 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003860fde6edf756e6-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.231 DEBUG:boto:path=/vwnfns-12/obj27 2024-04-13T21:29:27.231 DEBUG:boto:auth_path=/vwnfns-12/obj27 2024-04-13T21:29:27.231 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.231 DEBUG:boto:Path: /vwnfns-12/obj27 2024-04-13T21:29:27.231 DEBUG:boto:Data: 2024-04-13T21:29:27.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.231 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.231 DEBUG:boto:Params: {} 2024-04-13T21:29:27.232 DEBUG:boto:Token: None 2024-04-13T21:29:27.232 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj27 2024-04-13T21:29:27.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ihh/fATerveizLduuIZ9Q0yhQw4= 2024-04-13T21:29:27.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ihh/fATerveizLduuIZ9Q0yhQw4='} 2024-04-13T21:29:27.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047de0b1465dc01bd-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.275 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.275 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.275 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.275 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.275 DEBUG:boto:Data: 2024-04-13T21:29:27.275 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.276 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.276 DEBUG:boto:Params: {} 2024-04-13T21:29:27.276 DEBUG:boto:Token: None 2024-04-13T21:29:27.276 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f360f920f998e0c-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.278 DEBUG:boto:path=/vwnfns-12/obj28 2024-04-13T21:29:27.279 DEBUG:boto:auth_path=/vwnfns-12/obj28 2024-04-13T21:29:27.279 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.279 DEBUG:boto:Path: /vwnfns-12/obj28 2024-04-13T21:29:27.279 DEBUG:boto:Data: 2024-04-13T21:29:27.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.279 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.279 DEBUG:boto:Params: {} 2024-04-13T21:29:27.279 DEBUG:boto:Token: None 2024-04-13T21:29:27.280 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj28 2024-04-13T21:29:27.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8BegRVyln7zfkH1ULFCZAahv2i4= 2024-04-13T21:29:27.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8BegRVyln7zfkH1ULFCZAahv2i4='} 2024-04-13T21:29:27.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084536773ee0e5528-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.283 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.284 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.284 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.284 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.284 DEBUG:boto:Data: 2024-04-13T21:29:27.284 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.284 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.284 DEBUG:boto:Params: {} 2024-04-13T21:29:27.284 DEBUG:boto:Token: None 2024-04-13T21:29:27.284 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.285 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f93b2d65536299a-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.287 DEBUG:boto:path=/vwnfns-12/obj29 2024-04-13T21:29:27.287 DEBUG:boto:auth_path=/vwnfns-12/obj29 2024-04-13T21:29:27.287 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.287 DEBUG:boto:Path: /vwnfns-12/obj29 2024-04-13T21:29:27.287 DEBUG:boto:Data: 2024-04-13T21:29:27.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.287 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.287 DEBUG:boto:Params: {} 2024-04-13T21:29:27.288 DEBUG:boto:Token: None 2024-04-13T21:29:27.288 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj29 2024-04-13T21:29:27.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vwr+kSdLl1wKIBRA8gwpzdGZbrQ= 2024-04-13T21:29:27.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vwr+kSdLl1wKIBRA8gwpzdGZbrQ='} 2024-04-13T21:29:27.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002aa3f51039ccb5a8-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.331 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.331 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.331 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.331 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.331 DEBUG:boto:Data: 2024-04-13T21:29:27.331 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.331 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.331 DEBUG:boto:Params: {} 2024-04-13T21:29:27.332 DEBUG:boto:Token: None 2024-04-13T21:29:27.332 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ec63ec797704818-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.334 DEBUG:boto:path=/vwnfns-12/obj30 2024-04-13T21:29:27.334 DEBUG:boto:auth_path=/vwnfns-12/obj30 2024-04-13T21:29:27.335 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.335 DEBUG:boto:Path: /vwnfns-12/obj30 2024-04-13T21:29:27.335 DEBUG:boto:Data: 2024-04-13T21:29:27.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.335 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.335 DEBUG:boto:Params: {} 2024-04-13T21:29:27.335 DEBUG:boto:Token: None 2024-04-13T21:29:27.335 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj30 2024-04-13T21:29:27.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VNKAG4c3fBFMtpGPCZNYZML8vPg= 2024-04-13T21:29:27.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VNKAG4c3fBFMtpGPCZNYZML8vPg='} 2024-04-13T21:29:27.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d0e6374a30dac37-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.339 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.339 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.339 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.339 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.339 DEBUG:boto:Data: 2024-04-13T21:29:27.339 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.340 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.340 DEBUG:boto:Params: {} 2024-04-13T21:29:27.340 DEBUG:boto:Token: None 2024-04-13T21:29:27.340 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008eec332c2e78ee8-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.344 DEBUG:boto:path=/vwnfns-12/obj31 2024-04-13T21:29:27.344 DEBUG:boto:auth_path=/vwnfns-12/obj31 2024-04-13T21:29:27.344 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.344 DEBUG:boto:Path: /vwnfns-12/obj31 2024-04-13T21:29:27.344 DEBUG:boto:Data: 2024-04-13T21:29:27.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.344 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.344 DEBUG:boto:Params: {} 2024-04-13T21:29:27.344 DEBUG:boto:Token: None 2024-04-13T21:29:27.345 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj31 2024-04-13T21:29:27.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2In76eumQYN4qb/H2cT+zJ1bfh4= 2024-04-13T21:29:27.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2In76eumQYN4qb/H2cT+zJ1bfh4='} 2024-04-13T21:29:27.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000029e6fe3550dc4f3-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.387 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.387 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.387 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.387 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.387 DEBUG:boto:Data: 2024-04-13T21:29:27.387 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.387 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.387 DEBUG:boto:Params: {} 2024-04-13T21:29:27.388 DEBUG:boto:Token: None 2024-04-13T21:29:27.388 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e88599499412ac63-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.390 DEBUG:boto:path=/vwnfns-12/obj32 2024-04-13T21:29:27.391 DEBUG:boto:auth_path=/vwnfns-12/obj32 2024-04-13T21:29:27.391 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.391 DEBUG:boto:Path: /vwnfns-12/obj32 2024-04-13T21:29:27.391 DEBUG:boto:Data: 2024-04-13T21:29:27.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.391 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.392 DEBUG:boto:Params: {} 2024-04-13T21:29:27.392 DEBUG:boto:Token: None 2024-04-13T21:29:27.392 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj32 2024-04-13T21:29:27.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWsrZBmNNK+WUe7Uq5ygaJXBido= 2024-04-13T21:29:27.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWsrZBmNNK+WUe7Uq5ygaJXBido='} 2024-04-13T21:29:27.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9b931e954166a39-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.396 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.396 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.396 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.396 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.396 DEBUG:boto:Data: 2024-04-13T21:29:27.396 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.397 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.397 DEBUG:boto:Params: {} 2024-04-13T21:29:27.397 DEBUG:boto:Token: None 2024-04-13T21:29:27.397 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bd55c3c79e16640-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.399 DEBUG:boto:path=/vwnfns-12/obj33 2024-04-13T21:29:27.399 DEBUG:boto:auth_path=/vwnfns-12/obj33 2024-04-13T21:29:27.399 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.399 DEBUG:boto:Path: /vwnfns-12/obj33 2024-04-13T21:29:27.400 DEBUG:boto:Data: 2024-04-13T21:29:27.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.400 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.400 DEBUG:boto:Params: {} 2024-04-13T21:29:27.400 DEBUG:boto:Token: None 2024-04-13T21:29:27.400 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj33 2024-04-13T21:29:27.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Og4bIBCmw+KOWUW/xTQtp5UlFLY= 2024-04-13T21:29:27.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Og4bIBCmw+KOWUW/xTQtp5UlFLY='} 2024-04-13T21:29:27.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2100cf82f496b85-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.443 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.443 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.443 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.443 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.443 DEBUG:boto:Data: 2024-04-13T21:29:27.443 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.443 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.443 DEBUG:boto:Params: {} 2024-04-13T21:29:27.444 DEBUG:boto:Token: None 2024-04-13T21:29:27.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020c393978d0fd532-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.446 DEBUG:boto:path=/vwnfns-12/obj34 2024-04-13T21:29:27.446 DEBUG:boto:auth_path=/vwnfns-12/obj34 2024-04-13T21:29:27.446 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.446 DEBUG:boto:Path: /vwnfns-12/obj34 2024-04-13T21:29:27.446 DEBUG:boto:Data: 2024-04-13T21:29:27.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.447 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.447 DEBUG:boto:Params: {} 2024-04-13T21:29:27.447 DEBUG:boto:Token: None 2024-04-13T21:29:27.447 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj34 2024-04-13T21:29:27.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xnc6QFsAVFgeaGUZ4WFO+en374E= 2024-04-13T21:29:27.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xnc6QFsAVFgeaGUZ4WFO+en374E='} 2024-04-13T21:29:27.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079318a39e20f6752-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.451 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.451 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.451 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.451 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.451 DEBUG:boto:Data: 2024-04-13T21:29:27.451 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.451 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.451 DEBUG:boto:Params: {} 2024-04-13T21:29:27.452 DEBUG:boto:Token: None 2024-04-13T21:29:27.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008373236fea6c9f1a-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.454 DEBUG:boto:path=/vwnfns-12/obj35 2024-04-13T21:29:27.454 DEBUG:boto:auth_path=/vwnfns-12/obj35 2024-04-13T21:29:27.454 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.454 DEBUG:boto:Path: /vwnfns-12/obj35 2024-04-13T21:29:27.454 DEBUG:boto:Data: 2024-04-13T21:29:27.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.454 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.455 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.455 DEBUG:boto:Params: {} 2024-04-13T21:29:27.455 DEBUG:boto:Token: None 2024-04-13T21:29:27.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj35 2024-04-13T21:29:27.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4XNV54uy36Bpno9x/ei7w9qxB/0= 2024-04-13T21:29:27.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4XNV54uy36Bpno9x/ei7w9qxB/0='} 2024-04-13T21:29:27.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55df966e3cc41f0-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.459 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.459 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.459 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.459 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.459 DEBUG:boto:Data: 2024-04-13T21:29:27.459 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.459 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.459 DEBUG:boto:Params: {} 2024-04-13T21:29:27.459 DEBUG:boto:Token: None 2024-04-13T21:29:27.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000662fbf8a78d2ee98-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.462 DEBUG:boto:path=/vwnfns-12/obj36 2024-04-13T21:29:27.462 DEBUG:boto:auth_path=/vwnfns-12/obj36 2024-04-13T21:29:27.462 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.462 DEBUG:boto:Path: /vwnfns-12/obj36 2024-04-13T21:29:27.462 DEBUG:boto:Data: 2024-04-13T21:29:27.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.462 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.462 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.463 DEBUG:boto:Params: {} 2024-04-13T21:29:27.463 DEBUG:boto:Token: None 2024-04-13T21:29:27.463 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj36 2024-04-13T21:29:27.463 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/YEKOkTH2BusWPJBE7NY/l4j+t4= 2024-04-13T21:29:27.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/YEKOkTH2BusWPJBE7NY/l4j+t4='} 2024-04-13T21:29:27.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b7ce4cb9b304de1-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.511 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.511 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.511 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.511 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.511 DEBUG:boto:Data: 2024-04-13T21:29:27.511 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.511 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.512 DEBUG:boto:Params: {} 2024-04-13T21:29:27.512 DEBUG:boto:Token: None 2024-04-13T21:29:27.512 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ced3a59817072c1-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.514 DEBUG:boto:path=/vwnfns-12/obj37 2024-04-13T21:29:27.514 DEBUG:boto:auth_path=/vwnfns-12/obj37 2024-04-13T21:29:27.514 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.515 DEBUG:boto:Path: /vwnfns-12/obj37 2024-04-13T21:29:27.515 DEBUG:boto:Data: 2024-04-13T21:29:27.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.515 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.515 DEBUG:boto:Params: {} 2024-04-13T21:29:27.515 DEBUG:boto:Token: None 2024-04-13T21:29:27.515 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj37 2024-04-13T21:29:27.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KI5y/uV+E2pUOo4zB7YaqVFbYNo= 2024-04-13T21:29:27.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KI5y/uV+E2pUOo4zB7YaqVFbYNo='} 2024-04-13T21:29:27.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f48a79b58c5b93f-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.519 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.519 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.519 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.519 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.520 DEBUG:boto:Data: 2024-04-13T21:29:27.520 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.520 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.520 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.520 DEBUG:boto:Params: {} 2024-04-13T21:29:27.520 DEBUG:boto:Token: None 2024-04-13T21:29:27.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b41e29557ffc663-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.522 DEBUG:boto:path=/vwnfns-12/obj38 2024-04-13T21:29:27.523 DEBUG:boto:auth_path=/vwnfns-12/obj38 2024-04-13T21:29:27.523 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.523 DEBUG:boto:Path: /vwnfns-12/obj38 2024-04-13T21:29:27.523 DEBUG:boto:Data: 2024-04-13T21:29:27.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.523 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.523 DEBUG:boto:Params: {} 2024-04-13T21:29:27.524 DEBUG:boto:Token: None 2024-04-13T21:29:27.524 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj38 2024-04-13T21:29:27.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PzuxdGNTVxIPlqDh06fd87tjPC0= 2024-04-13T21:29:27.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PzuxdGNTVxIPlqDh06fd87tjPC0='} 2024-04-13T21:29:27.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fdd6974a2f8d429-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.571 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.571 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.571 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.571 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.571 DEBUG:boto:Data: 2024-04-13T21:29:27.571 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.571 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.571 DEBUG:boto:Params: {} 2024-04-13T21:29:27.572 DEBUG:boto:Token: None 2024-04-13T21:29:27.572 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077045dd62349642e-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.574 DEBUG:boto:path=/vwnfns-12/obj39 2024-04-13T21:29:27.574 DEBUG:boto:auth_path=/vwnfns-12/obj39 2024-04-13T21:29:27.574 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.574 DEBUG:boto:Path: /vwnfns-12/obj39 2024-04-13T21:29:27.575 DEBUG:boto:Data: 2024-04-13T21:29:27.575 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.575 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.575 DEBUG:boto:Params: {} 2024-04-13T21:29:27.575 DEBUG:boto:Token: None 2024-04-13T21:29:27.575 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj39 2024-04-13T21:29:27.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7MbuZ6bXurlW8Ut2J874KNqDG3Y= 2024-04-13T21:29:27.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7MbuZ6bXurlW8Ut2J874KNqDG3Y='} 2024-04-13T21:29:27.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010108dc89d56e795-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.579 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.579 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.579 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.579 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.579 DEBUG:boto:Data: 2024-04-13T21:29:27.579 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.580 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.580 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.580 DEBUG:boto:Params: {} 2024-04-13T21:29:27.580 DEBUG:boto:Token: None 2024-04-13T21:29:27.580 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cea192ce68d77fa1-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.582 DEBUG:boto:path=/vwnfns-12/obj40 2024-04-13T21:29:27.582 DEBUG:boto:auth_path=/vwnfns-12/obj40 2024-04-13T21:29:27.582 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.582 DEBUG:boto:Path: /vwnfns-12/obj40 2024-04-13T21:29:27.583 DEBUG:boto:Data: 2024-04-13T21:29:27.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.583 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.583 DEBUG:boto:Params: {} 2024-04-13T21:29:27.583 DEBUG:boto:Token: None 2024-04-13T21:29:27.583 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj40 2024-04-13T21:29:27.583 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W6ixUNIYY4ftRF7DN9JVUc1fmk0= 2024-04-13T21:29:27.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W6ixUNIYY4ftRF7DN9JVUc1fmk0='} 2024-04-13T21:29:27.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dee24e9ebba98c12-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.587 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.587 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.587 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.587 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.587 DEBUG:boto:Data: 2024-04-13T21:29:27.587 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.587 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.587 DEBUG:boto:Params: {} 2024-04-13T21:29:27.588 DEBUG:boto:Token: None 2024-04-13T21:29:27.588 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015c38ba45afbe306-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.590 DEBUG:boto:path=/vwnfns-12/obj41 2024-04-13T21:29:27.590 DEBUG:boto:auth_path=/vwnfns-12/obj41 2024-04-13T21:29:27.591 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.591 DEBUG:boto:Path: /vwnfns-12/obj41 2024-04-13T21:29:27.591 DEBUG:boto:Data: 2024-04-13T21:29:27.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.591 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.591 DEBUG:boto:Params: {} 2024-04-13T21:29:27.591 DEBUG:boto:Token: None 2024-04-13T21:29:27.591 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj41 2024-04-13T21:29:27.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tgW7sLlcwEfR4JUVj6+OeSjRNtE= 2024-04-13T21:29:27.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tgW7sLlcwEfR4JUVj6+OeSjRNtE='} 2024-04-13T21:29:27.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028c04e76afbdfed9-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.639 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.639 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.639 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.639 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.639 DEBUG:boto:Data: 2024-04-13T21:29:27.639 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.640 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.640 DEBUG:boto:Params: {} 2024-04-13T21:29:27.640 DEBUG:boto:Token: None 2024-04-13T21:29:27.640 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098c8450d253c46a9-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.642 DEBUG:boto:path=/vwnfns-12/obj42 2024-04-13T21:29:27.643 DEBUG:boto:auth_path=/vwnfns-12/obj42 2024-04-13T21:29:27.643 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.643 DEBUG:boto:Path: /vwnfns-12/obj42 2024-04-13T21:29:27.643 DEBUG:boto:Data: 2024-04-13T21:29:27.643 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.643 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.643 DEBUG:boto:Params: {} 2024-04-13T21:29:27.643 DEBUG:boto:Token: None 2024-04-13T21:29:27.644 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj42 2024-04-13T21:29:27.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/pe4u15kd9RA0AabMnDHTkoTWLo= 2024-04-13T21:29:27.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/pe4u15kd9RA0AabMnDHTkoTWLo='} 2024-04-13T21:29:27.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000984db9abcc7d592-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.647 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.648 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.648 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.648 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.648 DEBUG:boto:Data: 2024-04-13T21:29:27.648 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.648 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.648 DEBUG:boto:Params: {} 2024-04-13T21:29:27.648 DEBUG:boto:Token: None 2024-04-13T21:29:27.648 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.649 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a389bd66c1d223b-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.651 DEBUG:boto:path=/vwnfns-12/obj43 2024-04-13T21:29:27.651 DEBUG:boto:auth_path=/vwnfns-12/obj43 2024-04-13T21:29:27.651 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.651 DEBUG:boto:Path: /vwnfns-12/obj43 2024-04-13T21:29:27.651 DEBUG:boto:Data: 2024-04-13T21:29:27.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.651 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.651 DEBUG:boto:Params: {} 2024-04-13T21:29:27.651 DEBUG:boto:Token: None 2024-04-13T21:29:27.652 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj43 2024-04-13T21:29:27.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LM+x/eG1yzESj5NQ+krH885KhBo= 2024-04-13T21:29:27.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LM+x/eG1yzESj5NQ+krH885KhBo='} 2024-04-13T21:29:27.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae02a5dc27a64545-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.699 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.699 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.699 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.699 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.699 DEBUG:boto:Data: 2024-04-13T21:29:27.699 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.699 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.700 DEBUG:boto:Params: {} 2024-04-13T21:29:27.700 DEBUG:boto:Token: None 2024-04-13T21:29:27.700 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a9b4d4f29aac1e-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.702 DEBUG:boto:path=/vwnfns-12/obj44 2024-04-13T21:29:27.702 DEBUG:boto:auth_path=/vwnfns-12/obj44 2024-04-13T21:29:27.703 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.703 DEBUG:boto:Path: /vwnfns-12/obj44 2024-04-13T21:29:27.703 DEBUG:boto:Data: 2024-04-13T21:29:27.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.703 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.703 DEBUG:boto:Params: {} 2024-04-13T21:29:27.703 DEBUG:boto:Token: None 2024-04-13T21:29:27.703 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj44 2024-04-13T21:29:27.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RPcmPLA3mR3bYNPlwvq2kaxC9Gc= 2024-04-13T21:29:27.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RPcmPLA3mR3bYNPlwvq2kaxC9Gc='} 2024-04-13T21:29:27.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e1d0b91d152d7ce-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.712 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.712 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.712 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.712 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.712 DEBUG:boto:Data: 2024-04-13T21:29:27.712 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.713 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.713 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.713 DEBUG:boto:Params: {} 2024-04-13T21:29:27.713 DEBUG:boto:Token: None 2024-04-13T21:29:27.713 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.713 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089e97b50d79a2fc5-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.715 DEBUG:boto:path=/vwnfns-12/obj45 2024-04-13T21:29:27.715 DEBUG:boto:auth_path=/vwnfns-12/obj45 2024-04-13T21:29:27.716 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.716 DEBUG:boto:Path: /vwnfns-12/obj45 2024-04-13T21:29:27.716 DEBUG:boto:Data: 2024-04-13T21:29:27.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.716 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.716 DEBUG:boto:Params: {} 2024-04-13T21:29:27.716 DEBUG:boto:Token: None 2024-04-13T21:29:27.716 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj45 2024-04-13T21:29:27.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4yy2USFfcVUZuSZ7zTN40gMR4oE= 2024-04-13T21:29:27.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4yy2USFfcVUZuSZ7zTN40gMR4oE='} 2024-04-13T21:29:27.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046326d1d986af1a0-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.759 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.759 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.759 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.759 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.759 DEBUG:boto:Data: 2024-04-13T21:29:27.759 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.760 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.760 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.760 DEBUG:boto:Params: {} 2024-04-13T21:29:27.760 DEBUG:boto:Token: None 2024-04-13T21:29:27.760 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.760 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec702628bbf585dd-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.762 DEBUG:boto:path=/vwnfns-12/obj46 2024-04-13T21:29:27.763 DEBUG:boto:auth_path=/vwnfns-12/obj46 2024-04-13T21:29:27.763 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.763 DEBUG:boto:Path: /vwnfns-12/obj46 2024-04-13T21:29:27.763 DEBUG:boto:Data: 2024-04-13T21:29:27.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.763 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.763 DEBUG:boto:Params: {} 2024-04-13T21:29:27.763 DEBUG:boto:Token: None 2024-04-13T21:29:27.764 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj46 2024-04-13T21:29:27.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MbEo7Zzu9Kjk4E/2oYlkXwkrdAk= 2024-04-13T21:29:27.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MbEo7Zzu9Kjk4E/2oYlkXwkrdAk='} 2024-04-13T21:29:27.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ed3b36cbe040c5f-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.767 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.767 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.768 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.768 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.768 DEBUG:boto:Data: 2024-04-13T21:29:27.768 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.768 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.768 DEBUG:boto:Params: {} 2024-04-13T21:29:27.768 DEBUG:boto:Token: None 2024-04-13T21:29:27.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.769 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f676425fc24d408-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.771 DEBUG:boto:path=/vwnfns-12/obj47 2024-04-13T21:29:27.771 DEBUG:boto:auth_path=/vwnfns-12/obj47 2024-04-13T21:29:27.771 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.771 DEBUG:boto:Path: /vwnfns-12/obj47 2024-04-13T21:29:27.771 DEBUG:boto:Data: 2024-04-13T21:29:27.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.772 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.772 DEBUG:boto:Params: {} 2024-04-13T21:29:27.772 DEBUG:boto:Token: None 2024-04-13T21:29:27.772 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj47 2024-04-13T21:29:27.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PieaNwV1eeQPCZESvJ0wyhk6lVU= 2024-04-13T21:29:27.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PieaNwV1eeQPCZESvJ0wyhk6lVU='} 2024-04-13T21:29:27.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dc492a5d1de083a-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.815 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.815 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.815 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.815 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.815 DEBUG:boto:Data: 2024-04-13T21:29:27.816 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.816 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.816 DEBUG:boto:Params: {} 2024-04-13T21:29:27.816 DEBUG:boto:Token: None 2024-04-13T21:29:27.816 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055312720473c4ee0-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.818 DEBUG:boto:path=/vwnfns-12/obj48 2024-04-13T21:29:27.819 DEBUG:boto:auth_path=/vwnfns-12/obj48 2024-04-13T21:29:27.819 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.819 DEBUG:boto:Path: /vwnfns-12/obj48 2024-04-13T21:29:27.819 DEBUG:boto:Data: 2024-04-13T21:29:27.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.819 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.819 DEBUG:boto:Params: {} 2024-04-13T21:29:27.819 DEBUG:boto:Token: None 2024-04-13T21:29:27.820 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj48 2024-04-13T21:29:27.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZOcnGstaSAfIep7FFFVRUl9fHN0= 2024-04-13T21:29:27.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZOcnGstaSAfIep7FFFVRUl9fHN0='} 2024-04-13T21:29:27.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd750a152801c0d5-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.823 DEBUG:boto:path=/vwnfns-12/ 2024-04-13T21:29:27.823 DEBUG:boto:auth_path=/vwnfns-12/ 2024-04-13T21:29:27.823 DEBUG:boto:Method: HEAD 2024-04-13T21:29:27.823 DEBUG:boto:Path: /vwnfns-12/ 2024-04-13T21:29:27.824 DEBUG:boto:Data: 2024-04-13T21:29:27.824 DEBUG:boto:Headers: {} 2024-04-13T21:29:27.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.824 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.824 DEBUG:boto:Params: {} 2024-04-13T21:29:27.824 DEBUG:boto:Token: None 2024-04-13T21:29:27.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/ 2024-04-13T21:29:27.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T21:29:27.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T21:29:27.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064b5e8edb2c88cb2-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.827 DEBUG:boto:path=/vwnfns-12/obj49 2024-04-13T21:29:27.827 DEBUG:boto:auth_path=/vwnfns-12/obj49 2024-04-13T21:29:27.827 DEBUG:boto:Method: PUT 2024-04-13T21:29:27.827 DEBUG:boto:Path: /vwnfns-12/obj49 2024-04-13T21:29:27.827 DEBUG:boto:Data: 2024-04-13T21:29:27.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T21:29:27.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:29:27.827 DEBUG:boto:Port: 8001 2024-04-13T21:29:27.827 DEBUG:boto:Params: {} 2024-04-13T21:29:27.827 DEBUG:boto:Token: None 2024-04-13T21:29:27.828 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 21:29:27 GMT /vwnfns-12/obj49 2024-04-13T21:29:27.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZTl5Bh5SKoOQR84W1IjnzHFRyU= 2024-04-13T21:29:27.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZTl5Bh5SKoOQR84W1IjnzHFRyU='} 2024-04-13T21:29:27.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003720876414b00abf-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:29:27.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:29:27.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:29:27.872 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:29:27.919 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:29:27.919 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:29:27.947 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:29:27.958 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043765.266939_90.1", 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:29:25.266939Z" 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:29:27.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:29:27.960 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:29:27.961 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713043765.266939_90.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:29:27.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:29:27.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:29:27.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:29:27.962 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:29:28.010 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:29:28.010 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:29:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:28.043+0000 7f0f6e706780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:29:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:28.044+0000 7f0f6e706780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:29:28.048 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:29:28.048 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:29:28.048 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:29:28.049 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043765.266939_90.1", 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:29:25.266939Z", 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:29:28.050 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:28.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:28.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:29:28.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:29:28.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:29:28.054 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:29:28.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:28.054 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:29:28.054 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:29:28.054 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:29:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:29:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:29:28Z" 2024-04-13T21:29:28.055 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:29:28.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713043765.266939_90.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:29:25.266939Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:29:28Z" } 2024-04-13T21:29:28.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043765.266939_90.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:29:25.266939Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:29:28.057 DEBUG:rgw_multi.tests:log_status={0: '1_1713043765.266939_90.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:29:28.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1713043765.266939_90.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:29:28.057 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:29:28.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:29:28.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:29:28.058 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:29:28.112 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:29:28.112 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:29:28.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:28.155+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:29:33.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:33.156+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:29:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:38.157+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:29:43.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:43.157+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:29:48.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:48.158+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:29:53.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:53.158+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:29:58.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:29:58.158+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:03.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:03.159+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:08.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:08.160+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:13.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:13.161+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:18.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:18.162+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:23.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:23.162+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:28.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:28.162+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:33.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:33.163+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:38.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:38.163+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:43.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:43.164+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:48.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:48.166+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:53.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:53.166+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:30:58.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:30:58.167+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:03.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:03.167+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:08.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:08.167+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:13.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:13.168+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:18.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:18.168+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:23.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:23.169+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:28.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:28.170+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:33.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:33.171+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:38.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:38.172+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:43.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:43.172+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:48.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:48.172+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:53.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:53.173+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:31:58.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:31:58.174+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:03.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:03.174+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:08.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:08.176+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:13.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:13.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:18.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:18.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:23.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:23.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:28.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:28.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:33.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:33.178+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:38.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:38.178+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:43.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:43.180+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:48.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:48.181+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:53.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:53.181+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:32:58.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:32:58.182+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:03.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:03.181+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:08.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:08.182+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:13.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:13.183+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:18.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:18.184+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:23.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:23.185+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:28.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:28.186+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:33.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:33.186+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:38.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:38.187+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:43.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:43.187+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:48.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:48.187+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:53.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:53.188+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:33:58.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:33:58.188+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:34:03.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:03.190+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:34:08.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:08.190+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:34:13.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:13.191+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:34:18.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:18.191+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T21:34:23.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:23.191+0000 7febb9dfa780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-13T21:34:23.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:23.191+0000 7febb9dfa780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T21:34:23.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:23.191+0000 7febb9dfa780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T21:34:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-13T21:34:23.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-04-13T21:34:23.208 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-13 2024-04-13T21:34:23.208 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:34:23.208 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:34:23.208 DEBUG:boto:Method: PUT 2024-04-13T21:34:23.208 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:34:23.209 DEBUG:boto:Data: 2024-04-13T21:34:23.209 DEBUG:boto:Headers: {} 2024-04-13T21:34:23.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:34:23.209 DEBUG:boto:Port: 8000 2024-04-13T21:34:23.209 DEBUG:boto:Params: {} 2024-04-13T21:34:23.209 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:34:23.210 DEBUG:boto:Token: None 2024-04-13T21:34:23.210 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:34:23 GMT /vwnfns-13/ 2024-04-13T21:34:23.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K8vA+hZff5mZxTBAjhtPzDQADtM= 2024-04-13T21:34:23.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:34:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K8vA+hZff5mZxTBAjhtPzDQADtM=', 'Content-Length': '0'} 2024-04-13T21:34:23.235 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000656ae0a701dc158b-00661afa5f-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:34:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:23.235 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-14 2024-04-13T21:34:23.235 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:34:23.235 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:34:23.235 DEBUG:boto:Method: PUT 2024-04-13T21:34:23.235 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:34:23.235 DEBUG:boto:Data: 2024-04-13T21:34:23.235 DEBUG:boto:Headers: {} 2024-04-13T21:34:23.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:34:23.236 DEBUG:boto:Port: 8001 2024-04-13T21:34:23.236 DEBUG:boto:Params: {} 2024-04-13T21:34:23.236 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:34:23.236 DEBUG:boto:Token: None 2024-04-13T21:34:23.236 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:34:23 GMT /vwnfns-14/ 2024-04-13T21:34:23.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gmFzry97spFptoTIkoffdLxWesA= 2024-04-13T21:34:23.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:34:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gmFzry97spFptoTIkoffdLxWesA=', 'Content-Length': '0'} 2024-04-13T21:34:23.271 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008096376a09cada5-00661afa5f-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:34:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:23.271 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:34:23.271 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:34:23.272 DEBUG:boto:path=/vwnfns-13/?versioning 2024-04-13T21:34:23.272 DEBUG:boto:auth_path=/vwnfns-13/?versioning 2024-04-13T21:34:23.272 DEBUG:boto:Method: PUT 2024-04-13T21:34:23.272 DEBUG:boto:Path: /vwnfns-13/?versioning 2024-04-13T21:34:23.272 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:34:23.272 DEBUG:boto:Headers: {} 2024-04-13T21:34:23.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:34:23.272 DEBUG:boto:Port: 8000 2024-04-13T21:34:23.272 DEBUG:boto:Params: {} 2024-04-13T21:34:23.272 DEBUG:boto:Token: None 2024-04-13T21:34:23.273 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:34:23 GMT /vwnfns-13/?versioning 2024-04-13T21:34:23.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JKjFBZcksx5JFtyblVUj8BwktyM= 2024-04-13T21:34:23.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:34:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JKjFBZcksx5JFtyblVUj8BwktyM=', 'Content-Length': '228'} 2024-04-13T21:34:23.277 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d53a6a2887414ac-00661afa5f-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:34:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:23.278 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:34:23.278 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:34:23.278 DEBUG:boto:path=/vwnfns-14/?versioning 2024-04-13T21:34:23.278 DEBUG:boto:auth_path=/vwnfns-14/?versioning 2024-04-13T21:34:23.278 DEBUG:boto:Method: PUT 2024-04-13T21:34:23.278 DEBUG:boto:Path: /vwnfns-14/?versioning 2024-04-13T21:34:23.278 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:34:23.278 DEBUG:boto:Headers: {} 2024-04-13T21:34:23.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:34:23.278 DEBUG:boto:Port: 8001 2024-04-13T21:34:23.279 DEBUG:boto:Params: {} 2024-04-13T21:34:23.279 DEBUG:boto:Token: None 2024-04-13T21:34:23.279 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:34:23 GMT /vwnfns-14/?versioning 2024-04-13T21:34:23.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lBn8s6FkC8KtYHJfkPST1M/SP+E= 2024-04-13T21:34:23.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:34:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lBn8s6FkC8KtYHJfkPST1M/SP+E=', 'Content-Length': '228'} 2024-04-13T21:34:23.286 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052a7bcc73a18038c-00661afa5f-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:34:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:23.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:34:23.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:34:23.287 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:34:23.342 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:34:23.342 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:34:23.375 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044063.292682_156.1", 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:34:23.292682Z" 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:34:23.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:23.390 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:34:23.391 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044063.292682_156.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:34:23.391 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:34:23.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:34:23.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:34:23.392 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:34:23.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:34:23.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:34:23.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:23.474+0000 7f9de9ccf780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:34:23.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:23.475+0000 7f9de9ccf780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:34:23.479 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:34:23.479 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:34:23.479 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:34:23.479 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044063.231459_145.1", 2024-04-13T21:34:23.480 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:34:23.231459Z", 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:34:23.481 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:23.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:23.483 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:34:23.484 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:34:23.485 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:34:23.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:23.485 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:34:23Z" 2024-04-13T21:34:23.485 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:34:23.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044063.231459_145.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:34:23.231459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:34:23Z" } 2024-04-13T21:34:23.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044063.231459_145.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:34:23.231459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:34:23.486 DEBUG:rgw_multi.tests:log_status={0: '1_1713044063.292682_156.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:34:23.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044063.231459_145.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:34:23.486 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044063.292682_156.1 target=1_1713044063.231459_145.1 2024-04-13T21:34:28.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:34:28.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:34:28.488 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:34:28.543 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:34:28.543 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:34:28.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:28.580+0000 7f79c7e63780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:34:28.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:28.581+0000 7f79c7e63780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:34:28.586 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:34:28.586 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:34:28.586 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:34:28.586 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:34:28.586 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044063.292682_156.1", 2024-04-13T21:34:28.587 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:34:23.292682Z", 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:34:28.588 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:28.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:28.590 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:34:28.591 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:34:28.592 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:34:28.593 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:34:28.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:34:28.593 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:34:28Z" 2024-04-13T21:34:28.593 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:34:28.594 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044063.292682_156.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:34:23.292682Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:34:28Z" } 2024-04-13T21:34:28.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044063.292682_156.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:34:23.292682Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:34:28.594 DEBUG:rgw_multi.tests:log_status={0: '1_1713044063.292682_156.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:34:28.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044063.292682_156.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:34:28.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:34:28.595 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:34:28.595 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:34:28.595 DEBUG:boto:Method: HEAD 2024-04-13T21:34:28.595 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:34:28.595 DEBUG:boto:Data: 2024-04-13T21:34:28.595 DEBUG:boto:Headers: {} 2024-04-13T21:34:28.596 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:34:28.596 DEBUG:boto:Port: 8000 2024-04-13T21:34:28.596 DEBUG:boto:Params: {} 2024-04-13T21:34:28.596 DEBUG:boto:Token: None 2024-04-13T21:34:28.596 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:34:28 GMT /vwnfns-13/ 2024-04-13T21:34:28.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qZuRHgJtCcES5ULS6Hp5VMJO2Xg= 2024-04-13T21:34:28.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:34:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qZuRHgJtCcES5ULS6Hp5VMJO2Xg=', 'Content-Length': '0'} 2024-04-13T21:34:28.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b52e627f9bd27e63-00661afa64-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:28.599 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:34:28.599 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:34:28.600 DEBUG:boto:Method: PUT 2024-04-13T21:34:28.600 DEBUG:boto:Path: /vwnfns-13/dummy 2024-04-13T21:34:28.600 DEBUG:boto:Data: 2024-04-13T21:34:28.600 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:34:28.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:34:28.600 DEBUG:boto:Port: 8000 2024-04-13T21:34:28.600 DEBUG:boto:Params: {} 2024-04-13T21:34:28.600 DEBUG:boto:Token: None 2024-04-13T21:34:28.601 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:34:28 GMT /vwnfns-13/dummy 2024-04-13T21:34:28.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JPn/2hKRwHNLoV9HIK/1OribbPw= 2024-04-13T21:34:28.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:34:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JPn/2hKRwHNLoV9HIK/1OribbPw='} 2024-04-13T21:34:28.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-amz-request-id', 'tx00000a303acb880750b3b-00661afa64-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:28.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:34:28.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:34:28.613 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:34:28.668 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:34:28.668 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:34:28.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:28.709+0000 7f11fa7f1780 1 waiting to reach incremental sync.. 2024-04-13T21:34:33.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:33.710+0000 7f11fa7f1780 1 waiting to reach incremental sync.. 2024-04-13T21:34:38.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:38.711+0000 7f11fa7f1780 1 waiting to reach incremental sync.. 2024-04-13T21:34:43.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:43.712+0000 7f11fa7f1780 1 waiting to reach incremental sync.. 2024-04-13T21:34:48.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:48.714+0000 7f11fa7f1780 1 bucket sync caught up with source: 2024-04-13T21:34:48.706 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-04-13T21:34:48.706 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.17.11, , , , , ] 2024-04-13T21:34:48.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:48.714+0000 7f11fa7f1780 0 bucket checkpoint complete 2024-04-13T21:34:48.722 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:34:48.722 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:34:48.722 DEBUG:boto:Method: HEAD 2024-04-13T21:34:48.722 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:34:48.722 DEBUG:boto:Data: 2024-04-13T21:34:48.722 DEBUG:boto:Headers: {} 2024-04-13T21:34:48.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:34:48.722 DEBUG:boto:Port: 8001 2024-04-13T21:34:48.723 DEBUG:boto:Params: {} 2024-04-13T21:34:48.723 DEBUG:boto:Token: None 2024-04-13T21:34:48.723 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:34:48 GMT /vwnfns-14/ 2024-04-13T21:34:48.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wZuJLt8+ajd5Ql6BoS2oNQ0V9UU= 2024-04-13T21:34:48.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:34:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wZuJLt8+ajd5Ql6BoS2oNQ0V9UU=', 'Content-Length': '0'} 2024-04-13T21:34:48.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e58cec4977cafda-00661afa78-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:48.727 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:34:48.727 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:34:48.727 DEBUG:boto:Method: PUT 2024-04-13T21:34:48.727 DEBUG:boto:Path: /vwnfns-14/dummy 2024-04-13T21:34:48.727 DEBUG:boto:Data: 2024-04-13T21:34:48.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:34:48.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:34:48.727 DEBUG:boto:Port: 8001 2024-04-13T21:34:48.727 DEBUG:boto:Params: {} 2024-04-13T21:34:48.727 DEBUG:boto:Token: None 2024-04-13T21:34:48.728 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:34:48 GMT /vwnfns-14/dummy 2024-04-13T21:34:48.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Doubo5ELf7MwM55nP+N8w+ZOHWc= 2024-04-13T21:34:48.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:34:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Doubo5ELf7MwM55nP+N8w+ZOHWc='} 2024-04-13T21:34:48.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-amz-request-id', 'tx000007520b02f159cbaf8-00661afa78-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:34:48.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:34:48.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:34:48.740 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:34:48.800 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:34:48.800 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:34:48.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:48.844+0000 7ff1a3116780 1 waiting to reach incremental sync.. 2024-04-13T21:34:53.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:53.844+0000 7ff1a3116780 1 waiting to reach incremental sync.. 2024-04-13T21:34:58.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:34:58.844+0000 7ff1a3116780 1 waiting to reach incremental sync.. 2024-04-13T21:35:03.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:03.844+0000 7ff1a3116780 1 waiting to reach incremental sync.. 2024-04-13T21:35:08.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:08.846+0000 7ff1a3116780 1 bucket sync caught up with source: 2024-04-13T21:35:08.840 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-04-13T21:35:08.840 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.31.11, , , , , ] 2024-04-13T21:35:08.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:08.846+0000 7ff1a3116780 0 bucket checkpoint complete 2024-04-13T21:35:08.855 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:08.856 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:08.856 DEBUG:boto:Method: HEAD 2024-04-13T21:35:08.856 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:35:08.856 DEBUG:boto:Data: 2024-04-13T21:35:08.856 DEBUG:boto:Headers: {} 2024-04-13T21:35:08.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:08.856 DEBUG:boto:Port: 8000 2024-04-13T21:35:08.857 DEBUG:boto:Params: {} 2024-04-13T21:35:08.857 DEBUG:boto:Token: None 2024-04-13T21:35:08.857 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/ 2024-04-13T21:35:08.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3l0HMHan8EsR/qN+DtzVV+aABSY= 2024-04-13T21:35:08.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3l0HMHan8EsR/qN+DtzVV+aABSY=', 'Content-Length': '0'} 2024-04-13T21:35:08.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d32ec5fffb8154-00661afa8c-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.861 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.861 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.861 DEBUG:boto:Method: PUT 2024-04-13T21:35:08.861 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:35:08.861 DEBUG:boto:Data: 2024-04-13T21:35:08.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:08.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:08.862 DEBUG:boto:Port: 8000 2024-04-13T21:35:08.862 DEBUG:boto:Params: {} 2024-04-13T21:35:08.862 DEBUG:boto:Token: None 2024-04-13T21:35:08.862 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a1 2024-04-13T21:35:08.862 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vHbbvmNfAGYQnjXOPCRGDElQMdo= 2024-04-13T21:35:08.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vHbbvmNfAGYQnjXOPCRGDElQMdo='} 2024-04-13T21:35:08.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-amz-request-id', 'tx000009f5d62015a2787f1-00661afa8c-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.874 DEBUG:rgw_multi.tests:version1 id=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:08.874 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.874 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.874 DEBUG:boto:Method: PUT 2024-04-13T21:35:08.874 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:35:08.874 DEBUG:boto:Data: 2024-04-13T21:35:08.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:08.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:08.875 DEBUG:boto:Port: 8000 2024-04-13T21:35:08.875 DEBUG:boto:Params: {} 2024-04-13T21:35:08.875 DEBUG:boto:Token: None 2024-04-13T21:35:08.875 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a1 2024-04-13T21:35:08.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e/fcakpZVLXw3j3pbwCIvK5p0Y4= 2024-04-13T21:35:08.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e/fcakpZVLXw3j3pbwCIvK5p0Y4='} 2024-04-13T21:35:08.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT'), ('x-amz-request-id', 'tx0000045e046e9771f4a44-00661afa8c-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.919 DEBUG:rgw_multi.tests:version2 id=.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT 2024-04-13T21:35:08.919 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.919 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.919 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT 2024-04-13T21:35:08.919 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT 2024-04-13T21:35:08.919 DEBUG:boto:Method: DELETE 2024-04-13T21:35:08.919 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT 2024-04-13T21:35:08.919 DEBUG:boto:Data: 2024-04-13T21:35:08.920 DEBUG:boto:Headers: {} 2024-04-13T21:35:08.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:08.920 DEBUG:boto:Port: 8000 2024-04-13T21:35:08.920 DEBUG:boto:Params: {} 2024-04-13T21:35:08.920 DEBUG:boto:Token: None 2024-04-13T21:35:08.920 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a1?versionId=.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT 2024-04-13T21:35:08.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SPFQg1y+cKSjWUxQ0JdcDOGN5Dc= 2024-04-13T21:35:08.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SPFQg1y+cKSjWUxQ0JdcDOGN5Dc=', 'Content-Length': '0'} 2024-04-13T21:35:08.930 DEBUG:boto:Response headers: [('x-amz-version-id', '.V1WdqvOMUwbwwqMYxnF0nh8jDISeZT'), ('x-amz-request-id', 'tx000000892cfaf9ab721ac-00661afa8c-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.930 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.930 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.930 DEBUG:boto:Method: PUT 2024-04-13T21:35:08.930 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:35:08.930 DEBUG:boto:Data: 2024-04-13T21:35:08.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:08.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:08.931 DEBUG:boto:Port: 8000 2024-04-13T21:35:08.931 DEBUG:boto:Params: {} 2024-04-13T21:35:08.931 DEBUG:boto:Token: None 2024-04-13T21:35:08.931 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a1 2024-04-13T21:35:08.931 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O3zZQIOYlVmgljHD4WsyqLzGfoA= 2024-04-13T21:35:08.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O3zZQIOYlVmgljHD4WsyqLzGfoA='} 2024-04-13T21:35:08.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'd44nYx1YRLu.D2SL3U5P1-60B7s6FX6'), ('x-amz-request-id', 'tx0000038ef7f358353e3a9-00661afa8c-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.940 DEBUG:rgw_multi.tests:version3 id=d44nYx1YRLu.D2SL3U5P1-60B7s6FX6 2024-04-13T21:35:08.941 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.941 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:08.941 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=d44nYx1YRLu.D2SL3U5P1-60B7s6FX6 2024-04-13T21:35:08.941 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=d44nYx1YRLu.D2SL3U5P1-60B7s6FX6 2024-04-13T21:35:08.941 DEBUG:boto:Method: DELETE 2024-04-13T21:35:08.941 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=d44nYx1YRLu.D2SL3U5P1-60B7s6FX6 2024-04-13T21:35:08.941 DEBUG:boto:Data: 2024-04-13T21:35:08.941 DEBUG:boto:Headers: {} 2024-04-13T21:35:08.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:08.941 DEBUG:boto:Port: 8000 2024-04-13T21:35:08.941 DEBUG:boto:Params: {} 2024-04-13T21:35:08.942 DEBUG:boto:Token: None 2024-04-13T21:35:08.942 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a1?versionId=d44nYx1YRLu.D2SL3U5P1-60B7s6FX6 2024-04-13T21:35:08.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+LScVHM0lu3zLvlHJ18lqIHXH4k= 2024-04-13T21:35:08.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+LScVHM0lu3zLvlHJ18lqIHXH4k=', 'Content-Length': '0'} 2024-04-13T21:35:08.951 DEBUG:boto:Response headers: [('x-amz-version-id', 'd44nYx1YRLu.D2SL3U5P1-60B7s6FX6'), ('x-amz-request-id', 'tx000003ee9350b858f3d4c-00661afa8c-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.951 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:08.951 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:08.951 DEBUG:boto:Method: HEAD 2024-04-13T21:35:08.951 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:35:08.952 DEBUG:boto:Data: 2024-04-13T21:35:08.952 DEBUG:boto:Headers: {} 2024-04-13T21:35:08.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:08.952 DEBUG:boto:Port: 8001 2024-04-13T21:35:08.952 DEBUG:boto:Params: {} 2024-04-13T21:35:08.952 DEBUG:boto:Token: None 2024-04-13T21:35:08.952 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/ 2024-04-13T21:35:08.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3l0HMHan8EsR/qN+DtzVV+aABSY= 2024-04-13T21:35:08.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3l0HMHan8EsR/qN+DtzVV+aABSY=', 'Content-Length': '0'} 2024-04-13T21:35:08.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a8504337b9b574d-00661afa8c-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.955 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:08.955 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:08.955 DEBUG:boto:Method: PUT 2024-04-13T21:35:08.955 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:35:08.955 DEBUG:boto:Data: 2024-04-13T21:35:08.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:08.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:08.955 DEBUG:boto:Port: 8001 2024-04-13T21:35:08.955 DEBUG:boto:Params: {} 2024-04-13T21:35:08.955 DEBUG:boto:Token: None 2024-04-13T21:35:08.956 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a2 2024-04-13T21:35:08.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pjZoQUxbD33XfpiQJWxRagFMhj0= 2024-04-13T21:35:08.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pjZoQUxbD33XfpiQJWxRagFMhj0='} 2024-04-13T21:35:08.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-amz-request-id', 'tx00000bba6d2e79c287cbc-00661afa8c-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:08.968 DEBUG:rgw_multi.tests:version1 id=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:08.969 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:08.969 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:08.969 DEBUG:boto:Method: PUT 2024-04-13T21:35:08.969 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:35:08.969 DEBUG:boto:Data: 2024-04-13T21:35:08.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:08.969 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:08.969 DEBUG:boto:Port: 8001 2024-04-13T21:35:08.969 DEBUG:boto:Params: {} 2024-04-13T21:35:08.969 DEBUG:boto:Token: None 2024-04-13T21:35:08.970 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Sat, 13 Apr 2024 21:35:08 GMT /vwnfns-13/obj-a2 2024-04-13T21:35:08.970 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8TVjFaq37Ts5HBVnPt2iHCKIFPg= 2024-04-13T21:35:08.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8TVjFaq37Ts5HBVnPt2iHCKIFPg='} 2024-04-13T21:35:09.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e'), ('x-amz-request-id', 'tx000009b559a0234903719-00661afa8c-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.019 DEBUG:rgw_multi.tests:version2 id=KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e 2024-04-13T21:35:09.019 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:09.019 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:09.019 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e 2024-04-13T21:35:09.019 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e 2024-04-13T21:35:09.019 DEBUG:boto:Method: DELETE 2024-04-13T21:35:09.019 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e 2024-04-13T21:35:09.019 DEBUG:boto:Data: 2024-04-13T21:35:09.019 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.020 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.020 DEBUG:boto:Params: {} 2024-04-13T21:35:09.020 DEBUG:boto:Token: None 2024-04-13T21:35:09.020 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-13/obj-a2?versionId=KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e 2024-04-13T21:35:09.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R3r1cwWwaE/wiYLxlS9JqJ2nel4= 2024-04-13T21:35:09.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R3r1cwWwaE/wiYLxlS9JqJ2nel4=', 'Content-Length': '0'} 2024-04-13T21:35:09.028 DEBUG:boto:Response headers: [('x-amz-version-id', 'KoGD3r3CRYtVl6rWv3WNVKDX9GcYx9e'), ('x-amz-request-id', 'tx000009e3b8a43b4f93181-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.029 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:09.029 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:09.029 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.029 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:35:09.029 DEBUG:boto:Data: 2024-04-13T21:35:09.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.030 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.030 DEBUG:boto:Params: {} 2024-04-13T21:35:09.030 DEBUG:boto:Token: None 2024-04-13T21:35:09.030 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-13/obj-a2 2024-04-13T21:35:09.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bg9ZcwhYkDz6wCPqq/Ir6XAqt5k= 2024-04-13T21:35:09.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bg9ZcwhYkDz6wCPqq/Ir6XAqt5k='} 2024-04-13T21:35:09.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA'), ('x-amz-request-id', 'tx000004f4e8d319ef67cc0-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.039 DEBUG:rgw_multi.tests:version3 id=4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA 2024-04-13T21:35:09.039 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:09.039 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:09.039 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA 2024-04-13T21:35:09.039 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA 2024-04-13T21:35:09.039 DEBUG:boto:Method: DELETE 2024-04-13T21:35:09.039 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA 2024-04-13T21:35:09.039 DEBUG:boto:Data: 2024-04-13T21:35:09.039 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.040 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.040 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.040 DEBUG:boto:Params: {} 2024-04-13T21:35:09.040 DEBUG:boto:Token: None 2024-04-13T21:35:09.040 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-13/obj-a2?versionId=4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA 2024-04-13T21:35:09.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:akWlKti1Ycz6dNBGUD/a1FI8XK4= 2024-04-13T21:35:09.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:akWlKti1Ycz6dNBGUD/a1FI8XK4=', 'Content-Length': '0'} 2024-04-13T21:35:09.049 DEBUG:boto:Response headers: [('x-amz-version-id', '4U6IXyyJP5C4h2Qy6V0KBl2cYLlcOmA'), ('x-amz-request-id', 'tx00000dc5d50587319f59b-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.050 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:09.050 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:09.050 DEBUG:boto:Method: HEAD 2024-04-13T21:35:09.050 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:35:09.051 DEBUG:boto:Data: 2024-04-13T21:35:09.051 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:09.052 DEBUG:boto:Port: 8000 2024-04-13T21:35:09.052 DEBUG:boto:Params: {} 2024-04-13T21:35:09.053 DEBUG:boto:Token: None 2024-04-13T21:35:09.053 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/ 2024-04-13T21:35:09.053 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:asDBpFL2TSONwDaxzupJ8TqL01o= 2024-04-13T21:35:09.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:asDBpFL2TSONwDaxzupJ8TqL01o=', 'Content-Length': '0'} 2024-04-13T21:35:09.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b753071b7aa5e9ae-00661afa8d-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.056 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.056 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.056 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.056 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:35:09.057 DEBUG:boto:Data: 2024-04-13T21:35:09.057 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:09.057 DEBUG:boto:Port: 8000 2024-04-13T21:35:09.057 DEBUG:boto:Params: {} 2024-04-13T21:35:09.058 DEBUG:boto:Token: None 2024-04-13T21:35:09.058 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a1 2024-04-13T21:35:09.058 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LqujEoRAKEyDcH/GET9L8zLNBUc= 2024-04-13T21:35:09.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LqujEoRAKEyDcH/GET9L8zLNBUc='} 2024-04-13T21:35:09.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-amz-request-id', 'tx0000013a66b7a571b16b7-00661afa8d-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.069 DEBUG:rgw_multi.tests:version1 id=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:09.069 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.070 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.070 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.070 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:35:09.070 DEBUG:boto:Data: 2024-04-13T21:35:09.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.070 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:09.070 DEBUG:boto:Port: 8000 2024-04-13T21:35:09.070 DEBUG:boto:Params: {} 2024-04-13T21:35:09.071 DEBUG:boto:Token: None 2024-04-13T21:35:09.071 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a1 2024-04-13T21:35:09.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+nyRvHzvD5Ekfq/XdnwDxBv4o4M= 2024-04-13T21:35:09.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+nyRvHzvD5Ekfq/XdnwDxBv4o4M='} 2024-04-13T21:35:09.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OnHl2wHwFf8iediskXf54s-.vSFi9Gf'), ('x-amz-request-id', 'tx00000ee238fc608dda7a3-00661afa8d-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.082 DEBUG:rgw_multi.tests:version2 id=OnHl2wHwFf8iediskXf54s-.vSFi9Gf 2024-04-13T21:35:09.082 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.082 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.082 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=OnHl2wHwFf8iediskXf54s-.vSFi9Gf 2024-04-13T21:35:09.082 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=OnHl2wHwFf8iediskXf54s-.vSFi9Gf 2024-04-13T21:35:09.083 DEBUG:boto:Method: DELETE 2024-04-13T21:35:09.083 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=OnHl2wHwFf8iediskXf54s-.vSFi9Gf 2024-04-13T21:35:09.083 DEBUG:boto:Data: 2024-04-13T21:35:09.083 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:09.083 DEBUG:boto:Port: 8000 2024-04-13T21:35:09.083 DEBUG:boto:Params: {} 2024-04-13T21:35:09.083 DEBUG:boto:Token: None 2024-04-13T21:35:09.084 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a1?versionId=OnHl2wHwFf8iediskXf54s-.vSFi9Gf 2024-04-13T21:35:09.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dgj6LswqoRFTHlOcf6tFef3I7Ps= 2024-04-13T21:35:09.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dgj6LswqoRFTHlOcf6tFef3I7Ps=', 'Content-Length': '0'} 2024-04-13T21:35:09.092 DEBUG:boto:Response headers: [('x-amz-version-id', 'OnHl2wHwFf8iediskXf54s-.vSFi9Gf'), ('x-amz-request-id', 'tx000004179307b70d83e4e-00661afa8d-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.093 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.093 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.093 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.093 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:35:09.093 DEBUG:boto:Data: 2024-04-13T21:35:09.093 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.093 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:09.093 DEBUG:boto:Port: 8000 2024-04-13T21:35:09.093 DEBUG:boto:Params: {} 2024-04-13T21:35:09.093 DEBUG:boto:Token: None 2024-04-13T21:35:09.094 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a1 2024-04-13T21:35:09.094 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EbcJjTO35mygLaChB1Vrlr96xzw= 2024-04-13T21:35:09.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EbcJjTO35mygLaChB1Vrlr96xzw='} 2024-04-13T21:35:09.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde'), ('x-amz-request-id', 'tx0000069f88e83d2c83173-00661afa8d-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.139 DEBUG:rgw_multi.tests:version3 id=w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde 2024-04-13T21:35:09.139 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.139 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:09.139 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde 2024-04-13T21:35:09.139 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde 2024-04-13T21:35:09.139 DEBUG:boto:Method: DELETE 2024-04-13T21:35:09.140 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde 2024-04-13T21:35:09.140 DEBUG:boto:Data: 2024-04-13T21:35:09.140 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:09.140 DEBUG:boto:Port: 8000 2024-04-13T21:35:09.140 DEBUG:boto:Params: {} 2024-04-13T21:35:09.140 DEBUG:boto:Token: None 2024-04-13T21:35:09.140 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a1?versionId=w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde 2024-04-13T21:35:09.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UIZfvk2RHH/PgwWkWvMKw9byqYI= 2024-04-13T21:35:09.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UIZfvk2RHH/PgwWkWvMKw9byqYI=', 'Content-Length': '0'} 2024-04-13T21:35:09.149 DEBUG:boto:Response headers: [('x-amz-version-id', 'w9vVHRuTrNkxX4ZqUozL3bsfOjWGvde'), ('x-amz-request-id', 'tx000004c73648225cae897-00661afa8d-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.150 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:09.150 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:09.150 DEBUG:boto:Method: HEAD 2024-04-13T21:35:09.150 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:35:09.150 DEBUG:boto:Data: 2024-04-13T21:35:09.151 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.151 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.151 DEBUG:boto:Params: {} 2024-04-13T21:35:09.151 DEBUG:boto:Token: None 2024-04-13T21:35:09.151 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/ 2024-04-13T21:35:09.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:asDBpFL2TSONwDaxzupJ8TqL01o= 2024-04-13T21:35:09.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:asDBpFL2TSONwDaxzupJ8TqL01o=', 'Content-Length': '0'} 2024-04-13T21:35:09.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df89aa889012dc62-00661afa8d-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.154 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.154 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.154 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.154 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:35:09.155 DEBUG:boto:Data: 2024-04-13T21:35:09.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.155 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.155 DEBUG:boto:Params: {} 2024-04-13T21:35:09.155 DEBUG:boto:Token: None 2024-04-13T21:35:09.155 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a2 2024-04-13T21:35:09.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+91P+yX5ffMnCy74OoM17F4VfQQ= 2024-04-13T21:35:09.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+91P+yX5ffMnCy74OoM17F4VfQQ='} 2024-04-13T21:35:09.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-amz-request-id', 'tx00000135b7827e6ed8911-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.164 DEBUG:rgw_multi.tests:version1 id=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:09.165 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.165 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.165 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.165 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:35:09.165 DEBUG:boto:Data: 2024-04-13T21:35:09.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.165 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.165 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.165 DEBUG:boto:Params: {} 2024-04-13T21:35:09.165 DEBUG:boto:Token: None 2024-04-13T21:35:09.165 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a2 2024-04-13T21:35:09.166 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wJ5FjvuOCypi+keqorsFWo+XJ3o= 2024-04-13T21:35:09.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wJ5FjvuOCypi+keqorsFWo+XJ3o='} 2024-04-13T21:35:09.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr'), ('x-amz-request-id', 'tx000002d22a238c5067f6c-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.173 DEBUG:rgw_multi.tests:version2 id=YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr 2024-04-13T21:35:09.173 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.173 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.173 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr 2024-04-13T21:35:09.174 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr 2024-04-13T21:35:09.174 DEBUG:boto:Method: DELETE 2024-04-13T21:35:09.174 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr 2024-04-13T21:35:09.174 DEBUG:boto:Data: 2024-04-13T21:35:09.174 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.174 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.174 DEBUG:boto:Params: {} 2024-04-13T21:35:09.174 DEBUG:boto:Token: None 2024-04-13T21:35:09.175 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a2?versionId=YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr 2024-04-13T21:35:09.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hsiJDhAxjIt6pixX45j5gJ0KrQY= 2024-04-13T21:35:09.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hsiJDhAxjIt6pixX45j5gJ0KrQY=', 'Content-Length': '0'} 2024-04-13T21:35:09.183 DEBUG:boto:Response headers: [('x-amz-version-id', 'YjpD2RjMVDRtrbD1bsSGzWBPjXyPzRr'), ('x-amz-request-id', 'tx000000ff6f634fd609c58-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.183 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.183 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.183 DEBUG:boto:Method: PUT 2024-04-13T21:35:09.183 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:35:09.184 DEBUG:boto:Data: 2024-04-13T21:35:09.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:35:09.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.184 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.184 DEBUG:boto:Params: {} 2024-04-13T21:35:09.184 DEBUG:boto:Token: None 2024-04-13T21:35:09.184 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a2 2024-04-13T21:35:09.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yozBT4zakROeJqsChm2kkxATS6s= 2024-04-13T21:35:09.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yozBT4zakROeJqsChm2kkxATS6s='} 2024-04-13T21:35:09.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd'), ('x-amz-request-id', 'tx000003c701b48565ebdbf-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.227 DEBUG:rgw_multi.tests:version3 id=BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd 2024-04-13T21:35:09.227 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.227 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:09.227 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd 2024-04-13T21:35:09.227 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd 2024-04-13T21:35:09.227 DEBUG:boto:Method: DELETE 2024-04-13T21:35:09.227 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd 2024-04-13T21:35:09.227 DEBUG:boto:Data: 2024-04-13T21:35:09.228 DEBUG:boto:Headers: {} 2024-04-13T21:35:09.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:09.228 DEBUG:boto:Port: 8001 2024-04-13T21:35:09.229 DEBUG:boto:Params: {} 2024-04-13T21:35:09.229 DEBUG:boto:Token: None 2024-04-13T21:35:09.229 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:35:09 GMT /vwnfns-14/obj-a2?versionId=BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd 2024-04-13T21:35:09.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Otq/FwJDEqR9zMC8p5PZg9L7euA= 2024-04-13T21:35:09.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Otq/FwJDEqR9zMC8p5PZg9L7euA=', 'Content-Length': '0'} 2024-04-13T21:35:09.238 DEBUG:boto:Response headers: [('x-amz-version-id', 'BIz6QF9ExfKtIpQx8ujfpr3a6Qf81pd'), ('x-amz-request-id', 'tx0000018552275193e2f6d-00661afa8d-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:09.238 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-13 2024-04-13T21:35:09.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:09.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:09.239 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:35:09.294 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:35:09.295 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:35:09.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:09.339+0000 7f7d0ce1a780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:09.332 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-04-13T21:35:09.332 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.39.6, ] 2024-04-13T21:35:14.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:14.340+0000 7f7d0ce1a780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:14.333 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-04-13T21:35:14.333 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.39.6, ] 2024-04-13T21:35:19.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:19.342+0000 7f7d0ce1a780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:19.335 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-04-13T21:35:19.335 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.39.6, ] 2024-04-13T21:35:24.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:24.344+0000 7f7d0ce1a780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:24.336 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-04-13T21:35:24.336 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.39.6, ] 2024-04-13T21:35:29.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:29.345+0000 7f7d0ce1a780 1 bucket sync caught up with source: 2024-04-13T21:35:29.337 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000010.39.6, ] 2024-04-13T21:35:29.337 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.39.6, ] 2024-04-13T21:35:29.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:29.345+0000 7f7d0ce1a780 0 bucket checkpoint complete 2024-04-13T21:35:29.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-13 2024-04-13T21:35:29.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:29.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:29.355 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:35:29.416 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:35:29.416 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:35:29.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:29.462+0000 7f48c7554780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:29.454 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000010.40.6, , ] 2024-04-13T21:35:29.454 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:35:34.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:34.462+0000 7f48c7554780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:34.456 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000010.40.6, , ] 2024-04-13T21:35:34.456 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:35:39.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:39.463+0000 7f48c7554780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:39.457 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000010.40.6, , ] 2024-04-13T21:35:39.457 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:35:44.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:44.465+0000 7f48c7554780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:44.458 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000010.40.6, , ] 2024-04-13T21:35:44.458 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:35:49.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:49.466+0000 7f48c7554780 1 bucket sync caught up with source: 2024-04-13T21:35:49.459 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:35:49.459 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:35:49.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:49.466+0000 7f48c7554780 0 bucket checkpoint complete 2024-04-13T21:35:49.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-13 zones={a2, a1} 2024-04-13T21:35:49.475 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.475 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.475 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.475 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:35:49.475 DEBUG:boto:Data: 2024-04-13T21:35:49.476 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.476 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.476 DEBUG:boto:Params: {} 2024-04-13T21:35:49.476 DEBUG:boto:Token: None 2024-04-13T21:35:49.476 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iq+hqvgmKAiAJBC/xcHmfGhgpj4= 2024-04-13T21:35:49.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iq+hqvgmKAiAJBC/xcHmfGhgpj4=', 'Content-Length': '0'} 2024-04-13T21:35:49.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e4c31819f5df16f-00661afab5-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.479 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.480 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.480 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.480 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:35:49.480 DEBUG:boto:Data: 2024-04-13T21:35:49.480 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.480 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.480 DEBUG:boto:Params: {} 2024-04-13T21:35:49.480 DEBUG:boto:Token: None 2024-04-13T21:35:49.480 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.481 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iq+hqvgmKAiAJBC/xcHmfGhgpj4= 2024-04-13T21:35:49.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iq+hqvgmKAiAJBC/xcHmfGhgpj4=', 'Content-Length': '0'} 2024-04-13T21:35:49.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d3abd1f01c3f4c7-00661afab5-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:35:49.483 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.483 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.483 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.483 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.483 DEBUG:boto:Method: GET 2024-04-13T21:35:49.483 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.483 DEBUG:boto:Data: 2024-04-13T21:35:49.484 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.484 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.484 DEBUG:boto:Params: {} 2024-04-13T21:35:49.484 DEBUG:boto:Token: None 2024-04-13T21:35:49.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/?versions 2024-04-13T21:35:49.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M= 2024-04-13T21:35:49.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M=', 'Content-Length': '0'} 2024-04-13T21:35:49.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009376731abd26799b-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.487 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.487 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:35:49.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:35:49.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:35:49.488 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:35:49.488 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.488 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.488 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.488 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.488 DEBUG:boto:Method: GET 2024-04-13T21:35:49.488 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.488 DEBUG:boto:Data: 2024-04-13T21:35:49.489 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.489 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.489 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.489 DEBUG:boto:Params: {} 2024-04-13T21:35:49.489 DEBUG:boto:Token: None 2024-04-13T21:35:49.489 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/?versions 2024-04-13T21:35:49.490 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M= 2024-04-13T21:35:49.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M=', 'Content-Length': '0'} 2024-04-13T21:35:49.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b516bbaafa9d875e-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.492 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.493 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:35:49.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:35:49.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:35:49.493 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.493 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.493 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.493 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.493 DEBUG:boto:Method: GET 2024-04-13T21:35:49.493 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.493 DEBUG:boto:Data: 2024-04-13T21:35:49.494 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.494 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.494 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.494 DEBUG:boto:Params: {} 2024-04-13T21:35:49.494 DEBUG:boto:Token: None 2024-04-13T21:35:49.494 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/?versions 2024-04-13T21:35:49.494 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M= 2024-04-13T21:35:49.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M=', 'Content-Length': '0'} 2024-04-13T21:35:49.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008caa1ee736ff7f99-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.497 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.497 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.497 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.498 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.498 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.498 DEBUG:boto:Method: GET 2024-04-13T21:35:49.498 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:35:49.498 DEBUG:boto:Data: 2024-04-13T21:35:49.499 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.499 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.499 DEBUG:boto:Params: {} 2024-04-13T21:35:49.499 DEBUG:boto:Token: None 2024-04-13T21:35:49.500 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/?versions 2024-04-13T21:35:49.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M= 2024-04-13T21:35:49.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:THaIFdqDQmfm0n1XtaZduXv3z3M=', 'Content-Length': '0'} 2024-04-13T21:35:49.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005127ff046d105ea4-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.503 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:35:49.504 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.504 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.504 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.504 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.505 DEBUG:boto:Method: GET 2024-04-13T21:35:49.505 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.505 DEBUG:boto:Data: 2024-04-13T21:35:49.505 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.506 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.506 DEBUG:boto:Params: {} 2024-04-13T21:35:49.506 DEBUG:boto:Token: None 2024-04-13T21:35:49.506 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk= 2024-04-13T21:35:49.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk=', 'Content-Length': '0'} 2024-04-13T21:35:49.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000064910000c4ebf10a-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.509 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.509 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.509 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.510 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.510 DEBUG:boto:Method: GET 2024-04-13T21:35:49.510 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.510 DEBUG:boto:Data: 2024-04-13T21:35:49.510 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.510 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.510 DEBUG:boto:Params: {} 2024-04-13T21:35:49.510 DEBUG:boto:Token: None 2024-04-13T21:35:49.510 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk= 2024-04-13T21:35:49.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk=', 'Content-Length': '0'} 2024-04-13T21:35:49.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e08daff7ea043c82-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.513 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.513 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.513 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.513 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.513 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.513 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.513 DEBUG:boto:Data: 2024-04-13T21:35:49.513 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.514 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.514 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.514 DEBUG:boto:Params: {} 2024-04-13T21:35:49.514 DEBUG:boto:Token: None 2024-04-13T21:35:49.514 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.514 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sla7KYWerxUJQYfpVo6LxJ3ujMU= 2024-04-13T21:35:49.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sla7KYWerxUJQYfpVo6LxJ3ujMU=', 'Content-Length': '0'} 2024-04-13T21:35:49.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000098fe68f86943e9ce-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.516 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.516 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.516 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.517 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.517 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.517 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.517 DEBUG:boto:Data: 2024-04-13T21:35:49.517 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.517 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.517 DEBUG:boto:Params: {} 2024-04-13T21:35:49.517 DEBUG:boto:Token: None 2024-04-13T21:35:49.517 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sla7KYWerxUJQYfpVo6LxJ3ujMU= 2024-04-13T21:35:49.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sla7KYWerxUJQYfpVo6LxJ3ujMU=', 'Content-Length': '0'} 2024-04-13T21:35:49.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020f00b45db2877ab-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:35:49.519 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.519 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.520 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.520 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.520 DEBUG:boto:Method: GET 2024-04-13T21:35:49.520 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.520 DEBUG:boto:Data: 2024-04-13T21:35:49.520 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.520 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.520 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.520 DEBUG:boto:Params: {} 2024-04-13T21:35:49.520 DEBUG:boto:Token: None 2024-04-13T21:35:49.521 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk= 2024-04-13T21:35:49.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk=', 'Content-Length': '0'} 2024-04-13T21:35:49.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db3d3a613496360e-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.523 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.523 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.523 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.523 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.523 DEBUG:boto:Method: GET 2024-04-13T21:35:49.523 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.523 DEBUG:boto:Data: 2024-04-13T21:35:49.523 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.524 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.524 DEBUG:boto:Params: {} 2024-04-13T21:35:49.524 DEBUG:boto:Token: None 2024-04-13T21:35:49.524 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk= 2024-04-13T21:35:49.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk=', 'Content-Length': '0'} 2024-04-13T21:35:49.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebfa030825d5bc7f-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.526 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.526 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.526 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.526 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.526 DEBUG:boto:Method: GET 2024-04-13T21:35:49.527 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.527 DEBUG:boto:Data: 2024-04-13T21:35:49.527 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.527 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.527 DEBUG:boto:Params: {} 2024-04-13T21:35:49.527 DEBUG:boto:Token: None 2024-04-13T21:35:49.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96f846c2b039edb-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.570 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.571 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.571 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.571 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.571 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.571 DEBUG:boto:Method: GET 2024-04-13T21:35:49.571 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.572 DEBUG:boto:Data: 2024-04-13T21:35:49.572 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.572 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.572 DEBUG:boto:Params: {} 2024-04-13T21:35:49.572 DEBUG:boto:Token: None 2024-04-13T21:35:49.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f414f8e2496fa579-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.575 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.575 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.575 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.575 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.575 DEBUG:boto:Path: /vwnfns-13/dummy 2024-04-13T21:35:49.576 DEBUG:boto:Data: 2024-04-13T21:35:49.576 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.576 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.576 DEBUG:boto:Params: {} 2024-04-13T21:35:49.576 DEBUG:boto:Token: None 2024-04-13T21:35:49.576 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy 2024-04-13T21:35:49.577 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7agbJhO1u9jmQJQHJblG6yWEGlU= 2024-04-13T21:35:49.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7agbJhO1u9jmQJQHJblG6yWEGlU=', 'Content-Length': '0'} 2024-04-13T21:35:49.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f72f867a45915e98-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.579 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.579 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.579 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.579 DEBUG:boto:Path: /vwnfns-13/dummy 2024-04-13T21:35:49.579 DEBUG:boto:Data: 2024-04-13T21:35:49.579 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.579 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.579 DEBUG:boto:Params: {} 2024-04-13T21:35:49.579 DEBUG:boto:Token: None 2024-04-13T21:35:49.580 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy 2024-04-13T21:35:49.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7agbJhO1u9jmQJQHJblG6yWEGlU= 2024-04-13T21:35:49.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7agbJhO1u9jmQJQHJblG6yWEGlU=', 'Content-Length': '0'} 2024-04-13T21:35:49.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f45ddc18aae2adf-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:35:49.582 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.582 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.582 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.582 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.583 DEBUG:boto:Method: GET 2024-04-13T21:35:49.583 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.583 DEBUG:boto:Data: 2024-04-13T21:35:49.583 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.583 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.583 DEBUG:boto:Params: {} 2024-04-13T21:35:49.583 DEBUG:boto:Token: None 2024-04-13T21:35:49.583 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk= 2024-04-13T21:35:49.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk=', 'Content-Length': '0'} 2024-04-13T21:35:49.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de286612e3ecaedb-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.586 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:35:49.586 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:35:49.586 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.586 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.586 DEBUG:boto:Method: GET 2024-04-13T21:35:49.586 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.586 DEBUG:boto:Data: 2024-04-13T21:35:49.586 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.586 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.587 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.587 DEBUG:boto:Params: {} 2024-04-13T21:35:49.587 DEBUG:boto:Token: None 2024-04-13T21:35:49.587 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:35:49.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk= 2024-04-13T21:35:49.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJ5ICt9Vc8RlzMVAr5Ne31tqNnk=', 'Content-Length': '0'} 2024-04-13T21:35:49.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1f387ce4ee75509-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.589 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.589 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.589 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.589 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.589 DEBUG:boto:Method: GET 2024-04-13T21:35:49.589 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.589 DEBUG:boto:Data: 2024-04-13T21:35:49.590 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.590 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.590 DEBUG:boto:Params: {} 2024-04-13T21:35:49.590 DEBUG:boto:Token: None 2024-04-13T21:35:49.590 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.590 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091299a624009686a-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.592 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.593 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.593 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.593 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.593 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.593 DEBUG:boto:Method: GET 2024-04-13T21:35:49.593 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:35:49.593 DEBUG:boto:Data: 2024-04-13T21:35:49.593 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.594 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.594 DEBUG:boto:Params: {} 2024-04-13T21:35:49.594 DEBUG:boto:Token: None 2024-04-13T21:35:49.594 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a84d5692b9aab7ab-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.596 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:35:49.597 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.597 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.597 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.597 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.597 DEBUG:boto:Method: GET 2024-04-13T21:35:49.597 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.597 DEBUG:boto:Data: 2024-04-13T21:35:49.597 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.597 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.598 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.598 DEBUG:boto:Params: {} 2024-04-13T21:35:49.598 DEBUG:boto:Token: None 2024-04-13T21:35:49.598 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.598 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc= 2024-04-13T21:35:49.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc=', 'Content-Length': '0'} 2024-04-13T21:35:49.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002db9346afd5ea75-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.600 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.600 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.600 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.600 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.601 DEBUG:boto:Method: GET 2024-04-13T21:35:49.601 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.601 DEBUG:boto:Data: 2024-04-13T21:35:49.601 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.601 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.601 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.601 DEBUG:boto:Params: {} 2024-04-13T21:35:49.601 DEBUG:boto:Token: None 2024-04-13T21:35:49.601 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc= 2024-04-13T21:35:49.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc=', 'Content-Length': '0'} 2024-04-13T21:35:49.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008323264ce4c489aa-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.603 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.603 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.603 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.603 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.604 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.604 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.604 DEBUG:boto:Data: 2024-04-13T21:35:49.604 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.604 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.604 DEBUG:boto:Params: {} 2024-04-13T21:35:49.604 DEBUG:boto:Token: None 2024-04-13T21:35:49.604 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LEnEqFiwiFp/FwOMypj8Yhs2DZM= 2024-04-13T21:35:49.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LEnEqFiwiFp/FwOMypj8Yhs2DZM=', 'Content-Length': '0'} 2024-04-13T21:35:49.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4b86adcfc01b7a0-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.606 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.606 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.606 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.607 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.607 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.607 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.607 DEBUG:boto:Data: 2024-04-13T21:35:49.607 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.607 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.607 DEBUG:boto:Params: {} 2024-04-13T21:35:49.607 DEBUG:boto:Token: None 2024-04-13T21:35:49.607 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LEnEqFiwiFp/FwOMypj8Yhs2DZM= 2024-04-13T21:35:49.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LEnEqFiwiFp/FwOMypj8Yhs2DZM=', 'Content-Length': '0'} 2024-04-13T21:35:49.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071468120d276f2f9-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:35:49.609 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.609 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.610 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.610 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.610 DEBUG:boto:Method: GET 2024-04-13T21:35:49.610 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.610 DEBUG:boto:Data: 2024-04-13T21:35:49.610 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.610 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.610 DEBUG:boto:Params: {} 2024-04-13T21:35:49.610 DEBUG:boto:Token: None 2024-04-13T21:35:49.611 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc= 2024-04-13T21:35:49.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc=', 'Content-Length': '0'} 2024-04-13T21:35:49.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000037d73b8b47b220f-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.654 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.655 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.655 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.655 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.655 DEBUG:boto:Method: GET 2024-04-13T21:35:49.655 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.655 DEBUG:boto:Data: 2024-04-13T21:35:49.655 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.655 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.655 DEBUG:boto:Params: {} 2024-04-13T21:35:49.656 DEBUG:boto:Token: None 2024-04-13T21:35:49.656 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc= 2024-04-13T21:35:49.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc=', 'Content-Length': '0'} 2024-04-13T21:35:49.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000908983eb2643c2ba-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.658 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.658 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.658 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.658 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.658 DEBUG:boto:Method: GET 2024-04-13T21:35:49.659 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.659 DEBUG:boto:Data: 2024-04-13T21:35:49.659 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.659 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.659 DEBUG:boto:Params: {} 2024-04-13T21:35:49.659 DEBUG:boto:Token: None 2024-04-13T21:35:49.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5a504fb7342eab5-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.702 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.703 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.703 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.703 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.703 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.703 DEBUG:boto:Method: GET 2024-04-13T21:35:49.703 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.703 DEBUG:boto:Data: 2024-04-13T21:35:49.703 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.704 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.704 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.704 DEBUG:boto:Params: {} 2024-04-13T21:35:49.704 DEBUG:boto:Token: None 2024-04-13T21:35:49.704 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006193144a77e9334-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.707 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.707 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.707 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.708 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.708 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:35:49.708 DEBUG:boto:Data: 2024-04-13T21:35:49.708 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.708 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.708 DEBUG:boto:Params: {} 2024-04-13T21:35:49.708 DEBUG:boto:Token: None 2024-04-13T21:35:49.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1 2024-04-13T21:35:49.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xU9cMvelIgORGiyz2UFfJkntKcg= 2024-04-13T21:35:49.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xU9cMvelIgORGiyz2UFfJkntKcg=', 'Content-Length': '0'} 2024-04-13T21:35:49.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d54fd4d783040a3-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.711 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.711 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.711 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.711 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:35:49.711 DEBUG:boto:Data: 2024-04-13T21:35:49.712 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.712 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.712 DEBUG:boto:Params: {} 2024-04-13T21:35:49.712 DEBUG:boto:Token: None 2024-04-13T21:35:49.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1 2024-04-13T21:35:49.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xU9cMvelIgORGiyz2UFfJkntKcg= 2024-04-13T21:35:49.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xU9cMvelIgORGiyz2UFfJkntKcg=', 'Content-Length': '0'} 2024-04-13T21:35:49.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000151ec281d86d16e4-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:35:49.714 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.714 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.715 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.715 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.715 DEBUG:boto:Method: GET 2024-04-13T21:35:49.715 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.715 DEBUG:boto:Data: 2024-04-13T21:35:49.715 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.715 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.715 DEBUG:boto:Params: {} 2024-04-13T21:35:49.715 DEBUG:boto:Token: None 2024-04-13T21:35:49.716 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc= 2024-04-13T21:35:49.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc=', 'Content-Length': '0'} 2024-04-13T21:35:49.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e67a4f9eb61cdddc-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.758 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.759 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:49.759 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.759 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.759 DEBUG:boto:Method: GET 2024-04-13T21:35:49.759 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.759 DEBUG:boto:Data: 2024-04-13T21:35:49.759 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.759 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.759 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.760 DEBUG:boto:Params: {} 2024-04-13T21:35:49.760 DEBUG:boto:Token: None 2024-04-13T21:35:49.760 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:35:49.760 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc= 2024-04-13T21:35:49.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8i4iA3g2RgK0N4HPaFYIPLFwdbc=', 'Content-Length': '0'} 2024-04-13T21:35:49.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000187188e729e06424-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.762 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.763 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.763 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.763 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.763 DEBUG:boto:Method: GET 2024-04-13T21:35:49.763 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.763 DEBUG:boto:Data: 2024-04-13T21:35:49.763 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.763 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.763 DEBUG:boto:Params: {} 2024-04-13T21:35:49.764 DEBUG:boto:Token: None 2024-04-13T21:35:49.764 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078e3f41a17cfa1c2-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.810 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.811 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.811 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.811 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.811 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.811 DEBUG:boto:Method: GET 2024-04-13T21:35:49.811 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:35:49.811 DEBUG:boto:Data: 2024-04-13T21:35:49.811 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.812 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.812 DEBUG:boto:Params: {} 2024-04-13T21:35:49.812 DEBUG:boto:Token: None 2024-04-13T21:35:49.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b53b15269ee997b6-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.815 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:35:49.815 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.815 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.815 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.815 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.816 DEBUG:boto:Method: GET 2024-04-13T21:35:49.816 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.816 DEBUG:boto:Data: 2024-04-13T21:35:49.816 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.816 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.816 DEBUG:boto:Params: {} 2024-04-13T21:35:49.816 DEBUG:boto:Token: None 2024-04-13T21:35:49.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY= 2024-04-13T21:35:49.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY=', 'Content-Length': '0'} 2024-04-13T21:35:49.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b8537b9c007bea6-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.819 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.819 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.819 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.819 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.819 DEBUG:boto:Method: GET 2024-04-13T21:35:49.820 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.820 DEBUG:boto:Data: 2024-04-13T21:35:49.820 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.820 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.820 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.820 DEBUG:boto:Params: {} 2024-04-13T21:35:49.820 DEBUG:boto:Token: None 2024-04-13T21:35:49.820 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.821 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY= 2024-04-13T21:35:49.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY=', 'Content-Length': '0'} 2024-04-13T21:35:49.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1f26841db6f6154-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.823 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.823 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.823 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.823 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.823 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.824 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.824 DEBUG:boto:Data: 2024-04-13T21:35:49.824 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.824 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.824 DEBUG:boto:Params: {} 2024-04-13T21:35:49.824 DEBUG:boto:Token: None 2024-04-13T21:35:49.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zEq26lFIV3MsRNTelyCm78zEIF4= 2024-04-13T21:35:49.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zEq26lFIV3MsRNTelyCm78zEIF4=', 'Content-Length': '0'} 2024-04-13T21:35:49.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005125049de6118980-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.826 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.826 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.826 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.827 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.827 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.827 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.827 DEBUG:boto:Data: 2024-04-13T21:35:49.827 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.827 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.827 DEBUG:boto:Params: {} 2024-04-13T21:35:49.827 DEBUG:boto:Token: None 2024-04-13T21:35:49.827 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zEq26lFIV3MsRNTelyCm78zEIF4= 2024-04-13T21:35:49.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zEq26lFIV3MsRNTelyCm78zEIF4=', 'Content-Length': '0'} 2024-04-13T21:35:49.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f24444dbf6fc463b-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:35:49.830 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.830 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.830 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.830 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.830 DEBUG:boto:Method: GET 2024-04-13T21:35:49.830 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.830 DEBUG:boto:Data: 2024-04-13T21:35:49.830 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.830 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.831 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.831 DEBUG:boto:Params: {} 2024-04-13T21:35:49.831 DEBUG:boto:Token: None 2024-04-13T21:35:49.831 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.831 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY= 2024-04-13T21:35:49.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY=', 'Content-Length': '0'} 2024-04-13T21:35:49.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000706054d2c7423981-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.833 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.833 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.834 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.834 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.834 DEBUG:boto:Method: GET 2024-04-13T21:35:49.834 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.834 DEBUG:boto:Data: 2024-04-13T21:35:49.834 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.834 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.834 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.834 DEBUG:boto:Params: {} 2024-04-13T21:35:49.834 DEBUG:boto:Token: None 2024-04-13T21:35:49.835 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.835 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY= 2024-04-13T21:35:49.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY=', 'Content-Length': '0'} 2024-04-13T21:35:49.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6dc33a3ca174649-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.878 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.879 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.879 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.879 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.879 DEBUG:boto:Method: GET 2024-04-13T21:35:49.879 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.879 DEBUG:boto:Data: 2024-04-13T21:35:49.879 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.879 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.879 DEBUG:boto:Params: {} 2024-04-13T21:35:49.880 DEBUG:boto:Token: None 2024-04-13T21:35:49.880 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9fd5509e5c17edb-00661afab5-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.883 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.883 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.883 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.883 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.883 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.883 DEBUG:boto:Method: GET 2024-04-13T21:35:49.883 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.883 DEBUG:boto:Data: 2024-04-13T21:35:49.884 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.884 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.884 DEBUG:boto:Params: {} 2024-04-13T21:35:49.884 DEBUG:boto:Token: None 2024-04-13T21:35:49.884 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081acfbfb80f4d9c1-00661afab5-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.931 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.931 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.931 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.931 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.931 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:35:49.932 DEBUG:boto:Data: 2024-04-13T21:35:49.932 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.932 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.932 DEBUG:boto:Params: {} 2024-04-13T21:35:49.932 DEBUG:boto:Token: None 2024-04-13T21:35:49.933 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2 2024-04-13T21:35:49.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2QxowdpOrBdWhwpDuNxv7AAtVbw= 2024-04-13T21:35:49.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2QxowdpOrBdWhwpDuNxv7AAtVbw=', 'Content-Length': '0'} 2024-04-13T21:35:49.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000241ee4093f9f2df2-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.935 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.936 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.936 DEBUG:boto:Method: HEAD 2024-04-13T21:35:49.936 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:35:49.936 DEBUG:boto:Data: 2024-04-13T21:35:49.936 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.936 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.936 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.936 DEBUG:boto:Params: {} 2024-04-13T21:35:49.936 DEBUG:boto:Token: None 2024-04-13T21:35:49.936 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2 2024-04-13T21:35:49.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2QxowdpOrBdWhwpDuNxv7AAtVbw= 2024-04-13T21:35:49.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2QxowdpOrBdWhwpDuNxv7AAtVbw=', 'Content-Length': '0'} 2024-04-13T21:35:49.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0e491509871fbff-00661afab5-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:35:49.939 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.940 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.940 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.940 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.940 DEBUG:boto:Method: GET 2024-04-13T21:35:49.940 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.940 DEBUG:boto:Data: 2024-04-13T21:35:49.940 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.940 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.940 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.940 DEBUG:boto:Params: {} 2024-04-13T21:35:49.940 DEBUG:boto:Token: None 2024-04-13T21:35:49.941 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.941 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY= 2024-04-13T21:35:49.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY=', 'Content-Length': '0'} 2024-04-13T21:35:49.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb07f063a0527c89-00661afab5-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.990 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.991 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:49.991 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.991 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.991 DEBUG:boto:Method: GET 2024-04-13T21:35:49.991 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.991 DEBUG:boto:Data: 2024-04-13T21:35:49.991 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.991 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.991 DEBUG:boto:Params: {} 2024-04-13T21:35:49.992 DEBUG:boto:Token: None 2024-04-13T21:35:49.992 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:35:49.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY= 2024-04-13T21:35:49.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+ekp7onwmQMflAnx3t0NrV+fdY=', 'Content-Length': '0'} 2024-04-13T21:35:49.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005204f081bc81664e-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.994 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.994 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.994 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.995 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.995 DEBUG:boto:Method: GET 2024-04-13T21:35:49.995 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.995 DEBUG:boto:Data: 2024-04-13T21:35:49.995 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:49.995 DEBUG:boto:Port: 8001 2024-04-13T21:35:49.995 DEBUG:boto:Params: {} 2024-04-13T21:35:49.995 DEBUG:boto:Token: None 2024-04-13T21:35:49.995 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:49.996 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:49.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:49.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c922504d3305fc0e-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:49.998 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:49.998 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:49.999 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:49.999 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.999 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.999 DEBUG:boto:Method: GET 2024-04-13T21:35:49.999 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:35:49.999 DEBUG:boto:Data: 2024-04-13T21:35:49.999 DEBUG:boto:Headers: {} 2024-04-13T21:35:49.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:49.999 DEBUG:boto:Port: 8000 2024-04-13T21:35:49.999 DEBUG:boto:Params: {} 2024-04-13T21:35:49.999 DEBUG:boto:Token: None 2024-04-13T21:35:50.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:49 GMT /vwnfns-13/ 2024-04-13T21:35:50.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA= 2024-04-13T21:35:50.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bU8MJSQmWM/g7hfQJS/RsiQaGoA=', 'Content-Length': '0'} 2024-04-13T21:35:50.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006faa421a670c65f1-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.002 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-13 zones={a2, a1} 2024-04-13T21:35:50.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-14 2024-04-13T21:35:50.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:50.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:50.003 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:35:50.065 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:35:50.065 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:35:50.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:50.109+0000 7efd9a6ff780 1 bucket sync caught up with source: 2024-04-13T21:35:50.102 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:35:50.102 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:35:50.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:50.109+0000 7efd9a6ff780 0 bucket checkpoint complete 2024-04-13T21:35:50.115 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-14 2024-04-13T21:35:50.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:50.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:50.115 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:35:50.171 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:35:50.171 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:35:50.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:50.214+0000 7f6296160780 1 bucket sync caught up with source: 2024-04-13T21:35:50.208 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.33.10, , , 00000000010.34.6, 00000000002.32.11, ] 2024-04-13T21:35:50.208 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.33.10, , , 00000000010.34.6, 00000000002.32.11, ] 2024-04-13T21:35:50.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:50.214+0000 7f6296160780 0 bucket checkpoint complete 2024-04-13T21:35:50.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-14 zones={a2, a1} 2024-04-13T21:35:50.220 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.220 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.220 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.220 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:35:50.220 DEBUG:boto:Data: 2024-04-13T21:35:50.220 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.221 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.221 DEBUG:boto:Params: {} 2024-04-13T21:35:50.221 DEBUG:boto:Token: None 2024-04-13T21:35:50.221 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw= 2024-04-13T21:35:50.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw=', 'Content-Length': '0'} 2024-04-13T21:35:50.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d9b64abf3717a83-00661afab6-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.224 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.224 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.224 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.224 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:35:50.224 DEBUG:boto:Data: 2024-04-13T21:35:50.224 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.224 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.224 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.224 DEBUG:boto:Params: {} 2024-04-13T21:35:50.224 DEBUG:boto:Token: None 2024-04-13T21:35:50.224 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.225 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw= 2024-04-13T21:35:50.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw=', 'Content-Length': '0'} 2024-04-13T21:35:50.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e27411a85a99ccfa-00661afab6-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.227 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:35:50.227 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.227 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.227 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.227 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.227 DEBUG:boto:Method: GET 2024-04-13T21:35:50.227 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.227 DEBUG:boto:Data: 2024-04-13T21:35:50.227 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.227 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.228 DEBUG:boto:Params: {} 2024-04-13T21:35:50.228 DEBUG:boto:Token: None 2024-04-13T21:35:50.228 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/?versions 2024-04-13T21:35:50.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI= 2024-04-13T21:35:50.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI=', 'Content-Length': '0'} 2024-04-13T21:35:50.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030d86114583a42d3-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.230 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.231 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:35:50.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:35:50.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:35:50.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:35:50.231 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.231 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.231 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.231 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.231 DEBUG:boto:Method: GET 2024-04-13T21:35:50.231 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.232 DEBUG:boto:Data: 2024-04-13T21:35:50.232 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.232 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.232 DEBUG:boto:Params: {} 2024-04-13T21:35:50.232 DEBUG:boto:Token: None 2024-04-13T21:35:50.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/?versions 2024-04-13T21:35:50.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI= 2024-04-13T21:35:50.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI=', 'Content-Length': '0'} 2024-04-13T21:35:50.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d9f9ab63c4f82e2-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.234 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:35:50.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:35:50.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:35:50.235 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.235 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.235 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.235 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.236 DEBUG:boto:Method: GET 2024-04-13T21:35:50.236 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.236 DEBUG:boto:Data: 2024-04-13T21:35:50.236 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.236 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.236 DEBUG:boto:Params: {} 2024-04-13T21:35:50.236 DEBUG:boto:Token: None 2024-04-13T21:35:50.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/?versions 2024-04-13T21:35:50.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI= 2024-04-13T21:35:50.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI=', 'Content-Length': '0'} 2024-04-13T21:35:50.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ba0b22e97d35a5-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.239 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.239 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.239 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.239 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.239 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.240 DEBUG:boto:Method: GET 2024-04-13T21:35:50.240 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:35:50.240 DEBUG:boto:Data: 2024-04-13T21:35:50.240 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.240 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.240 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.240 DEBUG:boto:Params: {} 2024-04-13T21:35:50.240 DEBUG:boto:Token: None 2024-04-13T21:35:50.240 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/?versions 2024-04-13T21:35:50.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI= 2024-04-13T21:35:50.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI=', 'Content-Length': '0'} 2024-04-13T21:35:50.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c345e899383ee003-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.243 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:35:50.243 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.243 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.243 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.243 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.244 DEBUG:boto:Method: GET 2024-04-13T21:35:50.244 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.244 DEBUG:boto:Data: 2024-04-13T21:35:50.244 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.244 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.244 DEBUG:boto:Params: {} 2024-04-13T21:35:50.244 DEBUG:boto:Token: None 2024-04-13T21:35:50.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.245 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q= 2024-04-13T21:35:50.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q=', 'Content-Length': '0'} 2024-04-13T21:35:50.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2012c178dea8fa9-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.247 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.247 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.247 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.247 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.247 DEBUG:boto:Method: GET 2024-04-13T21:35:50.247 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.247 DEBUG:boto:Data: 2024-04-13T21:35:50.247 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.247 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.247 DEBUG:boto:Params: {} 2024-04-13T21:35:50.248 DEBUG:boto:Token: None 2024-04-13T21:35:50.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q= 2024-04-13T21:35:50.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q=', 'Content-Length': '0'} 2024-04-13T21:35:50.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c29e1b3529bdffd-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.250 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.250 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.250 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.250 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.250 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.251 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.251 DEBUG:boto:Data: 2024-04-13T21:35:50.251 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.251 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.251 DEBUG:boto:Params: {} 2024-04-13T21:35:50.251 DEBUG:boto:Token: None 2024-04-13T21:35:50.251 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6cKfqzWiGZTCFuMh5gv4FHeTaNU= 2024-04-13T21:35:50.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6cKfqzWiGZTCFuMh5gv4FHeTaNU=', 'Content-Length': '0'} 2024-04-13T21:35:50.253 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e54135baa37c87b-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.253 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.253 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.253 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.253 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.253 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.254 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.254 DEBUG:boto:Data: 2024-04-13T21:35:50.254 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.254 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.254 DEBUG:boto:Params: {} 2024-04-13T21:35:50.254 DEBUG:boto:Token: None 2024-04-13T21:35:50.254 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6cKfqzWiGZTCFuMh5gv4FHeTaNU= 2024-04-13T21:35:50.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6cKfqzWiGZTCFuMh5gv4FHeTaNU=', 'Content-Length': '0'} 2024-04-13T21:35:50.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000127e1dd661b9b81f-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:35:50.256 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.256 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.256 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.256 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.256 DEBUG:boto:Method: GET 2024-04-13T21:35:50.257 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.257 DEBUG:boto:Data: 2024-04-13T21:35:50.257 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.257 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.257 DEBUG:boto:Params: {} 2024-04-13T21:35:50.257 DEBUG:boto:Token: None 2024-04-13T21:35:50.257 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q= 2024-04-13T21:35:50.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q=', 'Content-Length': '0'} 2024-04-13T21:35:50.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb03ac6b104df8a8-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.259 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.259 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.259 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.259 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.259 DEBUG:boto:Method: GET 2024-04-13T21:35:50.259 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.260 DEBUG:boto:Data: 2024-04-13T21:35:50.260 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.260 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.260 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.260 DEBUG:boto:Params: {} 2024-04-13T21:35:50.260 DEBUG:boto:Token: None 2024-04-13T21:35:50.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q= 2024-04-13T21:35:50.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q=', 'Content-Length': '0'} 2024-04-13T21:35:50.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000477583c24a65abf8-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.262 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.262 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.262 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.262 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.262 DEBUG:boto:Method: GET 2024-04-13T21:35:50.263 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.263 DEBUG:boto:Data: 2024-04-13T21:35:50.263 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.263 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.263 DEBUG:boto:Params: {} 2024-04-13T21:35:50.263 DEBUG:boto:Token: None 2024-04-13T21:35:50.263 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9090b31bd8cf0f-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.306 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.307 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.307 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.307 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.307 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.307 DEBUG:boto:Method: GET 2024-04-13T21:35:50.307 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.307 DEBUG:boto:Data: 2024-04-13T21:35:50.307 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.308 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.308 DEBUG:boto:Params: {} 2024-04-13T21:35:50.308 DEBUG:boto:Token: None 2024-04-13T21:35:50.308 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b901d11f6931c3-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.310 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.311 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.311 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.311 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.311 DEBUG:boto:Path: /vwnfns-14/dummy 2024-04-13T21:35:50.311 DEBUG:boto:Data: 2024-04-13T21:35:50.311 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.311 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.311 DEBUG:boto:Params: {} 2024-04-13T21:35:50.312 DEBUG:boto:Token: None 2024-04-13T21:35:50.312 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy 2024-04-13T21:35:50.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E4/XX8P7SCUmFHFBMDBSlmMvezA= 2024-04-13T21:35:50.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E4/XX8P7SCUmFHFBMDBSlmMvezA=', 'Content-Length': '0'} 2024-04-13T21:35:50.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f4f36a1b5dc2999-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.314 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.314 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.314 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.314 DEBUG:boto:Path: /vwnfns-14/dummy 2024-04-13T21:35:50.314 DEBUG:boto:Data: 2024-04-13T21:35:50.315 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.315 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.315 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.315 DEBUG:boto:Params: {} 2024-04-13T21:35:50.315 DEBUG:boto:Token: None 2024-04-13T21:35:50.315 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy 2024-04-13T21:35:50.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E4/XX8P7SCUmFHFBMDBSlmMvezA= 2024-04-13T21:35:50.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E4/XX8P7SCUmFHFBMDBSlmMvezA=', 'Content-Length': '0'} 2024-04-13T21:35:50.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c43489fc19cf3096-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:35:50.317 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.317 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.317 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.317 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.317 DEBUG:boto:Method: GET 2024-04-13T21:35:50.318 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.318 DEBUG:boto:Data: 2024-04-13T21:35:50.318 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.318 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.318 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.318 DEBUG:boto:Params: {} 2024-04-13T21:35:50.318 DEBUG:boto:Token: None 2024-04-13T21:35:50.318 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.318 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q= 2024-04-13T21:35:50.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q=', 'Content-Length': '0'} 2024-04-13T21:35:50.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cc6dcd3347c7f37-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.320 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:35:50.320 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:35:50.321 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.321 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.321 DEBUG:boto:Method: GET 2024-04-13T21:35:50.321 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.321 DEBUG:boto:Data: 2024-04-13T21:35:50.321 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.321 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.321 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.321 DEBUG:boto:Params: {} 2024-04-13T21:35:50.321 DEBUG:boto:Token: None 2024-04-13T21:35:50.321 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:35:50.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q= 2024-04-13T21:35:50.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CFpXlDZHOHYFntUHHxtZQl0J+7Q=', 'Content-Length': '0'} 2024-04-13T21:35:50.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001291e3db05c54300-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.323 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.323 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.323 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.324 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.324 DEBUG:boto:Method: GET 2024-04-13T21:35:50.324 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.324 DEBUG:boto:Data: 2024-04-13T21:35:50.324 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.324 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.324 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.324 DEBUG:boto:Params: {} 2024-04-13T21:35:50.324 DEBUG:boto:Token: None 2024-04-13T21:35:50.324 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.325 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6a2297f2632b46-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.327 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.327 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.328 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.328 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.328 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.328 DEBUG:boto:Method: GET 2024-04-13T21:35:50.328 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:35:50.328 DEBUG:boto:Data: 2024-04-13T21:35:50.328 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.328 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.328 DEBUG:boto:Params: {} 2024-04-13T21:35:50.329 DEBUG:boto:Token: None 2024-04-13T21:35:50.329 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d59a7523c89ab3ff-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.331 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:35:50.331 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.331 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.332 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.332 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.332 DEBUG:boto:Method: GET 2024-04-13T21:35:50.332 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.332 DEBUG:boto:Data: 2024-04-13T21:35:50.332 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.332 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.332 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.333 DEBUG:boto:Params: {} 2024-04-13T21:35:50.333 DEBUG:boto:Token: None 2024-04-13T21:35:50.333 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.333 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM= 2024-04-13T21:35:50.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM=', 'Content-Length': '0'} 2024-04-13T21:35:50.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072ed3bd11e30fb8a-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.335 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.335 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.335 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.335 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.335 DEBUG:boto:Method: GET 2024-04-13T21:35:50.335 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.336 DEBUG:boto:Data: 2024-04-13T21:35:50.336 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.336 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.336 DEBUG:boto:Params: {} 2024-04-13T21:35:50.336 DEBUG:boto:Token: None 2024-04-13T21:35:50.337 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM= 2024-04-13T21:35:50.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM=', 'Content-Length': '0'} 2024-04-13T21:35:50.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a7e3d7941f34e72-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.382 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.383 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.383 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.383 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.383 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.383 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.383 DEBUG:boto:Data: 2024-04-13T21:35:50.383 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.383 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.384 DEBUG:boto:Params: {} 2024-04-13T21:35:50.384 DEBUG:boto:Token: None 2024-04-13T21:35:50.384 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q5ei1rCQwR9lc9zMj4dYeaH/GSM= 2024-04-13T21:35:50.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q5ei1rCQwR9lc9zMj4dYeaH/GSM=', 'Content-Length': '0'} 2024-04-13T21:35:50.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000386e5728467fa863-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.386 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.386 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.386 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.386 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.386 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.386 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.386 DEBUG:boto:Data: 2024-04-13T21:35:50.387 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.387 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.387 DEBUG:boto:Params: {} 2024-04-13T21:35:50.387 DEBUG:boto:Token: None 2024-04-13T21:35:50.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q5ei1rCQwR9lc9zMj4dYeaH/GSM= 2024-04-13T21:35:50.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q5ei1rCQwR9lc9zMj4dYeaH/GSM=', 'Content-Length': '0'} 2024-04-13T21:35:50.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b55440a08772693-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:35:50.389 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.389 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.389 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.390 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.390 DEBUG:boto:Method: GET 2024-04-13T21:35:50.390 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.390 DEBUG:boto:Data: 2024-04-13T21:35:50.390 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.390 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.390 DEBUG:boto:Params: {} 2024-04-13T21:35:50.390 DEBUG:boto:Token: None 2024-04-13T21:35:50.390 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM= 2024-04-13T21:35:50.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM=', 'Content-Length': '0'} 2024-04-13T21:35:50.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd0f61de0589b349-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.434 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.434 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.435 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.435 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.435 DEBUG:boto:Method: GET 2024-04-13T21:35:50.435 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.435 DEBUG:boto:Data: 2024-04-13T21:35:50.435 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.435 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.435 DEBUG:boto:Params: {} 2024-04-13T21:35:50.435 DEBUG:boto:Token: None 2024-04-13T21:35:50.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM= 2024-04-13T21:35:50.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM=', 'Content-Length': '0'} 2024-04-13T21:35:50.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2094e4079bf0813-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.438 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.438 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.438 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.438 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.438 DEBUG:boto:Method: GET 2024-04-13T21:35:50.438 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.438 DEBUG:boto:Data: 2024-04-13T21:35:50.438 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.439 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.439 DEBUG:boto:Params: {} 2024-04-13T21:35:50.439 DEBUG:boto:Token: None 2024-04-13T21:35:50.439 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.439 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c2ee81ea68d2eb-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.486 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.487 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.487 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.487 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.487 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.487 DEBUG:boto:Method: GET 2024-04-13T21:35:50.487 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.487 DEBUG:boto:Data: 2024-04-13T21:35:50.487 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.488 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.488 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.488 DEBUG:boto:Params: {} 2024-04-13T21:35:50.488 DEBUG:boto:Token: None 2024-04-13T21:35:50.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d17de8d3fc92d4e-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.491 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.491 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.491 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.491 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.491 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:35:50.491 DEBUG:boto:Data: 2024-04-13T21:35:50.491 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.492 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.492 DEBUG:boto:Params: {} 2024-04-13T21:35:50.492 DEBUG:boto:Token: None 2024-04-13T21:35:50.492 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1 2024-04-13T21:35:50.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3rlNZktsjX4Kq7ZAFH1Wz56ZIQA= 2024-04-13T21:35:50.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3rlNZktsjX4Kq7ZAFH1Wz56ZIQA=', 'Content-Length': '0'} 2024-04-13T21:35:50.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6648833aac4c34f-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.495 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.495 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.495 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.495 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:35:50.495 DEBUG:boto:Data: 2024-04-13T21:35:50.495 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.496 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.496 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.496 DEBUG:boto:Params: {} 2024-04-13T21:35:50.496 DEBUG:boto:Token: None 2024-04-13T21:35:50.496 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1 2024-04-13T21:35:50.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3rlNZktsjX4Kq7ZAFH1Wz56ZIQA= 2024-04-13T21:35:50.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3rlNZktsjX4Kq7ZAFH1Wz56ZIQA=', 'Content-Length': '0'} 2024-04-13T21:35:50.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f16aa836dcec8eaa-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:35:50.498 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.498 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.498 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.499 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.499 DEBUG:boto:Method: GET 2024-04-13T21:35:50.499 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.499 DEBUG:boto:Data: 2024-04-13T21:35:50.499 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.499 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.499 DEBUG:boto:Params: {} 2024-04-13T21:35:50.499 DEBUG:boto:Token: None 2024-04-13T21:35:50.500 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM= 2024-04-13T21:35:50.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM=', 'Content-Length': '0'} 2024-04-13T21:35:50.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca6ee28141f8dc6b-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.542 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.542 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.543 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.543 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.543 DEBUG:boto:Method: GET 2024-04-13T21:35:50.543 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.543 DEBUG:boto:Data: 2024-04-13T21:35:50.543 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.543 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.543 DEBUG:boto:Params: {} 2024-04-13T21:35:50.543 DEBUG:boto:Token: None 2024-04-13T21:35:50.544 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:35:50.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM= 2024-04-13T21:35:50.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V4/TBPF1zYiZTQvkumN+XY20lpM=', 'Content-Length': '0'} 2024-04-13T21:35:50.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014c7719190c49f85-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.546 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.546 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.546 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.546 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.546 DEBUG:boto:Method: GET 2024-04-13T21:35:50.546 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.546 DEBUG:boto:Data: 2024-04-13T21:35:50.546 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.547 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.547 DEBUG:boto:Params: {} 2024-04-13T21:35:50.547 DEBUG:boto:Token: None 2024-04-13T21:35:50.547 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df8cfee745359b36-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.590 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.591 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.591 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.591 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.591 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.591 DEBUG:boto:Method: GET 2024-04-13T21:35:50.591 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:35:50.591 DEBUG:boto:Data: 2024-04-13T21:35:50.592 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.592 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.592 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.592 DEBUG:boto:Params: {} 2024-04-13T21:35:50.592 DEBUG:boto:Token: None 2024-04-13T21:35:50.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc4d86901a509f4-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.594 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:35:50.595 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.595 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.595 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.595 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.595 DEBUG:boto:Method: GET 2024-04-13T21:35:50.595 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.596 DEBUG:boto:Data: 2024-04-13T21:35:50.596 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.596 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.596 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.596 DEBUG:boto:Params: {} 2024-04-13T21:35:50.596 DEBUG:boto:Token: None 2024-04-13T21:35:50.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc= 2024-04-13T21:35:50.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc=', 'Content-Length': '0'} 2024-04-13T21:35:50.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3e9b845faddcb8e-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.598 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.598 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.598 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.598 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.598 DEBUG:boto:Method: GET 2024-04-13T21:35:50.599 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.599 DEBUG:boto:Data: 2024-04-13T21:35:50.599 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.599 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.599 DEBUG:boto:Params: {} 2024-04-13T21:35:50.599 DEBUG:boto:Token: None 2024-04-13T21:35:50.599 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.599 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc= 2024-04-13T21:35:50.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc=', 'Content-Length': '0'} 2024-04-13T21:35:50.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012fb8e40b682b6e9-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.642 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.643 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.643 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.643 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.643 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.643 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.643 DEBUG:boto:Data: 2024-04-13T21:35:50.643 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.643 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.643 DEBUG:boto:Params: {} 2024-04-13T21:35:50.644 DEBUG:boto:Token: None 2024-04-13T21:35:50.644 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wtWA5ecXyg7aCfrC1jfrAjwl5n4= 2024-04-13T21:35:50.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wtWA5ecXyg7aCfrC1jfrAjwl5n4=', 'Content-Length': '0'} 2024-04-13T21:35:50.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000091589752590e5c9-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.646 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.646 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.646 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.647 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.647 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.647 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.647 DEBUG:boto:Data: 2024-04-13T21:35:50.647 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.647 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.647 DEBUG:boto:Params: {} 2024-04-13T21:35:50.647 DEBUG:boto:Token: None 2024-04-13T21:35:50.647 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wtWA5ecXyg7aCfrC1jfrAjwl5n4= 2024-04-13T21:35:50.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wtWA5ecXyg7aCfrC1jfrAjwl5n4=', 'Content-Length': '0'} 2024-04-13T21:35:50.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000594c93227f6e91aa-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:35:50.650 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.650 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.650 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.650 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.650 DEBUG:boto:Method: GET 2024-04-13T21:35:50.650 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.650 DEBUG:boto:Data: 2024-04-13T21:35:50.650 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.651 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.651 DEBUG:boto:Params: {} 2024-04-13T21:35:50.651 DEBUG:boto:Token: None 2024-04-13T21:35:50.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc= 2024-04-13T21:35:50.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc=', 'Content-Length': '0'} 2024-04-13T21:35:50.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a3ed3345c3e9fe6-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.653 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.653 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.653 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.653 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.653 DEBUG:boto:Method: GET 2024-04-13T21:35:50.653 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.653 DEBUG:boto:Data: 2024-04-13T21:35:50.653 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.654 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.654 DEBUG:boto:Params: {} 2024-04-13T21:35:50.654 DEBUG:boto:Token: None 2024-04-13T21:35:50.654 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.654 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc= 2024-04-13T21:35:50.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc=', 'Content-Length': '0'} 2024-04-13T21:35:50.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c287ae5da975978d-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.698 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.699 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.699 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.699 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.699 DEBUG:boto:Method: GET 2024-04-13T21:35:50.699 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.699 DEBUG:boto:Data: 2024-04-13T21:35:50.699 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.699 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.699 DEBUG:boto:Params: {} 2024-04-13T21:35:50.700 DEBUG:boto:Token: None 2024-04-13T21:35:50.700 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000173abf79c6f9f2e-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.702 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.703 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.703 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.703 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.703 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.703 DEBUG:boto:Method: GET 2024-04-13T21:35:50.703 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.703 DEBUG:boto:Data: 2024-04-13T21:35:50.703 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.704 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.704 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.704 DEBUG:boto:Params: {} 2024-04-13T21:35:50.704 DEBUG:boto:Token: None 2024-04-13T21:35:50.704 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4fe0534ce3f12e-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.706 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.707 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.707 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.707 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.707 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:35:50.707 DEBUG:boto:Data: 2024-04-13T21:35:50.707 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.707 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.707 DEBUG:boto:Params: {} 2024-04-13T21:35:50.708 DEBUG:boto:Token: None 2024-04-13T21:35:50.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2 2024-04-13T21:35:50.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9nixvMgOxhe0NS145MBlXyTnlEg= 2024-04-13T21:35:50.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9nixvMgOxhe0NS145MBlXyTnlEg=', 'Content-Length': '0'} 2024-04-13T21:35:50.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d59385a9ba0e0cb4-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.710 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.710 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.710 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.710 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:35:50.710 DEBUG:boto:Data: 2024-04-13T21:35:50.710 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.711 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.711 DEBUG:boto:Params: {} 2024-04-13T21:35:50.711 DEBUG:boto:Token: None 2024-04-13T21:35:50.711 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2 2024-04-13T21:35:50.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9nixvMgOxhe0NS145MBlXyTnlEg= 2024-04-13T21:35:50.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9nixvMgOxhe0NS145MBlXyTnlEg=', 'Content-Length': '0'} 2024-04-13T21:35:50.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c03ae7f235bb379e-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:35:50.713 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.713 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.713 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.713 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.713 DEBUG:boto:Method: GET 2024-04-13T21:35:50.713 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.713 DEBUG:boto:Data: 2024-04-13T21:35:50.714 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.714 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.714 DEBUG:boto:Params: {} 2024-04-13T21:35:50.714 DEBUG:boto:Token: None 2024-04-13T21:35:50.714 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc= 2024-04-13T21:35:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc=', 'Content-Length': '0'} 2024-04-13T21:35:50.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065e61c82351c7457-00661afab6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.716 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.716 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.716 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.716 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.716 DEBUG:boto:Method: GET 2024-04-13T21:35:50.717 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.717 DEBUG:boto:Data: 2024-04-13T21:35:50.717 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.717 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.717 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.717 DEBUG:boto:Params: {} 2024-04-13T21:35:50.717 DEBUG:boto:Token: None 2024-04-13T21:35:50.717 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:35:50.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc= 2024-04-13T21:35:50.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1QCAPs3LMk3YQNiREuHMFxQJkc=', 'Content-Length': '0'} 2024-04-13T21:35:50.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071ee48d1de263fcf-00661afab6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.763 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.763 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.763 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.763 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.763 DEBUG:boto:Method: GET 2024-04-13T21:35:50.763 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.763 DEBUG:boto:Data: 2024-04-13T21:35:50.763 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.764 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.764 DEBUG:boto:Params: {} 2024-04-13T21:35:50.764 DEBUG:boto:Token: None 2024-04-13T21:35:50.764 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def0967d8e916f97-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.766 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.767 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.767 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.767 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.767 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.767 DEBUG:boto:Method: GET 2024-04-13T21:35:50.767 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:35:50.767 DEBUG:boto:Data: 2024-04-13T21:35:50.767 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.767 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.768 DEBUG:boto:Params: {} 2024-04-13T21:35:50.768 DEBUG:boto:Token: None 2024-04-13T21:35:50.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs= 2024-04-13T21:35:50.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FR7cA+hQuukACp6g7KKnDsHepMs=', 'Content-Length': '0'} 2024-04-13T21:35:50.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037500f7df512b074-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.770 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.770 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-14 zones={a2, a1} 2024-04-13T21:35:50.771 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:50.771 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:50.771 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.771 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:35:50.771 DEBUG:boto:Data: 2024-04-13T21:35:50.771 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.771 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.771 DEBUG:boto:Params: {} 2024-04-13T21:35:50.772 DEBUG:boto:Token: None 2024-04-13T21:35:50.772 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-13/ 2024-04-13T21:35:50.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X0uaN9anCAU2A5bIo2SsgiGPr5k= 2024-04-13T21:35:50.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X0uaN9anCAU2A5bIo2SsgiGPr5k=', 'Content-Length': '0'} 2024-04-13T21:35:50.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043acb944f2bd28f5-00661afab6-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.774 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:50.774 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:50.774 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999&prefix=obj-a1 2024-04-13T21:35:50.774 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999&prefix=obj-a1 2024-04-13T21:35:50.774 DEBUG:boto:Method: GET 2024-04-13T21:35:50.774 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999&prefix=obj-a1 2024-04-13T21:35:50.774 DEBUG:boto:Data: 2024-04-13T21:35:50.775 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.775 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.775 DEBUG:boto:Params: {} 2024-04-13T21:35:50.775 DEBUG:boto:Token: None 2024-04-13T21:35:50.775 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-13/?versions 2024-04-13T21:35:50.775 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SeKgTklPS0jv+MElX4HIv9F3Y+U= 2024-04-13T21:35:50.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SeKgTklPS0jv+MElX4HIv9F3Y+U=', 'Content-Length': '0'} 2024-04-13T21:35:50.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f1c3e3d7204de6e-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.777 DEBUG:boto:b'vwnfns-13obj-a1999falseobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.778 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:35:50.778 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:35:50.778 DEBUG:boto:path=/vwnfns-13/obj-a1?acl 2024-04-13T21:35:50.778 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?acl 2024-04-13T21:35:50.778 DEBUG:boto:Method: PUT 2024-04-13T21:35:50.778 DEBUG:boto:Path: /vwnfns-13/obj-a1?acl 2024-04-13T21:35:50.778 DEBUG:boto:Data: 2024-04-13T21:35:50.778 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:35:50.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.779 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.779 DEBUG:boto:Params: {} 2024-04-13T21:35:50.779 DEBUG:boto:Token: None 2024-04-13T21:35:50.779 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:35:50 GMT x-amz-acl:public-read /vwnfns-13/obj-a1?acl 2024-04-13T21:35:50.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UGxrYc2R5By1JHZUPtWZaDL7DF0= 2024-04-13T21:35:50.779 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UGxrYc2R5By1JHZUPtWZaDL7DF0=', 'Content-Length': '0'} 2024-04-13T21:35:50.785 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd84a7cfea3b65d5-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.785 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:50.785 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:50.785 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.785 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:35:50.785 DEBUG:boto:Data: 2024-04-13T21:35:50.785 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.786 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.786 DEBUG:boto:Params: {} 2024-04-13T21:35:50.786 DEBUG:boto:Token: None 2024-04-13T21:35:50.786 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-13/ 2024-04-13T21:35:50.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X0uaN9anCAU2A5bIo2SsgiGPr5k= 2024-04-13T21:35:50.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X0uaN9anCAU2A5bIo2SsgiGPr5k=', 'Content-Length': '0'} 2024-04-13T21:35:50.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013988703b72e68e8-00661afab6-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.788 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:35:50.789 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:35:50.789 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999&prefix=obj-a2 2024-04-13T21:35:50.789 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999&prefix=obj-a2 2024-04-13T21:35:50.789 DEBUG:boto:Method: GET 2024-04-13T21:35:50.789 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999&prefix=obj-a2 2024-04-13T21:35:50.789 DEBUG:boto:Data: 2024-04-13T21:35:50.789 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.789 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.789 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.789 DEBUG:boto:Params: {} 2024-04-13T21:35:50.790 DEBUG:boto:Token: None 2024-04-13T21:35:50.790 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-13/?versions 2024-04-13T21:35:50.790 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SeKgTklPS0jv+MElX4HIv9F3Y+U= 2024-04-13T21:35:50.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SeKgTklPS0jv+MElX4HIv9F3Y+U=', 'Content-Length': '0'} 2024-04-13T21:35:50.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003be775e3423c3f01-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.834 DEBUG:boto:b'vwnfns-13obj-a2999falseobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.835 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:35:50.835 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:35:50.835 DEBUG:boto:path=/vwnfns-13/obj-a2?acl 2024-04-13T21:35:50.835 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?acl 2024-04-13T21:35:50.835 DEBUG:boto:Method: PUT 2024-04-13T21:35:50.835 DEBUG:boto:Path: /vwnfns-13/obj-a2?acl 2024-04-13T21:35:50.835 DEBUG:boto:Data: 2024-04-13T21:35:50.836 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:35:50.836 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.836 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.836 DEBUG:boto:Params: {} 2024-04-13T21:35:50.836 DEBUG:boto:Token: None 2024-04-13T21:35:50.836 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:35:50 GMT x-amz-acl:public-read /vwnfns-13/obj-a2?acl 2024-04-13T21:35:50.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4RQJVHA9cNQbDm6OPd7M47U7CGE= 2024-04-13T21:35:50.837 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4RQJVHA9cNQbDm6OPd7M47U7CGE=', 'Content-Length': '0'} 2024-04-13T21:35:50.842 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006eaa1f5f0b3c6837-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.842 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.843 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.843 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.843 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:35:50.843 DEBUG:boto:Data: 2024-04-13T21:35:50.843 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.843 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.844 DEBUG:boto:Params: {} 2024-04-13T21:35:50.844 DEBUG:boto:Token: None 2024-04-13T21:35:50.844 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw= 2024-04-13T21:35:50.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw=', 'Content-Length': '0'} 2024-04-13T21:35:50.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006048e212a0c27bda-00661afab6-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.847 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.847 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.847 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999&prefix=obj-a1 2024-04-13T21:35:50.847 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999&prefix=obj-a1 2024-04-13T21:35:50.847 DEBUG:boto:Method: GET 2024-04-13T21:35:50.848 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999&prefix=obj-a1 2024-04-13T21:35:50.848 DEBUG:boto:Data: 2024-04-13T21:35:50.848 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.848 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.848 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.848 DEBUG:boto:Params: {} 2024-04-13T21:35:50.848 DEBUG:boto:Token: None 2024-04-13T21:35:50.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/?versions 2024-04-13T21:35:50.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI= 2024-04-13T21:35:50.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI=', 'Content-Length': '0'} 2024-04-13T21:35:50.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021f061a8d1761a50-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.894 DEBUG:boto:b'vwnfns-14obj-a1999falseobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.895 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.895 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:35:50.895 DEBUG:boto:path=/vwnfns-14/obj-a1?acl 2024-04-13T21:35:50.895 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?acl 2024-04-13T21:35:50.895 DEBUG:boto:Method: PUT 2024-04-13T21:35:50.895 DEBUG:boto:Path: /vwnfns-14/obj-a1?acl 2024-04-13T21:35:50.895 DEBUG:boto:Data: 2024-04-13T21:35:50.895 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:35:50.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:35:50.896 DEBUG:boto:Port: 8000 2024-04-13T21:35:50.896 DEBUG:boto:Params: {} 2024-04-13T21:35:50.896 DEBUG:boto:Token: None 2024-04-13T21:35:50.896 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:35:50 GMT x-amz-acl:public-read /vwnfns-14/obj-a1?acl 2024-04-13T21:35:50.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WMubv67fOBO18sPoQ2KEH1i/nhM= 2024-04-13T21:35:50.896 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WMubv67fOBO18sPoQ2KEH1i/nhM=', 'Content-Length': '0'} 2024-04-13T21:35:50.902 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000597f3db37b5c12c8-00661afab6-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.902 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.902 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.902 DEBUG:boto:Method: HEAD 2024-04-13T21:35:50.902 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:35:50.902 DEBUG:boto:Data: 2024-04-13T21:35:50.902 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.902 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.903 DEBUG:boto:Params: {} 2024-04-13T21:35:50.903 DEBUG:boto:Token: None 2024-04-13T21:35:50.903 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/ 2024-04-13T21:35:50.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw= 2024-04-13T21:35:50.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wI0PkIgajvfgo7AzskZcC7CX9fw=', 'Content-Length': '0'} 2024-04-13T21:35:50.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef9f5ac14f85085c-00661afab6-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.905 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:35:50.905 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:35:50.905 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999&prefix=obj-a2 2024-04-13T21:35:50.905 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999&prefix=obj-a2 2024-04-13T21:35:50.905 DEBUG:boto:Method: GET 2024-04-13T21:35:50.905 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999&prefix=obj-a2 2024-04-13T21:35:50.906 DEBUG:boto:Data: 2024-04-13T21:35:50.906 DEBUG:boto:Headers: {} 2024-04-13T21:35:50.906 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.906 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.906 DEBUG:boto:Params: {} 2024-04-13T21:35:50.906 DEBUG:boto:Token: None 2024-04-13T21:35:50.906 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:35:50 GMT /vwnfns-14/?versions 2024-04-13T21:35:50.906 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI= 2024-04-13T21:35:50.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ooe393DGAX9t/7AyszNEXHl/LsI=', 'Content-Length': '0'} 2024-04-13T21:35:50.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac663a35a558f1fd-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.908 DEBUG:boto:b'vwnfns-14obj-a2999falseobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:35:50.908 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.909 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:35:50.909 DEBUG:boto:path=/vwnfns-14/obj-a2?acl 2024-04-13T21:35:50.909 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?acl 2024-04-13T21:35:50.909 DEBUG:boto:Method: PUT 2024-04-13T21:35:50.909 DEBUG:boto:Path: /vwnfns-14/obj-a2?acl 2024-04-13T21:35:50.909 DEBUG:boto:Data: 2024-04-13T21:35:50.909 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:35:50.909 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:35:50.909 DEBUG:boto:Port: 8001 2024-04-13T21:35:50.909 DEBUG:boto:Params: {} 2024-04-13T21:35:50.910 DEBUG:boto:Token: None 2024-04-13T21:35:50.910 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:35:50 GMT x-amz-acl:public-read /vwnfns-14/obj-a2?acl 2024-04-13T21:35:50.910 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jLsuBTJbBbrI+gDrbdxd3MPBSRI= 2024-04-13T21:35:50.910 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:35:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jLsuBTJbBbrI+gDrbdxd3MPBSRI=', 'Content-Length': '0'} 2024-04-13T21:35:50.915 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000605c47b2935ebc2e-00661afab6-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:35:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:35:50.915 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-13 2024-04-13T21:35:50.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:50.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:35:50.916 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:35:50.970 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:35:50.970 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:35:51.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:51.011+0000 7f2f5a31c780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:51.004 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000010.39.6, ] 2024-04-13T21:35:51.004 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000011.42.6, ] 2024-04-13T21:35:56.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:35:56.013+0000 7f2f5a31c780 1 waiting for incremental sync to catch up: 2024-04-13T21:35:56.005 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000010.39.6, ] 2024-04-13T21:35:56.005 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000011.42.6, ] 2024-04-13T21:36:01.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:01.014+0000 7f2f5a31c780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:01.006 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000010.39.6, ] 2024-04-13T21:36:01.006 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000011.42.6, ] 2024-04-13T21:36:06.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:06.015+0000 7f2f5a31c780 1 bucket sync caught up with source: 2024-04-13T21:36:06.007 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000011.42.6, ] 2024-04-13T21:36:06.007 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.8.11, 00000000011.42.6, ] 2024-04-13T21:36:06.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:06.015+0000 7f2f5a31c780 0 bucket checkpoint complete 2024-04-13T21:36:06.021 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-13 2024-04-13T21:36:06.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:36:06.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:36:06.021 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:36:06.078 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:36:06.078 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:36:06.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:06.121+0000 7f2b73659780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:06.113 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000010.40.6, 00000000002.32.11, ] 2024-04-13T21:36:06.113 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000011.43.6, 00000000002.32.11, ] 2024-04-13T21:36:11.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:11.122+0000 7f2b73659780 1 bucket sync caught up with source: 2024-04-13T21:36:11.114 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.28.11, , , 00000000011.43.6, 00000000002.32.11, ] 2024-04-13T21:36:11.115 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.28.11, , , 00000000011.43.6, 00000000002.32.11, ] 2024-04-13T21:36:11.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:11.122+0000 7f2b73659780 0 bucket checkpoint complete 2024-04-13T21:36:11.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-13 zones={a2, a1} 2024-04-13T21:36:11.130 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.130 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.130 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.130 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:36:11.130 DEBUG:boto:Data: 2024-04-13T21:36:11.131 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.131 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.131 DEBUG:boto:Params: {} 2024-04-13T21:36:11.131 DEBUG:boto:Token: None 2024-04-13T21:36:11.131 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.131 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZwzdSbpfDKVrMkpPLUuZEdVV8k= 2024-04-13T21:36:11.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZwzdSbpfDKVrMkpPLUuZEdVV8k=', 'Content-Length': '0'} 2024-04-13T21:36:11.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c38d197abad665fb-00661afacb-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.134 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.134 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.135 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.135 DEBUG:boto:Path: /vwnfns-13/ 2024-04-13T21:36:11.135 DEBUG:boto:Data: 2024-04-13T21:36:11.135 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.135 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.135 DEBUG:boto:Params: {} 2024-04-13T21:36:11.135 DEBUG:boto:Token: None 2024-04-13T21:36:11.135 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZwzdSbpfDKVrMkpPLUuZEdVV8k= 2024-04-13T21:36:11.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZwzdSbpfDKVrMkpPLUuZEdVV8k=', 'Content-Length': '0'} 2024-04-13T21:36:11.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e3f0e75a2a207ca-00661afacb-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.138 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:36:11.138 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.138 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.138 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.138 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.138 DEBUG:boto:Method: GET 2024-04-13T21:36:11.139 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.139 DEBUG:boto:Data: 2024-04-13T21:36:11.139 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.139 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.139 DEBUG:boto:Params: {} 2024-04-13T21:36:11.139 DEBUG:boto:Token: None 2024-04-13T21:36:11.139 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/?versions 2024-04-13T21:36:11.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA= 2024-04-13T21:36:11.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA=', 'Content-Length': '0'} 2024-04-13T21:36:11.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d10dd5bf0263ed22-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.142 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.143 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:36:11.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:36:11.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:36:11.143 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:36:11.143 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.143 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.143 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.143 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.143 DEBUG:boto:Method: GET 2024-04-13T21:36:11.144 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.144 DEBUG:boto:Data: 2024-04-13T21:36:11.144 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.144 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.144 DEBUG:boto:Params: {} 2024-04-13T21:36:11.144 DEBUG:boto:Token: None 2024-04-13T21:36:11.144 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/?versions 2024-04-13T21:36:11.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA= 2024-04-13T21:36:11.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA=', 'Content-Length': '0'} 2024-04-13T21:36:11.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac346430968f23e-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.147 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.148 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:36:11.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:36:11.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:36:11.148 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.148 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.148 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.148 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.148 DEBUG:boto:Method: GET 2024-04-13T21:36:11.148 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.149 DEBUG:boto:Data: 2024-04-13T21:36:11.149 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.149 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.149 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.149 DEBUG:boto:Params: {} 2024-04-13T21:36:11.149 DEBUG:boto:Token: None 2024-04-13T21:36:11.149 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/?versions 2024-04-13T21:36:11.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA= 2024-04-13T21:36:11.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA=', 'Content-Length': '0'} 2024-04-13T21:36:11.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b10e7f6a1966d571-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.152 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.152 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.152 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.152 DEBUG:boto:path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.152 DEBUG:boto:auth_path=/vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.152 DEBUG:boto:Method: GET 2024-04-13T21:36:11.152 DEBUG:boto:Path: /vwnfns-13/?versions&max-keys=999 2024-04-13T21:36:11.153 DEBUG:boto:Data: 2024-04-13T21:36:11.153 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.153 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.153 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.153 DEBUG:boto:Params: {} 2024-04-13T21:36:11.153 DEBUG:boto:Token: None 2024-04-13T21:36:11.153 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/?versions 2024-04-13T21:36:11.153 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA= 2024-04-13T21:36:11.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFqOvlY2/6e/k0tJ6Hs6+z4OQdA=', 'Content-Length': '0'} 2024-04-13T21:36:11.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193e9cb7d64bd0bb-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.156 DEBUG:boto:b'vwnfns-13999falsedummyUE-6VjGEvakKflwLsB85XvcXx7.WAXutrue2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hHN8DBGbwxHORHE-cQZPwkqECLVqXn2true2024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2c3GMP6xszEoVPbSZOTFyjE9frkzW5Ejtrue2024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:36:11.157 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.157 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.157 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.157 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.157 DEBUG:boto:Method: GET 2024-04-13T21:36:11.157 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.158 DEBUG:boto:Data: 2024-04-13T21:36:11.158 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.158 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.158 DEBUG:boto:Params: {} 2024-04-13T21:36:11.158 DEBUG:boto:Token: None 2024-04-13T21:36:11.158 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.158 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU= 2024-04-13T21:36:11.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU=', 'Content-Length': '0'} 2024-04-13T21:36:11.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6f5b4511e4e5b45-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.160 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.161 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.161 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.161 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.161 DEBUG:boto:Method: GET 2024-04-13T21:36:11.161 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.161 DEBUG:boto:Data: 2024-04-13T21:36:11.161 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.161 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.161 DEBUG:boto:Params: {} 2024-04-13T21:36:11.161 DEBUG:boto:Token: None 2024-04-13T21:36:11.162 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.162 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU= 2024-04-13T21:36:11.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU=', 'Content-Length': '0'} 2024-04-13T21:36:11.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b49eb412ec088cf3-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.163 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.164 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.164 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.164 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.164 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.164 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.164 DEBUG:boto:Data: 2024-04-13T21:36:11.164 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.164 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.164 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.164 DEBUG:boto:Params: {} 2024-04-13T21:36:11.165 DEBUG:boto:Token: None 2024-04-13T21:36:11.165 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:chxqkkqh6hRVxsgiHbZLUMOklPs= 2024-04-13T21:36:11.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:chxqkkqh6hRVxsgiHbZLUMOklPs=', 'Content-Length': '0'} 2024-04-13T21:36:11.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b80e2875f2cede17-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.167 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.167 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.167 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.167 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.167 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.167 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.167 DEBUG:boto:Data: 2024-04-13T21:36:11.167 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.167 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.167 DEBUG:boto:Params: {} 2024-04-13T21:36:11.168 DEBUG:boto:Token: None 2024-04-13T21:36:11.168 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:chxqkkqh6hRVxsgiHbZLUMOklPs= 2024-04-13T21:36:11.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:chxqkkqh6hRVxsgiHbZLUMOklPs=', 'Content-Length': '0'} 2024-04-13T21:36:11.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000664bc4fe57b65af-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:36:11.170 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.170 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.170 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.170 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.170 DEBUG:boto:Method: GET 2024-04-13T21:36:11.170 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.171 DEBUG:boto:Data: 2024-04-13T21:36:11.171 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.171 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.171 DEBUG:boto:Params: {} 2024-04-13T21:36:11.171 DEBUG:boto:Token: None 2024-04-13T21:36:11.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU= 2024-04-13T21:36:11.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU=', 'Content-Length': '0'} 2024-04-13T21:36:11.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000305432250e002290-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.173 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.173 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.173 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.173 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.173 DEBUG:boto:Method: GET 2024-04-13T21:36:11.174 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.174 DEBUG:boto:Data: 2024-04-13T21:36:11.174 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.174 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.174 DEBUG:boto:Params: {} 2024-04-13T21:36:11.174 DEBUG:boto:Token: None 2024-04-13T21:36:11.174 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.174 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU= 2024-04-13T21:36:11.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU=', 'Content-Length': '0'} 2024-04-13T21:36:11.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6bf666524bd0704-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.176 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.176 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.177 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.177 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.177 DEBUG:boto:Method: GET 2024-04-13T21:36:11.177 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.177 DEBUG:boto:Data: 2024-04-13T21:36:11.177 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.177 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.177 DEBUG:boto:Params: {} 2024-04-13T21:36:11.178 DEBUG:boto:Token: None 2024-04-13T21:36:11.178 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d084f812518ac8d-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.226 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.227 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.227 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.227 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.227 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.227 DEBUG:boto:Method: GET 2024-04-13T21:36:11.227 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.227 DEBUG:boto:Data: 2024-04-13T21:36:11.227 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.228 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.228 DEBUG:boto:Params: {} 2024-04-13T21:36:11.228 DEBUG:boto:Token: None 2024-04-13T21:36:11.228 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000686b39ed0da4b8db-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.231 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.231 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.231 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.231 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.232 DEBUG:boto:Path: /vwnfns-13/dummy 2024-04-13T21:36:11.232 DEBUG:boto:Data: 2024-04-13T21:36:11.232 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.232 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.232 DEBUG:boto:Params: {} 2024-04-13T21:36:11.232 DEBUG:boto:Token: None 2024-04-13T21:36:11.232 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy 2024-04-13T21:36:11.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xHdmzGN7OIvJ+nYY3MxT0wKuoA4= 2024-04-13T21:36:11.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xHdmzGN7OIvJ+nYY3MxT0wKuoA4=', 'Content-Length': '0'} 2024-04-13T21:36:11.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbd4368fe3474a12-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.235 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.235 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.235 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.235 DEBUG:boto:Path: /vwnfns-13/dummy 2024-04-13T21:36:11.235 DEBUG:boto:Data: 2024-04-13T21:36:11.235 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.235 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.235 DEBUG:boto:Params: {} 2024-04-13T21:36:11.236 DEBUG:boto:Token: None 2024-04-13T21:36:11.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy 2024-04-13T21:36:11.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xHdmzGN7OIvJ+nYY3MxT0wKuoA4= 2024-04-13T21:36:11.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xHdmzGN7OIvJ+nYY3MxT0wKuoA4=', 'Content-Length': '0'} 2024-04-13T21:36:11.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0702c4d271b9b8c-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:36:11.238 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.238 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.239 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.239 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.239 DEBUG:boto:Method: GET 2024-04-13T21:36:11.239 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.239 DEBUG:boto:Data: 2024-04-13T21:36:11.239 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.239 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.239 DEBUG:boto:Params: {} 2024-04-13T21:36:11.239 DEBUG:boto:Token: None 2024-04-13T21:36:11.240 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU= 2024-04-13T21:36:11.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU=', 'Content-Length': '0'} 2024-04-13T21:36:11.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090771cdeb636938a-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.242 DEBUG:boto:path=/vwnfns-13/dummy 2024-04-13T21:36:11.242 DEBUG:boto:auth_path=/vwnfns-13/dummy 2024-04-13T21:36:11.242 DEBUG:boto:path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.242 DEBUG:boto:auth_path=/vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.242 DEBUG:boto:Method: GET 2024-04-13T21:36:11.242 DEBUG:boto:Path: /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.242 DEBUG:boto:Data: 2024-04-13T21:36:11.243 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.243 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.243 DEBUG:boto:Params: {} 2024-04-13T21:36:11.243 DEBUG:boto:Token: None 2024-04-13T21:36:11.243 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/dummy?versionId=UE-6VjGEvakKflwLsB85XvcXx7.WAXu 2024-04-13T21:36:11.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU= 2024-04-13T21:36:11.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SG5e7m/oYckjTuWvzZezqUwmCVU=', 'Content-Length': '0'} 2024-04-13T21:36:11.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:28 GMT'), ('x-amz-version-id', 'UE-6VjGEvakKflwLsB85XvcXx7.WAXu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b05bea10835adc5b-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.245 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.245 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.246 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.246 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.246 DEBUG:boto:Method: GET 2024-04-13T21:36:11.246 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.246 DEBUG:boto:Data: 2024-04-13T21:36:11.246 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.246 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.246 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.246 DEBUG:boto:Params: {} 2024-04-13T21:36:11.246 DEBUG:boto:Token: None 2024-04-13T21:36:11.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383d5c4027f26921-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.249 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.249 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.250 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.250 DEBUG:boto:path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.250 DEBUG:boto:auth_path=/vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.250 DEBUG:boto:Method: GET 2024-04-13T21:36:11.250 DEBUG:boto:Path: /vwnfns-13/?prefix=dummy 2024-04-13T21:36:11.250 DEBUG:boto:Data: 2024-04-13T21:36:11.250 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.250 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.250 DEBUG:boto:Params: {} 2024-04-13T21:36:11.251 DEBUG:boto:Token: None 2024-04-13T21:36:11.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c57aa4bcd7cdf29-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.253 DEBUG:boto:b'vwnfns-13dummy1000falsedummy2024-04-13T21:34:28.611Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:36:11.254 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.254 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.254 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.254 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.254 DEBUG:boto:Method: GET 2024-04-13T21:36:11.254 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.254 DEBUG:boto:Data: 2024-04-13T21:36:11.255 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.255 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.255 DEBUG:boto:Params: {} 2024-04-13T21:36:11.255 DEBUG:boto:Token: None 2024-04-13T21:36:11.255 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA= 2024-04-13T21:36:11.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA=', 'Content-Length': '0'} 2024-04-13T21:36:11.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000baaed529eda0e8d1-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.257 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.258 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.258 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.258 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.258 DEBUG:boto:Method: GET 2024-04-13T21:36:11.258 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.258 DEBUG:boto:Data: 2024-04-13T21:36:11.258 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.258 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.258 DEBUG:boto:Params: {} 2024-04-13T21:36:11.259 DEBUG:boto:Token: None 2024-04-13T21:36:11.259 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA= 2024-04-13T21:36:11.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA=', 'Content-Length': '0'} 2024-04-13T21:36:11.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006178c13df7cc007f-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.261 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.261 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.261 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.261 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.261 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.261 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.261 DEBUG:boto:Data: 2024-04-13T21:36:11.261 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.262 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.262 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.262 DEBUG:boto:Params: {} 2024-04-13T21:36:11.262 DEBUG:boto:Token: None 2024-04-13T21:36:11.262 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8KpiDx5y+PRTtcKr9LaXzo3cnYE= 2024-04-13T21:36:11.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8KpiDx5y+PRTtcKr9LaXzo3cnYE=', 'Content-Length': '0'} 2024-04-13T21:36:11.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f41888161cbe64df-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.265 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.265 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.265 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.265 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.265 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.265 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.266 DEBUG:boto:Data: 2024-04-13T21:36:11.266 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.266 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.266 DEBUG:boto:Params: {} 2024-04-13T21:36:11.266 DEBUG:boto:Token: None 2024-04-13T21:36:11.266 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8KpiDx5y+PRTtcKr9LaXzo3cnYE= 2024-04-13T21:36:11.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8KpiDx5y+PRTtcKr9LaXzo3cnYE=', 'Content-Length': '0'} 2024-04-13T21:36:11.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd0e5df0803c0821-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:36:11.269 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.269 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.269 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.269 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.269 DEBUG:boto:Method: GET 2024-04-13T21:36:11.269 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.269 DEBUG:boto:Data: 2024-04-13T21:36:11.269 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.270 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.270 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.270 DEBUG:boto:Params: {} 2024-04-13T21:36:11.270 DEBUG:boto:Token: None 2024-04-13T21:36:11.270 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.270 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA= 2024-04-13T21:36:11.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA=', 'Content-Length': '0'} 2024-04-13T21:36:11.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ef037cc7dfe7a0e-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.318 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.319 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.319 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.319 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.319 DEBUG:boto:Method: GET 2024-04-13T21:36:11.319 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.319 DEBUG:boto:Data: 2024-04-13T21:36:11.319 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.319 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.319 DEBUG:boto:Params: {} 2024-04-13T21:36:11.320 DEBUG:boto:Token: None 2024-04-13T21:36:11.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA= 2024-04-13T21:36:11.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA=', 'Content-Length': '0'} 2024-04-13T21:36:11.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068c01a4594fadc47-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.322 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.322 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.322 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.322 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.323 DEBUG:boto:Method: GET 2024-04-13T21:36:11.323 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.323 DEBUG:boto:Data: 2024-04-13T21:36:11.323 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.323 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.323 DEBUG:boto:Params: {} 2024-04-13T21:36:11.323 DEBUG:boto:Token: None 2024-04-13T21:36:11.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054226527ebb61275-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.366 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.367 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.367 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.367 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.367 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.367 DEBUG:boto:Method: GET 2024-04-13T21:36:11.367 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.367 DEBUG:boto:Data: 2024-04-13T21:36:11.367 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.368 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.368 DEBUG:boto:Params: {} 2024-04-13T21:36:11.368 DEBUG:boto:Token: None 2024-04-13T21:36:11.368 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d95cd13bd3e99eca-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.371 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.371 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.371 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.371 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.371 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:36:11.372 DEBUG:boto:Data: 2024-04-13T21:36:11.372 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.372 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.372 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.372 DEBUG:boto:Params: {} 2024-04-13T21:36:11.372 DEBUG:boto:Token: None 2024-04-13T21:36:11.372 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1 2024-04-13T21:36:11.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bE/W2jw/f+hgOKAqw5vHRe7Kols= 2024-04-13T21:36:11.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bE/W2jw/f+hgOKAqw5vHRe7Kols=', 'Content-Length': '0'} 2024-04-13T21:36:11.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aca5aef9b070fee6-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.375 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.375 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.375 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.375 DEBUG:boto:Path: /vwnfns-13/obj-a1 2024-04-13T21:36:11.375 DEBUG:boto:Data: 2024-04-13T21:36:11.375 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.375 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.376 DEBUG:boto:Params: {} 2024-04-13T21:36:11.376 DEBUG:boto:Token: None 2024-04-13T21:36:11.376 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1 2024-04-13T21:36:11.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bE/W2jw/f+hgOKAqw5vHRe7Kols= 2024-04-13T21:36:11.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bE/W2jw/f+hgOKAqw5vHRe7Kols=', 'Content-Length': '0'} 2024-04-13T21:36:11.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acc833ba645a21d4-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:36:11.379 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.379 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.379 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.379 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.379 DEBUG:boto:Method: GET 2024-04-13T21:36:11.379 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.379 DEBUG:boto:Data: 2024-04-13T21:36:11.379 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.380 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.380 DEBUG:boto:Params: {} 2024-04-13T21:36:11.380 DEBUG:boto:Token: None 2024-04-13T21:36:11.380 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA= 2024-04-13T21:36:11.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA=', 'Content-Length': '0'} 2024-04-13T21:36:11.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d8db1c40ced5044-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.422 DEBUG:boto:path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.422 DEBUG:boto:auth_path=/vwnfns-13/obj-a1 2024-04-13T21:36:11.423 DEBUG:boto:path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.423 DEBUG:boto:auth_path=/vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.423 DEBUG:boto:Method: GET 2024-04-13T21:36:11.423 DEBUG:boto:Path: /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.423 DEBUG:boto:Data: 2024-04-13T21:36:11.423 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.423 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.423 DEBUG:boto:Params: {} 2024-04-13T21:36:11.423 DEBUG:boto:Token: None 2024-04-13T21:36:11.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a1?versionId=hHN8DBGbwxHORHE-cQZPwkqECLVqXn2 2024-04-13T21:36:11.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA= 2024-04-13T21:36:11.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BCKHz1igEDfUfE+Z0RxIGottnLA=', 'Content-Length': '0'} 2024-04-13T21:36:11.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'hHN8DBGbwxHORHE-cQZPwkqECLVqXn2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bba9b4b49689a901-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.426 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.426 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.426 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.426 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.426 DEBUG:boto:Method: GET 2024-04-13T21:36:11.426 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.426 DEBUG:boto:Data: 2024-04-13T21:36:11.427 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.427 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.427 DEBUG:boto:Params: {} 2024-04-13T21:36:11.427 DEBUG:boto:Token: None 2024-04-13T21:36:11.427 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edcb8ed066efbfd2-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.470 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.471 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.471 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.471 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.471 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.471 DEBUG:boto:Method: GET 2024-04-13T21:36:11.471 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a1 2024-04-13T21:36:11.471 DEBUG:boto:Data: 2024-04-13T21:36:11.471 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.472 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.472 DEBUG:boto:Params: {} 2024-04-13T21:36:11.472 DEBUG:boto:Token: None 2024-04-13T21:36:11.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f9ff7910017f84-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.474 DEBUG:boto:b'vwnfns-13obj-a11000falseobj-a12024-04-13T21:35:08.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:36:11.475 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.475 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.475 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.475 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.475 DEBUG:boto:Method: GET 2024-04-13T21:36:11.475 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.476 DEBUG:boto:Data: 2024-04-13T21:36:11.476 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.476 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.476 DEBUG:boto:Params: {} 2024-04-13T21:36:11.476 DEBUG:boto:Token: None 2024-04-13T21:36:11.476 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA= 2024-04-13T21:36:11.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA=', 'Content-Length': '0'} 2024-04-13T21:36:11.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000759017b3b7c9b6dd-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.478 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.479 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.479 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.479 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.479 DEBUG:boto:Method: GET 2024-04-13T21:36:11.479 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.479 DEBUG:boto:Data: 2024-04-13T21:36:11.479 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.479 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.479 DEBUG:boto:Params: {} 2024-04-13T21:36:11.480 DEBUG:boto:Token: None 2024-04-13T21:36:11.480 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA= 2024-04-13T21:36:11.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA=', 'Content-Length': '0'} 2024-04-13T21:36:11.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2da9e99b25dbd3a-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.522 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.523 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.523 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.523 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.523 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.523 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.523 DEBUG:boto:Data: 2024-04-13T21:36:11.523 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.523 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.523 DEBUG:boto:Params: {} 2024-04-13T21:36:11.524 DEBUG:boto:Token: None 2024-04-13T21:36:11.524 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tFheprWi39U9J1WU8LWysY5W4UU= 2024-04-13T21:36:11.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tFheprWi39U9J1WU8LWysY5W4UU=', 'Content-Length': '0'} 2024-04-13T21:36:11.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000152c8dadeb6155aa-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.526 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.526 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.526 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.526 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.526 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.526 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.527 DEBUG:boto:Data: 2024-04-13T21:36:11.527 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.527 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.527 DEBUG:boto:Params: {} 2024-04-13T21:36:11.527 DEBUG:boto:Token: None 2024-04-13T21:36:11.527 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tFheprWi39U9J1WU8LWysY5W4UU= 2024-04-13T21:36:11.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tFheprWi39U9J1WU8LWysY5W4UU=', 'Content-Length': '0'} 2024-04-13T21:36:11.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0fc028e1e2a29c6-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:36:11.530 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.530 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.530 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.530 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.530 DEBUG:boto:Method: GET 2024-04-13T21:36:11.530 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.530 DEBUG:boto:Data: 2024-04-13T21:36:11.530 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.530 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.530 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.531 DEBUG:boto:Params: {} 2024-04-13T21:36:11.531 DEBUG:boto:Token: None 2024-04-13T21:36:11.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA= 2024-04-13T21:36:11.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA=', 'Content-Length': '0'} 2024-04-13T21:36:11.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047bc0daef75dd1c3-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.533 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.533 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.533 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.533 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.533 DEBUG:boto:Method: GET 2024-04-13T21:36:11.533 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.533 DEBUG:boto:Data: 2024-04-13T21:36:11.534 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.534 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.534 DEBUG:boto:Params: {} 2024-04-13T21:36:11.534 DEBUG:boto:Token: None 2024-04-13T21:36:11.534 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA= 2024-04-13T21:36:11.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA=', 'Content-Length': '0'} 2024-04-13T21:36:11.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000303cc28a891dd04f-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.582 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.583 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.583 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.583 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.583 DEBUG:boto:Method: GET 2024-04-13T21:36:11.583 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.583 DEBUG:boto:Data: 2024-04-13T21:36:11.583 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.583 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.583 DEBUG:boto:Params: {} 2024-04-13T21:36:11.584 DEBUG:boto:Token: None 2024-04-13T21:36:11.584 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739e7f828712620f-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.586 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.587 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.587 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.587 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.587 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.587 DEBUG:boto:Method: GET 2024-04-13T21:36:11.587 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.587 DEBUG:boto:Data: 2024-04-13T21:36:11.587 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.587 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.588 DEBUG:boto:Params: {} 2024-04-13T21:36:11.588 DEBUG:boto:Token: None 2024-04-13T21:36:11.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4dc390e39079de-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.590 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.591 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.591 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.591 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.591 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:36:11.591 DEBUG:boto:Data: 2024-04-13T21:36:11.591 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.591 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.592 DEBUG:boto:Params: {} 2024-04-13T21:36:11.592 DEBUG:boto:Token: None 2024-04-13T21:36:11.592 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2 2024-04-13T21:36:11.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KvqDaWMxdJG6XQSTQ/n686kA1to= 2024-04-13T21:36:11.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KvqDaWMxdJG6XQSTQ/n686kA1to=', 'Content-Length': '0'} 2024-04-13T21:36:11.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dc121752df495b3-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.594 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.595 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.595 DEBUG:boto:Method: HEAD 2024-04-13T21:36:11.595 DEBUG:boto:Path: /vwnfns-13/obj-a2 2024-04-13T21:36:11.595 DEBUG:boto:Data: 2024-04-13T21:36:11.595 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.595 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.595 DEBUG:boto:Params: {} 2024-04-13T21:36:11.595 DEBUG:boto:Token: None 2024-04-13T21:36:11.596 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2 2024-04-13T21:36:11.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KvqDaWMxdJG6XQSTQ/n686kA1to= 2024-04-13T21:36:11.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KvqDaWMxdJG6XQSTQ/n686kA1to=', 'Content-Length': '0'} 2024-04-13T21:36:11.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000763883bf17c293b7-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:36:11.598 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.598 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.598 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.598 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.599 DEBUG:boto:Method: GET 2024-04-13T21:36:11.599 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.599 DEBUG:boto:Data: 2024-04-13T21:36:11.599 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.599 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.599 DEBUG:boto:Params: {} 2024-04-13T21:36:11.599 DEBUG:boto:Token: None 2024-04-13T21:36:11.599 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA= 2024-04-13T21:36:11.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA=', 'Content-Length': '0'} 2024-04-13T21:36:11.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e958fb53fc579ed-00661afacb-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.642 DEBUG:boto:path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.642 DEBUG:boto:auth_path=/vwnfns-13/obj-a2 2024-04-13T21:36:11.643 DEBUG:boto:path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.643 DEBUG:boto:auth_path=/vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.643 DEBUG:boto:Method: GET 2024-04-13T21:36:11.643 DEBUG:boto:Path: /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.643 DEBUG:boto:Data: 2024-04-13T21:36:11.643 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.643 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.643 DEBUG:boto:Params: {} 2024-04-13T21:36:11.643 DEBUG:boto:Token: None 2024-04-13T21:36:11.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/obj-a2?versionId=c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej 2024-04-13T21:36:11.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA= 2024-04-13T21:36:11.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yPOyJVbzZsYhvefsvWq+gEbX4tA=', 'Content-Length': '0'} 2024-04-13T21:36:11.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:08 GMT'), ('x-amz-version-id', 'c3GMP6xszEoVPbSZOTFyjE9frkzW5Ej'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056b0244eb1804b5a-00661afacb-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.690 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.691 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.691 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.691 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.691 DEBUG:boto:Method: GET 2024-04-13T21:36:11.691 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.691 DEBUG:boto:Data: 2024-04-13T21:36:11.691 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:36:11.691 DEBUG:boto:Port: 8001 2024-04-13T21:36:11.691 DEBUG:boto:Params: {} 2024-04-13T21:36:11.692 DEBUG:boto:Token: None 2024-04-13T21:36:11.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3dc4b6798477e47-00661afacb-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.694 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.695 DEBUG:boto:path=/vwnfns-13/ 2024-04-13T21:36:11.695 DEBUG:boto:auth_path=/vwnfns-13/ 2024-04-13T21:36:11.695 DEBUG:boto:path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.695 DEBUG:boto:auth_path=/vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.695 DEBUG:boto:Method: GET 2024-04-13T21:36:11.695 DEBUG:boto:Path: /vwnfns-13/?prefix=obj-a2 2024-04-13T21:36:11.695 DEBUG:boto:Data: 2024-04-13T21:36:11.695 DEBUG:boto:Headers: {} 2024-04-13T21:36:11.696 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:36:11.696 DEBUG:boto:Port: 8000 2024-04-13T21:36:11.696 DEBUG:boto:Params: {} 2024-04-13T21:36:11.696 DEBUG:boto:Token: None 2024-04-13T21:36:11.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:36:11 GMT /vwnfns-13/ 2024-04-13T21:36:11.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY= 2024-04-13T21:36:11.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:36:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgAqwsNPxNyen0xsvyi0p/OjbxY=', 'Content-Length': '0'} 2024-04-13T21:36:11.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4f2d1fb2fac9f0-00661afacb-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:36:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:36:11.698 DEBUG:boto:b'vwnfns-13obj-a21000falseobj-a22024-04-13T21:35:08.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:36:11.699 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-13 zones={a2, a1} 2024-04-13T21:36:11.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-14 2024-04-13T21:36:11.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:36:11.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:36:11.699 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:36:11.762 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:36:11.762 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:36:11.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:11.808+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:11.800 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:11.800 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:16.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:16.808+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:16.801 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:16.801 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:21.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:21.809+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:21.803 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:21.803 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:26.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:26.811+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:26.804 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:26.804 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:31.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:31.812+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:31.806 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:31.806 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:36.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:36.814+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:36.807 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:36.807 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:41.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:41.815+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:41.808 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:41.808 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:46.809 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:46.817+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:46.809 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:46.809 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:51.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:51.817+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:51.810 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:51.811 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:36:56.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:36:56.818+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:36:56.811 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:36:56.811 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:01.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:01.820+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:37:01.813 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:37:01.813 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:06.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:06.821+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:37:06.814 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:37:06.814 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:11.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:11.823+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:37:11.815 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:37:11.815 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:16.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:16.824+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:37:16.817 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:37:16.817 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:21.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:21.826+0000 7fa1ce3ae780 1 waiting for incremental sync to catch up: 2024-04-13T21:37:21.818 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000010.29.6, ] 2024-04-13T21:37:21.818 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:26.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:26.827+0000 7fa1ce3ae780 1 bucket sync caught up with source: 2024-04-13T21:37:26.819 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:26.819 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000002.5.11, 00000000011.32.6, ] 2024-04-13T21:37:26.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:26.827+0000 7fa1ce3ae780 0 bucket checkpoint complete 2024-04-13T21:37:26.835 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-14 2024-04-13T21:37:26.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:37:26.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:37:26.836 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:37:26.898 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:37:26.898 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:37:26.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:26.949+0000 7f31ef841780 1 bucket sync caught up with source: 2024-04-13T21:37:26.940 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.33.10, , , 00000000011.37.6, 00000000002.32.11, ] 2024-04-13T21:37:26.941 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.33.10, , , 00000000011.37.6, 00000000002.32.11, ] 2024-04-13T21:37:26.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:26.949+0000 7f31ef841780 0 bucket checkpoint complete 2024-04-13T21:37:26.954 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-14 zones={a2, a1} 2024-04-13T21:37:26.954 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:26.954 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:26.954 DEBUG:boto:Method: HEAD 2024-04-13T21:37:26.955 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:37:26.955 DEBUG:boto:Data: 2024-04-13T21:37:26.955 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:26.955 DEBUG:boto:Port: 8001 2024-04-13T21:37:26.955 DEBUG:boto:Params: {} 2024-04-13T21:37:26.956 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:37:26.956 DEBUG:boto:Token: None 2024-04-13T21:37:26.956 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/ 2024-04-13T21:37:26.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pyXX89O0j1DFZtcb6IK9QH4jIL0= 2024-04-13T21:37:26.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pyXX89O0j1DFZtcb6IK9QH4jIL0=', 'Content-Length': '0'} 2024-04-13T21:37:26.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0d2aeae2c87ea1-00661afb16-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.962 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:26.962 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:26.962 DEBUG:boto:Method: HEAD 2024-04-13T21:37:26.962 DEBUG:boto:Path: /vwnfns-14/ 2024-04-13T21:37:26.962 DEBUG:boto:Data: 2024-04-13T21:37:26.962 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.962 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:26.962 DEBUG:boto:Port: 8000 2024-04-13T21:37:26.963 DEBUG:boto:Params: {} 2024-04-13T21:37:26.963 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:37:26.963 DEBUG:boto:Token: None 2024-04-13T21:37:26.963 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/ 2024-04-13T21:37:26.963 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pyXX89O0j1DFZtcb6IK9QH4jIL0= 2024-04-13T21:37:26.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pyXX89O0j1DFZtcb6IK9QH4jIL0=', 'Content-Length': '0'} 2024-04-13T21:37:26.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006521043a20b696bd-00661afb16-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:37:26.967 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:26.967 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:26.967 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.967 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.967 DEBUG:boto:Method: GET 2024-04-13T21:37:26.967 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.967 DEBUG:boto:Data: 2024-04-13T21:37:26.967 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:26.967 DEBUG:boto:Port: 8001 2024-04-13T21:37:26.968 DEBUG:boto:Params: {} 2024-04-13T21:37:26.968 DEBUG:boto:Token: None 2024-04-13T21:37:26.968 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/?versions 2024-04-13T21:37:26.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g= 2024-04-13T21:37:26.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g=', 'Content-Length': '0'} 2024-04-13T21:37:26.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f32266fae3f104-00661afb16-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.970 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:26.971 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:37:26.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:37:26.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:37:26.972 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:37:26.972 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:26.972 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:26.972 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.972 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.972 DEBUG:boto:Method: GET 2024-04-13T21:37:26.972 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.972 DEBUG:boto:Data: 2024-04-13T21:37:26.972 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.972 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:26.972 DEBUG:boto:Port: 8000 2024-04-13T21:37:26.973 DEBUG:boto:Params: {} 2024-04-13T21:37:26.973 DEBUG:boto:Token: None 2024-04-13T21:37:26.973 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/?versions 2024-04-13T21:37:26.973 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g= 2024-04-13T21:37:26.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g=', 'Content-Length': '0'} 2024-04-13T21:37:26.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d306f9fd91f464b9-00661afb16-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.975 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:26.976 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:37:26.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-13T21:37:26.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-13T21:37:26.976 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:26.976 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:26.976 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.976 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.976 DEBUG:boto:Method: GET 2024-04-13T21:37:26.976 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.977 DEBUG:boto:Data: 2024-04-13T21:37:26.977 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.977 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:26.977 DEBUG:boto:Port: 8001 2024-04-13T21:37:26.977 DEBUG:boto:Params: {} 2024-04-13T21:37:26.977 DEBUG:boto:Token: None 2024-04-13T21:37:26.977 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/?versions 2024-04-13T21:37:26.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g= 2024-04-13T21:37:26.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g=', 'Content-Length': '0'} 2024-04-13T21:37:26.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f2a913ccaa09440-00661afb16-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.979 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:26.980 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:26.980 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:26.980 DEBUG:boto:path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.980 DEBUG:boto:auth_path=/vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.980 DEBUG:boto:Method: GET 2024-04-13T21:37:26.980 DEBUG:boto:Path: /vwnfns-14/?versions&max-keys=999 2024-04-13T21:37:26.981 DEBUG:boto:Data: 2024-04-13T21:37:26.981 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:26.981 DEBUG:boto:Port: 8000 2024-04-13T21:37:26.981 DEBUG:boto:Params: {} 2024-04-13T21:37:26.981 DEBUG:boto:Token: None 2024-04-13T21:37:26.981 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/?versions 2024-04-13T21:37:26.981 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g= 2024-04-13T21:37:26.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mbEvyobHTM1AoqQfWKDnNcLVb5g=', 'Content-Length': '0'} 2024-04-13T21:37:26.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faaa858237a0327a-00661afb16-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.984 DEBUG:boto:b'vwnfns-14999falsedummyVEwPrF-pba.VgxyolxEKbOqpqWC-tAKtrue2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Dgg9IHP3eMDsJrFb99SQwF949kbUtqptrue2024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24zoIHxDyYrStev-dcMN8LcM4DCtogOitrue2024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:26.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:37:26.984 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:26.984 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:26.984 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.984 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.985 DEBUG:boto:Method: GET 2024-04-13T21:37:26.985 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.985 DEBUG:boto:Data: 2024-04-13T21:37:26.985 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.985 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:26.985 DEBUG:boto:Port: 8001 2024-04-13T21:37:26.985 DEBUG:boto:Params: {} 2024-04-13T21:37:26.985 DEBUG:boto:Token: None 2024-04-13T21:37:26.985 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PoZXWAuA0/yKXlkpzueI1cZdNzI= 2024-04-13T21:37:26.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PoZXWAuA0/yKXlkpzueI1cZdNzI=', 'Content-Length': '0'} 2024-04-13T21:37:26.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007da79dcd9835d0ae-00661afb16-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.987 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:26.988 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:26.988 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.988 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.988 DEBUG:boto:Method: GET 2024-04-13T21:37:26.988 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.988 DEBUG:boto:Data: 2024-04-13T21:37:26.988 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:26.988 DEBUG:boto:Port: 8000 2024-04-13T21:37:26.988 DEBUG:boto:Params: {} 2024-04-13T21:37:26.988 DEBUG:boto:Token: None 2024-04-13T21:37:26.989 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.989 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PoZXWAuA0/yKXlkpzueI1cZdNzI= 2024-04-13T21:37:26.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PoZXWAuA0/yKXlkpzueI1cZdNzI=', 'Content-Length': '0'} 2024-04-13T21:37:26.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003c57373a127bf67-00661afb16-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.990 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:26.991 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:26.991 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.991 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.991 DEBUG:boto:Method: HEAD 2024-04-13T21:37:26.991 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.991 DEBUG:boto:Data: 2024-04-13T21:37:26.991 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.992 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:26.992 DEBUG:boto:Port: 8001 2024-04-13T21:37:26.992 DEBUG:boto:Params: {} 2024-04-13T21:37:26.992 DEBUG:boto:Token: None 2024-04-13T21:37:26.992 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TYLsl/F3Rlh/c9CZX7WS0mXNuCg= 2024-04-13T21:37:26.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TYLsl/F3Rlh/c9CZX7WS0mXNuCg=', 'Content-Length': '0'} 2024-04-13T21:37:26.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f7a0c650da1f4f7-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.994 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:26.994 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:26.994 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.994 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.995 DEBUG:boto:Method: HEAD 2024-04-13T21:37:26.995 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.995 DEBUG:boto:Data: 2024-04-13T21:37:26.995 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:26.995 DEBUG:boto:Port: 8000 2024-04-13T21:37:26.995 DEBUG:boto:Params: {} 2024-04-13T21:37:26.995 DEBUG:boto:Token: None 2024-04-13T21:37:26.995 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TYLsl/F3Rlh/c9CZX7WS0mXNuCg= 2024-04-13T21:37:26.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TYLsl/F3Rlh/c9CZX7WS0mXNuCg=', 'Content-Length': '0'} 2024-04-13T21:37:26.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003287fd1031cf0dc0-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:26.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:37:26.998 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:26.998 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:26.998 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.998 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.998 DEBUG:boto:Method: GET 2024-04-13T21:37:26.998 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.998 DEBUG:boto:Data: 2024-04-13T21:37:26.998 DEBUG:boto:Headers: {} 2024-04-13T21:37:26.998 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:26.998 DEBUG:boto:Port: 8001 2024-04-13T21:37:26.999 DEBUG:boto:Params: {} 2024-04-13T21:37:26.999 DEBUG:boto:Token: None 2024-04-13T21:37:26.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:26 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:26.999 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PoZXWAuA0/yKXlkpzueI1cZdNzI= 2024-04-13T21:37:26.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PoZXWAuA0/yKXlkpzueI1cZdNzI=', 'Content-Length': '0'} 2024-04-13T21:37:27.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003facbf84cdf1300e-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.001 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:27.001 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:27.001 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.001 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.001 DEBUG:boto:Method: GET 2024-04-13T21:37:27.001 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.001 DEBUG:boto:Data: 2024-04-13T21:37:27.001 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.001 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.002 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.002 DEBUG:boto:Params: {} 2024-04-13T21:37:27.002 DEBUG:boto:Token: None 2024-04-13T21:37:27.002 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.002 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pZvQ44KTfOSkBVt15YSIAWZ4ZhY= 2024-04-13T21:37:27.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pZvQ44KTfOSkBVt15YSIAWZ4ZhY=', 'Content-Length': '0'} 2024-04-13T21:37:27.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bc820dd50abeb35-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.004 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.004 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.004 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.004 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.004 DEBUG:boto:Method: GET 2024-04-13T21:37:27.004 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.004 DEBUG:boto:Data: 2024-04-13T21:37:27.005 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.005 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.005 DEBUG:boto:Params: {} 2024-04-13T21:37:27.005 DEBUG:boto:Token: None 2024-04-13T21:37:27.005 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.005 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c78af4a5ca996c-00661afb17-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.007 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.008 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.008 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.008 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.008 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.008 DEBUG:boto:Method: GET 2024-04-13T21:37:27.008 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.008 DEBUG:boto:Data: 2024-04-13T21:37:27.009 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.009 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.009 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.009 DEBUG:boto:Params: {} 2024-04-13T21:37:27.009 DEBUG:boto:Token: None 2024-04-13T21:37:27.009 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f700a9f53d2d90f6-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.054 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.055 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:27.055 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:27.055 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.056 DEBUG:boto:Path: /vwnfns-14/dummy 2024-04-13T21:37:27.056 DEBUG:boto:Data: 2024-04-13T21:37:27.056 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.056 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.056 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.056 DEBUG:boto:Params: {} 2024-04-13T21:37:27.057 DEBUG:boto:Token: None 2024-04-13T21:37:27.057 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/dummy 2024-04-13T21:37:27.057 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EjY5RI7aDZwTdysrFXY8tt0+yEM= 2024-04-13T21:37:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EjY5RI7aDZwTdysrFXY8tt0+yEM=', 'Content-Length': '0'} 2024-04-13T21:37:27.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071c2914ff29bae0e-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.060 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:27.060 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:27.060 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.060 DEBUG:boto:Path: /vwnfns-14/dummy 2024-04-13T21:37:27.060 DEBUG:boto:Data: 2024-04-13T21:37:27.060 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.060 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.061 DEBUG:boto:Params: {} 2024-04-13T21:37:27.061 DEBUG:boto:Token: None 2024-04-13T21:37:27.061 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/dummy 2024-04-13T21:37:27.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EjY5RI7aDZwTdysrFXY8tt0+yEM= 2024-04-13T21:37:27.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EjY5RI7aDZwTdysrFXY8tt0+yEM=', 'Content-Length': '0'} 2024-04-13T21:37:27.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d90856060073ef5-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:37:27.063 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:27.063 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:27.063 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.063 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.063 DEBUG:boto:Method: GET 2024-04-13T21:37:27.063 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.064 DEBUG:boto:Data: 2024-04-13T21:37:27.064 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.064 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.064 DEBUG:boto:Params: {} 2024-04-13T21:37:27.064 DEBUG:boto:Token: None 2024-04-13T21:37:27.064 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.064 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pZvQ44KTfOSkBVt15YSIAWZ4ZhY= 2024-04-13T21:37:27.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pZvQ44KTfOSkBVt15YSIAWZ4ZhY=', 'Content-Length': '0'} 2024-04-13T21:37:27.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f082e7ea8287d21f-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.066 DEBUG:boto:path=/vwnfns-14/dummy 2024-04-13T21:37:27.066 DEBUG:boto:auth_path=/vwnfns-14/dummy 2024-04-13T21:37:27.066 DEBUG:boto:path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.067 DEBUG:boto:auth_path=/vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.067 DEBUG:boto:Method: GET 2024-04-13T21:37:27.067 DEBUG:boto:Path: /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.067 DEBUG:boto:Data: 2024-04-13T21:37:27.067 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.067 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.067 DEBUG:boto:Params: {} 2024-04-13T21:37:27.067 DEBUG:boto:Token: None 2024-04-13T21:37:27.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/dummy?versionId=VEwPrF-pba.VgxyolxEKbOqpqWC-tAK 2024-04-13T21:37:27.068 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pZvQ44KTfOSkBVt15YSIAWZ4ZhY= 2024-04-13T21:37:27.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pZvQ44KTfOSkBVt15YSIAWZ4ZhY=', 'Content-Length': '0'} 2024-04-13T21:37:27.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:34:48 GMT'), ('x-amz-version-id', 'VEwPrF-pba.VgxyolxEKbOqpqWC-tAK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d85a36df342ba984-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.070 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.070 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.070 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.070 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.070 DEBUG:boto:Method: GET 2024-04-13T21:37:27.070 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.070 DEBUG:boto:Data: 2024-04-13T21:37:27.070 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.070 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.070 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.071 DEBUG:boto:Params: {} 2024-04-13T21:37:27.071 DEBUG:boto:Token: None 2024-04-13T21:37:27.071 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037dfa52b182b7c09-00661afb17-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.118 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.119 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.119 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.119 DEBUG:boto:path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.119 DEBUG:boto:auth_path=/vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.119 DEBUG:boto:Method: GET 2024-04-13T21:37:27.119 DEBUG:boto:Path: /vwnfns-14/?prefix=dummy 2024-04-13T21:37:27.120 DEBUG:boto:Data: 2024-04-13T21:37:27.120 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.120 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.120 DEBUG:boto:Params: {} 2024-04-13T21:37:27.120 DEBUG:boto:Token: None 2024-04-13T21:37:27.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089bc06300786b2b2-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.123 DEBUG:boto:b'vwnfns-14dummy1000falsedummy2024-04-13T21:34:48.739Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:37:27.123 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.123 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.123 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.123 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.123 DEBUG:boto:Method: GET 2024-04-13T21:37:27.124 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.124 DEBUG:boto:Data: 2024-04-13T21:37:27.124 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.124 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.124 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.124 DEBUG:boto:Params: {} 2024-04-13T21:37:27.124 DEBUG:boto:Token: None 2024-04-13T21:37:27.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA= 2024-04-13T21:37:27.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA=', 'Content-Length': '0'} 2024-04-13T21:37:27.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007454269530136748-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.126 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.126 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.127 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.127 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.127 DEBUG:boto:Method: GET 2024-04-13T21:37:27.127 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.127 DEBUG:boto:Data: 2024-04-13T21:37:27.127 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.127 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.127 DEBUG:boto:Params: {} 2024-04-13T21:37:27.127 DEBUG:boto:Token: None 2024-04-13T21:37:27.128 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.128 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA= 2024-04-13T21:37:27.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA=', 'Content-Length': '0'} 2024-04-13T21:37:27.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf5d0e45ef36c3b6-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.170 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.171 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.171 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.171 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.171 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.171 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.171 DEBUG:boto:Data: 2024-04-13T21:37:27.171 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.171 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.171 DEBUG:boto:Params: {} 2024-04-13T21:37:27.172 DEBUG:boto:Token: None 2024-04-13T21:37:27.172 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oPeka3iuGZrWDY7FVgPdph57LKE= 2024-04-13T21:37:27.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oPeka3iuGZrWDY7FVgPdph57LKE=', 'Content-Length': '0'} 2024-04-13T21:37:27.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce34732072164ce5-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.174 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.174 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.174 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.175 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.175 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.175 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.175 DEBUG:boto:Data: 2024-04-13T21:37:27.175 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.175 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.175 DEBUG:boto:Params: {} 2024-04-13T21:37:27.175 DEBUG:boto:Token: None 2024-04-13T21:37:27.175 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oPeka3iuGZrWDY7FVgPdph57LKE= 2024-04-13T21:37:27.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oPeka3iuGZrWDY7FVgPdph57LKE=', 'Content-Length': '0'} 2024-04-13T21:37:27.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c8d54b6f3357704-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:37:27.177 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.178 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.178 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.178 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.178 DEBUG:boto:Method: GET 2024-04-13T21:37:27.178 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.178 DEBUG:boto:Data: 2024-04-13T21:37:27.178 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.178 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.178 DEBUG:boto:Params: {} 2024-04-13T21:37:27.179 DEBUG:boto:Token: None 2024-04-13T21:37:27.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA= 2024-04-13T21:37:27.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA=', 'Content-Length': '0'} 2024-04-13T21:37:27.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9aa5a34776a5005-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.222 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.222 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.223 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.223 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.223 DEBUG:boto:Method: GET 2024-04-13T21:37:27.223 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.223 DEBUG:boto:Data: 2024-04-13T21:37:27.223 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.223 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.223 DEBUG:boto:Params: {} 2024-04-13T21:37:27.223 DEBUG:boto:Token: None 2024-04-13T21:37:27.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA= 2024-04-13T21:37:27.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA=', 'Content-Length': '0'} 2024-04-13T21:37:27.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084466caac5f8fc01-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.226 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.226 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.226 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.226 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.226 DEBUG:boto:Method: GET 2024-04-13T21:37:27.227 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.227 DEBUG:boto:Data: 2024-04-13T21:37:27.227 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.227 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.227 DEBUG:boto:Params: {} 2024-04-13T21:37:27.227 DEBUG:boto:Token: None 2024-04-13T21:37:27.227 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.227 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa752bee5b9702b-00661afb17-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.230 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.230 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.230 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.231 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.231 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.231 DEBUG:boto:Method: GET 2024-04-13T21:37:27.231 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.231 DEBUG:boto:Data: 2024-04-13T21:37:27.231 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.231 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.231 DEBUG:boto:Params: {} 2024-04-13T21:37:27.231 DEBUG:boto:Token: None 2024-04-13T21:37:27.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931dccb5c660b10e-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.234 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.234 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.234 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.235 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.235 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:37:27.235 DEBUG:boto:Data: 2024-04-13T21:37:27.235 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.235 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.235 DEBUG:boto:Params: {} 2024-04-13T21:37:27.235 DEBUG:boto:Token: None 2024-04-13T21:37:27.235 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1 2024-04-13T21:37:27.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AKTRUVwaPJvpm9uUaYX0S1/sZuc= 2024-04-13T21:37:27.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AKTRUVwaPJvpm9uUaYX0S1/sZuc=', 'Content-Length': '0'} 2024-04-13T21:37:27.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebcf773a53ff2a6d-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.237 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.238 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.238 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.238 DEBUG:boto:Path: /vwnfns-14/obj-a1 2024-04-13T21:37:27.238 DEBUG:boto:Data: 2024-04-13T21:37:27.238 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.238 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.238 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.238 DEBUG:boto:Params: {} 2024-04-13T21:37:27.238 DEBUG:boto:Token: None 2024-04-13T21:37:27.238 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1 2024-04-13T21:37:27.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AKTRUVwaPJvpm9uUaYX0S1/sZuc= 2024-04-13T21:37:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AKTRUVwaPJvpm9uUaYX0S1/sZuc=', 'Content-Length': '0'} 2024-04-13T21:37:27.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001560ac712630033d-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-13T21:37:27.241 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.241 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.241 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.241 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.241 DEBUG:boto:Method: GET 2024-04-13T21:37:27.241 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.241 DEBUG:boto:Data: 2024-04-13T21:37:27.241 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.241 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.242 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.242 DEBUG:boto:Params: {} 2024-04-13T21:37:27.242 DEBUG:boto:Token: None 2024-04-13T21:37:27.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA= 2024-04-13T21:37:27.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA=', 'Content-Length': '0'} 2024-04-13T21:37:27.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d52e8f2072bddbd-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.286 DEBUG:boto:path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.287 DEBUG:boto:auth_path=/vwnfns-14/obj-a1 2024-04-13T21:37:27.287 DEBUG:boto:path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.287 DEBUG:boto:auth_path=/vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.287 DEBUG:boto:Method: GET 2024-04-13T21:37:27.287 DEBUG:boto:Path: /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.287 DEBUG:boto:Data: 2024-04-13T21:37:27.287 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.287 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.287 DEBUG:boto:Params: {} 2024-04-13T21:37:27.288 DEBUG:boto:Token: None 2024-04-13T21:37:27.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a1?versionId=Dgg9IHP3eMDsJrFb99SQwF949kbUtqp 2024-04-13T21:37:27.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA= 2024-04-13T21:37:27.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RSLffsJOGsZvRskuRMYUezQwWUA=', 'Content-Length': '0'} 2024-04-13T21:37:27.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', 'Dgg9IHP3eMDsJrFb99SQwF949kbUtqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039065129375c7049-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.290 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.290 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.290 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.290 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.290 DEBUG:boto:Method: GET 2024-04-13T21:37:27.290 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.291 DEBUG:boto:Data: 2024-04-13T21:37:27.291 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.291 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.291 DEBUG:boto:Params: {} 2024-04-13T21:37:27.291 DEBUG:boto:Token: None 2024-04-13T21:37:27.291 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b6e90a801fd2cf1-00661afb17-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.338 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.339 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.339 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.339 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.339 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.339 DEBUG:boto:Method: GET 2024-04-13T21:37:27.339 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a1 2024-04-13T21:37:27.340 DEBUG:boto:Data: 2024-04-13T21:37:27.340 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.340 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.340 DEBUG:boto:Params: {} 2024-04-13T21:37:27.340 DEBUG:boto:Token: None 2024-04-13T21:37:27.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a358f84e3eca6549-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.343 DEBUG:boto:b'vwnfns-14obj-a11000falseobj-a12024-04-13T21:35:09.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:37:27.343 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.344 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.344 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.344 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.344 DEBUG:boto:Method: GET 2024-04-13T21:37:27.344 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.344 DEBUG:boto:Data: 2024-04-13T21:37:27.344 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.344 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.344 DEBUG:boto:Params: {} 2024-04-13T21:37:27.345 DEBUG:boto:Token: None 2024-04-13T21:37:27.345 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8= 2024-04-13T21:37:27.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8=', 'Content-Length': '0'} 2024-04-13T21:37:27.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000467b6e3a89171b4d-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.346 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.347 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.347 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.347 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.347 DEBUG:boto:Method: GET 2024-04-13T21:37:27.347 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.347 DEBUG:boto:Data: 2024-04-13T21:37:27.347 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.347 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.347 DEBUG:boto:Params: {} 2024-04-13T21:37:27.348 DEBUG:boto:Token: None 2024-04-13T21:37:27.348 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8= 2024-04-13T21:37:27.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8=', 'Content-Length': '0'} 2024-04-13T21:37:27.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a54a11d0ae898e80-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.350 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.350 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.350 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.350 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.350 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.351 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.351 DEBUG:boto:Data: 2024-04-13T21:37:27.351 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.351 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.351 DEBUG:boto:Params: {} 2024-04-13T21:37:27.351 DEBUG:boto:Token: None 2024-04-13T21:37:27.351 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BzcHOf9BObNhuaIpq2t6qLSSSLI= 2024-04-13T21:37:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BzcHOf9BObNhuaIpq2t6qLSSSLI=', 'Content-Length': '0'} 2024-04-13T21:37:27.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009389a60015be5ee0-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.353 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.353 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.354 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.354 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.354 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.354 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.354 DEBUG:boto:Data: 2024-04-13T21:37:27.354 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.354 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.354 DEBUG:boto:Params: {} 2024-04-13T21:37:27.354 DEBUG:boto:Token: None 2024-04-13T21:37:27.355 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.355 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BzcHOf9BObNhuaIpq2t6qLSSSLI= 2024-04-13T21:37:27.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BzcHOf9BObNhuaIpq2t6qLSSSLI=', 'Content-Length': '0'} 2024-04-13T21:37:27.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf5f523c18a58c7a-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:37:27.357 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.357 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.357 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.357 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.357 DEBUG:boto:Method: GET 2024-04-13T21:37:27.357 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.357 DEBUG:boto:Data: 2024-04-13T21:37:27.357 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.358 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.358 DEBUG:boto:Params: {} 2024-04-13T21:37:27.358 DEBUG:boto:Token: None 2024-04-13T21:37:27.358 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8= 2024-04-13T21:37:27.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8=', 'Content-Length': '0'} 2024-04-13T21:37:27.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008aa125b408e25bc5-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.360 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.360 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.360 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.360 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.360 DEBUG:boto:Method: GET 2024-04-13T21:37:27.361 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.361 DEBUG:boto:Data: 2024-04-13T21:37:27.361 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.361 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.361 DEBUG:boto:Params: {} 2024-04-13T21:37:27.361 DEBUG:boto:Token: None 2024-04-13T21:37:27.361 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.361 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8= 2024-04-13T21:37:27.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8=', 'Content-Length': '0'} 2024-04-13T21:37:27.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0e47bc0cd62ef8c-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.406 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.406 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.407 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.407 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.407 DEBUG:boto:Method: GET 2024-04-13T21:37:27.407 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.407 DEBUG:boto:Data: 2024-04-13T21:37:27.407 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.407 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.407 DEBUG:boto:Params: {} 2024-04-13T21:37:27.407 DEBUG:boto:Token: None 2024-04-13T21:37:27.408 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665fed13ae0d3363-00661afb17-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.410 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.410 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.411 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.411 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.411 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.411 DEBUG:boto:Method: GET 2024-04-13T21:37:27.411 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.411 DEBUG:boto:Data: 2024-04-13T21:37:27.411 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.411 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.411 DEBUG:boto:Params: {} 2024-04-13T21:37:27.412 DEBUG:boto:Token: None 2024-04-13T21:37:27.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf7706b260d8ac64-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.454 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.455 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.455 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.455 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.455 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:37:27.455 DEBUG:boto:Data: 2024-04-13T21:37:27.455 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.455 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.455 DEBUG:boto:Params: {} 2024-04-13T21:37:27.455 DEBUG:boto:Token: None 2024-04-13T21:37:27.456 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2 2024-04-13T21:37:27.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VuJ1cN0GaxdDbo8bOozd+SOYjbU= 2024-04-13T21:37:27.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VuJ1cN0GaxdDbo8bOozd+SOYjbU=', 'Content-Length': '0'} 2024-04-13T21:37:27.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4fca234b80b1b38-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.458 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.458 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.458 DEBUG:boto:Method: HEAD 2024-04-13T21:37:27.458 DEBUG:boto:Path: /vwnfns-14/obj-a2 2024-04-13T21:37:27.458 DEBUG:boto:Data: 2024-04-13T21:37:27.458 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.459 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.459 DEBUG:boto:Params: {} 2024-04-13T21:37:27.459 DEBUG:boto:Token: None 2024-04-13T21:37:27.459 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2 2024-04-13T21:37:27.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VuJ1cN0GaxdDbo8bOozd+SOYjbU= 2024-04-13T21:37:27.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VuJ1cN0GaxdDbo8bOozd+SOYjbU=', 'Content-Length': '0'} 2024-04-13T21:37:27.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb1533ab8d9512df-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-13T21:37:27.461 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.461 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.461 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.461 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.461 DEBUG:boto:Method: GET 2024-04-13T21:37:27.461 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.462 DEBUG:boto:Data: 2024-04-13T21:37:27.462 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.462 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.462 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.462 DEBUG:boto:Params: {} 2024-04-13T21:37:27.462 DEBUG:boto:Token: None 2024-04-13T21:37:27.462 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.462 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8= 2024-04-13T21:37:27.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8=', 'Content-Length': '0'} 2024-04-13T21:37:27.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6385805a1579de6-00661afb17-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.464 DEBUG:boto:path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.464 DEBUG:boto:auth_path=/vwnfns-14/obj-a2 2024-04-13T21:37:27.464 DEBUG:boto:path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.464 DEBUG:boto:auth_path=/vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.464 DEBUG:boto:Method: GET 2024-04-13T21:37:27.465 DEBUG:boto:Path: /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.465 DEBUG:boto:Data: 2024-04-13T21:37:27.465 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.465 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.465 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.465 DEBUG:boto:Params: {} 2024-04-13T21:37:27.465 DEBUG:boto:Token: None 2024-04-13T21:37:27.465 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/obj-a2?versionId=4zoIHxDyYrStev-dcMN8LcM4DCtogOi 2024-04-13T21:37:27.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8= 2024-04-13T21:37:27.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JmAJxB2wUUn0JsjOsAKvcjcEoG8=', 'Content-Length': '0'} 2024-04-13T21:37:27.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:35:09 GMT'), ('x-amz-version-id', '4zoIHxDyYrStev-dcMN8LcM4DCtogOi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076520e63b1171684-00661afb17-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.510 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.510 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.511 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.511 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.511 DEBUG:boto:Method: GET 2024-04-13T21:37:27.511 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.511 DEBUG:boto:Data: 2024-04-13T21:37:27.511 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:37:27.511 DEBUG:boto:Port: 8001 2024-04-13T21:37:27.511 DEBUG:boto:Params: {} 2024-04-13T21:37:27.511 DEBUG:boto:Token: None 2024-04-13T21:37:27.511 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda2cc251f8a09a4-00661afb17-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.514 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.514 DEBUG:boto:path=/vwnfns-14/ 2024-04-13T21:37:27.514 DEBUG:boto:auth_path=/vwnfns-14/ 2024-04-13T21:37:27.514 DEBUG:boto:path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.515 DEBUG:boto:auth_path=/vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.515 DEBUG:boto:Method: GET 2024-04-13T21:37:27.515 DEBUG:boto:Path: /vwnfns-14/?prefix=obj-a2 2024-04-13T21:37:27.515 DEBUG:boto:Data: 2024-04-13T21:37:27.515 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.515 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.515 DEBUG:boto:Params: {} 2024-04-13T21:37:27.515 DEBUG:boto:Token: None 2024-04-13T21:37:27.515 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-14/ 2024-04-13T21:37:27.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0= 2024-04-13T21:37:27.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cmbvy7WjAw1o2FKyyewCcrUuaY0=', 'Content-Length': '0'} 2024-04-13T21:37:27.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd3e46f0327a3f6-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.518 DEBUG:boto:b'vwnfns-14obj-a21000falseobj-a22024-04-13T21:35:09.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:37:27.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-14 zones={a2, a1} 2024-04-13T21:37:27.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-04-13T21:37:27.520 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:37:27.520 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:37:27.521 DEBUG:boto:Method: PUT 2024-04-13T21:37:27.521 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:37:27.521 DEBUG:boto:Data: 2024-04-13T21:37:27.521 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.521 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.521 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.521 DEBUG:boto:Params: {} 2024-04-13T21:37:27.521 DEBUG:boto:Token: None 2024-04-13T21:37:27.521 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-15/ 2024-04-13T21:37:27.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FzT0sYIY2sbfKPiyWaJNufo+K74= 2024-04-13T21:37:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FzT0sYIY2sbfKPiyWaJNufo+K74=', 'Content-Length': '0'} 2024-04-13T21:37:27.542 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ec3d8ae7c7f4554-00661afb17-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.542 DEBUG:rgw_multi.tests:created bucket=vwnfns-15 2024-04-13T21:37:27.543 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:37:27.543 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:37:27.543 DEBUG:boto:path=/vwnfns-15/?versioning 2024-04-13T21:37:27.543 DEBUG:boto:auth_path=/vwnfns-15/?versioning 2024-04-13T21:37:27.543 DEBUG:boto:Method: PUT 2024-04-13T21:37:27.543 DEBUG:boto:Path: /vwnfns-15/?versioning 2024-04-13T21:37:27.543 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:37:27.543 DEBUG:boto:Headers: {} 2024-04-13T21:37:27.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:27.543 DEBUG:boto:Port: 8000 2024-04-13T21:37:27.544 DEBUG:boto:Params: {} 2024-04-13T21:37:27.544 DEBUG:boto:Token: None 2024-04-13T21:37:27.544 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:37:27 GMT /vwnfns-15/?versioning 2024-04-13T21:37:27.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k6/iBL//OHTWQsGC+RnhnTdC68A= 2024-04-13T21:37:27.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k6/iBL//OHTWQsGC+RnhnTdC68A=', 'Content-Length': '228'} 2024-04-13T21:37:27.548 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006d8a0395c9bc4250-00661afb17-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:37:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:27.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:37:27.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:37:27.549 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:37:27.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:37:27.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:37:27.628 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044247.556946_181.1", 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:37:27.556946Z" 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.640 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:27.641 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:37:27.642 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044247.556946_181.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:37:27.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:37:27.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:37:27.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:37:27.643 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:37:27.691 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:37:27.691 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:37:27.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:27.727+0000 7f22feff4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:37:27.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:27.727+0000 7f22feff4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:37:27.730 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:37:27.731 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044247.540320_176.1", 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:37:27.540320Z", 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.732 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:27.734 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:37:27.735 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:37:27Z" 2024-04-13T21:37:27.736 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:37:27.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044247.540320_176.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:37:27.540320Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:37:27Z" } 2024-04-13T21:37:27.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044247.540320_176.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:37:27.540320Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:37:27.738 DEBUG:rgw_multi.tests:log_status={0: '1_1713044247.556946_181.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:37:27.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044247.540320_176.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:37:27.738 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044247.556946_181.1 target=1_1713044247.540320_176.1 2024-04-13T21:37:32.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:37:32.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:37:32.740 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:37:32.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:37:32.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:37:32.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:32.834+0000 7f332a0ea780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:37:32.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:32.835+0000 7f332a0ea780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:37:32.840 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:37:32.840 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:37:32.840 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:37:32.840 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:37:32.840 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044247.556946_181.1", 2024-04-13T21:37:32.841 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:37:27.556946Z", 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:32.842 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:37:32.843 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:37:32.844 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:37:32.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:32.846 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:37:32.846 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:37:32.846 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:37:32.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:37:32.846 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:37:32Z" 2024-04-13T21:37:32.846 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:37:32.847 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044247.556946_181.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:37:27.556946Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:37:32Z" } 2024-04-13T21:37:32.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044247.556946_181.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:37:27.556946Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:37:32.847 DEBUG:rgw_multi.tests:log_status={0: '1_1713044247.556946_181.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:37:32.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044247.556946_181.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:37:32.848 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:37:32.848 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:37:32.848 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:37:32.848 DEBUG:boto:Method: HEAD 2024-04-13T21:37:32.848 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:37:32.848 DEBUG:boto:Data: 2024-04-13T21:37:32.848 DEBUG:boto:Headers: {} 2024-04-13T21:37:32.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:32.849 DEBUG:boto:Port: 8000 2024-04-13T21:37:32.849 DEBUG:boto:Params: {} 2024-04-13T21:37:32.849 DEBUG:boto:Token: None 2024-04-13T21:37:32.849 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:37:32 GMT /vwnfns-15/ 2024-04-13T21:37:32.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GTR5n4yTff04qOLPYKDZCjadmHE= 2024-04-13T21:37:32.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:37:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GTR5n4yTff04qOLPYKDZCjadmHE=', 'Content-Length': '0'} 2024-04-13T21:37:32.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1f1a62977e3b626-00661afb1c-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:32.852 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:37:32.853 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:37:32.853 DEBUG:boto:Method: PUT 2024-04-13T21:37:32.853 DEBUG:boto:Path: /vwnfns-15/dummy 2024-04-13T21:37:32.853 DEBUG:boto:Data: 2024-04-13T21:37:32.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:37:32.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:37:32.853 DEBUG:boto:Port: 8000 2024-04-13T21:37:32.853 DEBUG:boto:Params: {} 2024-04-13T21:37:32.853 DEBUG:boto:Token: None 2024-04-13T21:37:32.853 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:37:32 GMT /vwnfns-15/dummy 2024-04-13T21:37:32.854 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/0W/15HRgbDqySEO04Z8OYysHNk= 2024-04-13T21:37:32.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:37:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/0W/15HRgbDqySEO04Z8OYysHNk='} 2024-04-13T21:37:32.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-amz-request-id', 'tx00000b3f1bb71b6ce4ab8-00661afb1c-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:37:32.864 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-15 2024-04-13T21:37:32.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:37:32.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:37:32.865 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:37:32.920 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:37:32.921 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:37:32.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:32.962+0000 7fd9f8a41780 1 waiting to reach incremental sync.. 2024-04-13T21:37:37.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:37.963+0000 7fd9f8a41780 1 waiting to reach incremental sync.. 2024-04-13T21:37:42.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:42.964+0000 7fd9f8a41780 1 waiting to reach incremental sync.. 2024-04-13T21:37:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:47.965+0000 7fd9f8a41780 1 bucket sync caught up with source: 2024-04-13T21:37:47.958 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.20.11, , , , , ] 2024-04-13T21:37:47.958 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.20.11, , , , , ] 2024-04-13T21:37:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:47.965+0000 7fd9f8a41780 0 bucket checkpoint complete 2024-04-13T21:37:47.974 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-15 2024-04-13T21:37:47.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:37:47.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:37:47.975 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:37:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:37:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:37:48.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:48.079+0000 7f67a3aa5780 1 waiting to reach incremental sync.. 2024-04-13T21:37:53.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:53.081+0000 7f67a3aa5780 1 waiting to reach incremental sync.. 2024-04-13T21:37:58.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:37:58.081+0000 7f67a3aa5780 1 waiting to reach incremental sync.. 2024-04-13T21:38:03.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:03.082+0000 7f67a3aa5780 1 waiting to reach incremental sync.. 2024-04-13T21:38:08.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:08.083+0000 7f67a3aa5780 1 bucket sync caught up with source: 2024-04-13T21:38:08.076 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.30.11, , , , , ] 2024-04-13T21:38:08.076 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.30.11, , , , , ] 2024-04-13T21:38:08.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:08.083+0000 7f67a3aa5780 0 bucket checkpoint complete 2024-04-13T21:38:08.091 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-15 zones={a2, a1} 2024-04-13T21:38:08.092 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.092 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.092 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.092 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.092 DEBUG:boto:Data: 2024-04-13T21:38:08.092 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.092 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.092 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.092 DEBUG:boto:Params: {} 2024-04-13T21:38:08.093 DEBUG:boto:Token: None 2024-04-13T21:38:08.093 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.093 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.097 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002eb132732c0a216e-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.097 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.097 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.097 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.098 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.098 DEBUG:boto:Data: 2024-04-13T21:38:08.098 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.098 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.098 DEBUG:boto:Params: {} 2024-04-13T21:38:08.098 DEBUG:boto:Token: None 2024-04-13T21:38:08.098 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000967e091cff1adf4a-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:38:08.101 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.101 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.101 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.102 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.102 DEBUG:boto:Method: GET 2024-04-13T21:38:08.102 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.102 DEBUG:boto:Data: 2024-04-13T21:38:08.102 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.102 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.102 DEBUG:boto:Params: {} 2024-04-13T21:38:08.102 DEBUG:boto:Token: None 2024-04-13T21:38:08.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/?versions 2024-04-13T21:38:08.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE= 2024-04-13T21:38:08.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE=', 'Content-Length': '0'} 2024-04-13T21:38:08.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c65db00b484eb32-00661afb40-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.105 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.106 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:38:08.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:38:08.107 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.107 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.107 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.107 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.107 DEBUG:boto:Method: GET 2024-04-13T21:38:08.107 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.107 DEBUG:boto:Data: 2024-04-13T21:38:08.107 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.107 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.108 DEBUG:boto:Params: {} 2024-04-13T21:38:08.108 DEBUG:boto:Token: None 2024-04-13T21:38:08.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/?versions 2024-04-13T21:38:08.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE= 2024-04-13T21:38:08.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE=', 'Content-Length': '0'} 2024-04-13T21:38:08.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007e302a871fb47f2-00661afb40-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.110 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.111 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:38:08.111 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.111 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.111 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.111 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.111 DEBUG:boto:Method: GET 2024-04-13T21:38:08.111 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.111 DEBUG:boto:Data: 2024-04-13T21:38:08.111 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.112 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.112 DEBUG:boto:Params: {} 2024-04-13T21:38:08.112 DEBUG:boto:Token: None 2024-04-13T21:38:08.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/?versions 2024-04-13T21:38:08.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE= 2024-04-13T21:38:08.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE=', 'Content-Length': '0'} 2024-04-13T21:38:08.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089571a1bc58bf6dd-00661afb40-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.115 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.115 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.115 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.115 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.116 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.116 DEBUG:boto:Method: GET 2024-04-13T21:38:08.116 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:08.116 DEBUG:boto:Data: 2024-04-13T21:38:08.116 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.116 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.116 DEBUG:boto:Params: {} 2024-04-13T21:38:08.116 DEBUG:boto:Token: None 2024-04-13T21:38:08.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/?versions 2024-04-13T21:38:08.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE= 2024-04-13T21:38:08.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PQCQmn/HBYYFf+vJ/p3bFjcRnWE=', 'Content-Length': '0'} 2024-04-13T21:38:08.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f360393a9722672-00661afb40-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.119 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:38:08.120 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.120 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.120 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.120 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.120 DEBUG:boto:Method: GET 2024-04-13T21:38:08.120 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.120 DEBUG:boto:Data: 2024-04-13T21:38:08.120 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.121 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.121 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.121 DEBUG:boto:Params: {} 2024-04-13T21:38:08.121 DEBUG:boto:Token: None 2024-04-13T21:38:08.121 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI= 2024-04-13T21:38:08.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI=', 'Content-Length': '0'} 2024-04-13T21:38:08.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006591f5a38ce55c30-00661afb40-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.123 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.123 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.123 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.123 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.123 DEBUG:boto:Method: GET 2024-04-13T21:38:08.123 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.124 DEBUG:boto:Data: 2024-04-13T21:38:08.124 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.124 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.124 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.124 DEBUG:boto:Params: {} 2024-04-13T21:38:08.124 DEBUG:boto:Token: None 2024-04-13T21:38:08.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI= 2024-04-13T21:38:08.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI=', 'Content-Length': '0'} 2024-04-13T21:38:08.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dad667edf92c45c1-00661afb40-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.126 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.126 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.126 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.126 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.126 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.126 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.127 DEBUG:boto:Data: 2024-04-13T21:38:08.127 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.127 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.127 DEBUG:boto:Params: {} 2024-04-13T21:38:08.127 DEBUG:boto:Token: None 2024-04-13T21:38:08.127 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.127 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jX5w/Zjg57+aYUiZZAqEd+RJP4M= 2024-04-13T21:38:08.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jX5w/Zjg57+aYUiZZAqEd+RJP4M=', 'Content-Length': '0'} 2024-04-13T21:38:08.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063c6853bed91a448-00661afb40-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.129 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.129 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.130 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.130 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.130 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.130 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.130 DEBUG:boto:Data: 2024-04-13T21:38:08.130 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.130 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.130 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.130 DEBUG:boto:Params: {} 2024-04-13T21:38:08.130 DEBUG:boto:Token: None 2024-04-13T21:38:08.131 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.131 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jX5w/Zjg57+aYUiZZAqEd+RJP4M= 2024-04-13T21:38:08.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jX5w/Zjg57+aYUiZZAqEd+RJP4M=', 'Content-Length': '0'} 2024-04-13T21:38:08.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033878d2442693234-00661afb40-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:38:08.133 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.133 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.133 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.133 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.133 DEBUG:boto:Method: GET 2024-04-13T21:38:08.133 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.133 DEBUG:boto:Data: 2024-04-13T21:38:08.133 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.133 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.134 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.134 DEBUG:boto:Params: {} 2024-04-13T21:38:08.134 DEBUG:boto:Token: None 2024-04-13T21:38:08.134 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI= 2024-04-13T21:38:08.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI=', 'Content-Length': '0'} 2024-04-13T21:38:08.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ab30cf6d764816f-00661afb40-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.136 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.136 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.136 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.136 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.136 DEBUG:boto:Method: GET 2024-04-13T21:38:08.136 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.136 DEBUG:boto:Data: 2024-04-13T21:38:08.137 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.137 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.137 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.137 DEBUG:boto:Params: {} 2024-04-13T21:38:08.137 DEBUG:boto:Token: None 2024-04-13T21:38:08.137 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI= 2024-04-13T21:38:08.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI=', 'Content-Length': '0'} 2024-04-13T21:38:08.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089cdc47f210a229a-00661afb40-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.139 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.139 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.139 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.140 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.140 DEBUG:boto:Method: GET 2024-04-13T21:38:08.140 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.140 DEBUG:boto:Data: 2024-04-13T21:38:08.140 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.140 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.140 DEBUG:boto:Params: {} 2024-04-13T21:38:08.140 DEBUG:boto:Token: None 2024-04-13T21:38:08.140 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8= 2024-04-13T21:38:08.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8=', 'Content-Length': '0'} 2024-04-13T21:38:08.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab7612b67bb78951-00661afb40-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.190 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.191 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.191 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.191 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.191 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.191 DEBUG:boto:Method: GET 2024-04-13T21:38:08.191 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.192 DEBUG:boto:Data: 2024-04-13T21:38:08.192 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.192 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.192 DEBUG:boto:Params: {} 2024-04-13T21:38:08.192 DEBUG:boto:Token: None 2024-04-13T21:38:08.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8= 2024-04-13T21:38:08.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8=', 'Content-Length': '0'} 2024-04-13T21:38:08.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2cb233fa1de59e-00661afb40-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.195 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.195 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.195 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.196 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.196 DEBUG:boto:Path: /vwnfns-15/dummy 2024-04-13T21:38:08.196 DEBUG:boto:Data: 2024-04-13T21:38:08.196 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.196 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.196 DEBUG:boto:Params: {} 2024-04-13T21:38:08.196 DEBUG:boto:Token: None 2024-04-13T21:38:08.196 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy 2024-04-13T21:38:08.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eiS3i939obuUX0UiARzJAIlR5Ik= 2024-04-13T21:38:08.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eiS3i939obuUX0UiARzJAIlR5Ik=', 'Content-Length': '0'} 2024-04-13T21:38:08.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088c64cbb029836e4-00661afb40-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.199 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.199 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.199 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.199 DEBUG:boto:Path: /vwnfns-15/dummy 2024-04-13T21:38:08.199 DEBUG:boto:Data: 2024-04-13T21:38:08.199 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.199 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.199 DEBUG:boto:Params: {} 2024-04-13T21:38:08.200 DEBUG:boto:Token: None 2024-04-13T21:38:08.200 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy 2024-04-13T21:38:08.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eiS3i939obuUX0UiARzJAIlR5Ik= 2024-04-13T21:38:08.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eiS3i939obuUX0UiARzJAIlR5Ik=', 'Content-Length': '0'} 2024-04-13T21:38:08.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047c159e7aee6c39d-00661afb40-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:38:08.202 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.202 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.202 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.202 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.202 DEBUG:boto:Method: GET 2024-04-13T21:38:08.203 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.203 DEBUG:boto:Data: 2024-04-13T21:38:08.203 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.203 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.203 DEBUG:boto:Params: {} 2024-04-13T21:38:08.203 DEBUG:boto:Token: None 2024-04-13T21:38:08.203 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.203 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI= 2024-04-13T21:38:08.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI=', 'Content-Length': '0'} 2024-04-13T21:38:08.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000108d4c4096dd5ec5-00661afb40-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.205 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:08.205 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:08.205 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.206 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.206 DEBUG:boto:Method: GET 2024-04-13T21:38:08.206 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.206 DEBUG:boto:Data: 2024-04-13T21:38:08.206 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.206 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.206 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.206 DEBUG:boto:Params: {} 2024-04-13T21:38:08.206 DEBUG:boto:Token: None 2024-04-13T21:38:08.206 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:08.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI= 2024-04-13T21:38:08.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPwmKAAnJpNuEAn4FmURKfk3hTI=', 'Content-Length': '0'} 2024-04-13T21:38:08.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059d1c8b6505acafb-00661afb40-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.208 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.209 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.209 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.209 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.209 DEBUG:boto:Method: GET 2024-04-13T21:38:08.209 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.209 DEBUG:boto:Data: 2024-04-13T21:38:08.209 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.209 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.209 DEBUG:boto:Params: {} 2024-04-13T21:38:08.209 DEBUG:boto:Token: None 2024-04-13T21:38:08.209 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8= 2024-04-13T21:38:08.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8=', 'Content-Length': '0'} 2024-04-13T21:38:08.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf98499e6fb4c8be-00661afb40-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.212 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.212 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.212 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.213 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.213 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.213 DEBUG:boto:Method: GET 2024-04-13T21:38:08.213 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:08.213 DEBUG:boto:Data: 2024-04-13T21:38:08.213 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.213 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.213 DEBUG:boto:Params: {} 2024-04-13T21:38:08.213 DEBUG:boto:Token: None 2024-04-13T21:38:08.214 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.214 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8= 2024-04-13T21:38:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FsJc/QtFWtjpIw3JnKvP8npbXj8=', 'Content-Length': '0'} 2024-04-13T21:38:08.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0c3259d74e635a-00661afb40-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.216 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:08.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-15 zones={a2, a1} 2024-04-13T21:38:08.217 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.217 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.217 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.217 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.217 DEBUG:boto:Data: 2024-04-13T21:38:08.217 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.217 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.217 DEBUG:boto:Params: {} 2024-04-13T21:38:08.217 DEBUG:boto:Token: None 2024-04-13T21:38:08.218 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c81a5983536f469b-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.220 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.220 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.220 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.221 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.221 DEBUG:boto:Data: 2024-04-13T21:38:08.221 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.221 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.221 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.221 DEBUG:boto:Params: {} 2024-04-13T21:38:08.221 DEBUG:boto:Token: None 2024-04-13T21:38:08.221 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-amz-request-id', 'tx00000ec37d570da604a19-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.232 DEBUG:rgw_multi.tests:zone=a1 version=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:08.232 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.232 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.232 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.232 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.233 DEBUG:boto:Data: 2024-04-13T21:38:08.233 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.233 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.233 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.233 DEBUG:boto:Params: {} 2024-04-13T21:38:08.233 DEBUG:boto:Token: None 2024-04-13T21:38:08.233 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.233 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000867b63d983c57edb-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.236 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.236 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.236 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.236 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.236 DEBUG:boto:Data: 2024-04-13T21:38:08.236 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.236 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.236 DEBUG:boto:Params: {} 2024-04-13T21:38:08.236 DEBUG:boto:Token: None 2024-04-13T21:38:08.237 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.237 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-amz-request-id', 'tx00000e9dba91c010bf81b-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.279 DEBUG:rgw_multi.tests:zone=a2 version=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:08.279 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.279 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.279 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.279 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.279 DEBUG:boto:Data: 2024-04-13T21:38:08.279 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.280 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.280 DEBUG:boto:Params: {} 2024-04-13T21:38:08.280 DEBUG:boto:Token: None 2024-04-13T21:38:08.280 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa39997cae33cf48-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.282 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.283 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.283 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.283 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.283 DEBUG:boto:Data: 2024-04-13T21:38:08.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.283 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.283 DEBUG:boto:Params: {} 2024-04-13T21:38:08.283 DEBUG:boto:Token: None 2024-04-13T21:38:08.283 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-amz-request-id', 'tx000009ee94acb50b82fde-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.292 DEBUG:rgw_multi.tests:zone=a1 version=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:08.292 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.292 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.292 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.293 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.293 DEBUG:boto:Data: 2024-04-13T21:38:08.293 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.293 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.293 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.293 DEBUG:boto:Params: {} 2024-04-13T21:38:08.293 DEBUG:boto:Token: None 2024-04-13T21:38:08.293 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.293 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049197cc1ac366602-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.296 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.296 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.296 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.296 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.296 DEBUG:boto:Data: 2024-04-13T21:38:08.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.296 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.297 DEBUG:boto:Params: {} 2024-04-13T21:38:08.297 DEBUG:boto:Token: None 2024-04-13T21:38:08.297 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.297 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-amz-request-id', 'tx0000015c315181e2fb8e8-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.307 DEBUG:rgw_multi.tests:zone=a2 version=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:08.307 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.307 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.307 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.307 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.308 DEBUG:boto:Data: 2024-04-13T21:38:08.308 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.308 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.308 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.308 DEBUG:boto:Params: {} 2024-04-13T21:38:08.308 DEBUG:boto:Token: None 2024-04-13T21:38:08.308 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035f99141cf797dbd-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.311 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.311 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.311 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.311 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.311 DEBUG:boto:Data: 2024-04-13T21:38:08.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.311 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.311 DEBUG:boto:Params: {} 2024-04-13T21:38:08.312 DEBUG:boto:Token: None 2024-04-13T21:38:08.312 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-amz-request-id', 'tx00000b238a91d02ed7d61-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.359 DEBUG:rgw_multi.tests:zone=a1 version=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:08.359 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.359 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.359 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.359 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.359 DEBUG:boto:Data: 2024-04-13T21:38:08.360 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.360 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.360 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.360 DEBUG:boto:Params: {} 2024-04-13T21:38:08.360 DEBUG:boto:Token: None 2024-04-13T21:38:08.360 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032c38869bb546402-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.363 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.363 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.363 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.363 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.363 DEBUG:boto:Data: 2024-04-13T21:38:08.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.364 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.364 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.364 DEBUG:boto:Params: {} 2024-04-13T21:38:08.364 DEBUG:boto:Token: None 2024-04-13T21:38:08.364 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-amz-request-id', 'tx0000071b6f89a001c6766-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.407 DEBUG:rgw_multi.tests:zone=a2 version=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:08.407 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.407 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.407 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.407 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.407 DEBUG:boto:Data: 2024-04-13T21:38:08.408 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.408 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.408 DEBUG:boto:Params: {} 2024-04-13T21:38:08.408 DEBUG:boto:Token: None 2024-04-13T21:38:08.408 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d25d52fb1073d5d2-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.411 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.411 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.411 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.411 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.411 DEBUG:boto:Data: 2024-04-13T21:38:08.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.411 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.411 DEBUG:boto:Params: {} 2024-04-13T21:38:08.411 DEBUG:boto:Token: None 2024-04-13T21:38:08.412 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-amz-request-id', 'tx00000d7a5c46d29723dd7-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.420 DEBUG:rgw_multi.tests:zone=a1 version=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:08.421 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.421 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.421 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.421 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.421 DEBUG:boto:Data: 2024-04-13T21:38:08.421 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.421 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.421 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.421 DEBUG:boto:Params: {} 2024-04-13T21:38:08.421 DEBUG:boto:Token: None 2024-04-13T21:38:08.422 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.422 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f6013b33bdcfb68-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.424 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.424 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.424 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.424 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.424 DEBUG:boto:Data: 2024-04-13T21:38:08.424 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.425 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.425 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.425 DEBUG:boto:Params: {} 2024-04-13T21:38:08.425 DEBUG:boto:Token: None 2024-04-13T21:38:08.425 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.425 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-amz-request-id', 'tx0000024628470ee4082a2-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.434 DEBUG:rgw_multi.tests:zone=a2 version=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:08.434 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.434 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.434 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.434 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.434 DEBUG:boto:Data: 2024-04-13T21:38:08.434 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.434 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.434 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.434 DEBUG:boto:Params: {} 2024-04-13T21:38:08.435 DEBUG:boto:Token: None 2024-04-13T21:38:08.435 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.435 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000898cb74c447b3899-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.437 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.437 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.437 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.437 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.437 DEBUG:boto:Data: 2024-04-13T21:38:08.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.438 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.438 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.438 DEBUG:boto:Params: {} 2024-04-13T21:38:08.438 DEBUG:boto:Token: None 2024-04-13T21:38:08.438 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-amz-request-id', 'tx00000f51d0ad036f19f54-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.446 DEBUG:rgw_multi.tests:zone=a1 version=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:08.446 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.446 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.447 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.447 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.447 DEBUG:boto:Data: 2024-04-13T21:38:08.447 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.447 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.447 DEBUG:boto:Params: {} 2024-04-13T21:38:08.447 DEBUG:boto:Token: None 2024-04-13T21:38:08.447 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca94adc8a3eb35bd-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.450 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.450 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.450 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.450 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.450 DEBUG:boto:Data: 2024-04-13T21:38:08.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.450 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.450 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.450 DEBUG:boto:Params: {} 2024-04-13T21:38:08.451 DEBUG:boto:Token: None 2024-04-13T21:38:08.451 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-amz-request-id', 'tx00000196240d59d949b8c-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.459 DEBUG:rgw_multi.tests:zone=a2 version=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:08.459 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.459 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.459 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.459 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.459 DEBUG:boto:Data: 2024-04-13T21:38:08.459 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.459 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.460 DEBUG:boto:Params: {} 2024-04-13T21:38:08.460 DEBUG:boto:Token: None 2024-04-13T21:38:08.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e887cd81b56e6c95-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.462 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.462 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.462 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.463 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.463 DEBUG:boto:Data: 2024-04-13T21:38:08.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.463 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.463 DEBUG:boto:Params: {} 2024-04-13T21:38:08.463 DEBUG:boto:Token: None 2024-04-13T21:38:08.463 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.463 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-amz-request-id', 'tx00000f3696dc6cfaa5c56-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.507 DEBUG:rgw_multi.tests:zone=a1 version=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:08.507 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.507 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.507 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.507 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.507 DEBUG:boto:Data: 2024-04-13T21:38:08.507 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.507 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.507 DEBUG:boto:Params: {} 2024-04-13T21:38:08.508 DEBUG:boto:Token: None 2024-04-13T21:38:08.508 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000576e588c3d36de64-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.510 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.510 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.511 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.511 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.511 DEBUG:boto:Data: 2024-04-13T21:38:08.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.511 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.511 DEBUG:boto:Params: {} 2024-04-13T21:38:08.511 DEBUG:boto:Token: None 2024-04-13T21:38:08.511 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-amz-request-id', 'tx00000e17beb5d97469af3-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.521 DEBUG:rgw_multi.tests:zone=a2 version=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:08.521 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.521 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.521 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.521 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.521 DEBUG:boto:Data: 2024-04-13T21:38:08.521 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.522 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.522 DEBUG:boto:Params: {} 2024-04-13T21:38:08.522 DEBUG:boto:Token: None 2024-04-13T21:38:08.522 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d80abf937327493a-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.524 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.525 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.525 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.525 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.525 DEBUG:boto:Data: 2024-04-13T21:38:08.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.525 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.525 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.525 DEBUG:boto:Params: {} 2024-04-13T21:38:08.525 DEBUG:boto:Token: None 2024-04-13T21:38:08.525 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.526 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-amz-request-id', 'tx000007aa85f6c9d822291-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.534 DEBUG:rgw_multi.tests:zone=a1 version=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:08.534 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.534 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.534 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.534 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.534 DEBUG:boto:Data: 2024-04-13T21:38:08.534 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.535 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.535 DEBUG:boto:Params: {} 2024-04-13T21:38:08.535 DEBUG:boto:Token: None 2024-04-13T21:38:08.535 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b74b752398c3cff6-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.537 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.537 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.537 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.537 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.537 DEBUG:boto:Data: 2024-04-13T21:38:08.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.538 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.538 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.538 DEBUG:boto:Params: {} 2024-04-13T21:38:08.538 DEBUG:boto:Token: None 2024-04-13T21:38:08.538 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.538 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-amz-request-id', 'tx00000a54b2d784002436c-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.587 DEBUG:rgw_multi.tests:zone=a2 version=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:08.587 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.587 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.587 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.587 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.587 DEBUG:boto:Data: 2024-04-13T21:38:08.587 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.588 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.588 DEBUG:boto:Params: {} 2024-04-13T21:38:08.588 DEBUG:boto:Token: None 2024-04-13T21:38:08.588 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cceac2cf5d91acb-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.591 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.591 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.591 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.591 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.591 DEBUG:boto:Data: 2024-04-13T21:38:08.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.591 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.591 DEBUG:boto:Params: {} 2024-04-13T21:38:08.592 DEBUG:boto:Token: None 2024-04-13T21:38:08.592 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-amz-request-id', 'tx000005f9cdbe067dcd910-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.600 DEBUG:rgw_multi.tests:zone=a1 version=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:08.600 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.600 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.600 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.600 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.601 DEBUG:boto:Data: 2024-04-13T21:38:08.601 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.601 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.601 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.601 DEBUG:boto:Params: {} 2024-04-13T21:38:08.601 DEBUG:boto:Token: None 2024-04-13T21:38:08.601 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069fb3f41b4008262-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.604 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.604 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.604 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.604 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.604 DEBUG:boto:Data: 2024-04-13T21:38:08.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.605 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.605 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.605 DEBUG:boto:Params: {} 2024-04-13T21:38:08.605 DEBUG:boto:Token: None 2024-04-13T21:38:08.605 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.605 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-amz-request-id', 'tx00000c0c4b94fe3b495b2-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.613 DEBUG:rgw_multi.tests:zone=a2 version=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:08.613 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.613 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.613 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.613 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.613 DEBUG:boto:Data: 2024-04-13T21:38:08.613 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.613 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.614 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.614 DEBUG:boto:Params: {} 2024-04-13T21:38:08.614 DEBUG:boto:Token: None 2024-04-13T21:38:08.614 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.614 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad371064bf50e969-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.616 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.617 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.617 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.617 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.617 DEBUG:boto:Data: 2024-04-13T21:38:08.617 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.617 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.617 DEBUG:boto:Params: {} 2024-04-13T21:38:08.617 DEBUG:boto:Token: None 2024-04-13T21:38:08.617 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.618 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-amz-request-id', 'tx000009d016efa9e36f225-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.663 DEBUG:rgw_multi.tests:zone=a1 version=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:08.663 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.663 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.663 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.663 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.663 DEBUG:boto:Data: 2024-04-13T21:38:08.663 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.663 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.663 DEBUG:boto:Params: {} 2024-04-13T21:38:08.663 DEBUG:boto:Token: None 2024-04-13T21:38:08.664 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bb985de7323f917-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.666 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.666 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.666 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.666 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.666 DEBUG:boto:Data: 2024-04-13T21:38:08.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.667 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.667 DEBUG:boto:Params: {} 2024-04-13T21:38:08.667 DEBUG:boto:Token: None 2024-04-13T21:38:08.667 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-amz-request-id', 'tx000001f36a546fc5f7d30-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.675 DEBUG:rgw_multi.tests:zone=a2 version=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:08.676 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.676 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.676 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.676 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.676 DEBUG:boto:Data: 2024-04-13T21:38:08.676 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.676 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.676 DEBUG:boto:Params: {} 2024-04-13T21:38:08.676 DEBUG:boto:Token: None 2024-04-13T21:38:08.677 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005787b9f3c164fce9-00661afb40-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.679 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.679 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.679 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.679 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.679 DEBUG:boto:Data: 2024-04-13T21:38:08.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:08.679 DEBUG:boto:Port: 8000 2024-04-13T21:38:08.679 DEBUG:boto:Params: {} 2024-04-13T21:38:08.680 DEBUG:boto:Token: None 2024-04-13T21:38:08.680 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-amz-request-id', 'tx00000ea902c081438950c-00661afb40-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.688 DEBUG:rgw_multi.tests:zone=a1 version=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:08.688 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:08.688 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:08.688 DEBUG:boto:Method: HEAD 2024-04-13T21:38:08.688 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:08.688 DEBUG:boto:Data: 2024-04-13T21:38:08.688 DEBUG:boto:Headers: {} 2024-04-13T21:38:08.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.688 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.688 DEBUG:boto:Params: {} 2024-04-13T21:38:08.688 DEBUG:boto:Token: None 2024-04-13T21:38:08.689 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/ 2024-04-13T21:38:08.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk= 2024-04-13T21:38:08.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arUaMcm7Ieldp6rO58lL74LB/Lk=', 'Content-Length': '0'} 2024-04-13T21:38:08.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f185c23943286f5-00661afb40-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.691 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:08.691 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:08.691 DEBUG:boto:Method: PUT 2024-04-13T21:38:08.691 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:08.691 DEBUG:boto:Data: 2024-04-13T21:38:08.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-13T21:38:08.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:08.691 DEBUG:boto:Port: 8001 2024-04-13T21:38:08.692 DEBUG:boto:Params: {} 2024-04-13T21:38:08.692 DEBUG:boto:Token: None 2024-04-13T21:38:08.692 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Sat, 13 Apr 2024 21:38:08 GMT /vwnfns-15/obj 2024-04-13T21:38:08.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g= 2024-04-13T21:38:08.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:38:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6NkIJX4zToDObJmTvfcdPZpN6g='} 2024-04-13T21:38:08.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-amz-request-id', 'tx000000533e1958d6efcb3-00661afb40-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:08.735 DEBUG:rgw_multi.tests:zone=a2 version=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:08.735 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-15 2024-04-13T21:38:08.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:38:08.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:38:08.735 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:38:08.790 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:38:08.790 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:38:08.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:08.831+0000 7feaae163780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:08.823 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.22.10, , , , , ] 2024-04-13T21:38:08.823 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.22.10, , , 00000000020.65.12, , ] 2024-04-13T21:38:13.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:13.832+0000 7feaae163780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:13.825 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.22.10, , , , , ] 2024-04-13T21:38:13.825 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.22.10, , , 00000000020.65.12, , ] 2024-04-13T21:38:18.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:18.833+0000 7feaae163780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:18.826 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.22.10, , , , , ] 2024-04-13T21:38:18.826 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.22.10, , , 00000000020.65.12, , ] 2024-04-13T21:38:23.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:23.835+0000 7feaae163780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:23.828 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.22.10, , , , , ] 2024-04-13T21:38:23.828 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.22.10, , , 00000000020.65.12, , ] 2024-04-13T21:38:28.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:28.837+0000 7feaae163780 1 bucket sync caught up with source: 2024-04-13T21:38:28.829 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000003.22.10, , , 00000000021.68.6, , ] 2024-04-13T21:38:28.829 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000003.22.10, , , 00000000020.65.12, , ] 2024-04-13T21:38:28.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:28.837+0000 7feaae163780 0 bucket checkpoint complete 2024-04-13T21:38:28.844 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-15 2024-04-13T21:38:28.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:38:28.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:38:28.845 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:38:28.904 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:38:28.904 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:38:28.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:28.950+0000 7f63e9372780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:28.942 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.30.11, , , 00000000003.44.6, , ] 2024-04-13T21:38:28.942 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.30.11, , , 00000000031.108.9, , ] 2024-04-13T21:38:33.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:33.952+0000 7f63e9372780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:33.944 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.30.11, , , 00000000003.44.6, , ] 2024-04-13T21:38:33.944 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.30.11, , , 00000000031.108.9, , ] 2024-04-13T21:38:38.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:38.953+0000 7f63e9372780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:38.945 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.30.11, , , 00000000003.44.6, , ] 2024-04-13T21:38:38.945 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.30.11, , , 00000000031.108.9, , ] 2024-04-13T21:38:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:43.954+0000 7f63e9372780 1 waiting for incremental sync to catch up: 2024-04-13T21:38:43.946 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.30.11, , , 00000000003.44.6, , ] 2024-04-13T21:38:43.947 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.30.11, , , 00000000031.108.9, , ] 2024-04-13T21:38:48.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:48.955+0000 7f63e9372780 1 bucket sync caught up with source: 2024-04-13T21:38:48.948 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000002.30.11, , , 00000000031.108.9, , ] 2024-04-13T21:38:48.948 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000002.30.11, , , 00000000031.108.9, , ] 2024-04-13T21:38:48.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:38:48.955+0000 7f63e9372780 0 bucket checkpoint complete 2024-04-13T21:38:48.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-15 zones={a2, a1} 2024-04-13T21:38:48.963 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:48.963 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:48.964 DEBUG:boto:Method: HEAD 2024-04-13T21:38:48.964 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:48.964 DEBUG:boto:Data: 2024-04-13T21:38:48.964 DEBUG:boto:Headers: {} 2024-04-13T21:38:48.964 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:48.964 DEBUG:boto:Port: 8001 2024-04-13T21:38:48.964 DEBUG:boto:Params: {} 2024-04-13T21:38:48.965 DEBUG:boto:Token: None 2024-04-13T21:38:48.965 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:48 GMT /vwnfns-15/ 2024-04-13T21:38:48.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpWkV7DBdEsi7IgiEQKzKPLPjv8= 2024-04-13T21:38:48.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpWkV7DBdEsi7IgiEQKzKPLPjv8=', 'Content-Length': '0'} 2024-04-13T21:38:48.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092d7b781d544f3be-00661afb68-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:48.968 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:48.968 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:48.968 DEBUG:boto:Method: HEAD 2024-04-13T21:38:48.968 DEBUG:boto:Path: /vwnfns-15/ 2024-04-13T21:38:48.968 DEBUG:boto:Data: 2024-04-13T21:38:48.968 DEBUG:boto:Headers: {} 2024-04-13T21:38:48.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:48.969 DEBUG:boto:Port: 8000 2024-04-13T21:38:48.969 DEBUG:boto:Params: {} 2024-04-13T21:38:48.969 DEBUG:boto:Token: None 2024-04-13T21:38:48.969 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:48 GMT /vwnfns-15/ 2024-04-13T21:38:48.969 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpWkV7DBdEsi7IgiEQKzKPLPjv8= 2024-04-13T21:38:48.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpWkV7DBdEsi7IgiEQKzKPLPjv8=', 'Content-Length': '0'} 2024-04-13T21:38:48.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075e6a5231b2e664b-00661afb68-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:38:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:48.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:38:48.972 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:48.972 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:48.972 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.972 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.972 DEBUG:boto:Method: GET 2024-04-13T21:38:48.972 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.972 DEBUG:boto:Data: 2024-04-13T21:38:48.972 DEBUG:boto:Headers: {} 2024-04-13T21:38:48.972 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:48.973 DEBUG:boto:Port: 8001 2024-04-13T21:38:48.973 DEBUG:boto:Params: {} 2024-04-13T21:38:48.973 DEBUG:boto:Token: None 2024-04-13T21:38:48.973 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:48 GMT /vwnfns-15/?versions 2024-04-13T21:38:48.973 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4= 2024-04-13T21:38:48.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4=', 'Content-Length': '0'} 2024-04-13T21:38:48.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da3114c02fe4a9c-00661afb68-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:48.976 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA3ZgYu.Obob4lazgpJ01ShY35PtFVXMtrue2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2false2024-04-13T21:38:08.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj019k12m187iEGtpVnMZTqfvWgxncrKZfalse2024-04-13T21:38:08.678Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHXIrwk7JSqUilmLyhyog.MC4rqQPfL3false2024-04-13T21:38:08.628Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJvku7JHOc-vMkfpIC8bwINCzCuWCiqFfalse2024-04-13T21:38:08.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju73pgnaQBi-XxafVIxtpwHUx5HMGrfIfalse2024-04-13T21:38:08.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14false2024-04-13T21:38:08.548Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJpv99ylZJsuQjpIg8bYZ0C1nGaBbiXCfalse2024-04-13T21:38:08.536Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjarsbDN8vjMAjqqXjJTyc0XH.3830UP2false2024-04-13T21:38:08.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPYfalse2024-04-13T21:38:08.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOoBjKMBXO0H5kpggzNatVsdQnIDNPgPfalse2024-04-13T21:38:08.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4false2024-04-13T21:38:08.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVTT69MXmX6XS..1R1xDYYnR3KtMzY4kfalse2024-04-13T21:38:08.422Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmuvbWmAiDOX3JDiJbLDSnvGOk5L8e92false2024-04-13T21:38:08.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkEONbY1fLwoEYq3ySy0dDwkmBgh6IP4false2024-04-13T21:38:08.322Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzq4c7U.CkxCgWVAAmHFw0dP9nKUV-vjfalse2024-04-13T21:38:08.374Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uLfalse2024-04-13T21:38:08.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdlqqJT0SU3uFQVk9dFJJxzvthMghGvifalse2024-04-13T21:38:08.307Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKLig-K680b4a5A8Q2HBvpyriK1de2Ljfalse2024-04-13T21:38:08.232Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwfWU7UeFtRE5fiy16I2Xtgd6dx-bhcbfalse2024-04-13T21:38:08.247Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.980 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:38:48.980 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:48.980 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:48.980 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.981 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.981 DEBUG:boto:Method: GET 2024-04-13T21:38:48.981 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.981 DEBUG:boto:Data: 2024-04-13T21:38:48.981 DEBUG:boto:Headers: {} 2024-04-13T21:38:48.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:48.981 DEBUG:boto:Port: 8000 2024-04-13T21:38:48.981 DEBUG:boto:Params: {} 2024-04-13T21:38:48.981 DEBUG:boto:Token: None 2024-04-13T21:38:48.981 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:48 GMT /vwnfns-15/?versions 2024-04-13T21:38:48.982 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4= 2024-04-13T21:38:48.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4=', 'Content-Length': '0'} 2024-04-13T21:38:48.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d20bcf14c2650303-00661afb68-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:48.985 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA3ZgYu.Obob4lazgpJ01ShY35PtFVXMtrue2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2false2024-04-13T21:38:08.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj019k12m187iEGtpVnMZTqfvWgxncrKZfalse2024-04-13T21:38:08.678Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHXIrwk7JSqUilmLyhyog.MC4rqQPfL3false2024-04-13T21:38:08.628Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJvku7JHOc-vMkfpIC8bwINCzCuWCiqFfalse2024-04-13T21:38:08.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju73pgnaQBi-XxafVIxtpwHUx5HMGrfIfalse2024-04-13T21:38:08.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14false2024-04-13T21:38:08.548Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJpv99ylZJsuQjpIg8bYZ0C1nGaBbiXCfalse2024-04-13T21:38:08.536Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjarsbDN8vjMAjqqXjJTyc0XH.3830UP2false2024-04-13T21:38:08.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPYfalse2024-04-13T21:38:08.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOoBjKMBXO0H5kpggzNatVsdQnIDNPgPfalse2024-04-13T21:38:08.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4false2024-04-13T21:38:08.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVTT69MXmX6XS..1R1xDYYnR3KtMzY4kfalse2024-04-13T21:38:08.422Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmuvbWmAiDOX3JDiJbLDSnvGOk5L8e92false2024-04-13T21:38:08.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkEONbY1fLwoEYq3ySy0dDwkmBgh6IP4false2024-04-13T21:38:08.322Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzq4c7U.CkxCgWVAAmHFw0dP9nKUV-vjfalse2024-04-13T21:38:08.374Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uLfalse2024-04-13T21:38:08.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdlqqJT0SU3uFQVk9dFJJxzvthMghGvifalse2024-04-13T21:38:08.307Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKLig-K680b4a5A8Q2HBvpyriK1de2Ljfalse2024-04-13T21:38:08.232Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwfWU7UeFtRE5fiy16I2Xtgd6dx-bhcbfalse2024-04-13T21:38:08.247Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:48.986 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T21:38:48.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:38:48.988 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:48.988 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:48.988 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.988 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.988 DEBUG:boto:Method: GET 2024-04-13T21:38:48.988 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.988 DEBUG:boto:Data: 2024-04-13T21:38:48.988 DEBUG:boto:Headers: {} 2024-04-13T21:38:48.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:48.989 DEBUG:boto:Port: 8001 2024-04-13T21:38:48.989 DEBUG:boto:Params: {} 2024-04-13T21:38:48.989 DEBUG:boto:Token: None 2024-04-13T21:38:48.989 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:48 GMT /vwnfns-15/?versions 2024-04-13T21:38:48.989 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4= 2024-04-13T21:38:48.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4=', 'Content-Length': '0'} 2024-04-13T21:38:48.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae5b43c29a10ddc1-00661afb68-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:48.992 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA3ZgYu.Obob4lazgpJ01ShY35PtFVXMtrue2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2false2024-04-13T21:38:08.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj019k12m187iEGtpVnMZTqfvWgxncrKZfalse2024-04-13T21:38:08.678Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHXIrwk7JSqUilmLyhyog.MC4rqQPfL3false2024-04-13T21:38:08.628Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJvku7JHOc-vMkfpIC8bwINCzCuWCiqFfalse2024-04-13T21:38:08.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju73pgnaQBi-XxafVIxtpwHUx5HMGrfIfalse2024-04-13T21:38:08.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14false2024-04-13T21:38:08.548Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJpv99ylZJsuQjpIg8bYZ0C1nGaBbiXCfalse2024-04-13T21:38:08.536Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjarsbDN8vjMAjqqXjJTyc0XH.3830UP2false2024-04-13T21:38:08.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPYfalse2024-04-13T21:38:08.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOoBjKMBXO0H5kpggzNatVsdQnIDNPgPfalse2024-04-13T21:38:08.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4false2024-04-13T21:38:08.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVTT69MXmX6XS..1R1xDYYnR3KtMzY4kfalse2024-04-13T21:38:08.422Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmuvbWmAiDOX3JDiJbLDSnvGOk5L8e92false2024-04-13T21:38:08.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkEONbY1fLwoEYq3ySy0dDwkmBgh6IP4false2024-04-13T21:38:08.322Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzq4c7U.CkxCgWVAAmHFw0dP9nKUV-vjfalse2024-04-13T21:38:08.374Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uLfalse2024-04-13T21:38:08.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdlqqJT0SU3uFQVk9dFJJxzvthMghGvifalse2024-04-13T21:38:08.307Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKLig-K680b4a5A8Q2HBvpyriK1de2Ljfalse2024-04-13T21:38:08.232Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwfWU7UeFtRE5fiy16I2Xtgd6dx-bhcbfalse2024-04-13T21:38:08.247Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:48.993 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:48.993 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:48.993 DEBUG:boto:path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.993 DEBUG:boto:auth_path=/vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.993 DEBUG:boto:Method: GET 2024-04-13T21:38:48.994 DEBUG:boto:Path: /vwnfns-15/?versions&max-keys=999 2024-04-13T21:38:48.994 DEBUG:boto:Data: 2024-04-13T21:38:48.994 DEBUG:boto:Headers: {} 2024-04-13T21:38:48.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:48.994 DEBUG:boto:Port: 8000 2024-04-13T21:38:48.994 DEBUG:boto:Params: {} 2024-04-13T21:38:48.994 DEBUG:boto:Token: None 2024-04-13T21:38:48.994 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:48 GMT /vwnfns-15/?versions 2024-04-13T21:38:48.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4= 2024-04-13T21:38:48.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rS8jtAX4nwoFzLQg+vBfn0Ot5n4=', 'Content-Length': '0'} 2024-04-13T21:38:48.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000722ee43f005f4a0-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:48.997 DEBUG:boto:b'vwnfns-15999falsedummyM8xL3OJnShdETrs77-etpSDbmBkBJVltrue2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA3ZgYu.Obob4lazgpJ01ShY35PtFVXMtrue2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2false2024-04-13T21:38:08.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj019k12m187iEGtpVnMZTqfvWgxncrKZfalse2024-04-13T21:38:08.678Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHXIrwk7JSqUilmLyhyog.MC4rqQPfL3false2024-04-13T21:38:08.628Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJvku7JHOc-vMkfpIC8bwINCzCuWCiqFfalse2024-04-13T21:38:08.602Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju73pgnaQBi-XxafVIxtpwHUx5HMGrfIfalse2024-04-13T21:38:08.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14false2024-04-13T21:38:08.548Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJpv99ylZJsuQjpIg8bYZ0C1nGaBbiXCfalse2024-04-13T21:38:08.536Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjarsbDN8vjMAjqqXjJTyc0XH.3830UP2false2024-04-13T21:38:08.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPYfalse2024-04-13T21:38:08.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOoBjKMBXO0H5kpggzNatVsdQnIDNPgPfalse2024-04-13T21:38:08.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4false2024-04-13T21:38:08.448Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVTT69MXmX6XS..1R1xDYYnR3KtMzY4kfalse2024-04-13T21:38:08.422Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmuvbWmAiDOX3JDiJbLDSnvGOk5L8e92false2024-04-13T21:38:08.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkEONbY1fLwoEYq3ySy0dDwkmBgh6IP4false2024-04-13T21:38:08.322Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzq4c7U.CkxCgWVAAmHFw0dP9nKUV-vjfalse2024-04-13T21:38:08.374Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uLfalse2024-04-13T21:38:08.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdlqqJT0SU3uFQVk9dFJJxzvthMghGvifalse2024-04-13T21:38:08.307Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKLig-K680b4a5A8Q2HBvpyriK1de2Ljfalse2024-04-13T21:38:08.232Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwfWU7UeFtRE5fiy16I2Xtgd6dx-bhcbfalse2024-04-13T21:38:08.247Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:48.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:38:48.999 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:48.999 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:48.999 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:48.999 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:48.999 DEBUG:boto:Method: GET 2024-04-13T21:38:48.999 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:48.999 DEBUG:boto:Data: 2024-04-13T21:38:48.999 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.000 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.000 DEBUG:boto:Params: {} 2024-04-13T21:38:49.000 DEBUG:boto:Token: None 2024-04-13T21:38:49.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY= 2024-04-13T21:38:49.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY=', 'Content-Length': '0'} 2024-04-13T21:38:49.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024c81fbe8a185197-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.002 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.002 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.002 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.002 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.002 DEBUG:boto:Method: GET 2024-04-13T21:38:49.002 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.003 DEBUG:boto:Data: 2024-04-13T21:38:49.003 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.003 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.003 DEBUG:boto:Params: {} 2024-04-13T21:38:49.003 DEBUG:boto:Token: None 2024-04-13T21:38:49.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY= 2024-04-13T21:38:49.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY=', 'Content-Length': '0'} 2024-04-13T21:38:49.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049a5fb62399f2277-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.005 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.005 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.005 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.005 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.005 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.006 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.006 DEBUG:boto:Data: 2024-04-13T21:38:49.006 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.006 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.006 DEBUG:boto:Params: {} 2024-04-13T21:38:49.006 DEBUG:boto:Token: None 2024-04-13T21:38:49.006 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oCllVQBNMNqddJB9hDV0lAt/OIY= 2024-04-13T21:38:49.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oCllVQBNMNqddJB9hDV0lAt/OIY=', 'Content-Length': '0'} 2024-04-13T21:38:49.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0821a4a3912cd57-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.009 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.009 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.009 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.009 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.009 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.009 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.009 DEBUG:boto:Data: 2024-04-13T21:38:49.009 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.009 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.009 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.010 DEBUG:boto:Params: {} 2024-04-13T21:38:49.010 DEBUG:boto:Token: None 2024-04-13T21:38:49.010 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.010 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oCllVQBNMNqddJB9hDV0lAt/OIY= 2024-04-13T21:38:49.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oCllVQBNMNqddJB9hDV0lAt/OIY=', 'Content-Length': '0'} 2024-04-13T21:38:49.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b02955e412e941ca-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:38:49.012 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.012 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.012 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.012 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.012 DEBUG:boto:Method: GET 2024-04-13T21:38:49.012 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.013 DEBUG:boto:Data: 2024-04-13T21:38:49.013 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.013 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.013 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.013 DEBUG:boto:Params: {} 2024-04-13T21:38:49.013 DEBUG:boto:Token: None 2024-04-13T21:38:49.013 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.013 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY= 2024-04-13T21:38:49.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY=', 'Content-Length': '0'} 2024-04-13T21:38:49.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072291fb50b1e86a9-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.015 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.015 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.015 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.015 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.015 DEBUG:boto:Method: GET 2024-04-13T21:38:49.015 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.016 DEBUG:boto:Data: 2024-04-13T21:38:49.016 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.016 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.016 DEBUG:boto:Params: {} 2024-04-13T21:38:49.016 DEBUG:boto:Token: None 2024-04-13T21:38:49.016 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY= 2024-04-13T21:38:49.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY=', 'Content-Length': '0'} 2024-04-13T21:38:49.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000deaa85701f9698e8-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.018 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.018 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.018 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.018 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.018 DEBUG:boto:Method: GET 2024-04-13T21:38:49.019 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.019 DEBUG:boto:Data: 2024-04-13T21:38:49.019 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.019 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.019 DEBUG:boto:Params: {} 2024-04-13T21:38:49.019 DEBUG:boto:Token: None 2024-04-13T21:38:49.019 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.019 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5fa089f6cb3daa3-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.066 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.067 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.067 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.067 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.067 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.067 DEBUG:boto:Method: GET 2024-04-13T21:38:49.067 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.067 DEBUG:boto:Data: 2024-04-13T21:38:49.067 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.068 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.068 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.068 DEBUG:boto:Params: {} 2024-04-13T21:38:49.068 DEBUG:boto:Token: None 2024-04-13T21:38:49.068 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.068 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b9283e1b1db5e7-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.071 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.071 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.071 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.071 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.072 DEBUG:boto:Path: /vwnfns-15/dummy 2024-04-13T21:38:49.072 DEBUG:boto:Data: 2024-04-13T21:38:49.072 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.072 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.072 DEBUG:boto:Params: {} 2024-04-13T21:38:49.072 DEBUG:boto:Token: None 2024-04-13T21:38:49.072 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy 2024-04-13T21:38:49.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nF3v1JN2wgyOW55rR6htUW6OdBs= 2024-04-13T21:38:49.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nF3v1JN2wgyOW55rR6htUW6OdBs=', 'Content-Length': '0'} 2024-04-13T21:38:49.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000274f71760406d9ba-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.075 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.075 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.075 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.075 DEBUG:boto:Path: /vwnfns-15/dummy 2024-04-13T21:38:49.075 DEBUG:boto:Data: 2024-04-13T21:38:49.075 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.075 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.075 DEBUG:boto:Params: {} 2024-04-13T21:38:49.076 DEBUG:boto:Token: None 2024-04-13T21:38:49.076 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy 2024-04-13T21:38:49.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nF3v1JN2wgyOW55rR6htUW6OdBs= 2024-04-13T21:38:49.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nF3v1JN2wgyOW55rR6htUW6OdBs=', 'Content-Length': '0'} 2024-04-13T21:38:49.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054365d06a952945a-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T21:38:49.078 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.078 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.079 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.079 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.079 DEBUG:boto:Method: GET 2024-04-13T21:38:49.079 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.079 DEBUG:boto:Data: 2024-04-13T21:38:49.079 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.079 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.079 DEBUG:boto:Params: {} 2024-04-13T21:38:49.079 DEBUG:boto:Token: None 2024-04-13T21:38:49.079 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY= 2024-04-13T21:38:49.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY=', 'Content-Length': '0'} 2024-04-13T21:38:49.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000feddf9cc4301b22d-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.081 DEBUG:boto:path=/vwnfns-15/dummy 2024-04-13T21:38:49.082 DEBUG:boto:auth_path=/vwnfns-15/dummy 2024-04-13T21:38:49.082 DEBUG:boto:path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.082 DEBUG:boto:auth_path=/vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.082 DEBUG:boto:Method: GET 2024-04-13T21:38:49.082 DEBUG:boto:Path: /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.082 DEBUG:boto:Data: 2024-04-13T21:38:49.082 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.082 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.082 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.082 DEBUG:boto:Params: {} 2024-04-13T21:38:49.083 DEBUG:boto:Token: None 2024-04-13T21:38:49.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/dummy?versionId=M8xL3OJnShdETrs77-etpSDbmBkBJVl 2024-04-13T21:38:49.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY= 2024-04-13T21:38:49.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BB5LUGz6xDHH8OOv5z5R2ko8CbY=', 'Content-Length': '0'} 2024-04-13T21:38:49.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:37:32 GMT'), ('x-amz-version-id', 'M8xL3OJnShdETrs77-etpSDbmBkBJVl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000187b880bca53d23b-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.085 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.085 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.085 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.085 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.085 DEBUG:boto:Method: GET 2024-04-13T21:38:49.085 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.085 DEBUG:boto:Data: 2024-04-13T21:38:49.086 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.086 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.086 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.086 DEBUG:boto:Params: {} 2024-04-13T21:38:49.086 DEBUG:boto:Token: None 2024-04-13T21:38:49.086 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.086 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b37aef25ca21af-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.089 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.089 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.089 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.089 DEBUG:boto:path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.089 DEBUG:boto:auth_path=/vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.090 DEBUG:boto:Method: GET 2024-04-13T21:38:49.090 DEBUG:boto:Path: /vwnfns-15/?prefix=dummy 2024-04-13T21:38:49.090 DEBUG:boto:Data: 2024-04-13T21:38:49.090 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.090 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.090 DEBUG:boto:Params: {} 2024-04-13T21:38:49.090 DEBUG:boto:Token: None 2024-04-13T21:38:49.090 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000878c647405fefdc-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.093 DEBUG:boto:b'vwnfns-15dummy1000falsedummy2024-04-13T21:37:32.865Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.093 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.093 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.094 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.094 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.094 DEBUG:boto:Method: GET 2024-04-13T21:38:49.094 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.094 DEBUG:boto:Data: 2024-04-13T21:38:49.094 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.094 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.094 DEBUG:boto:Params: {} 2024-04-13T21:38:49.094 DEBUG:boto:Token: None 2024-04-13T21:38:49.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6a5f1c856b4980f-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.097 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.097 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.097 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.097 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.097 DEBUG:boto:Method: GET 2024-04-13T21:38:49.097 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.097 DEBUG:boto:Data: 2024-04-13T21:38:49.097 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.097 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.097 DEBUG:boto:Params: {} 2024-04-13T21:38:49.098 DEBUG:boto:Token: None 2024-04-13T21:38:49.098 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cf582a4e0d48ed3-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.100 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.100 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.100 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.100 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.101 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.101 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.101 DEBUG:boto:Data: 2024-04-13T21:38:49.101 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.101 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.101 DEBUG:boto:Params: {} 2024-04-13T21:38:49.101 DEBUG:boto:Token: None 2024-04-13T21:38:49.101 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.101 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3jWihMkYltICwPkfDnTwv/p1vto= 2024-04-13T21:38:49.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3jWihMkYltICwPkfDnTwv/p1vto=', 'Content-Length': '0'} 2024-04-13T21:38:49.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000413b6af842eb1d31-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.103 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.103 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.104 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.104 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.104 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.104 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.104 DEBUG:boto:Data: 2024-04-13T21:38:49.104 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.104 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.104 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.104 DEBUG:boto:Params: {} 2024-04-13T21:38:49.105 DEBUG:boto:Token: None 2024-04-13T21:38:49.105 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.105 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3jWihMkYltICwPkfDnTwv/p1vto= 2024-04-13T21:38:49.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3jWihMkYltICwPkfDnTwv/p1vto=', 'Content-Length': '0'} 2024-04-13T21:38:49.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011bbcb43b94ce17f-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.107 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.107 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.107 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.107 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.107 DEBUG:boto:Method: GET 2024-04-13T21:38:49.107 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.108 DEBUG:boto:Data: 2024-04-13T21:38:49.108 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.108 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.108 DEBUG:boto:Params: {} 2024-04-13T21:38:49.108 DEBUG:boto:Token: None 2024-04-13T21:38:49.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002959c98366133839-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.150 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.150 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.151 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.151 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.151 DEBUG:boto:Method: GET 2024-04-13T21:38:49.151 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.151 DEBUG:boto:Data: 2024-04-13T21:38:49.151 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.151 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.151 DEBUG:boto:Params: {} 2024-04-13T21:38:49.151 DEBUG:boto:Token: None 2024-04-13T21:38:49.152 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046ca672d094f2937-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.194 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.195 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.195 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.195 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.195 DEBUG:boto:Method: GET 2024-04-13T21:38:49.195 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.195 DEBUG:boto:Data: 2024-04-13T21:38:49.195 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.195 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.195 DEBUG:boto:Params: {} 2024-04-13T21:38:49.196 DEBUG:boto:Token: None 2024-04-13T21:38:49.196 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f233d347881c73bd-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.199 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.199 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.199 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.199 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.199 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.199 DEBUG:boto:Method: GET 2024-04-13T21:38:49.199 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.200 DEBUG:boto:Data: 2024-04-13T21:38:49.200 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.200 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.200 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.200 DEBUG:boto:Params: {} 2024-04-13T21:38:49.200 DEBUG:boto:Token: None 2024-04-13T21:38:49.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1ebbef270dec1d2-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.203 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.203 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.203 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.204 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.204 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.204 DEBUG:boto:Data: 2024-04-13T21:38:49.204 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.204 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.204 DEBUG:boto:Params: {} 2024-04-13T21:38:49.204 DEBUG:boto:Token: None 2024-04-13T21:38:49.204 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000198dc994f123dfee-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.207 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.207 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.208 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.208 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.208 DEBUG:boto:Data: 2024-04-13T21:38:49.208 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.208 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.208 DEBUG:boto:Params: {} 2024-04-13T21:38:49.208 DEBUG:boto:Token: None 2024-04-13T21:38:49.208 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085e272ae1426852b-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.212 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.212 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.212 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.212 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.212 DEBUG:boto:Method: GET 2024-04-13T21:38:49.212 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.212 DEBUG:boto:Data: 2024-04-13T21:38:49.212 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.212 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.213 DEBUG:boto:Params: {} 2024-04-13T21:38:49.213 DEBUG:boto:Token: None 2024-04-13T21:38:49.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e095dfd19cde9f8f-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.215 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.215 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.215 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.215 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.215 DEBUG:boto:Method: GET 2024-04-13T21:38:49.216 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.216 DEBUG:boto:Data: 2024-04-13T21:38:49.216 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.216 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.216 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.216 DEBUG:boto:Params: {} 2024-04-13T21:38:49.216 DEBUG:boto:Token: None 2024-04-13T21:38:49.216 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029bb24e9d3fcd618-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.262 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.263 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.263 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.263 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.263 DEBUG:boto:Method: GET 2024-04-13T21:38:49.263 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.263 DEBUG:boto:Data: 2024-04-13T21:38:49.263 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.263 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.263 DEBUG:boto:Params: {} 2024-04-13T21:38:49.264 DEBUG:boto:Token: None 2024-04-13T21:38:49.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e674d47ee686283c-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.266 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.267 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.267 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.267 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.267 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.267 DEBUG:boto:Method: GET 2024-04-13T21:38:49.267 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.267 DEBUG:boto:Data: 2024-04-13T21:38:49.268 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.268 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.268 DEBUG:boto:Params: {} 2024-04-13T21:38:49.268 DEBUG:boto:Token: None 2024-04-13T21:38:49.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072997404a6bb9497-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.271 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.271 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.271 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.271 DEBUG:boto:path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.272 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.272 DEBUG:boto:Method: GET 2024-04-13T21:38:49.272 DEBUG:boto:Path: /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.272 DEBUG:boto:Data: 2024-04-13T21:38:49.272 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.272 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.272 DEBUG:boto:Params: {} 2024-04-13T21:38:49.272 DEBUG:boto:Token: None 2024-04-13T21:38:49.272 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE= 2024-04-13T21:38:49.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE=', 'Content-Length': '0'} 2024-04-13T21:38:49.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000346250aef3b5989d-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.322 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.323 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.323 DEBUG:boto:path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.323 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.323 DEBUG:boto:Method: GET 2024-04-13T21:38:49.323 DEBUG:boto:Path: /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.323 DEBUG:boto:Data: 2024-04-13T21:38:49.323 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.323 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.324 DEBUG:boto:Params: {} 2024-04-13T21:38:49.324 DEBUG:boto:Token: None 2024-04-13T21:38:49.324 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE= 2024-04-13T21:38:49.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE=', 'Content-Length': '0'} 2024-04-13T21:38:49.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c8fb5564b57140b-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.326 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.327 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.327 DEBUG:boto:path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.327 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.327 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.327 DEBUG:boto:Path: /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.327 DEBUG:boto:Data: 2024-04-13T21:38:49.327 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.327 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.327 DEBUG:boto:Params: {} 2024-04-13T21:38:49.327 DEBUG:boto:Token: None 2024-04-13T21:38:49.328 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X64+5cBdWZZR1gkpDS96T3hY23g= 2024-04-13T21:38:49.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X64+5cBdWZZR1gkpDS96T3hY23g=', 'Content-Length': '0'} 2024-04-13T21:38:49.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000291e5695e91476d5-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.330 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.330 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.330 DEBUG:boto:path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.331 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.331 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.331 DEBUG:boto:Path: /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.331 DEBUG:boto:Data: 2024-04-13T21:38:49.331 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.331 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.331 DEBUG:boto:Params: {} 2024-04-13T21:38:49.331 DEBUG:boto:Token: None 2024-04-13T21:38:49.332 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X64+5cBdWZZR1gkpDS96T3hY23g= 2024-04-13T21:38:49.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X64+5cBdWZZR1gkpDS96T3hY23g=', 'Content-Length': '0'} 2024-04-13T21:38:49.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c4406623af007ae-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.334 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.334 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.334 DEBUG:boto:path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.334 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.335 DEBUG:boto:Method: GET 2024-04-13T21:38:49.335 DEBUG:boto:Path: /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.335 DEBUG:boto:Data: 2024-04-13T21:38:49.335 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.335 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.335 DEBUG:boto:Params: {} 2024-04-13T21:38:49.335 DEBUG:boto:Token: None 2024-04-13T21:38:49.335 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE= 2024-04-13T21:38:49.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE=', 'Content-Length': '0'} 2024-04-13T21:38:49.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028b5b9a9c28a148c-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.378 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.379 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.379 DEBUG:boto:path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.379 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.379 DEBUG:boto:Method: GET 2024-04-13T21:38:49.379 DEBUG:boto:Path: /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.379 DEBUG:boto:Data: 2024-04-13T21:38:49.379 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.379 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.379 DEBUG:boto:Params: {} 2024-04-13T21:38:49.380 DEBUG:boto:Token: None 2024-04-13T21:38:49.380 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2 2024-04-13T21:38:49.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE= 2024-04-13T21:38:49.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QvT4KPgE/NOHB2kyFKakqDAEBkE=', 'Content-Length': '0'} 2024-04-13T21:38:49.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'N9Ft7Mo3kB5tuGv30Ln-aCGVnrvBUc2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dca146264d9a25c-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.382 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.382 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.382 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.382 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.383 DEBUG:boto:Method: GET 2024-04-13T21:38:49.383 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.383 DEBUG:boto:Data: 2024-04-13T21:38:49.383 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.383 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.383 DEBUG:boto:Params: {} 2024-04-13T21:38:49.383 DEBUG:boto:Token: None 2024-04-13T21:38:49.383 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fefe7e71f05f8d5-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.386 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.386 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.386 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.386 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.386 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.387 DEBUG:boto:Method: GET 2024-04-13T21:38:49.387 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.387 DEBUG:boto:Data: 2024-04-13T21:38:49.387 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.387 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.387 DEBUG:boto:Params: {} 2024-04-13T21:38:49.387 DEBUG:boto:Token: None 2024-04-13T21:38:49.387 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020a26fdd0e3311a7-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.390 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.390 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.390 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.391 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.391 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.391 DEBUG:boto:Data: 2024-04-13T21:38:49.391 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.391 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.391 DEBUG:boto:Params: {} 2024-04-13T21:38:49.391 DEBUG:boto:Token: None 2024-04-13T21:38:49.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004a371127a1b6d41-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.395 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.395 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.395 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.395 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.395 DEBUG:boto:Data: 2024-04-13T21:38:49.395 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.395 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.396 DEBUG:boto:Params: {} 2024-04-13T21:38:49.396 DEBUG:boto:Token: None 2024-04-13T21:38:49.396 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000627351cd78337b2-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.399 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.399 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.399 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.399 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.399 DEBUG:boto:Method: GET 2024-04-13T21:38:49.399 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.399 DEBUG:boto:Data: 2024-04-13T21:38:49.400 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.400 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.400 DEBUG:boto:Params: {} 2024-04-13T21:38:49.400 DEBUG:boto:Token: None 2024-04-13T21:38:49.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008098cdc8dd7b64e7-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.442 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.443 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.443 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.443 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.443 DEBUG:boto:Method: GET 2024-04-13T21:38:49.443 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.443 DEBUG:boto:Data: 2024-04-13T21:38:49.443 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.443 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.443 DEBUG:boto:Params: {} 2024-04-13T21:38:49.444 DEBUG:boto:Token: None 2024-04-13T21:38:49.444 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba48372ecd1fccda-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.486 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.486 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.487 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.487 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.487 DEBUG:boto:Method: GET 2024-04-13T21:38:49.487 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.487 DEBUG:boto:Data: 2024-04-13T21:38:49.487 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.487 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.487 DEBUG:boto:Params: {} 2024-04-13T21:38:49.487 DEBUG:boto:Token: None 2024-04-13T21:38:49.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee88bc7aec3976b-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.490 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.491 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.491 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.491 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.491 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.491 DEBUG:boto:Method: GET 2024-04-13T21:38:49.491 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.491 DEBUG:boto:Data: 2024-04-13T21:38:49.491 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.492 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.492 DEBUG:boto:Params: {} 2024-04-13T21:38:49.492 DEBUG:boto:Token: None 2024-04-13T21:38:49.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfbb07f6c12717f7-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.494 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.495 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.495 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.495 DEBUG:boto:path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.495 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.495 DEBUG:boto:Method: GET 2024-04-13T21:38:49.495 DEBUG:boto:Path: /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.496 DEBUG:boto:Data: 2024-04-13T21:38:49.496 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.496 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.496 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.496 DEBUG:boto:Params: {} 2024-04-13T21:38:49.496 DEBUG:boto:Token: None 2024-04-13T21:38:49.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o= 2024-04-13T21:38:49.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o=', 'Content-Length': '0'} 2024-04-13T21:38:49.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058c4822a376ac16f-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.498 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.499 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.499 DEBUG:boto:path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.499 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.499 DEBUG:boto:Method: GET 2024-04-13T21:38:49.499 DEBUG:boto:Path: /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.499 DEBUG:boto:Data: 2024-04-13T21:38:49.499 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.499 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.499 DEBUG:boto:Params: {} 2024-04-13T21:38:49.500 DEBUG:boto:Token: None 2024-04-13T21:38:49.500 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o= 2024-04-13T21:38:49.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o=', 'Content-Length': '0'} 2024-04-13T21:38:49.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6c9f5f90f4343d7-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.502 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.502 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.503 DEBUG:boto:path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.503 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.503 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.503 DEBUG:boto:Path: /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.503 DEBUG:boto:Data: 2024-04-13T21:38:49.503 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.503 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.503 DEBUG:boto:Params: {} 2024-04-13T21:38:49.503 DEBUG:boto:Token: None 2024-04-13T21:38:49.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LL7Ibx0dekUxODY+JYM48zu3S18= 2024-04-13T21:38:49.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LL7Ibx0dekUxODY+JYM48zu3S18=', 'Content-Length': '0'} 2024-04-13T21:38:49.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aac4505d51e18e3b-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.506 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.506 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.506 DEBUG:boto:path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.506 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.506 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.506 DEBUG:boto:Path: /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.506 DEBUG:boto:Data: 2024-04-13T21:38:49.507 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.507 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.507 DEBUG:boto:Params: {} 2024-04-13T21:38:49.507 DEBUG:boto:Token: None 2024-04-13T21:38:49.507 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LL7Ibx0dekUxODY+JYM48zu3S18= 2024-04-13T21:38:49.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LL7Ibx0dekUxODY+JYM48zu3S18=', 'Content-Length': '0'} 2024-04-13T21:38:49.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c6ec32065df6e4d-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.509 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.509 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.509 DEBUG:boto:path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.509 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.509 DEBUG:boto:Method: GET 2024-04-13T21:38:49.510 DEBUG:boto:Path: /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.510 DEBUG:boto:Data: 2024-04-13T21:38:49.510 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.510 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.510 DEBUG:boto:Params: {} 2024-04-13T21:38:49.510 DEBUG:boto:Token: None 2024-04-13T21:38:49.510 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.510 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o= 2024-04-13T21:38:49.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o=', 'Content-Length': '0'} 2024-04-13T21:38:49.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000380d298e7012ab7b-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.512 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.513 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.513 DEBUG:boto:path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.513 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.513 DEBUG:boto:Method: GET 2024-04-13T21:38:49.513 DEBUG:boto:Path: /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.513 DEBUG:boto:Data: 2024-04-13T21:38:49.513 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.513 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.513 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.513 DEBUG:boto:Params: {} 2024-04-13T21:38:49.513 DEBUG:boto:Token: None 2024-04-13T21:38:49.514 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=019k12m187iEGtpVnMZTqfvWgxncrKZ 2024-04-13T21:38:49.514 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o= 2024-04-13T21:38:49.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6g4/w3cqKmgEZ6W0KHhVpRYbK8o=', 'Content-Length': '0'} 2024-04-13T21:38:49.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '019k12m187iEGtpVnMZTqfvWgxncrKZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f5bf5827f391dd0-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.558 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.558 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.559 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.559 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.559 DEBUG:boto:Method: GET 2024-04-13T21:38:49.559 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.559 DEBUG:boto:Data: 2024-04-13T21:38:49.559 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.560 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.560 DEBUG:boto:Params: {} 2024-04-13T21:38:49.560 DEBUG:boto:Token: None 2024-04-13T21:38:49.560 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3cc9fab8edd9f47-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.563 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.563 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.564 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.564 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.564 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.564 DEBUG:boto:Method: GET 2024-04-13T21:38:49.564 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.564 DEBUG:boto:Data: 2024-04-13T21:38:49.564 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.564 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.565 DEBUG:boto:Params: {} 2024-04-13T21:38:49.565 DEBUG:boto:Token: None 2024-04-13T21:38:49.565 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ef3de7c1d70f4b-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.610 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.611 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.611 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.611 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.611 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.611 DEBUG:boto:Data: 2024-04-13T21:38:49.611 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.611 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.611 DEBUG:boto:Params: {} 2024-04-13T21:38:49.612 DEBUG:boto:Token: None 2024-04-13T21:38:49.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000049fa56687c98985-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.615 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.615 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.615 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.615 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.615 DEBUG:boto:Data: 2024-04-13T21:38:49.615 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.615 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.616 DEBUG:boto:Params: {} 2024-04-13T21:38:49.616 DEBUG:boto:Token: None 2024-04-13T21:38:49.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000139f29749e38eef3-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.619 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.619 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.619 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.619 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.619 DEBUG:boto:Method: GET 2024-04-13T21:38:49.620 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.620 DEBUG:boto:Data: 2024-04-13T21:38:49.620 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.620 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.620 DEBUG:boto:Params: {} 2024-04-13T21:38:49.620 DEBUG:boto:Token: None 2024-04-13T21:38:49.620 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ffe1177c263e457-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.662 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.663 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.663 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.663 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.663 DEBUG:boto:Method: GET 2024-04-13T21:38:49.663 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.663 DEBUG:boto:Data: 2024-04-13T21:38:49.663 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.663 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.663 DEBUG:boto:Params: {} 2024-04-13T21:38:49.663 DEBUG:boto:Token: None 2024-04-13T21:38:49.664 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000301444f2c524fbdb-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.706 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.707 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.707 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.707 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.707 DEBUG:boto:Method: GET 2024-04-13T21:38:49.707 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.707 DEBUG:boto:Data: 2024-04-13T21:38:49.707 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.707 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.708 DEBUG:boto:Params: {} 2024-04-13T21:38:49.708 DEBUG:boto:Token: None 2024-04-13T21:38:49.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8403308fcaf155f-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.711 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.711 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.711 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.711 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.712 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.712 DEBUG:boto:Method: GET 2024-04-13T21:38:49.712 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.712 DEBUG:boto:Data: 2024-04-13T21:38:49.712 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.712 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.712 DEBUG:boto:Params: {} 2024-04-13T21:38:49.712 DEBUG:boto:Token: None 2024-04-13T21:38:49.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.713 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e512667221c1b00e-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.715 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.716 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.716 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.716 DEBUG:boto:path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.716 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.716 DEBUG:boto:Method: GET 2024-04-13T21:38:49.716 DEBUG:boto:Path: /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.716 DEBUG:boto:Data: 2024-04-13T21:38:49.716 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.716 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.717 DEBUG:boto:Params: {} 2024-04-13T21:38:49.717 DEBUG:boto:Token: None 2024-04-13T21:38:49.717 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE= 2024-04-13T21:38:49.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE=', 'Content-Length': '0'} 2024-04-13T21:38:49.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019775d997166f6a4-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.719 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.719 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.720 DEBUG:boto:path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.720 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.720 DEBUG:boto:Method: GET 2024-04-13T21:38:49.720 DEBUG:boto:Path: /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.720 DEBUG:boto:Data: 2024-04-13T21:38:49.720 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.720 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.720 DEBUG:boto:Params: {} 2024-04-13T21:38:49.720 DEBUG:boto:Token: None 2024-04-13T21:38:49.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE= 2024-04-13T21:38:49.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE=', 'Content-Length': '0'} 2024-04-13T21:38:49.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a31f66e2b8bc597b-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.770 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.771 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.771 DEBUG:boto:path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.771 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.771 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.771 DEBUG:boto:Path: /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.771 DEBUG:boto:Data: 2024-04-13T21:38:49.771 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.772 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.772 DEBUG:boto:Params: {} 2024-04-13T21:38:49.772 DEBUG:boto:Token: None 2024-04-13T21:38:49.772 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VElrkmu7VaAl3/+L1qg7nUwwNyo= 2024-04-13T21:38:49.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VElrkmu7VaAl3/+L1qg7nUwwNyo=', 'Content-Length': '0'} 2024-04-13T21:38:49.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041d5c53a6bb8fae3-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.774 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.774 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.774 DEBUG:boto:path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.775 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.775 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.775 DEBUG:boto:Path: /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.775 DEBUG:boto:Data: 2024-04-13T21:38:49.775 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.775 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.775 DEBUG:boto:Params: {} 2024-04-13T21:38:49.775 DEBUG:boto:Token: None 2024-04-13T21:38:49.776 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VElrkmu7VaAl3/+L1qg7nUwwNyo= 2024-04-13T21:38:49.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VElrkmu7VaAl3/+L1qg7nUwwNyo=', 'Content-Length': '0'} 2024-04-13T21:38:49.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fdae437180f7567-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.778 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.778 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.778 DEBUG:boto:path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.778 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.778 DEBUG:boto:Method: GET 2024-04-13T21:38:49.778 DEBUG:boto:Path: /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.779 DEBUG:boto:Data: 2024-04-13T21:38:49.779 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.779 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.779 DEBUG:boto:Params: {} 2024-04-13T21:38:49.779 DEBUG:boto:Token: None 2024-04-13T21:38:49.779 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE= 2024-04-13T21:38:49.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE=', 'Content-Length': '0'} 2024-04-13T21:38:49.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7ab820a3b216e10-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.822 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.822 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.822 DEBUG:boto:path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.823 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.823 DEBUG:boto:Method: GET 2024-04-13T21:38:49.823 DEBUG:boto:Path: /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.823 DEBUG:boto:Data: 2024-04-13T21:38:49.823 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.823 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.823 DEBUG:boto:Params: {} 2024-04-13T21:38:49.823 DEBUG:boto:Token: None 2024-04-13T21:38:49.823 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=HXIrwk7JSqUilmLyhyog.MC4rqQPfL3 2024-04-13T21:38:49.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE= 2024-04-13T21:38:49.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t0OT+VA4Q9QXvSlGl6iYck7uPuE=', 'Content-Length': '0'} 2024-04-13T21:38:49.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'HXIrwk7JSqUilmLyhyog.MC4rqQPfL3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095f71ee34e3fbda1-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.826 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.826 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.826 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.826 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.826 DEBUG:boto:Method: GET 2024-04-13T21:38:49.826 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.826 DEBUG:boto:Data: 2024-04-13T21:38:49.826 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.827 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.827 DEBUG:boto:Params: {} 2024-04-13T21:38:49.827 DEBUG:boto:Token: None 2024-04-13T21:38:49.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000058551d251686403-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.830 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.830 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.830 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.830 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.830 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.830 DEBUG:boto:Method: GET 2024-04-13T21:38:49.831 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.831 DEBUG:boto:Data: 2024-04-13T21:38:49.831 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.831 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.831 DEBUG:boto:Params: {} 2024-04-13T21:38:49.831 DEBUG:boto:Token: None 2024-04-13T21:38:49.831 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.831 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003084d0b1fd301e84-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.834 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.834 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.835 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.835 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.835 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.835 DEBUG:boto:Data: 2024-04-13T21:38:49.835 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.835 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.835 DEBUG:boto:Params: {} 2024-04-13T21:38:49.835 DEBUG:boto:Token: None 2024-04-13T21:38:49.836 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bc76d467490f79c-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.839 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.839 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.839 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.839 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:49.839 DEBUG:boto:Data: 2024-04-13T21:38:49.839 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.840 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.840 DEBUG:boto:Params: {} 2024-04-13T21:38:49.840 DEBUG:boto:Token: None 2024-04-13T21:38:49.840 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj 2024-04-13T21:38:49.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo= 2024-04-13T21:38:49.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vg3aTG5O9vWwGacdcaC0YgXJgVo=', 'Content-Length': '0'} 2024-04-13T21:38:49.843 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073196a51c6c27002-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.843 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.843 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.844 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.844 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.844 DEBUG:boto:Method: GET 2024-04-13T21:38:49.844 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.844 DEBUG:boto:Data: 2024-04-13T21:38:49.844 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.844 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.844 DEBUG:boto:Params: {} 2024-04-13T21:38:49.844 DEBUG:boto:Token: None 2024-04-13T21:38:49.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000625ca635d7d71919-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.886 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.886 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.887 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.887 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.887 DEBUG:boto:Method: GET 2024-04-13T21:38:49.887 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.887 DEBUG:boto:Data: 2024-04-13T21:38:49.887 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.887 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.887 DEBUG:boto:Params: {} 2024-04-13T21:38:49.887 DEBUG:boto:Token: None 2024-04-13T21:38:49.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:49.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM= 2024-04-13T21:38:49.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxR60u9v9iyO+OV+DWRcRERa/dM=', 'Content-Length': '0'} 2024-04-13T21:38:49.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b37f9cc830206f1e-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.930 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.931 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.931 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.931 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.931 DEBUG:boto:Method: GET 2024-04-13T21:38:49.931 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.931 DEBUG:boto:Data: 2024-04-13T21:38:49.931 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.931 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.931 DEBUG:boto:Params: {} 2024-04-13T21:38:49.931 DEBUG:boto:Token: None 2024-04-13T21:38:49.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ddb39414fac784d-00661afb69-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.934 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.935 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:49.935 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:49.935 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.935 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:49.935 DEBUG:boto:Method: GET 2024-04-13T21:38:49.935 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:49.935 DEBUG:boto:Data: 2024-04-13T21:38:49.936 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.936 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.936 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.936 DEBUG:boto:Params: {} 2024-04-13T21:38:49.936 DEBUG:boto:Token: None 2024-04-13T21:38:49.936 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/ 2024-04-13T21:38:49.936 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8= 2024-04-13T21:38:49.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:43hGbdNak1qA3/GH3KMy1Q9KVz8=', 'Content-Length': '0'} 2024-04-13T21:38:49.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056d55bd2c7aae3a8-00661afb69-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.939 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:49.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.939 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.939 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.939 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.939 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.940 DEBUG:boto:Method: GET 2024-04-13T21:38:49.940 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.940 DEBUG:boto:Data: 2024-04-13T21:38:49.940 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.940 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.940 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.940 DEBUG:boto:Params: {} 2024-04-13T21:38:49.940 DEBUG:boto:Token: None 2024-04-13T21:38:49.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G2rB/3jhHP3uxQBMovf1sTuxL24= 2024-04-13T21:38:49.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G2rB/3jhHP3uxQBMovf1sTuxL24=', 'Content-Length': '0'} 2024-04-13T21:38:49.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000564d9a8ba885af97-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.943 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.943 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.943 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.943 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.943 DEBUG:boto:Method: GET 2024-04-13T21:38:49.943 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.943 DEBUG:boto:Data: 2024-04-13T21:38:49.943 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.943 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.943 DEBUG:boto:Params: {} 2024-04-13T21:38:49.944 DEBUG:boto:Token: None 2024-04-13T21:38:49.944 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.944 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G2rB/3jhHP3uxQBMovf1sTuxL24= 2024-04-13T21:38:49.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G2rB/3jhHP3uxQBMovf1sTuxL24=', 'Content-Length': '0'} 2024-04-13T21:38:49.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a776d314b65f3a63-00661afb69-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.986 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.987 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.987 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.987 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.987 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.987 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.987 DEBUG:boto:Data: 2024-04-13T21:38:49.987 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.987 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.987 DEBUG:boto:Params: {} 2024-04-13T21:38:49.987 DEBUG:boto:Token: None 2024-04-13T21:38:49.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4/ll2HQO18kCZPpo87JmSjtWoa4= 2024-04-13T21:38:49.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4/ll2HQO18kCZPpo87JmSjtWoa4=', 'Content-Length': '0'} 2024-04-13T21:38:49.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a768dd050d70c90-00661afb69-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.990 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.990 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.990 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.990 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.990 DEBUG:boto:Method: HEAD 2024-04-13T21:38:49.991 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.991 DEBUG:boto:Data: 2024-04-13T21:38:49.991 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:49.991 DEBUG:boto:Port: 8000 2024-04-13T21:38:49.991 DEBUG:boto:Params: {} 2024-04-13T21:38:49.991 DEBUG:boto:Token: None 2024-04-13T21:38:49.991 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4/ll2HQO18kCZPpo87JmSjtWoa4= 2024-04-13T21:38:49.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4/ll2HQO18kCZPpo87JmSjtWoa4=', 'Content-Length': '0'} 2024-04-13T21:38:49.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000475922f3bca804aa-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:49.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:49.994 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:49.994 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:49.994 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.994 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.994 DEBUG:boto:Method: GET 2024-04-13T21:38:49.994 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.994 DEBUG:boto:Data: 2024-04-13T21:38:49.994 DEBUG:boto:Headers: {} 2024-04-13T21:38:49.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:49.994 DEBUG:boto:Port: 8001 2024-04-13T21:38:49.994 DEBUG:boto:Params: {} 2024-04-13T21:38:49.995 DEBUG:boto:Token: None 2024-04-13T21:38:49.995 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:49 GMT /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:49.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G2rB/3jhHP3uxQBMovf1sTuxL24= 2024-04-13T21:38:49.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G2rB/3jhHP3uxQBMovf1sTuxL24=', 'Content-Length': '0'} 2024-04-13T21:38:49.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007afa6167ac3a2673-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.038 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.038 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.039 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:50.039 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:50.039 DEBUG:boto:Method: GET 2024-04-13T21:38:50.039 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:50.039 DEBUG:boto:Data: 2024-04-13T21:38:50.039 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.039 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.039 DEBUG:boto:Params: {} 2024-04-13T21:38:50.039 DEBUG:boto:Token: None 2024-04-13T21:38:50.039 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF 2024-04-13T21:38:50.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r9tL6/PrHco/6FGZvrh37D/eqYg= 2024-04-13T21:38:50.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r9tL6/PrHco/6FGZvrh37D/eqYg=', 'Content-Length': '0'} 2024-04-13T21:38:50.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jvku7JHOc-vMkfpIC8bwINCzCuWCiqF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005de412f859a40c3c-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.042 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.042 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.042 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.042 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.042 DEBUG:boto:Method: GET 2024-04-13T21:38:50.042 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.043 DEBUG:boto:Data: 2024-04-13T21:38:50.043 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.043 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.043 DEBUG:boto:Params: {} 2024-04-13T21:38:50.043 DEBUG:boto:Token: None 2024-04-13T21:38:50.043 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017f4b41a24e5f962-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.046 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.047 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.047 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.047 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.047 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.047 DEBUG:boto:Method: GET 2024-04-13T21:38:50.047 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.047 DEBUG:boto:Data: 2024-04-13T21:38:50.047 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.047 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.047 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.048 DEBUG:boto:Params: {} 2024-04-13T21:38:50.048 DEBUG:boto:Token: None 2024-04-13T21:38:50.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a09a07b9420af4-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.051 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.051 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.051 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.051 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.051 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.051 DEBUG:boto:Data: 2024-04-13T21:38:50.051 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.052 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.052 DEBUG:boto:Params: {} 2024-04-13T21:38:50.052 DEBUG:boto:Token: None 2024-04-13T21:38:50.052 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b869b064e3d3768b-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.055 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.055 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.055 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.055 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.056 DEBUG:boto:Data: 2024-04-13T21:38:50.056 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.056 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.056 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.056 DEBUG:boto:Params: {} 2024-04-13T21:38:50.056 DEBUG:boto:Token: None 2024-04-13T21:38:50.056 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002ffba7b3a591d47-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.059 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.059 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.060 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.060 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.060 DEBUG:boto:Method: GET 2024-04-13T21:38:50.060 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.060 DEBUG:boto:Data: 2024-04-13T21:38:50.060 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.060 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.060 DEBUG:boto:Params: {} 2024-04-13T21:38:50.060 DEBUG:boto:Token: None 2024-04-13T21:38:50.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000250eaaccb7a4adeb-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.110 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.111 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.111 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.111 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.111 DEBUG:boto:Method: GET 2024-04-13T21:38:50.111 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.111 DEBUG:boto:Data: 2024-04-13T21:38:50.111 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.111 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.111 DEBUG:boto:Params: {} 2024-04-13T21:38:50.112 DEBUG:boto:Token: None 2024-04-13T21:38:50.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fe7f232c9fe7ae5-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.158 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.159 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.159 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.159 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.159 DEBUG:boto:Method: GET 2024-04-13T21:38:50.159 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.159 DEBUG:boto:Data: 2024-04-13T21:38:50.159 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.159 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.159 DEBUG:boto:Params: {} 2024-04-13T21:38:50.159 DEBUG:boto:Token: None 2024-04-13T21:38:50.160 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001795618ec21aba2d-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.163 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.163 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.163 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.163 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.163 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.163 DEBUG:boto:Method: GET 2024-04-13T21:38:50.164 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.164 DEBUG:boto:Data: 2024-04-13T21:38:50.164 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.164 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.164 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.164 DEBUG:boto:Params: {} 2024-04-13T21:38:50.164 DEBUG:boto:Token: None 2024-04-13T21:38:50.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ad84bec3178d27-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.210 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.211 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.211 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.211 DEBUG:boto:path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.211 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.211 DEBUG:boto:Method: GET 2024-04-13T21:38:50.211 DEBUG:boto:Path: /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.211 DEBUG:boto:Data: 2024-04-13T21:38:50.212 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.212 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.212 DEBUG:boto:Params: {} 2024-04-13T21:38:50.212 DEBUG:boto:Token: None 2024-04-13T21:38:50.212 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA= 2024-04-13T21:38:50.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA=', 'Content-Length': '0'} 2024-04-13T21:38:50.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc043252fef9d309-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.214 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.215 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.215 DEBUG:boto:path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.215 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.215 DEBUG:boto:Method: GET 2024-04-13T21:38:50.215 DEBUG:boto:Path: /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.215 DEBUG:boto:Data: 2024-04-13T21:38:50.215 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.215 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.215 DEBUG:boto:Params: {} 2024-04-13T21:38:50.216 DEBUG:boto:Token: None 2024-04-13T21:38:50.216 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA= 2024-04-13T21:38:50.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA=', 'Content-Length': '0'} 2024-04-13T21:38:50.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d1fc16b0b27f90a-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.218 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.218 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.218 DEBUG:boto:path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.219 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.219 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.219 DEBUG:boto:Path: /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.219 DEBUG:boto:Data: 2024-04-13T21:38:50.219 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.219 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.219 DEBUG:boto:Params: {} 2024-04-13T21:38:50.219 DEBUG:boto:Token: None 2024-04-13T21:38:50.219 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZlfZgij4e2qT8/589KRD3Bo4EpY= 2024-04-13T21:38:50.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZlfZgij4e2qT8/589KRD3Bo4EpY=', 'Content-Length': '0'} 2024-04-13T21:38:50.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d225af230786f129-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.222 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.222 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.222 DEBUG:boto:path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.222 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.222 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.222 DEBUG:boto:Path: /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.222 DEBUG:boto:Data: 2024-04-13T21:38:50.222 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.222 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.222 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.223 DEBUG:boto:Params: {} 2024-04-13T21:38:50.223 DEBUG:boto:Token: None 2024-04-13T21:38:50.223 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.223 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZlfZgij4e2qT8/589KRD3Bo4EpY= 2024-04-13T21:38:50.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZlfZgij4e2qT8/589KRD3Bo4EpY=', 'Content-Length': '0'} 2024-04-13T21:38:50.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042dc135171815396-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.225 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.225 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.225 DEBUG:boto:path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.225 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.225 DEBUG:boto:Method: GET 2024-04-13T21:38:50.225 DEBUG:boto:Path: /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.225 DEBUG:boto:Data: 2024-04-13T21:38:50.226 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.226 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.226 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.226 DEBUG:boto:Params: {} 2024-04-13T21:38:50.226 DEBUG:boto:Token: None 2024-04-13T21:38:50.226 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.226 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA= 2024-04-13T21:38:50.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA=', 'Content-Length': '0'} 2024-04-13T21:38:50.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f8e7c88e8ec7965-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.228 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.229 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.229 DEBUG:boto:path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.229 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.229 DEBUG:boto:Method: GET 2024-04-13T21:38:50.229 DEBUG:boto:Path: /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.229 DEBUG:boto:Data: 2024-04-13T21:38:50.229 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.229 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.229 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.229 DEBUG:boto:Params: {} 2024-04-13T21:38:50.229 DEBUG:boto:Token: None 2024-04-13T21:38:50.230 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=u73pgnaQBi-XxafVIxtpwHUx5HMGrfI 2024-04-13T21:38:50.230 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA= 2024-04-13T21:38:50.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MtL5OgmPWog2qkpiMUghJ7og7cA=', 'Content-Length': '0'} 2024-04-13T21:38:50.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'u73pgnaQBi-XxafVIxtpwHUx5HMGrfI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f73fb2b110e4f66-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.274 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.274 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.274 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.275 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.275 DEBUG:boto:Method: GET 2024-04-13T21:38:50.275 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.275 DEBUG:boto:Data: 2024-04-13T21:38:50.275 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.275 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.275 DEBUG:boto:Params: {} 2024-04-13T21:38:50.275 DEBUG:boto:Token: None 2024-04-13T21:38:50.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046f5879f2760e057-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.278 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.279 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.279 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.279 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.279 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.279 DEBUG:boto:Method: GET 2024-04-13T21:38:50.279 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.279 DEBUG:boto:Data: 2024-04-13T21:38:50.279 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.279 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.280 DEBUG:boto:Params: {} 2024-04-13T21:38:50.280 DEBUG:boto:Token: None 2024-04-13T21:38:50.280 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e7a46cea1952838-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.282 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.283 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.283 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.283 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.283 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.283 DEBUG:boto:Data: 2024-04-13T21:38:50.283 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.283 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.283 DEBUG:boto:Params: {} 2024-04-13T21:38:50.284 DEBUG:boto:Token: None 2024-04-13T21:38:50.284 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030b0a4b9f75b8101-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.287 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.287 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.287 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.287 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.287 DEBUG:boto:Data: 2024-04-13T21:38:50.287 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.288 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.288 DEBUG:boto:Params: {} 2024-04-13T21:38:50.288 DEBUG:boto:Token: None 2024-04-13T21:38:50.288 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025e1bef277f7f3c0-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.291 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.291 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.292 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.292 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.292 DEBUG:boto:Method: GET 2024-04-13T21:38:50.292 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.292 DEBUG:boto:Data: 2024-04-13T21:38:50.292 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.292 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.292 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.292 DEBUG:boto:Params: {} 2024-04-13T21:38:50.292 DEBUG:boto:Token: None 2024-04-13T21:38:50.293 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.293 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070cc73f03a6f0f51-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.338 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.339 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.339 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.339 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.339 DEBUG:boto:Method: GET 2024-04-13T21:38:50.339 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.339 DEBUG:boto:Data: 2024-04-13T21:38:50.339 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.339 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.339 DEBUG:boto:Params: {} 2024-04-13T21:38:50.340 DEBUG:boto:Token: None 2024-04-13T21:38:50.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f314ba01df51aab-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.386 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.387 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.387 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.387 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.387 DEBUG:boto:Method: GET 2024-04-13T21:38:50.387 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.387 DEBUG:boto:Data: 2024-04-13T21:38:50.387 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.388 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.388 DEBUG:boto:Params: {} 2024-04-13T21:38:50.388 DEBUG:boto:Token: None 2024-04-13T21:38:50.388 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d060cd7dc88a94-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.391 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.392 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.392 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.392 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.392 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.392 DEBUG:boto:Method: GET 2024-04-13T21:38:50.392 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.392 DEBUG:boto:Data: 2024-04-13T21:38:50.392 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.393 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.393 DEBUG:boto:Params: {} 2024-04-13T21:38:50.393 DEBUG:boto:Token: None 2024-04-13T21:38:50.393 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.393 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046c5d2b73003dd16-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.396 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.396 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.396 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.396 DEBUG:boto:path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.396 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.397 DEBUG:boto:Method: GET 2024-04-13T21:38:50.397 DEBUG:boto:Path: /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.397 DEBUG:boto:Data: 2024-04-13T21:38:50.397 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.397 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.397 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.397 DEBUG:boto:Params: {} 2024-04-13T21:38:50.397 DEBUG:boto:Token: None 2024-04-13T21:38:50.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8= 2024-04-13T21:38:50.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8=', 'Content-Length': '0'} 2024-04-13T21:38:50.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043b40fe4362f8ff2-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.400 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.400 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.400 DEBUG:boto:path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.400 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.400 DEBUG:boto:Method: GET 2024-04-13T21:38:50.400 DEBUG:boto:Path: /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.401 DEBUG:boto:Data: 2024-04-13T21:38:50.401 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.401 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.401 DEBUG:boto:Params: {} 2024-04-13T21:38:50.401 DEBUG:boto:Token: None 2024-04-13T21:38:50.402 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.402 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8= 2024-04-13T21:38:50.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8=', 'Content-Length': '0'} 2024-04-13T21:38:50.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059515e29f4b6185a-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.405 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.405 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.405 DEBUG:boto:path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.405 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.405 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.406 DEBUG:boto:Path: /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.406 DEBUG:boto:Data: 2024-04-13T21:38:50.406 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.406 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.406 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.406 DEBUG:boto:Params: {} 2024-04-13T21:38:50.406 DEBUG:boto:Token: None 2024-04-13T21:38:50.406 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/eDrYB3CNrvfQGKCY8by/ziOXG0= 2024-04-13T21:38:50.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/eDrYB3CNrvfQGKCY8by/ziOXG0=', 'Content-Length': '0'} 2024-04-13T21:38:50.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eda6c3b7085efdcd-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.408 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.409 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.409 DEBUG:boto:path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.409 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.409 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.409 DEBUG:boto:Path: /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.409 DEBUG:boto:Data: 2024-04-13T21:38:50.409 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.409 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.409 DEBUG:boto:Params: {} 2024-04-13T21:38:50.410 DEBUG:boto:Token: None 2024-04-13T21:38:50.410 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/eDrYB3CNrvfQGKCY8by/ziOXG0= 2024-04-13T21:38:50.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/eDrYB3CNrvfQGKCY8by/ziOXG0=', 'Content-Length': '0'} 2024-04-13T21:38:50.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000444104a5802d54f2-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.412 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.412 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.412 DEBUG:boto:path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.412 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.412 DEBUG:boto:Method: GET 2024-04-13T21:38:50.412 DEBUG:boto:Path: /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.412 DEBUG:boto:Data: 2024-04-13T21:38:50.412 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.413 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.413 DEBUG:boto:Params: {} 2024-04-13T21:38:50.413 DEBUG:boto:Token: None 2024-04-13T21:38:50.413 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8= 2024-04-13T21:38:50.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8=', 'Content-Length': '0'} 2024-04-13T21:38:50.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000084526ea46e747c5-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.415 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.415 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.415 DEBUG:boto:path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.415 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.416 DEBUG:boto:Method: GET 2024-04-13T21:38:50.416 DEBUG:boto:Path: /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.416 DEBUG:boto:Data: 2024-04-13T21:38:50.416 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.416 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.416 DEBUG:boto:Params: {} 2024-04-13T21:38:50.416 DEBUG:boto:Token: None 2024-04-13T21:38:50.416 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14 2024-04-13T21:38:50.417 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8= 2024-04-13T21:38:50.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ruK22qF6oH5B6cWux7DT7aoUaj8=', 'Content-Length': '0'} 2024-04-13T21:38:50.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '8kNGdSrfZVxqeFcaAJzOYrFoNVQQG14'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8ec58c745c238fb-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.462 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.462 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.463 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.463 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.463 DEBUG:boto:Method: GET 2024-04-13T21:38:50.463 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.463 DEBUG:boto:Data: 2024-04-13T21:38:50.463 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.463 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.463 DEBUG:boto:Params: {} 2024-04-13T21:38:50.463 DEBUG:boto:Token: None 2024-04-13T21:38:50.463 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000211c3a86041c21fe-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.466 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.467 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.467 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.467 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.467 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.467 DEBUG:boto:Method: GET 2024-04-13T21:38:50.467 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.467 DEBUG:boto:Data: 2024-04-13T21:38:50.467 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.467 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.468 DEBUG:boto:Params: {} 2024-04-13T21:38:50.468 DEBUG:boto:Token: None 2024-04-13T21:38:50.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7568aa207f86632-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.514 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.515 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.515 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.515 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.515 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.515 DEBUG:boto:Data: 2024-04-13T21:38:50.515 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.515 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.516 DEBUG:boto:Params: {} 2024-04-13T21:38:50.516 DEBUG:boto:Token: None 2024-04-13T21:38:50.516 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad5e96b9047afb7f-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.519 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.520 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.520 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.520 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.520 DEBUG:boto:Data: 2024-04-13T21:38:50.520 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.520 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.520 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.520 DEBUG:boto:Params: {} 2024-04-13T21:38:50.520 DEBUG:boto:Token: None 2024-04-13T21:38:50.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002181faca32e8820e-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.524 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.524 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.524 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.524 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.524 DEBUG:boto:Method: GET 2024-04-13T21:38:50.525 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.525 DEBUG:boto:Data: 2024-04-13T21:38:50.525 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.525 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.525 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.525 DEBUG:boto:Params: {} 2024-04-13T21:38:50.525 DEBUG:boto:Token: None 2024-04-13T21:38:50.525 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.525 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000821a259f7cd1cf99-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.570 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.570 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.571 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.571 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.571 DEBUG:boto:Method: GET 2024-04-13T21:38:50.571 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.571 DEBUG:boto:Data: 2024-04-13T21:38:50.571 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.571 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.571 DEBUG:boto:Params: {} 2024-04-13T21:38:50.571 DEBUG:boto:Token: None 2024-04-13T21:38:50.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009a59989877009ca-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.614 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.614 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.615 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.615 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.615 DEBUG:boto:Method: GET 2024-04-13T21:38:50.615 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.615 DEBUG:boto:Data: 2024-04-13T21:38:50.615 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.615 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.615 DEBUG:boto:Params: {} 2024-04-13T21:38:50.615 DEBUG:boto:Token: None 2024-04-13T21:38:50.616 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029cdd420690ca112-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.618 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.619 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.619 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.619 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.619 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.619 DEBUG:boto:Method: GET 2024-04-13T21:38:50.619 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.619 DEBUG:boto:Data: 2024-04-13T21:38:50.620 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.620 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.620 DEBUG:boto:Params: {} 2024-04-13T21:38:50.620 DEBUG:boto:Token: None 2024-04-13T21:38:50.620 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531b49e7fcb670e7-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.623 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.623 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.623 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.624 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.624 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.624 DEBUG:boto:Method: GET 2024-04-13T21:38:50.624 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.624 DEBUG:boto:Data: 2024-04-13T21:38:50.624 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.624 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.624 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.624 DEBUG:boto:Params: {} 2024-04-13T21:38:50.625 DEBUG:boto:Token: None 2024-04-13T21:38:50.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM= 2024-04-13T21:38:50.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM=', 'Content-Length': '0'} 2024-04-13T21:38:50.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e5c0afdc03d4357-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.627 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.627 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.628 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.628 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.628 DEBUG:boto:Method: GET 2024-04-13T21:38:50.628 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.628 DEBUG:boto:Data: 2024-04-13T21:38:50.628 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.628 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.628 DEBUG:boto:Params: {} 2024-04-13T21:38:50.629 DEBUG:boto:Token: None 2024-04-13T21:38:50.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM= 2024-04-13T21:38:50.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM=', 'Content-Length': '0'} 2024-04-13T21:38:50.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000921680653e376f3b-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.678 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.678 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.679 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.679 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.679 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.679 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.679 DEBUG:boto:Data: 2024-04-13T21:38:50.679 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.679 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.679 DEBUG:boto:Params: {} 2024-04-13T21:38:50.679 DEBUG:boto:Token: None 2024-04-13T21:38:50.679 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4/Ni1xVqUEMD5mbnitzQAwHN+9c= 2024-04-13T21:38:50.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4/Ni1xVqUEMD5mbnitzQAwHN+9c=', 'Content-Length': '0'} 2024-04-13T21:38:50.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8cf3a4b76be4714-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.682 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.682 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.682 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.682 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.682 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.683 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.683 DEBUG:boto:Data: 2024-04-13T21:38:50.683 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.683 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.683 DEBUG:boto:Params: {} 2024-04-13T21:38:50.683 DEBUG:boto:Token: None 2024-04-13T21:38:50.683 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.683 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4/Ni1xVqUEMD5mbnitzQAwHN+9c= 2024-04-13T21:38:50.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4/Ni1xVqUEMD5mbnitzQAwHN+9c=', 'Content-Length': '0'} 2024-04-13T21:38:50.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e16089678c5ecb3-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.685 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.686 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.686 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.686 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.686 DEBUG:boto:Method: GET 2024-04-13T21:38:50.686 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.686 DEBUG:boto:Data: 2024-04-13T21:38:50.686 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.686 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.686 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.686 DEBUG:boto:Params: {} 2024-04-13T21:38:50.686 DEBUG:boto:Token: None 2024-04-13T21:38:50.687 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM= 2024-04-13T21:38:50.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM=', 'Content-Length': '0'} 2024-04-13T21:38:50.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a2ef3c5d8528ece-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.730 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.731 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.731 DEBUG:boto:path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.731 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.731 DEBUG:boto:Method: GET 2024-04-13T21:38:50.731 DEBUG:boto:Path: /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.731 DEBUG:boto:Data: 2024-04-13T21:38:50.731 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.731 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.731 DEBUG:boto:Params: {} 2024-04-13T21:38:50.732 DEBUG:boto:Token: None 2024-04-13T21:38:50.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC 2024-04-13T21:38:50.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM= 2024-04-13T21:38:50.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hnH56TqB8kZZv2lnx7E+qjDrMiM=', 'Content-Length': '0'} 2024-04-13T21:38:50.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'Jpv99ylZJsuQjpIg8bYZ0C1nGaBbiXC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080da64860093f072-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.734 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.734 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.735 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.735 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.735 DEBUG:boto:Method: GET 2024-04-13T21:38:50.735 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.735 DEBUG:boto:Data: 2024-04-13T21:38:50.735 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.735 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.735 DEBUG:boto:Params: {} 2024-04-13T21:38:50.735 DEBUG:boto:Token: None 2024-04-13T21:38:50.736 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009946e8129742f59c-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.738 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.739 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.739 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.739 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.739 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.739 DEBUG:boto:Method: GET 2024-04-13T21:38:50.739 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.740 DEBUG:boto:Data: 2024-04-13T21:38:50.740 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.740 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.740 DEBUG:boto:Params: {} 2024-04-13T21:38:50.740 DEBUG:boto:Token: None 2024-04-13T21:38:50.740 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8386677fcf34bc-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.743 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.744 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.744 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.744 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.744 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.744 DEBUG:boto:Data: 2024-04-13T21:38:50.744 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.744 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.744 DEBUG:boto:Params: {} 2024-04-13T21:38:50.744 DEBUG:boto:Token: None 2024-04-13T21:38:50.745 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.745 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e6ba0b61e5ba22c-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.748 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.748 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.748 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.749 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:50.749 DEBUG:boto:Data: 2024-04-13T21:38:50.749 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.749 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.749 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.749 DEBUG:boto:Params: {} 2024-04-13T21:38:50.749 DEBUG:boto:Token: None 2024-04-13T21:38:50.749 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj 2024-04-13T21:38:50.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w= 2024-04-13T21:38:50.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YwDj7GT5Hw6HFk/0X34o8oES06w=', 'Content-Length': '0'} 2024-04-13T21:38:50.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f88f28859b21d82e-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.753 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.753 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.753 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.753 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.753 DEBUG:boto:Method: GET 2024-04-13T21:38:50.753 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.754 DEBUG:boto:Data: 2024-04-13T21:38:50.754 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.754 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.754 DEBUG:boto:Params: {} 2024-04-13T21:38:50.754 DEBUG:boto:Token: None 2024-04-13T21:38:50.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b699d9d302f2b87b-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.798 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.799 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.799 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.799 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.799 DEBUG:boto:Method: GET 2024-04-13T21:38:50.799 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.799 DEBUG:boto:Data: 2024-04-13T21:38:50.799 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.799 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.799 DEBUG:boto:Params: {} 2024-04-13T21:38:50.800 DEBUG:boto:Token: None 2024-04-13T21:38:50.800 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:50.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM= 2024-04-13T21:38:50.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5NMSAKwS0ZJ1jBLowENGbTsBaWM=', 'Content-Length': '0'} 2024-04-13T21:38:50.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003332b3b63e9f0e11-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.842 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.843 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.843 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.843 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.843 DEBUG:boto:Method: GET 2024-04-13T21:38:50.843 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.843 DEBUG:boto:Data: 2024-04-13T21:38:50.843 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.843 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.843 DEBUG:boto:Params: {} 2024-04-13T21:38:50.844 DEBUG:boto:Token: None 2024-04-13T21:38:50.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0518c72727496c4-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.847 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.847 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.847 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.847 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.847 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.848 DEBUG:boto:Method: GET 2024-04-13T21:38:50.848 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.848 DEBUG:boto:Data: 2024-04-13T21:38:50.848 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.848 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.848 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.848 DEBUG:boto:Params: {} 2024-04-13T21:38:50.848 DEBUG:boto:Token: None 2024-04-13T21:38:50.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b28b764af9db5702-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.894 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.895 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.895 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.895 DEBUG:boto:path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.895 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.895 DEBUG:boto:Method: GET 2024-04-13T21:38:50.896 DEBUG:boto:Path: /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.896 DEBUG:boto:Data: 2024-04-13T21:38:50.896 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.896 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.896 DEBUG:boto:Params: {} 2024-04-13T21:38:50.896 DEBUG:boto:Token: None 2024-04-13T21:38:50.896 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0= 2024-04-13T21:38:50.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0=', 'Content-Length': '0'} 2024-04-13T21:38:50.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cab54283598d554-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.899 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.899 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.899 DEBUG:boto:path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.899 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.899 DEBUG:boto:Method: GET 2024-04-13T21:38:50.899 DEBUG:boto:Path: /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.899 DEBUG:boto:Data: 2024-04-13T21:38:50.900 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.900 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.900 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.900 DEBUG:boto:Params: {} 2024-04-13T21:38:50.900 DEBUG:boto:Token: None 2024-04-13T21:38:50.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0= 2024-04-13T21:38:50.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0=', 'Content-Length': '0'} 2024-04-13T21:38:50.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000929b5c846142edb9-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.902 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.903 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.903 DEBUG:boto:path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.903 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.903 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.903 DEBUG:boto:Path: /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.903 DEBUG:boto:Data: 2024-04-13T21:38:50.903 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.903 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.903 DEBUG:boto:Params: {} 2024-04-13T21:38:50.904 DEBUG:boto:Token: None 2024-04-13T21:38:50.904 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lozY2tFRVinl1JFRD4wz3fkLaDw= 2024-04-13T21:38:50.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lozY2tFRVinl1JFRD4wz3fkLaDw=', 'Content-Length': '0'} 2024-04-13T21:38:50.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f833012dd85cb205-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.906 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.906 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.906 DEBUG:boto:path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.906 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.906 DEBUG:boto:Method: HEAD 2024-04-13T21:38:50.907 DEBUG:boto:Path: /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.907 DEBUG:boto:Data: 2024-04-13T21:38:50.907 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.907 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.907 DEBUG:boto:Params: {} 2024-04-13T21:38:50.907 DEBUG:boto:Token: None 2024-04-13T21:38:50.907 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lozY2tFRVinl1JFRD4wz3fkLaDw= 2024-04-13T21:38:50.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lozY2tFRVinl1JFRD4wz3fkLaDw=', 'Content-Length': '0'} 2024-04-13T21:38:50.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6c25afec8c299f5-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:50.910 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.910 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.910 DEBUG:boto:path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.910 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.910 DEBUG:boto:Method: GET 2024-04-13T21:38:50.910 DEBUG:boto:Path: /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.910 DEBUG:boto:Data: 2024-04-13T21:38:50.910 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.910 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.911 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.911 DEBUG:boto:Params: {} 2024-04-13T21:38:50.911 DEBUG:boto:Token: None 2024-04-13T21:38:50.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0= 2024-04-13T21:38:50.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0=', 'Content-Length': '0'} 2024-04-13T21:38:50.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c387b598f7521cd-00661afb6a-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.913 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:50.913 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:50.914 DEBUG:boto:path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.914 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.914 DEBUG:boto:Method: GET 2024-04-13T21:38:50.914 DEBUG:boto:Path: /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.914 DEBUG:boto:Data: 2024-04-13T21:38:50.914 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.914 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.914 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.914 DEBUG:boto:Params: {} 2024-04-13T21:38:50.914 DEBUG:boto:Token: None 2024-04-13T21:38:50.915 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/obj?versionId=arsbDN8vjMAjqqXjJTyc0XH.3830UP2 2024-04-13T21:38:50.915 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0= 2024-04-13T21:38:50.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xfnZjDd0pC3J6QQdebg/Jl/AfK0=', 'Content-Length': '0'} 2024-04-13T21:38:50.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'arsbDN8vjMAjqqXjJTyc0XH.3830UP2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010e21f321753ed6b-00661afb6a-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.958 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.959 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.959 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.959 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.959 DEBUG:boto:Method: GET 2024-04-13T21:38:50.959 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.959 DEBUG:boto:Data: 2024-04-13T21:38:50.959 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:50.959 DEBUG:boto:Port: 8001 2024-04-13T21:38:50.959 DEBUG:boto:Params: {} 2024-04-13T21:38:50.959 DEBUG:boto:Token: None 2024-04-13T21:38:50.960 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d2bb01c29dd8c1-00661afb6a-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:50.963 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:50.963 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:50.963 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:50.964 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.964 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:50.964 DEBUG:boto:Method: GET 2024-04-13T21:38:50.964 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:50.964 DEBUG:boto:Data: 2024-04-13T21:38:50.964 DEBUG:boto:Headers: {} 2024-04-13T21:38:50.964 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:50.964 DEBUG:boto:Port: 8000 2024-04-13T21:38:50.964 DEBUG:boto:Params: {} 2024-04-13T21:38:50.964 DEBUG:boto:Token: None 2024-04-13T21:38:50.965 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:50 GMT /vwnfns-15/ 2024-04-13T21:38:50.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU= 2024-04-13T21:38:50.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqs9xzLa6xR50c1GupLj6VQ8FsU=', 'Content-Length': '0'} 2024-04-13T21:38:50.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032fa54bb55b92849-00661afb6a-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.010 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.011 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.011 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.011 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.011 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.011 DEBUG:boto:Data: 2024-04-13T21:38:51.012 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.012 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.012 DEBUG:boto:Params: {} 2024-04-13T21:38:51.012 DEBUG:boto:Token: None 2024-04-13T21:38:51.012 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000190d15a86fddd78d-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.016 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.016 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.016 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.016 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.016 DEBUG:boto:Data: 2024-04-13T21:38:51.016 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.016 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.016 DEBUG:boto:Params: {} 2024-04-13T21:38:51.017 DEBUG:boto:Token: None 2024-04-13T21:38:51.017 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5ee91562b580e4f-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.020 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.020 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.020 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.020 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.021 DEBUG:boto:Method: GET 2024-04-13T21:38:51.021 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.021 DEBUG:boto:Data: 2024-04-13T21:38:51.021 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.021 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.021 DEBUG:boto:Params: {} 2024-04-13T21:38:51.021 DEBUG:boto:Token: None 2024-04-13T21:38:51.021 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000573d78112c3709a4-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.066 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.128 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.128 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.128 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.128 DEBUG:boto:Method: GET 2024-04-13T21:38:51.128 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.128 DEBUG:boto:Data: 2024-04-13T21:38:51.128 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.128 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.128 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.128 DEBUG:boto:Params: {} 2024-04-13T21:38:51.129 DEBUG:boto:Token: None 2024-04-13T21:38:51.129 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c53998af7beef12-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.174 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.175 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.175 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.175 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.175 DEBUG:boto:Method: GET 2024-04-13T21:38:51.175 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.175 DEBUG:boto:Data: 2024-04-13T21:38:51.175 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.175 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.175 DEBUG:boto:Params: {} 2024-04-13T21:38:51.176 DEBUG:boto:Token: None 2024-04-13T21:38:51.176 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001394bfe3c73179a4-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.179 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.179 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.179 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.179 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.180 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.180 DEBUG:boto:Method: GET 2024-04-13T21:38:51.180 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.180 DEBUG:boto:Data: 2024-04-13T21:38:51.180 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.180 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.180 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.180 DEBUG:boto:Params: {} 2024-04-13T21:38:51.180 DEBUG:boto:Token: None 2024-04-13T21:38:51.180 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2f7823d833476c0-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.183 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.184 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.184 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.184 DEBUG:boto:path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.184 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.184 DEBUG:boto:Method: GET 2024-04-13T21:38:51.184 DEBUG:boto:Path: /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.184 DEBUG:boto:Data: 2024-04-13T21:38:51.184 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.184 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.184 DEBUG:boto:Params: {} 2024-04-13T21:38:51.185 DEBUG:boto:Token: None 2024-04-13T21:38:51.185 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU= 2024-04-13T21:38:51.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU=', 'Content-Length': '0'} 2024-04-13T21:38:51.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047d788d50f53dcdb-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.187 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.187 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.187 DEBUG:boto:path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.187 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.188 DEBUG:boto:Method: GET 2024-04-13T21:38:51.188 DEBUG:boto:Path: /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.188 DEBUG:boto:Data: 2024-04-13T21:38:51.188 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.188 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.188 DEBUG:boto:Params: {} 2024-04-13T21:38:51.188 DEBUG:boto:Token: None 2024-04-13T21:38:51.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU= 2024-04-13T21:38:51.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU=', 'Content-Length': '0'} 2024-04-13T21:38:51.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060fccfc0e7653ade-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.234 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.235 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.235 DEBUG:boto:path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.235 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.235 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.235 DEBUG:boto:Path: /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.235 DEBUG:boto:Data: 2024-04-13T21:38:51.235 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.235 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.235 DEBUG:boto:Params: {} 2024-04-13T21:38:51.236 DEBUG:boto:Token: None 2024-04-13T21:38:51.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yxPSPPYX6OTwXURzEk7VecVSiig= 2024-04-13T21:38:51.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yxPSPPYX6OTwXURzEk7VecVSiig=', 'Content-Length': '0'} 2024-04-13T21:38:51.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002894617ca8c79068-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.238 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.238 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.238 DEBUG:boto:path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.238 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.238 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.239 DEBUG:boto:Path: /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.239 DEBUG:boto:Data: 2024-04-13T21:38:51.239 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.239 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.239 DEBUG:boto:Params: {} 2024-04-13T21:38:51.239 DEBUG:boto:Token: None 2024-04-13T21:38:51.239 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yxPSPPYX6OTwXURzEk7VecVSiig= 2024-04-13T21:38:51.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yxPSPPYX6OTwXURzEk7VecVSiig=', 'Content-Length': '0'} 2024-04-13T21:38:51.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec0ea2b1b97fa013-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.241 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.241 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.242 DEBUG:boto:path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.242 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.242 DEBUG:boto:Method: GET 2024-04-13T21:38:51.242 DEBUG:boto:Path: /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.242 DEBUG:boto:Data: 2024-04-13T21:38:51.242 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.242 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.242 DEBUG:boto:Params: {} 2024-04-13T21:38:51.242 DEBUG:boto:Token: None 2024-04-13T21:38:51.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU= 2024-04-13T21:38:51.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU=', 'Content-Length': '0'} 2024-04-13T21:38:51.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093928593a2b77a0c-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.286 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.287 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.287 DEBUG:boto:path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.287 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.287 DEBUG:boto:Method: GET 2024-04-13T21:38:51.287 DEBUG:boto:Path: /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.287 DEBUG:boto:Data: 2024-04-13T21:38:51.287 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.287 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.287 DEBUG:boto:Params: {} 2024-04-13T21:38:51.288 DEBUG:boto:Token: None 2024-04-13T21:38:51.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=m0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY 2024-04-13T21:38:51.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU= 2024-04-13T21:38:51.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IIvdbUEduKxiD2oHsbTgX8uurYU=', 'Content-Length': '0'} 2024-04-13T21:38:51.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'm0Rb-mS7o7tZ6v3aWRH2wJp.u3GksPY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066d9bc34a640f66a-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.290 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.290 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.290 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.291 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.291 DEBUG:boto:Method: GET 2024-04-13T21:38:51.291 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.291 DEBUG:boto:Data: 2024-04-13T21:38:51.291 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.291 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.291 DEBUG:boto:Params: {} 2024-04-13T21:38:51.291 DEBUG:boto:Token: None 2024-04-13T21:38:51.291 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d274002031bf351-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.294 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.295 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.295 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.295 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.295 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.295 DEBUG:boto:Method: GET 2024-04-13T21:38:51.295 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.295 DEBUG:boto:Data: 2024-04-13T21:38:51.295 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.296 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.296 DEBUG:boto:Params: {} 2024-04-13T21:38:51.296 DEBUG:boto:Token: None 2024-04-13T21:38:51.296 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d5ba1207ff1ed4f-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.299 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.299 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.299 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.299 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.299 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.299 DEBUG:boto:Data: 2024-04-13T21:38:51.300 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.300 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.300 DEBUG:boto:Params: {} 2024-04-13T21:38:51.300 DEBUG:boto:Token: None 2024-04-13T21:38:51.300 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb14206360671458-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.304 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.304 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.304 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.304 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.304 DEBUG:boto:Data: 2024-04-13T21:38:51.304 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.304 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.304 DEBUG:boto:Params: {} 2024-04-13T21:38:51.304 DEBUG:boto:Token: None 2024-04-13T21:38:51.305 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009569fab96bf1caec-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.308 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.308 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.308 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.308 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.308 DEBUG:boto:Method: GET 2024-04-13T21:38:51.308 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.309 DEBUG:boto:Data: 2024-04-13T21:38:51.309 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.309 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.309 DEBUG:boto:Params: {} 2024-04-13T21:38:51.309 DEBUG:boto:Token: None 2024-04-13T21:38:51.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.309 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b12210f8a28ad436-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.354 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.355 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.355 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.355 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.355 DEBUG:boto:Method: GET 2024-04-13T21:38:51.355 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.355 DEBUG:boto:Data: 2024-04-13T21:38:51.355 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.355 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.355 DEBUG:boto:Params: {} 2024-04-13T21:38:51.356 DEBUG:boto:Token: None 2024-04-13T21:38:51.356 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de9a521650a0b2d3-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.398 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.399 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.399 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.399 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.399 DEBUG:boto:Method: GET 2024-04-13T21:38:51.399 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.399 DEBUG:boto:Data: 2024-04-13T21:38:51.399 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.399 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.399 DEBUG:boto:Params: {} 2024-04-13T21:38:51.400 DEBUG:boto:Token: None 2024-04-13T21:38:51.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002221e0493393ea3-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.403 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.403 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.403 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.403 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.403 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.403 DEBUG:boto:Method: GET 2024-04-13T21:38:51.404 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.404 DEBUG:boto:Data: 2024-04-13T21:38:51.404 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.404 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.404 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.404 DEBUG:boto:Params: {} 2024-04-13T21:38:51.404 DEBUG:boto:Token: None 2024-04-13T21:38:51.404 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b836bd3342a6d588-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.450 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.451 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.451 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.451 DEBUG:boto:path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.451 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.451 DEBUG:boto:Method: GET 2024-04-13T21:38:51.451 DEBUG:boto:Path: /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.451 DEBUG:boto:Data: 2024-04-13T21:38:51.451 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.452 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.452 DEBUG:boto:Params: {} 2024-04-13T21:38:51.452 DEBUG:boto:Token: None 2024-04-13T21:38:51.452 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78= 2024-04-13T21:38:51.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78=', 'Content-Length': '0'} 2024-04-13T21:38:51.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3b786d69d102728-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.454 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.455 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.455 DEBUG:boto:path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.455 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.455 DEBUG:boto:Method: GET 2024-04-13T21:38:51.455 DEBUG:boto:Path: /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.455 DEBUG:boto:Data: 2024-04-13T21:38:51.455 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.455 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.455 DEBUG:boto:Params: {} 2024-04-13T21:38:51.456 DEBUG:boto:Token: None 2024-04-13T21:38:51.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78= 2024-04-13T21:38:51.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78=', 'Content-Length': '0'} 2024-04-13T21:38:51.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1ee2be36f513006-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.458 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.458 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.458 DEBUG:boto:path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.459 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.459 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.459 DEBUG:boto:Path: /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.459 DEBUG:boto:Data: 2024-04-13T21:38:51.459 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.459 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.459 DEBUG:boto:Params: {} 2024-04-13T21:38:51.459 DEBUG:boto:Token: None 2024-04-13T21:38:51.459 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8yDMTmtD70qIkPGIPtODC8ru50A= 2024-04-13T21:38:51.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8yDMTmtD70qIkPGIPtODC8ru50A=', 'Content-Length': '0'} 2024-04-13T21:38:51.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f63fae1944832e98-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.461 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.461 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.462 DEBUG:boto:path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.462 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.462 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.462 DEBUG:boto:Path: /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.462 DEBUG:boto:Data: 2024-04-13T21:38:51.462 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.462 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.462 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.462 DEBUG:boto:Params: {} 2024-04-13T21:38:51.465 DEBUG:boto:Token: None 2024-04-13T21:38:51.465 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8yDMTmtD70qIkPGIPtODC8ru50A= 2024-04-13T21:38:51.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8yDMTmtD70qIkPGIPtODC8ru50A=', 'Content-Length': '0'} 2024-04-13T21:38:51.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bb9d9871e0995c2-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.468 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.468 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.468 DEBUG:boto:path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.468 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.468 DEBUG:boto:Method: GET 2024-04-13T21:38:51.468 DEBUG:boto:Path: /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.468 DEBUG:boto:Data: 2024-04-13T21:38:51.468 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.468 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.469 DEBUG:boto:Params: {} 2024-04-13T21:38:51.469 DEBUG:boto:Token: None 2024-04-13T21:38:51.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78= 2024-04-13T21:38:51.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78=', 'Content-Length': '0'} 2024-04-13T21:38:51.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b66cab96573a6550-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.471 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.471 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.471 DEBUG:boto:path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.471 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.471 DEBUG:boto:Method: GET 2024-04-13T21:38:51.471 DEBUG:boto:Path: /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.471 DEBUG:boto:Data: 2024-04-13T21:38:51.471 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.472 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.472 DEBUG:boto:Params: {} 2024-04-13T21:38:51.472 DEBUG:boto:Token: None 2024-04-13T21:38:51.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=OoBjKMBXO0H5kpggzNatVsdQnIDNPgP 2024-04-13T21:38:51.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78= 2024-04-13T21:38:51.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UmXOvtlquSu1ANB7IAaJNefUR78=', 'Content-Length': '0'} 2024-04-13T21:38:51.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'OoBjKMBXO0H5kpggzNatVsdQnIDNPgP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d90c8998280d86f-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.514 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.515 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.515 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.515 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.515 DEBUG:boto:Method: GET 2024-04-13T21:38:51.515 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.515 DEBUG:boto:Data: 2024-04-13T21:38:51.515 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.515 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.515 DEBUG:boto:Params: {} 2024-04-13T21:38:51.516 DEBUG:boto:Token: None 2024-04-13T21:38:51.516 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b20319a38b06d78b-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.519 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.519 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.519 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.519 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.519 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.519 DEBUG:boto:Method: GET 2024-04-13T21:38:51.519 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.520 DEBUG:boto:Data: 2024-04-13T21:38:51.520 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.520 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.520 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.520 DEBUG:boto:Params: {} 2024-04-13T21:38:51.520 DEBUG:boto:Token: None 2024-04-13T21:38:51.520 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca46cf04b48a92d-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.570 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.571 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.571 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.571 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.571 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.571 DEBUG:boto:Data: 2024-04-13T21:38:51.571 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.572 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.572 DEBUG:boto:Params: {} 2024-04-13T21:38:51.572 DEBUG:boto:Token: None 2024-04-13T21:38:51.572 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070a519148b907936-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.575 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.576 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.576 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.576 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.576 DEBUG:boto:Data: 2024-04-13T21:38:51.576 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.576 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.576 DEBUG:boto:Params: {} 2024-04-13T21:38:51.576 DEBUG:boto:Token: None 2024-04-13T21:38:51.576 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.577 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000257448e3d4ea25f9-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.580 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.580 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.581 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.581 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.581 DEBUG:boto:Method: GET 2024-04-13T21:38:51.581 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.581 DEBUG:boto:Data: 2024-04-13T21:38:51.581 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.581 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.581 DEBUG:boto:Params: {} 2024-04-13T21:38:51.581 DEBUG:boto:Token: None 2024-04-13T21:38:51.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008215334b736285a6-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.630 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.631 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.631 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.631 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.631 DEBUG:boto:Method: GET 2024-04-13T21:38:51.631 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.631 DEBUG:boto:Data: 2024-04-13T21:38:51.631 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.631 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.631 DEBUG:boto:Params: {} 2024-04-13T21:38:51.632 DEBUG:boto:Token: None 2024-04-13T21:38:51.632 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4e524a66c0146e7-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.678 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.679 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.679 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.679 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.679 DEBUG:boto:Method: GET 2024-04-13T21:38:51.679 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.679 DEBUG:boto:Data: 2024-04-13T21:38:51.679 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.679 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.679 DEBUG:boto:Params: {} 2024-04-13T21:38:51.680 DEBUG:boto:Token: None 2024-04-13T21:38:51.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa1a48669ea81ccc-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.683 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.683 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.683 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.683 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.683 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.683 DEBUG:boto:Method: GET 2024-04-13T21:38:51.684 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.684 DEBUG:boto:Data: 2024-04-13T21:38:51.684 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.684 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.684 DEBUG:boto:Params: {} 2024-04-13T21:38:51.684 DEBUG:boto:Token: None 2024-04-13T21:38:51.684 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000747af9b71956939e-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.687 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.687 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.687 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.687 DEBUG:boto:path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.687 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.688 DEBUG:boto:Method: GET 2024-04-13T21:38:51.688 DEBUG:boto:Path: /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.688 DEBUG:boto:Data: 2024-04-13T21:38:51.688 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.688 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.688 DEBUG:boto:Params: {} 2024-04-13T21:38:51.688 DEBUG:boto:Token: None 2024-04-13T21:38:51.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg= 2024-04-13T21:38:51.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg=', 'Content-Length': '0'} 2024-04-13T21:38:51.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f7b8bd5d2465665-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.691 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.691 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.691 DEBUG:boto:path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.691 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.691 DEBUG:boto:Method: GET 2024-04-13T21:38:51.692 DEBUG:boto:Path: /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.692 DEBUG:boto:Data: 2024-04-13T21:38:51.692 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.692 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.692 DEBUG:boto:Params: {} 2024-04-13T21:38:51.692 DEBUG:boto:Token: None 2024-04-13T21:38:51.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg= 2024-04-13T21:38:51.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg=', 'Content-Length': '0'} 2024-04-13T21:38:51.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034334c5e56e927a8-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.734 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.735 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.735 DEBUG:boto:path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.735 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.735 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.735 DEBUG:boto:Path: /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.735 DEBUG:boto:Data: 2024-04-13T21:38:51.735 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.735 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.736 DEBUG:boto:Params: {} 2024-04-13T21:38:51.736 DEBUG:boto:Token: None 2024-04-13T21:38:51.736 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jGK7DzVaKfSdfV2IRIY9ntokR7w= 2024-04-13T21:38:51.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jGK7DzVaKfSdfV2IRIY9ntokR7w=', 'Content-Length': '0'} 2024-04-13T21:38:51.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006093a368f74d6102-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.738 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.738 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.738 DEBUG:boto:path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.738 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.739 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.739 DEBUG:boto:Path: /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.739 DEBUG:boto:Data: 2024-04-13T21:38:51.739 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.739 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.739 DEBUG:boto:Params: {} 2024-04-13T21:38:51.739 DEBUG:boto:Token: None 2024-04-13T21:38:51.739 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jGK7DzVaKfSdfV2IRIY9ntokR7w= 2024-04-13T21:38:51.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jGK7DzVaKfSdfV2IRIY9ntokR7w=', 'Content-Length': '0'} 2024-04-13T21:38:51.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b69133ed938e539b-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.742 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.742 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.742 DEBUG:boto:path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.742 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.742 DEBUG:boto:Method: GET 2024-04-13T21:38:51.742 DEBUG:boto:Path: /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.743 DEBUG:boto:Data: 2024-04-13T21:38:51.743 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.743 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.743 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.743 DEBUG:boto:Params: {} 2024-04-13T21:38:51.743 DEBUG:boto:Token: None 2024-04-13T21:38:51.743 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg= 2024-04-13T21:38:51.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg=', 'Content-Length': '0'} 2024-04-13T21:38:51.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c85a7a7136d295a9-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.786 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.787 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.787 DEBUG:boto:path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.787 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.787 DEBUG:boto:Method: GET 2024-04-13T21:38:51.787 DEBUG:boto:Path: /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.787 DEBUG:boto:Data: 2024-04-13T21:38:51.787 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.787 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.787 DEBUG:boto:Params: {} 2024-04-13T21:38:51.788 DEBUG:boto:Token: None 2024-04-13T21:38:51.788 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4 2024-04-13T21:38:51.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg= 2024-04-13T21:38:51.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4aOTfnDTDT1TxpDefQ5TyJHwvtg=', 'Content-Length': '0'} 2024-04-13T21:38:51.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'mMCpvrLcgVsrieRl9bzAzr4AkJ3Gni4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b0452aa658f7eec-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.791 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.791 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.791 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.791 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.791 DEBUG:boto:Method: GET 2024-04-13T21:38:51.791 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.791 DEBUG:boto:Data: 2024-04-13T21:38:51.791 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.791 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.792 DEBUG:boto:Params: {} 2024-04-13T21:38:51.792 DEBUG:boto:Token: None 2024-04-13T21:38:51.792 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e387ebf45321982-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.794 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.795 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.795 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.795 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.795 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.795 DEBUG:boto:Method: GET 2024-04-13T21:38:51.795 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.795 DEBUG:boto:Data: 2024-04-13T21:38:51.795 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.796 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.796 DEBUG:boto:Params: {} 2024-04-13T21:38:51.796 DEBUG:boto:Token: None 2024-04-13T21:38:51.796 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d6c0ada0acd0a6-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.799 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.799 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.799 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.799 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.799 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.799 DEBUG:boto:Data: 2024-04-13T21:38:51.800 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.800 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.800 DEBUG:boto:Params: {} 2024-04-13T21:38:51.800 DEBUG:boto:Token: None 2024-04-13T21:38:51.800 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3de02b5599312c6-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.803 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.803 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.803 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.803 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:51.804 DEBUG:boto:Data: 2024-04-13T21:38:51.804 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.804 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.804 DEBUG:boto:Params: {} 2024-04-13T21:38:51.804 DEBUG:boto:Token: None 2024-04-13T21:38:51.804 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj 2024-04-13T21:38:51.805 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24= 2024-04-13T21:38:51.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KHwpTwVTsemk1iM0wR5u4dx5k24=', 'Content-Length': '0'} 2024-04-13T21:38:51.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c550caace114363e-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.808 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.808 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.808 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.808 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.808 DEBUG:boto:Method: GET 2024-04-13T21:38:51.808 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.808 DEBUG:boto:Data: 2024-04-13T21:38:51.808 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.809 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.809 DEBUG:boto:Params: {} 2024-04-13T21:38:51.809 DEBUG:boto:Token: None 2024-04-13T21:38:51.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e8779f9a694913b-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.854 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.854 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.855 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.855 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.855 DEBUG:boto:Method: GET 2024-04-13T21:38:51.855 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.855 DEBUG:boto:Data: 2024-04-13T21:38:51.855 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.855 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.855 DEBUG:boto:Params: {} 2024-04-13T21:38:51.856 DEBUG:boto:Token: None 2024-04-13T21:38:51.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:51.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA= 2024-04-13T21:38:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e5suqvRcI+GwJ2lYumnmxW12ibA=', 'Content-Length': '0'} 2024-04-13T21:38:51.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4419d950e9d81be-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.898 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.899 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.899 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.899 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.899 DEBUG:boto:Method: GET 2024-04-13T21:38:51.899 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.899 DEBUG:boto:Data: 2024-04-13T21:38:51.899 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.899 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.899 DEBUG:boto:Params: {} 2024-04-13T21:38:51.900 DEBUG:boto:Token: None 2024-04-13T21:38:51.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae0a05976fce60f-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.903 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.903 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.903 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.903 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.903 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.903 DEBUG:boto:Method: GET 2024-04-13T21:38:51.904 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.904 DEBUG:boto:Data: 2024-04-13T21:38:51.904 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.904 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.904 DEBUG:boto:Params: {} 2024-04-13T21:38:51.904 DEBUG:boto:Token: None 2024-04-13T21:38:51.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000238eea04691058ec-00661afb6b-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.907 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:51.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.907 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.907 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.908 DEBUG:boto:path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.908 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.908 DEBUG:boto:Method: GET 2024-04-13T21:38:51.908 DEBUG:boto:Path: /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.908 DEBUG:boto:Data: 2024-04-13T21:38:51.908 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.908 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.908 DEBUG:boto:Params: {} 2024-04-13T21:38:51.909 DEBUG:boto:Token: None 2024-04-13T21:38:51.909 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms= 2024-04-13T21:38:51.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms=', 'Content-Length': '0'} 2024-04-13T21:38:51.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e0a76e169dd1854-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.911 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.912 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.912 DEBUG:boto:path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.912 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.912 DEBUG:boto:Method: GET 2024-04-13T21:38:51.912 DEBUG:boto:Path: /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.912 DEBUG:boto:Data: 2024-04-13T21:38:51.912 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.912 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.912 DEBUG:boto:Params: {} 2024-04-13T21:38:51.913 DEBUG:boto:Token: None 2024-04-13T21:38:51.913 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms= 2024-04-13T21:38:51.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms=', 'Content-Length': '0'} 2024-04-13T21:38:51.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001003b3b30606dcd1-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.915 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.915 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.915 DEBUG:boto:path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.915 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.915 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.915 DEBUG:boto:Path: /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.915 DEBUG:boto:Data: 2024-04-13T21:38:51.916 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.916 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.916 DEBUG:boto:Params: {} 2024-04-13T21:38:51.916 DEBUG:boto:Token: None 2024-04-13T21:38:51.916 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKVK6tuBC1pUxjRN8b3oHnE1K0E= 2024-04-13T21:38:51.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKVK6tuBC1pUxjRN8b3oHnE1K0E=', 'Content-Length': '0'} 2024-04-13T21:38:51.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c68580c7b392e15-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.918 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.918 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.918 DEBUG:boto:path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.918 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.918 DEBUG:boto:Method: HEAD 2024-04-13T21:38:51.918 DEBUG:boto:Path: /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.919 DEBUG:boto:Data: 2024-04-13T21:38:51.919 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.919 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.919 DEBUG:boto:Params: {} 2024-04-13T21:38:51.919 DEBUG:boto:Token: None 2024-04-13T21:38:51.919 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKVK6tuBC1pUxjRN8b3oHnE1K0E= 2024-04-13T21:38:51.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKVK6tuBC1pUxjRN8b3oHnE1K0E=', 'Content-Length': '0'} 2024-04-13T21:38:51.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9a696640920043c-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:51.921 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.921 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.921 DEBUG:boto:path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.921 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.922 DEBUG:boto:Method: GET 2024-04-13T21:38:51.922 DEBUG:boto:Path: /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.922 DEBUG:boto:Data: 2024-04-13T21:38:51.922 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.923 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.923 DEBUG:boto:Params: {} 2024-04-13T21:38:51.923 DEBUG:boto:Token: None 2024-04-13T21:38:51.923 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms= 2024-04-13T21:38:51.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms=', 'Content-Length': '0'} 2024-04-13T21:38:51.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005afea5509c35a059-00661afb6b-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.966 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:51.967 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:51.967 DEBUG:boto:path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.967 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.967 DEBUG:boto:Method: GET 2024-04-13T21:38:51.967 DEBUG:boto:Path: /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.967 DEBUG:boto:Data: 2024-04-13T21:38:51.967 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:51.967 DEBUG:boto:Port: 8000 2024-04-13T21:38:51.967 DEBUG:boto:Params: {} 2024-04-13T21:38:51.968 DEBUG:boto:Token: None 2024-04-13T21:38:51.968 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/obj?versionId=VTT69MXmX6XS..1R1xDYYnR3KtMzY4k 2024-04-13T21:38:51.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms= 2024-04-13T21:38:51.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZBWNer6C3zw/ySlBCyCUCWfKAms=', 'Content-Length': '0'} 2024-04-13T21:38:51.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'VTT69MXmX6XS..1R1xDYYnR3KtMzY4k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054e3e17b26427c39-00661afb6b-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:51.970 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:51.971 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:51.971 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.971 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:51.971 DEBUG:boto:Method: GET 2024-04-13T21:38:51.972 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:51.972 DEBUG:boto:Data: 2024-04-13T21:38:51.972 DEBUG:boto:Headers: {} 2024-04-13T21:38:51.972 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:51.972 DEBUG:boto:Port: 8001 2024-04-13T21:38:51.972 DEBUG:boto:Params: {} 2024-04-13T21:38:51.972 DEBUG:boto:Token: None 2024-04-13T21:38:51.972 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:51 GMT /vwnfns-15/ 2024-04-13T21:38:51.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14= 2024-04-13T21:38:51.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1/zaqPQvO2QlA6s2ZL+Gq6cwD14=', 'Content-Length': '0'} 2024-04-13T21:38:51.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08ff8c235ebf05f-00661afb6b-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.018 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.019 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.019 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.019 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.019 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.019 DEBUG:boto:Method: GET 2024-04-13T21:38:52.019 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.019 DEBUG:boto:Data: 2024-04-13T21:38:52.020 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.020 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.020 DEBUG:boto:Params: {} 2024-04-13T21:38:52.020 DEBUG:boto:Token: None 2024-04-13T21:38:52.020 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5b32461316f997-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.023 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.023 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.023 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.023 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.024 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.024 DEBUG:boto:Data: 2024-04-13T21:38:52.024 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.024 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.024 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.024 DEBUG:boto:Params: {} 2024-04-13T21:38:52.024 DEBUG:boto:Token: None 2024-04-13T21:38:52.024 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048ba001c33d3ad49-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.027 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.027 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.028 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.028 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.028 DEBUG:boto:Data: 2024-04-13T21:38:52.028 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.028 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.028 DEBUG:boto:Params: {} 2024-04-13T21:38:52.028 DEBUG:boto:Token: None 2024-04-13T21:38:52.028 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5a097140dbd9edc-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.032 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.032 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.032 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.032 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.032 DEBUG:boto:Method: GET 2024-04-13T21:38:52.032 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.033 DEBUG:boto:Data: 2024-04-13T21:38:52.033 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.033 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.033 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.033 DEBUG:boto:Params: {} 2024-04-13T21:38:52.033 DEBUG:boto:Token: None 2024-04-13T21:38:52.033 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.033 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be9f2c178d95d7a8-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.035 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.035 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.035 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.036 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.036 DEBUG:boto:Method: GET 2024-04-13T21:38:52.036 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.036 DEBUG:boto:Data: 2024-04-13T21:38:52.036 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.036 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.036 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.036 DEBUG:boto:Params: {} 2024-04-13T21:38:52.036 DEBUG:boto:Token: None 2024-04-13T21:38:52.036 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013df73c896d7677d-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.082 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.083 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.083 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.083 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.083 DEBUG:boto:Method: GET 2024-04-13T21:38:52.083 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.083 DEBUG:boto:Data: 2024-04-13T21:38:52.083 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.083 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.083 DEBUG:boto:Params: {} 2024-04-13T21:38:52.084 DEBUG:boto:Token: None 2024-04-13T21:38:52.084 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff68707245ca393-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.087 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.087 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.087 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.087 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.088 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.088 DEBUG:boto:Method: GET 2024-04-13T21:38:52.088 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.088 DEBUG:boto:Data: 2024-04-13T21:38:52.088 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.088 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.088 DEBUG:boto:Params: {} 2024-04-13T21:38:52.088 DEBUG:boto:Token: None 2024-04-13T21:38:52.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a03c2db51fb5c3-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.134 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.135 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.135 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.135 DEBUG:boto:path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.135 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.135 DEBUG:boto:Method: GET 2024-04-13T21:38:52.135 DEBUG:boto:Path: /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.136 DEBUG:boto:Data: 2024-04-13T21:38:52.136 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.136 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.136 DEBUG:boto:Params: {} 2024-04-13T21:38:52.137 DEBUG:boto:Token: None 2024-04-13T21:38:52.137 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU= 2024-04-13T21:38:52.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU=', 'Content-Length': '0'} 2024-04-13T21:38:52.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000752750e4fbc4519b-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.182 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.183 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.183 DEBUG:boto:path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.183 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.183 DEBUG:boto:Method: GET 2024-04-13T21:38:52.183 DEBUG:boto:Path: /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.183 DEBUG:boto:Data: 2024-04-13T21:38:52.183 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.183 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.183 DEBUG:boto:Params: {} 2024-04-13T21:38:52.184 DEBUG:boto:Token: None 2024-04-13T21:38:52.184 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU= 2024-04-13T21:38:52.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU=', 'Content-Length': '0'} 2024-04-13T21:38:52.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e544fd9e13bce6fa-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.186 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.187 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.187 DEBUG:boto:path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.187 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.187 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.187 DEBUG:boto:Path: /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.187 DEBUG:boto:Data: 2024-04-13T21:38:52.187 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.187 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.187 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.187 DEBUG:boto:Params: {} 2024-04-13T21:38:52.187 DEBUG:boto:Token: None 2024-04-13T21:38:52.188 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dxQ6/aoutBZXJFSNLtN80zEcXJI= 2024-04-13T21:38:52.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dxQ6/aoutBZXJFSNLtN80zEcXJI=', 'Content-Length': '0'} 2024-04-13T21:38:52.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036e8b6b574c46c95-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.190 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.190 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.190 DEBUG:boto:path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.190 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.190 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.190 DEBUG:boto:Path: /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.190 DEBUG:boto:Data: 2024-04-13T21:38:52.190 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.190 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.190 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.191 DEBUG:boto:Params: {} 2024-04-13T21:38:52.191 DEBUG:boto:Token: None 2024-04-13T21:38:52.191 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.191 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dxQ6/aoutBZXJFSNLtN80zEcXJI= 2024-04-13T21:38:52.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dxQ6/aoutBZXJFSNLtN80zEcXJI=', 'Content-Length': '0'} 2024-04-13T21:38:52.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002685e3f2265964c3-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.193 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.193 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.193 DEBUG:boto:path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.193 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.194 DEBUG:boto:Method: GET 2024-04-13T21:38:52.194 DEBUG:boto:Path: /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.195 DEBUG:boto:Data: 2024-04-13T21:38:52.195 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.195 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.195 DEBUG:boto:Params: {} 2024-04-13T21:38:52.195 DEBUG:boto:Token: None 2024-04-13T21:38:52.196 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU= 2024-04-13T21:38:52.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU=', 'Content-Length': '0'} 2024-04-13T21:38:52.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a78155c59a499498-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.198 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.198 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.198 DEBUG:boto:path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.198 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.198 DEBUG:boto:Method: GET 2024-04-13T21:38:52.198 DEBUG:boto:Path: /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.198 DEBUG:boto:Data: 2024-04-13T21:38:52.198 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.199 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.199 DEBUG:boto:Params: {} 2024-04-13T21:38:52.199 DEBUG:boto:Token: None 2024-04-13T21:38:52.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=muvbWmAiDOX3JDiJbLDSnvGOk5L8e92 2024-04-13T21:38:52.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU= 2024-04-13T21:38:52.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gqu7klZyeClHe3+lsgCO3EkROFU=', 'Content-Length': '0'} 2024-04-13T21:38:52.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'muvbWmAiDOX3JDiJbLDSnvGOk5L8e92'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007214dd0c2726c6ba-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.242 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.243 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.243 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.243 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.243 DEBUG:boto:Method: GET 2024-04-13T21:38:52.243 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.243 DEBUG:boto:Data: 2024-04-13T21:38:52.243 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.243 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.243 DEBUG:boto:Params: {} 2024-04-13T21:38:52.244 DEBUG:boto:Token: None 2024-04-13T21:38:52.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3d65e8c8e66314-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.247 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.247 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.247 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.247 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.247 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.247 DEBUG:boto:Method: GET 2024-04-13T21:38:52.247 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.248 DEBUG:boto:Data: 2024-04-13T21:38:52.248 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.248 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.248 DEBUG:boto:Params: {} 2024-04-13T21:38:52.248 DEBUG:boto:Token: None 2024-04-13T21:38:52.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a76df07ba9877b-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.298 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.299 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.299 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.299 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.299 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.299 DEBUG:boto:Data: 2024-04-13T21:38:52.299 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.300 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.300 DEBUG:boto:Params: {} 2024-04-13T21:38:52.300 DEBUG:boto:Token: None 2024-04-13T21:38:52.300 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab68f5bac218346b-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.304 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.304 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.304 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.304 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.304 DEBUG:boto:Data: 2024-04-13T21:38:52.304 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.304 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.304 DEBUG:boto:Params: {} 2024-04-13T21:38:52.304 DEBUG:boto:Token: None 2024-04-13T21:38:52.305 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e29c6faa0e5fffb8-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.308 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.308 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.308 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.309 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.309 DEBUG:boto:Method: GET 2024-04-13T21:38:52.309 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.309 DEBUG:boto:Data: 2024-04-13T21:38:52.309 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.309 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.309 DEBUG:boto:Params: {} 2024-04-13T21:38:52.309 DEBUG:boto:Token: None 2024-04-13T21:38:52.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.309 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cfca9ee2c009703-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.354 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.355 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.355 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.355 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.355 DEBUG:boto:Method: GET 2024-04-13T21:38:52.355 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.355 DEBUG:boto:Data: 2024-04-13T21:38:52.355 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.355 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.355 DEBUG:boto:Params: {} 2024-04-13T21:38:52.356 DEBUG:boto:Token: None 2024-04-13T21:38:52.356 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000574e1670b20fafd0-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.398 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.399 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.399 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.399 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.399 DEBUG:boto:Method: GET 2024-04-13T21:38:52.399 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.399 DEBUG:boto:Data: 2024-04-13T21:38:52.399 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.399 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.399 DEBUG:boto:Params: {} 2024-04-13T21:38:52.400 DEBUG:boto:Token: None 2024-04-13T21:38:52.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070b7e683b01fa85f-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.402 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.403 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.403 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.403 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.403 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.403 DEBUG:boto:Method: GET 2024-04-13T21:38:52.403 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.404 DEBUG:boto:Data: 2024-04-13T21:38:52.404 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.404 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.404 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.404 DEBUG:boto:Params: {} 2024-04-13T21:38:52.404 DEBUG:boto:Token: None 2024-04-13T21:38:52.404 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010af5aef0e38e8d9-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.407 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.408 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.408 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.408 DEBUG:boto:path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.408 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.408 DEBUG:boto:Method: GET 2024-04-13T21:38:52.408 DEBUG:boto:Path: /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.408 DEBUG:boto:Data: 2024-04-13T21:38:52.409 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.409 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.409 DEBUG:boto:Params: {} 2024-04-13T21:38:52.409 DEBUG:boto:Token: None 2024-04-13T21:38:52.410 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0= 2024-04-13T21:38:52.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0=', 'Content-Length': '0'} 2024-04-13T21:38:52.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e753305e6fe8b47-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.412 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.413 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.413 DEBUG:boto:path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.413 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.413 DEBUG:boto:Method: GET 2024-04-13T21:38:52.413 DEBUG:boto:Path: /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.413 DEBUG:boto:Data: 2024-04-13T21:38:52.413 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.414 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.414 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.414 DEBUG:boto:Params: {} 2024-04-13T21:38:52.414 DEBUG:boto:Token: None 2024-04-13T21:38:52.414 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.414 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0= 2024-04-13T21:38:52.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0=', 'Content-Length': '0'} 2024-04-13T21:38:52.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a212c3a7738db6b4-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.458 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.459 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.459 DEBUG:boto:path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.459 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.459 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.459 DEBUG:boto:Path: /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.460 DEBUG:boto:Data: 2024-04-13T21:38:52.460 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.460 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.460 DEBUG:boto:Params: {} 2024-04-13T21:38:52.460 DEBUG:boto:Token: None 2024-04-13T21:38:52.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CYhD4m/OHg0WRtdy4mvgC6akI5Q= 2024-04-13T21:38:52.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CYhD4m/OHg0WRtdy4mvgC6akI5Q=', 'Content-Length': '0'} 2024-04-13T21:38:52.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0bf6bc08b7b69a1-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.463 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.463 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.463 DEBUG:boto:path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.463 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.463 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.464 DEBUG:boto:Path: /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.464 DEBUG:boto:Data: 2024-04-13T21:38:52.464 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.464 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.464 DEBUG:boto:Params: {} 2024-04-13T21:38:52.464 DEBUG:boto:Token: None 2024-04-13T21:38:52.464 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CYhD4m/OHg0WRtdy4mvgC6akI5Q= 2024-04-13T21:38:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CYhD4m/OHg0WRtdy4mvgC6akI5Q=', 'Content-Length': '0'} 2024-04-13T21:38:52.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000233b475004a1348f-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.467 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.467 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.467 DEBUG:boto:path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.467 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.467 DEBUG:boto:Method: GET 2024-04-13T21:38:52.468 DEBUG:boto:Path: /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.468 DEBUG:boto:Data: 2024-04-13T21:38:52.468 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.468 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.469 DEBUG:boto:Params: {} 2024-04-13T21:38:52.469 DEBUG:boto:Token: None 2024-04-13T21:38:52.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0= 2024-04-13T21:38:52.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0=', 'Content-Length': '0'} 2024-04-13T21:38:52.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054ff217a9fe2bab0-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.514 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.515 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.515 DEBUG:boto:path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.515 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.515 DEBUG:boto:Method: GET 2024-04-13T21:38:52.515 DEBUG:boto:Path: /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.515 DEBUG:boto:Data: 2024-04-13T21:38:52.515 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.515 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.515 DEBUG:boto:Params: {} 2024-04-13T21:38:52.516 DEBUG:boto:Token: None 2024-04-13T21:38:52.516 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4 2024-04-13T21:38:52.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0= 2024-04-13T21:38:52.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHsZyN9CIL+P3n/qtW24VVMxBk0=', 'Content-Length': '0'} 2024-04-13T21:38:52.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'kEONbY1fLwoEYq3ySy0dDwkmBgh6IP4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be57eef904b74f68-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.519 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.519 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.519 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.519 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.519 DEBUG:boto:Method: GET 2024-04-13T21:38:52.519 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.521 DEBUG:boto:Data: 2024-04-13T21:38:52.521 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.521 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.521 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.521 DEBUG:boto:Params: {} 2024-04-13T21:38:52.521 DEBUG:boto:Token: None 2024-04-13T21:38:52.522 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009715e80162103b73-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.570 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.571 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.571 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.571 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.571 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.571 DEBUG:boto:Method: GET 2024-04-13T21:38:52.571 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.571 DEBUG:boto:Data: 2024-04-13T21:38:52.572 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.572 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.572 DEBUG:boto:Params: {} 2024-04-13T21:38:52.572 DEBUG:boto:Token: None 2024-04-13T21:38:52.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340ebe2d9794fa58-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.575 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.575 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.575 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.575 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.576 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.576 DEBUG:boto:Data: 2024-04-13T21:38:52.576 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.576 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.576 DEBUG:boto:Params: {} 2024-04-13T21:38:52.576 DEBUG:boto:Token: None 2024-04-13T21:38:52.576 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f011477a3c95dfa5-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.579 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.579 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.580 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.580 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.580 DEBUG:boto:Data: 2024-04-13T21:38:52.580 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.580 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.580 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.580 DEBUG:boto:Params: {} 2024-04-13T21:38:52.580 DEBUG:boto:Token: None 2024-04-13T21:38:52.581 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.581 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8da78cb7afa7c3f-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.584 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.584 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.584 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.584 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.584 DEBUG:boto:Method: GET 2024-04-13T21:38:52.585 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.585 DEBUG:boto:Data: 2024-04-13T21:38:52.585 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.585 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.585 DEBUG:boto:Params: {} 2024-04-13T21:38:52.585 DEBUG:boto:Token: None 2024-04-13T21:38:52.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.586 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087bf43b39cf0c32b-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.588 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.588 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.588 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.588 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.588 DEBUG:boto:Method: GET 2024-04-13T21:38:52.588 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.588 DEBUG:boto:Data: 2024-04-13T21:38:52.588 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.588 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.588 DEBUG:boto:Params: {} 2024-04-13T21:38:52.589 DEBUG:boto:Token: None 2024-04-13T21:38:52.589 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c11666ffc731dae6-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.638 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.639 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.639 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.639 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.639 DEBUG:boto:Method: GET 2024-04-13T21:38:52.639 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.639 DEBUG:boto:Data: 2024-04-13T21:38:52.639 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.639 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.639 DEBUG:boto:Params: {} 2024-04-13T21:38:52.640 DEBUG:boto:Token: None 2024-04-13T21:38:52.640 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a1265201316290-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.643 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.643 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.643 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.643 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.643 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.644 DEBUG:boto:Method: GET 2024-04-13T21:38:52.644 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.644 DEBUG:boto:Data: 2024-04-13T21:38:52.644 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.644 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.644 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.644 DEBUG:boto:Params: {} 2024-04-13T21:38:52.644 DEBUG:boto:Token: None 2024-04-13T21:38:52.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e51f11c32115a885-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.647 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.648 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.648 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.648 DEBUG:boto:path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.648 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.648 DEBUG:boto:Method: GET 2024-04-13T21:38:52.648 DEBUG:boto:Path: /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.648 DEBUG:boto:Data: 2024-04-13T21:38:52.648 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.648 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.649 DEBUG:boto:Params: {} 2024-04-13T21:38:52.649 DEBUG:boto:Token: None 2024-04-13T21:38:52.649 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.649 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c= 2024-04-13T21:38:52.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c=', 'Content-Length': '0'} 2024-04-13T21:38:52.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d9d3006ff04dda4-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.694 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.695 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.695 DEBUG:boto:path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.695 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.695 DEBUG:boto:Method: GET 2024-04-13T21:38:52.695 DEBUG:boto:Path: /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.695 DEBUG:boto:Data: 2024-04-13T21:38:52.695 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.695 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.695 DEBUG:boto:Params: {} 2024-04-13T21:38:52.696 DEBUG:boto:Token: None 2024-04-13T21:38:52.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c= 2024-04-13T21:38:52.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c=', 'Content-Length': '0'} 2024-04-13T21:38:52.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4f70005546beaf7-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.698 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.699 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.699 DEBUG:boto:path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.699 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.699 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.699 DEBUG:boto:Path: /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.699 DEBUG:boto:Data: 2024-04-13T21:38:52.699 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.699 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.699 DEBUG:boto:Params: {} 2024-04-13T21:38:52.700 DEBUG:boto:Token: None 2024-04-13T21:38:52.700 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tSdWTfAfX+ptqi1IvLGoyfGMunA= 2024-04-13T21:38:52.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tSdWTfAfX+ptqi1IvLGoyfGMunA=', 'Content-Length': '0'} 2024-04-13T21:38:52.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014359925c4dd0d5d-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.702 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.702 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.702 DEBUG:boto:path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.702 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.702 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.702 DEBUG:boto:Path: /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.702 DEBUG:boto:Data: 2024-04-13T21:38:52.702 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.703 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.703 DEBUG:boto:Params: {} 2024-04-13T21:38:52.703 DEBUG:boto:Token: None 2024-04-13T21:38:52.703 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.703 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tSdWTfAfX+ptqi1IvLGoyfGMunA= 2024-04-13T21:38:52.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tSdWTfAfX+ptqi1IvLGoyfGMunA=', 'Content-Length': '0'} 2024-04-13T21:38:52.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae2c31e3f650d7c0-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.705 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.705 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.705 DEBUG:boto:path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.705 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.705 DEBUG:boto:Method: GET 2024-04-13T21:38:52.705 DEBUG:boto:Path: /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.706 DEBUG:boto:Data: 2024-04-13T21:38:52.706 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.706 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.706 DEBUG:boto:Params: {} 2024-04-13T21:38:52.706 DEBUG:boto:Token: None 2024-04-13T21:38:52.706 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c= 2024-04-13T21:38:52.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c=', 'Content-Length': '0'} 2024-04-13T21:38:52.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bce624245f07ef00-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.709 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.709 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.709 DEBUG:boto:path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.709 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.709 DEBUG:boto:Method: GET 2024-04-13T21:38:52.709 DEBUG:boto:Path: /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.709 DEBUG:boto:Data: 2024-04-13T21:38:52.709 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.709 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.709 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.710 DEBUG:boto:Params: {} 2024-04-13T21:38:52.710 DEBUG:boto:Token: None 2024-04-13T21:38:52.710 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj 2024-04-13T21:38:52.710 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c= 2024-04-13T21:38:52.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ip9Z+Eqm3jOnTOVsZGSYOcN2P0c=', 'Content-Length': '0'} 2024-04-13T21:38:52.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'zq4c7U.CkxCgWVAAmHFw0dP9nKUV-vj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feb3ec1831d69ffa-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.754 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.755 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.755 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.755 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.755 DEBUG:boto:Method: GET 2024-04-13T21:38:52.755 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.755 DEBUG:boto:Data: 2024-04-13T21:38:52.755 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.755 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.755 DEBUG:boto:Params: {} 2024-04-13T21:38:52.756 DEBUG:boto:Token: None 2024-04-13T21:38:52.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce4b320a1c66c4c-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.759 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.759 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.759 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.759 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.760 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.760 DEBUG:boto:Method: GET 2024-04-13T21:38:52.760 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.760 DEBUG:boto:Data: 2024-04-13T21:38:52.760 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.760 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.760 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.760 DEBUG:boto:Params: {} 2024-04-13T21:38:52.760 DEBUG:boto:Token: None 2024-04-13T21:38:52.760 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d19d05def16836f5-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.763 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.763 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.763 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.764 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.764 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.764 DEBUG:boto:Data: 2024-04-13T21:38:52.764 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.764 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.764 DEBUG:boto:Params: {} 2024-04-13T21:38:52.764 DEBUG:boto:Token: None 2024-04-13T21:38:52.764 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004195ec112d2439c8-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.768 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.768 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.768 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.769 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:52.769 DEBUG:boto:Data: 2024-04-13T21:38:52.769 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.769 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.769 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.769 DEBUG:boto:Params: {} 2024-04-13T21:38:52.769 DEBUG:boto:Token: None 2024-04-13T21:38:52.769 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj 2024-04-13T21:38:52.769 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg= 2024-04-13T21:38:52.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScbipxDVENijgSxUwODoBlm9arg=', 'Content-Length': '0'} 2024-04-13T21:38:52.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e97fe6b07208c22a-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.773 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.773 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.773 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.773 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.773 DEBUG:boto:Method: GET 2024-04-13T21:38:52.773 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.773 DEBUG:boto:Data: 2024-04-13T21:38:52.773 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.773 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.773 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.773 DEBUG:boto:Params: {} 2024-04-13T21:38:52.774 DEBUG:boto:Token: None 2024-04-13T21:38:52.774 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.774 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000754fa77f2bbe69ee-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.818 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.819 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.819 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.819 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.819 DEBUG:boto:Method: GET 2024-04-13T21:38:52.819 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.819 DEBUG:boto:Data: 2024-04-13T21:38:52.819 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.819 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.819 DEBUG:boto:Params: {} 2024-04-13T21:38:52.820 DEBUG:boto:Token: None 2024-04-13T21:38:52.820 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:52.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE= 2024-04-13T21:38:52.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uOgjOo2YVui3zpAzKiAG3LpPHRE=', 'Content-Length': '0'} 2024-04-13T21:38:52.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac502e7156f43f3f-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.866 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.867 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.867 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.867 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.867 DEBUG:boto:Method: GET 2024-04-13T21:38:52.867 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.867 DEBUG:boto:Data: 2024-04-13T21:38:52.867 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.867 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.867 DEBUG:boto:Params: {} 2024-04-13T21:38:52.868 DEBUG:boto:Token: None 2024-04-13T21:38:52.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea8dbadc53cff2d-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.870 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.871 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.871 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.871 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.871 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.871 DEBUG:boto:Method: GET 2024-04-13T21:38:52.871 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.871 DEBUG:boto:Data: 2024-04-13T21:38:52.872 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.872 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.872 DEBUG:boto:Params: {} 2024-04-13T21:38:52.872 DEBUG:boto:Token: None 2024-04-13T21:38:52.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.873 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1d6ca43f19a5953-00661afb6c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.918 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:52.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.919 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.919 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.919 DEBUG:boto:path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.919 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.919 DEBUG:boto:Method: GET 2024-04-13T21:38:52.919 DEBUG:boto:Path: /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.919 DEBUG:boto:Data: 2024-04-13T21:38:52.920 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.920 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.920 DEBUG:boto:Params: {} 2024-04-13T21:38:52.920 DEBUG:boto:Token: None 2024-04-13T21:38:52.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE= 2024-04-13T21:38:52.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE=', 'Content-Length': '0'} 2024-04-13T21:38:52.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4ba51d7b5c04912-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.923 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.923 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.923 DEBUG:boto:path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.923 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.923 DEBUG:boto:Method: GET 2024-04-13T21:38:52.923 DEBUG:boto:Path: /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.923 DEBUG:boto:Data: 2024-04-13T21:38:52.923 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.923 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.923 DEBUG:boto:Params: {} 2024-04-13T21:38:52.924 DEBUG:boto:Token: None 2024-04-13T21:38:52.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE= 2024-04-13T21:38:52.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE=', 'Content-Length': '0'} 2024-04-13T21:38:52.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a087db95ecba711-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.926 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.926 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.926 DEBUG:boto:path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.926 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.926 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.926 DEBUG:boto:Path: /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.926 DEBUG:boto:Data: 2024-04-13T21:38:52.926 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.927 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.927 DEBUG:boto:Params: {} 2024-04-13T21:38:52.927 DEBUG:boto:Token: None 2024-04-13T21:38:52.927 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.927 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UZMmVXf6IiPDjmu05eFI6fkh37o= 2024-04-13T21:38:52.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UZMmVXf6IiPDjmu05eFI6fkh37o=', 'Content-Length': '0'} 2024-04-13T21:38:52.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007e44b9f9ab0c459-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.929 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.929 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.929 DEBUG:boto:path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.929 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.929 DEBUG:boto:Method: HEAD 2024-04-13T21:38:52.929 DEBUG:boto:Path: /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.929 DEBUG:boto:Data: 2024-04-13T21:38:52.929 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.930 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.930 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.930 DEBUG:boto:Params: {} 2024-04-13T21:38:52.930 DEBUG:boto:Token: None 2024-04-13T21:38:52.930 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UZMmVXf6IiPDjmu05eFI6fkh37o= 2024-04-13T21:38:52.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UZMmVXf6IiPDjmu05eFI6fkh37o=', 'Content-Length': '0'} 2024-04-13T21:38:52.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc548d3074b41402-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:52.932 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.932 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.932 DEBUG:boto:path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.932 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.932 DEBUG:boto:Method: GET 2024-04-13T21:38:52.933 DEBUG:boto:Path: /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.933 DEBUG:boto:Data: 2024-04-13T21:38:52.933 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.933 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.933 DEBUG:boto:Params: {} 2024-04-13T21:38:52.933 DEBUG:boto:Token: None 2024-04-13T21:38:52.933 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE= 2024-04-13T21:38:52.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE=', 'Content-Length': '0'} 2024-04-13T21:38:52.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eed67f144d0c008c-00661afb6c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.982 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:52.982 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:52.983 DEBUG:boto:path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.983 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.983 DEBUG:boto:Method: GET 2024-04-13T21:38:52.983 DEBUG:boto:Path: /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.983 DEBUG:boto:Data: 2024-04-13T21:38:52.983 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:52.983 DEBUG:boto:Port: 8000 2024-04-13T21:38:52.983 DEBUG:boto:Params: {} 2024-04-13T21:38:52.983 DEBUG:boto:Token: None 2024-04-13T21:38:52.984 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/obj?versionId=-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL 2024-04-13T21:38:52.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE= 2024-04-13T21:38:52.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgKNEtwAZr2JeXUv+2MFGzCpYvE=', 'Content-Length': '0'} 2024-04-13T21:38:52.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', '-l.ZswlBqB6bw1PzzLtfu0mMr9jb3uL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d9aaaa37408c6d7-00661afb6c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:52.986 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:52.986 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:52.986 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.986 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:52.987 DEBUG:boto:Method: GET 2024-04-13T21:38:52.987 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:52.987 DEBUG:boto:Data: 2024-04-13T21:38:52.987 DEBUG:boto:Headers: {} 2024-04-13T21:38:52.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:52.987 DEBUG:boto:Port: 8001 2024-04-13T21:38:52.987 DEBUG:boto:Params: {} 2024-04-13T21:38:52.987 DEBUG:boto:Token: None 2024-04-13T21:38:52.987 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:52 GMT /vwnfns-15/ 2024-04-13T21:38:52.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o= 2024-04-13T21:38:52.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PXCIcipmNXmxw1RK5830JV6sE6o=', 'Content-Length': '0'} 2024-04-13T21:38:52.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4aa78ec152a8f2-00661afb6c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.030 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.031 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.031 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.031 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.031 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.031 DEBUG:boto:Method: GET 2024-04-13T21:38:53.031 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.032 DEBUG:boto:Data: 2024-04-13T21:38:53.032 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.032 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.032 DEBUG:boto:Params: {} 2024-04-13T21:38:53.032 DEBUG:boto:Token: None 2024-04-13T21:38:53.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbd4fa66c7b8b6c-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.035 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.035 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.035 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.036 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.036 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.036 DEBUG:boto:Data: 2024-04-13T21:38:53.036 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.036 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.036 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.036 DEBUG:boto:Params: {} 2024-04-13T21:38:53.036 DEBUG:boto:Token: None 2024-04-13T21:38:53.037 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.037 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003769483664637f63-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.040 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.040 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.040 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.040 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.040 DEBUG:boto:Data: 2024-04-13T21:38:53.040 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.041 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.041 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.041 DEBUG:boto:Params: {} 2024-04-13T21:38:53.041 DEBUG:boto:Token: None 2024-04-13T21:38:53.041 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.041 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d0dee54e6fdb934-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.044 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.045 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.045 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.045 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.045 DEBUG:boto:Method: GET 2024-04-13T21:38:53.045 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.045 DEBUG:boto:Data: 2024-04-13T21:38:53.045 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.045 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.046 DEBUG:boto:Params: {} 2024-04-13T21:38:53.046 DEBUG:boto:Token: None 2024-04-13T21:38:53.046 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000288bc15fbc5a70fd-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.048 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.048 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.048 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.048 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.048 DEBUG:boto:Method: GET 2024-04-13T21:38:53.048 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.049 DEBUG:boto:Data: 2024-04-13T21:38:53.049 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.049 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.049 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.049 DEBUG:boto:Params: {} 2024-04-13T21:38:53.050 DEBUG:boto:Token: None 2024-04-13T21:38:53.050 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.050 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8b78c4151d7e0b0-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.094 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.095 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.095 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.095 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.095 DEBUG:boto:Method: GET 2024-04-13T21:38:53.095 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.095 DEBUG:boto:Data: 2024-04-13T21:38:53.095 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.095 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.095 DEBUG:boto:Params: {} 2024-04-13T21:38:53.095 DEBUG:boto:Token: None 2024-04-13T21:38:53.096 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d6d0f02b047e7a-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.099 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.101 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.101 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.101 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.102 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.102 DEBUG:boto:Method: GET 2024-04-13T21:38:53.102 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.102 DEBUG:boto:Data: 2024-04-13T21:38:53.102 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.103 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.103 DEBUG:boto:Params: {} 2024-04-13T21:38:53.103 DEBUG:boto:Token: None 2024-04-13T21:38:53.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ada1b2c7f06430-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.106 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.106 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.106 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.106 DEBUG:boto:path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.106 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.107 DEBUG:boto:Method: GET 2024-04-13T21:38:53.107 DEBUG:boto:Path: /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.107 DEBUG:boto:Data: 2024-04-13T21:38:53.107 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.107 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.107 DEBUG:boto:Params: {} 2024-04-13T21:38:53.107 DEBUG:boto:Token: None 2024-04-13T21:38:53.107 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU= 2024-04-13T21:38:53.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU=', 'Content-Length': '0'} 2024-04-13T21:38:53.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d4c08436036544e-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.150 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.151 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.151 DEBUG:boto:path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.151 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.151 DEBUG:boto:Method: GET 2024-04-13T21:38:53.151 DEBUG:boto:Path: /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.151 DEBUG:boto:Data: 2024-04-13T21:38:53.151 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.151 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.151 DEBUG:boto:Params: {} 2024-04-13T21:38:53.152 DEBUG:boto:Token: None 2024-04-13T21:38:53.152 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU= 2024-04-13T21:38:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU=', 'Content-Length': '0'} 2024-04-13T21:38:53.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c227de8c8680574-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.155 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.155 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.155 DEBUG:boto:path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.155 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.155 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.155 DEBUG:boto:Path: /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.155 DEBUG:boto:Data: 2024-04-13T21:38:53.155 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.155 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.155 DEBUG:boto:Params: {} 2024-04-13T21:38:53.156 DEBUG:boto:Token: None 2024-04-13T21:38:53.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XnUX/ahgbG4Mx4bp7DDKnHrx3z0= 2024-04-13T21:38:53.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XnUX/ahgbG4Mx4bp7DDKnHrx3z0=', 'Content-Length': '0'} 2024-04-13T21:38:53.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000488709af10feaa93-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.158 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.158 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.158 DEBUG:boto:path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.158 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.158 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.158 DEBUG:boto:Path: /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.158 DEBUG:boto:Data: 2024-04-13T21:38:53.158 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.159 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.159 DEBUG:boto:Params: {} 2024-04-13T21:38:53.159 DEBUG:boto:Token: None 2024-04-13T21:38:53.159 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.159 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XnUX/ahgbG4Mx4bp7DDKnHrx3z0= 2024-04-13T21:38:53.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XnUX/ahgbG4Mx4bp7DDKnHrx3z0=', 'Content-Length': '0'} 2024-04-13T21:38:53.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2beb6d1c2b219d5-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.161 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.162 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.162 DEBUG:boto:path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.162 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.162 DEBUG:boto:Method: GET 2024-04-13T21:38:53.162 DEBUG:boto:Path: /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.162 DEBUG:boto:Data: 2024-04-13T21:38:53.162 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.162 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.162 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.162 DEBUG:boto:Params: {} 2024-04-13T21:38:53.162 DEBUG:boto:Token: None 2024-04-13T21:38:53.163 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.163 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU= 2024-04-13T21:38:53.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU=', 'Content-Length': '0'} 2024-04-13T21:38:53.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dd70470e76dde64-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.165 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.165 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.165 DEBUG:boto:path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.165 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.165 DEBUG:boto:Method: GET 2024-04-13T21:38:53.165 DEBUG:boto:Path: /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.166 DEBUG:boto:Data: 2024-04-13T21:38:53.166 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.166 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.166 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.166 DEBUG:boto:Params: {} 2024-04-13T21:38:53.166 DEBUG:boto:Token: None 2024-04-13T21:38:53.166 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=dlqqJT0SU3uFQVk9dFJJxzvthMghGvi 2024-04-13T21:38:53.166 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU= 2024-04-13T21:38:53.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7WaeEfTojOm411ZhJb0gWjibMSU=', 'Content-Length': '0'} 2024-04-13T21:38:53.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'dlqqJT0SU3uFQVk9dFJJxzvthMghGvi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d2a075c2feb3846-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.211 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.211 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.211 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.211 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.212 DEBUG:boto:Method: GET 2024-04-13T21:38:53.212 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.212 DEBUG:boto:Data: 2024-04-13T21:38:53.212 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.212 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.212 DEBUG:boto:Params: {} 2024-04-13T21:38:53.212 DEBUG:boto:Token: None 2024-04-13T21:38:53.212 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3748b7be3143fa4-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.216 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.216 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.216 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.216 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.216 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.216 DEBUG:boto:Method: GET 2024-04-13T21:38:53.217 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.217 DEBUG:boto:Data: 2024-04-13T21:38:53.217 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.217 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.217 DEBUG:boto:Params: {} 2024-04-13T21:38:53.217 DEBUG:boto:Token: None 2024-04-13T21:38:53.217 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b7245160862ea38-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.266 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.267 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.267 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.267 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.267 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.267 DEBUG:boto:Data: 2024-04-13T21:38:53.267 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.268 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.268 DEBUG:boto:Params: {} 2024-04-13T21:38:53.268 DEBUG:boto:Token: None 2024-04-13T21:38:53.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056f312c404df1870-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.271 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.271 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.272 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.272 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.272 DEBUG:boto:Data: 2024-04-13T21:38:53.272 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.272 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.272 DEBUG:boto:Params: {} 2024-04-13T21:38:53.272 DEBUG:boto:Token: None 2024-04-13T21:38:53.272 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000671b596a4caac8a4-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.276 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.276 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.276 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.276 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.276 DEBUG:boto:Method: GET 2024-04-13T21:38:53.276 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.276 DEBUG:boto:Data: 2024-04-13T21:38:53.276 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.277 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.277 DEBUG:boto:Params: {} 2024-04-13T21:38:53.277 DEBUG:boto:Token: None 2024-04-13T21:38:53.277 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.277 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053506e5fbdbc63ce-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.322 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.323 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.323 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.323 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.323 DEBUG:boto:Method: GET 2024-04-13T21:38:53.323 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.323 DEBUG:boto:Data: 2024-04-13T21:38:53.323 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.323 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.323 DEBUG:boto:Params: {} 2024-04-13T21:38:53.324 DEBUG:boto:Token: None 2024-04-13T21:38:53.324 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e88db2c4bb73faf9-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.366 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.367 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.367 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.367 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.367 DEBUG:boto:Method: GET 2024-04-13T21:38:53.367 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.367 DEBUG:boto:Data: 2024-04-13T21:38:53.367 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.367 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.367 DEBUG:boto:Params: {} 2024-04-13T21:38:53.368 DEBUG:boto:Token: None 2024-04-13T21:38:53.368 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f9adb2240e81cab-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.371 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.371 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.371 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.371 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.371 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.371 DEBUG:boto:Method: GET 2024-04-13T21:38:53.372 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.372 DEBUG:boto:Data: 2024-04-13T21:38:53.372 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.372 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.372 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.372 DEBUG:boto:Params: {} 2024-04-13T21:38:53.372 DEBUG:boto:Token: None 2024-04-13T21:38:53.372 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004795424692108be6-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.375 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.375 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.375 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.375 DEBUG:boto:path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.375 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.376 DEBUG:boto:Method: GET 2024-04-13T21:38:53.376 DEBUG:boto:Path: /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.376 DEBUG:boto:Data: 2024-04-13T21:38:53.376 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.376 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.376 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.376 DEBUG:boto:Params: {} 2024-04-13T21:38:53.376 DEBUG:boto:Token: None 2024-04-13T21:38:53.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY= 2024-04-13T21:38:53.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY=', 'Content-Length': '0'} 2024-04-13T21:38:53.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003141af25599d8729-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.379 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.379 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.379 DEBUG:boto:path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.379 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.379 DEBUG:boto:Method: GET 2024-04-13T21:38:53.379 DEBUG:boto:Path: /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.379 DEBUG:boto:Data: 2024-04-13T21:38:53.380 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.380 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.380 DEBUG:boto:Params: {} 2024-04-13T21:38:53.380 DEBUG:boto:Token: None 2024-04-13T21:38:53.380 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY= 2024-04-13T21:38:53.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY=', 'Content-Length': '0'} 2024-04-13T21:38:53.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dd0f014a9419c4f-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.422 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.423 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.423 DEBUG:boto:path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.423 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.423 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.423 DEBUG:boto:Path: /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.423 DEBUG:boto:Data: 2024-04-13T21:38:53.423 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.423 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.423 DEBUG:boto:Params: {} 2024-04-13T21:38:53.424 DEBUG:boto:Token: None 2024-04-13T21:38:53.424 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pkNI+L81Ojj953We48nyDBPjVis= 2024-04-13T21:38:53.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pkNI+L81Ojj953We48nyDBPjVis=', 'Content-Length': '0'} 2024-04-13T21:38:53.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ee2d3131f946fe1-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.426 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.426 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.426 DEBUG:boto:path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.426 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.426 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.426 DEBUG:boto:Path: /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.427 DEBUG:boto:Data: 2024-04-13T21:38:53.427 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.427 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.427 DEBUG:boto:Params: {} 2024-04-13T21:38:53.427 DEBUG:boto:Token: None 2024-04-13T21:38:53.427 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pkNI+L81Ojj953We48nyDBPjVis= 2024-04-13T21:38:53.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pkNI+L81Ojj953We48nyDBPjVis=', 'Content-Length': '0'} 2024-04-13T21:38:53.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cffa9127d6b8d362-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.429 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.429 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.429 DEBUG:boto:path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.430 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.430 DEBUG:boto:Method: GET 2024-04-13T21:38:53.430 DEBUG:boto:Path: /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.430 DEBUG:boto:Data: 2024-04-13T21:38:53.430 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.430 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.430 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.430 DEBUG:boto:Params: {} 2024-04-13T21:38:53.430 DEBUG:boto:Token: None 2024-04-13T21:38:53.430 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.431 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY= 2024-04-13T21:38:53.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY=', 'Content-Length': '0'} 2024-04-13T21:38:53.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081691fa8b398b07d-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.478 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.479 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.479 DEBUG:boto:path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.479 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.479 DEBUG:boto:Method: GET 2024-04-13T21:38:53.479 DEBUG:boto:Path: /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.480 DEBUG:boto:Data: 2024-04-13T21:38:53.480 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.480 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.480 DEBUG:boto:Params: {} 2024-04-13T21:38:53.481 DEBUG:boto:Token: None 2024-04-13T21:38:53.481 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=KLig-K680b4a5A8Q2HBvpyriK1de2Lj 2024-04-13T21:38:53.482 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY= 2024-04-13T21:38:53.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0krtXLJOg6HJguR2i09cKPZ5BfY=', 'Content-Length': '0'} 2024-04-13T21:38:53.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'KLig-K680b4a5A8Q2HBvpyriK1de2Lj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ca57b6135fa56c8-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.485 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.485 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.485 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.485 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.485 DEBUG:boto:Method: GET 2024-04-13T21:38:53.486 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.486 DEBUG:boto:Data: 2024-04-13T21:38:53.486 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.486 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.486 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.487 DEBUG:boto:Params: {} 2024-04-13T21:38:53.487 DEBUG:boto:Token: None 2024-04-13T21:38:53.487 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.487 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdb9a456f6c569a8-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.491 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.491 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.492 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.492 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.492 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.492 DEBUG:boto:Method: GET 2024-04-13T21:38:53.492 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.492 DEBUG:boto:Data: 2024-04-13T21:38:53.492 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.492 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.492 DEBUG:boto:Params: {} 2024-04-13T21:38:53.493 DEBUG:boto:Token: None 2024-04-13T21:38:53.493 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.493 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca7ee7d97967e00-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.495 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.496 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.496 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.496 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.496 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.496 DEBUG:boto:Data: 2024-04-13T21:38:53.496 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.496 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.496 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.496 DEBUG:boto:Params: {} 2024-04-13T21:38:53.497 DEBUG:boto:Token: None 2024-04-13T21:38:53.497 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.497 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e3ff008765229d2-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.500 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.500 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.500 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.500 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.500 DEBUG:boto:Data: 2024-04-13T21:38:53.500 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.501 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.501 DEBUG:boto:Params: {} 2024-04-13T21:38:53.501 DEBUG:boto:Token: None 2024-04-13T21:38:53.501 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007977e8e678d90af7-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.506 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.506 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.506 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.506 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.506 DEBUG:boto:Method: GET 2024-04-13T21:38:53.506 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.506 DEBUG:boto:Data: 2024-04-13T21:38:53.506 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.506 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.506 DEBUG:boto:Params: {} 2024-04-13T21:38:53.507 DEBUG:boto:Token: None 2024-04-13T21:38:53.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cfc5c1484117ace-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.550 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.551 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.551 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.551 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.551 DEBUG:boto:Method: GET 2024-04-13T21:38:53.551 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.551 DEBUG:boto:Data: 2024-04-13T21:38:53.551 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.551 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.551 DEBUG:boto:Params: {} 2024-04-13T21:38:53.552 DEBUG:boto:Token: None 2024-04-13T21:38:53.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000469f710c00f12081-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.594 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.595 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.595 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.595 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.595 DEBUG:boto:Method: GET 2024-04-13T21:38:53.595 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.595 DEBUG:boto:Data: 2024-04-13T21:38:53.595 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.595 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.595 DEBUG:boto:Params: {} 2024-04-13T21:38:53.596 DEBUG:boto:Token: None 2024-04-13T21:38:53.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c4703db03829a25-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.599 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.599 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.599 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.599 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.600 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.600 DEBUG:boto:Method: GET 2024-04-13T21:38:53.600 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.600 DEBUG:boto:Data: 2024-04-13T21:38:53.600 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.600 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.600 DEBUG:boto:Params: {} 2024-04-13T21:38:53.600 DEBUG:boto:Token: None 2024-04-13T21:38:53.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd34a35b60d27e56-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.646 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.647 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.648 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.648 DEBUG:boto:path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.648 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.648 DEBUG:boto:Method: GET 2024-04-13T21:38:53.648 DEBUG:boto:Path: /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.648 DEBUG:boto:Data: 2024-04-13T21:38:53.648 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.649 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.649 DEBUG:boto:Params: {} 2024-04-13T21:38:53.649 DEBUG:boto:Token: None 2024-04-13T21:38:53.650 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.650 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg= 2024-04-13T21:38:53.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg=', 'Content-Length': '0'} 2024-04-13T21:38:53.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008234b9851e2f008b-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.652 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.653 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.653 DEBUG:boto:path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.653 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.653 DEBUG:boto:Method: GET 2024-04-13T21:38:53.653 DEBUG:boto:Path: /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.654 DEBUG:boto:Data: 2024-04-13T21:38:53.654 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.654 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.654 DEBUG:boto:Params: {} 2024-04-13T21:38:53.654 DEBUG:boto:Token: None 2024-04-13T21:38:53.654 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg= 2024-04-13T21:38:53.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg=', 'Content-Length': '0'} 2024-04-13T21:38:53.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d8fbb7a587da9f2-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.657 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.657 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.657 DEBUG:boto:path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.657 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.658 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.658 DEBUG:boto:Path: /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.658 DEBUG:boto:Data: 2024-04-13T21:38:53.658 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.658 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.658 DEBUG:boto:Params: {} 2024-04-13T21:38:53.658 DEBUG:boto:Token: None 2024-04-13T21:38:53.658 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKDMFplCmjZaXFBdPXgqhDS0Rs0= 2024-04-13T21:38:53.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKDMFplCmjZaXFBdPXgqhDS0Rs0=', 'Content-Length': '0'} 2024-04-13T21:38:53.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000335321a66040a592-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.661 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.661 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.661 DEBUG:boto:path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.661 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.661 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.662 DEBUG:boto:Path: /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.662 DEBUG:boto:Data: 2024-04-13T21:38:53.662 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.662 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.662 DEBUG:boto:Params: {} 2024-04-13T21:38:53.662 DEBUG:boto:Token: None 2024-04-13T21:38:53.663 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKDMFplCmjZaXFBdPXgqhDS0Rs0= 2024-04-13T21:38:53.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKDMFplCmjZaXFBdPXgqhDS0Rs0=', 'Content-Length': '0'} 2024-04-13T21:38:53.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d84930bdf930bfd-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.666 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.666 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.666 DEBUG:boto:path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.666 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.666 DEBUG:boto:Method: GET 2024-04-13T21:38:53.667 DEBUG:boto:Path: /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.667 DEBUG:boto:Data: 2024-04-13T21:38:53.667 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.667 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.667 DEBUG:boto:Params: {} 2024-04-13T21:38:53.668 DEBUG:boto:Token: None 2024-04-13T21:38:53.668 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.668 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg= 2024-04-13T21:38:53.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg=', 'Content-Length': '0'} 2024-04-13T21:38:53.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f86dce2f8043b930-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.670 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.671 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.671 DEBUG:boto:path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.671 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.671 DEBUG:boto:Method: GET 2024-04-13T21:38:53.671 DEBUG:boto:Path: /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.672 DEBUG:boto:Data: 2024-04-13T21:38:53.672 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.672 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.672 DEBUG:boto:Params: {} 2024-04-13T21:38:53.672 DEBUG:boto:Token: None 2024-04-13T21:38:53.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb 2024-04-13T21:38:53.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg= 2024-04-13T21:38:53.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OZxBLGArHtt5f3tXgPwiZgq0ztg=', 'Content-Length': '0'} 2024-04-13T21:38:53.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'wfWU7UeFtRE5fiy16I2Xtgd6dx-bhcb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000be488ff622aa634-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.718 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.719 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.719 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.719 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.719 DEBUG:boto:Method: GET 2024-04-13T21:38:53.719 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.719 DEBUG:boto:Data: 2024-04-13T21:38:53.719 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.720 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.720 DEBUG:boto:Params: {} 2024-04-13T21:38:53.720 DEBUG:boto:Token: None 2024-04-13T21:38:53.720 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c2591591c68a64-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.723 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.724 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.724 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.724 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.724 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.724 DEBUG:boto:Method: GET 2024-04-13T21:38:53.724 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.725 DEBUG:boto:Data: 2024-04-13T21:38:53.725 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.725 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.725 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.725 DEBUG:boto:Params: {} 2024-04-13T21:38:53.725 DEBUG:boto:Token: None 2024-04-13T21:38:53.725 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.725 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000771fd597b9b657-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.728 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.728 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.728 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.729 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.729 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.729 DEBUG:boto:Data: 2024-04-13T21:38:53.729 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.729 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.729 DEBUG:boto:Params: {} 2024-04-13T21:38:53.729 DEBUG:boto:Token: None 2024-04-13T21:38:53.729 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fb600b80cfc6a13-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.733 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.733 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.733 DEBUG:boto:Method: HEAD 2024-04-13T21:38:53.733 DEBUG:boto:Path: /vwnfns-15/obj 2024-04-13T21:38:53.733 DEBUG:boto:Data: 2024-04-13T21:38:53.733 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.733 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.733 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.733 DEBUG:boto:Params: {} 2024-04-13T21:38:53.734 DEBUG:boto:Token: None 2024-04-13T21:38:53.734 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj 2024-04-13T21:38:53.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI= 2024-04-13T21:38:53.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FRZapGdUlRg1XPuk0vQODLsYRaI=', 'Content-Length': '0'} 2024-04-13T21:38:53.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048f83fc6a0665452-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:38:53.738 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.738 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.738 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.738 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.738 DEBUG:boto:Method: GET 2024-04-13T21:38:53.738 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.738 DEBUG:boto:Data: 2024-04-13T21:38:53.739 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.739 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.739 DEBUG:boto:Params: {} 2024-04-13T21:38:53.739 DEBUG:boto:Token: None 2024-04-13T21:38:53.739 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000355161910e7420e6-00661afb6d-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.782 DEBUG:boto:path=/vwnfns-15/obj 2024-04-13T21:38:53.783 DEBUG:boto:auth_path=/vwnfns-15/obj 2024-04-13T21:38:53.783 DEBUG:boto:path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.783 DEBUG:boto:auth_path=/vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.783 DEBUG:boto:Method: GET 2024-04-13T21:38:53.783 DEBUG:boto:Path: /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.783 DEBUG:boto:Data: 2024-04-13T21:38:53.783 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.784 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.784 DEBUG:boto:Params: {} 2024-04-13T21:38:53.784 DEBUG:boto:Token: None 2024-04-13T21:38:53.785 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/obj?versionId=A3ZgYu.Obob4lazgpJ01ShY35PtFVXM 2024-04-13T21:38:53.785 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8= 2024-04-13T21:38:53.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zzp68MfkiqQiuc11S7eFlEEv5L8=', 'Content-Length': '0'} 2024-04-13T21:38:53.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:38:08 GMT'), ('x-amz-version-id', 'A3ZgYu.Obob4lazgpJ01ShY35PtFVXM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2863308a6ebd472-00661afb6d-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.831 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.831 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.831 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.832 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.832 DEBUG:boto:Method: GET 2024-04-13T21:38:53.832 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.832 DEBUG:boto:Data: 2024-04-13T21:38:53.832 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.833 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:38:53.833 DEBUG:boto:Port: 8001 2024-04-13T21:38:53.833 DEBUG:boto:Params: {} 2024-04-13T21:38:53.833 DEBUG:boto:Token: None 2024-04-13T21:38:53.833 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.834 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71432f181ebbb69-00661afb6d-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.837 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.837 DEBUG:boto:path=/vwnfns-15/ 2024-04-13T21:38:53.838 DEBUG:boto:auth_path=/vwnfns-15/ 2024-04-13T21:38:53.838 DEBUG:boto:path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.838 DEBUG:boto:auth_path=/vwnfns-15/?prefix=obj 2024-04-13T21:38:53.838 DEBUG:boto:Method: GET 2024-04-13T21:38:53.838 DEBUG:boto:Path: /vwnfns-15/?prefix=obj 2024-04-13T21:38:53.838 DEBUG:boto:Data: 2024-04-13T21:38:53.839 DEBUG:boto:Headers: {} 2024-04-13T21:38:53.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:38:53.839 DEBUG:boto:Port: 8000 2024-04-13T21:38:53.839 DEBUG:boto:Params: {} 2024-04-13T21:38:53.839 DEBUG:boto:Token: None 2024-04-13T21:38:53.840 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:38:53 GMT /vwnfns-15/ 2024-04-13T21:38:53.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI= 2024-04-13T21:38:53.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:38:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7J1MMhElMAOnZ/HhmHa1zL1pbjI=', 'Content-Length': '0'} 2024-04-13T21:38:53.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062d4d8e77c89ead-00661afb6d-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:38:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:38:53.843 DEBUG:boto:b'vwnfns-15obj1000falseobj2024-04-13T21:38:08.702Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:38:53.843 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-15 zones={a2, a1} 2024-04-13T21:38:53.844 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-13T21:38:53.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:38:53.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:38:53.844 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:38:53.899 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:38:53.899 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:38:09.538757Z" 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T21:38:53.949 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:30:05.522018Z" 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:30:05.523190Z" 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003747", 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:38:53.544885Z" 2024-04-13T21:38:53.950 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:53.951 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:38:53.952 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003747'} 2024-04-13T21:38:53.952 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T21:38:53.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:38:53.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:38:53.952 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:38:54.003 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:38:54.004 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:38:54.045 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:38:54.045 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:38:54.045 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:54.046 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:09.538757Z" 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:54.047 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:54.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:54.049 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:38:54.049 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:54.049 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:54.049 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T21:38:54.050 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:54.050 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:54.050 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:54.050 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T21:38:54.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:54.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003606", 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:31.543192Z" 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:54.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:38:54Z" 2024-04-13T21:38:54.052 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:38:54.053 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:09.538757Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:31.543192Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T21:38:54Z" } 2024-04-13T21:38:54.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:09.538757Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:31.543192Z'}}] 2024-04-13T21:38:54.054 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003747'} 2024-04-13T21:38:54.054 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003606'} 2024-04-13T21:38:54.054 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000003747 target=00000000000000000000:00000000000000003606 2024-04-13T21:38:59.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:38:59.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:38:59.056 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:38:59.114 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:38:59.114 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:38:59.158 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:59.159 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:09.538757Z" 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:59.160 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:59.161 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:38:59.162 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003606", 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:31.543192Z" 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:59.163 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:38:59.164 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T21:38:59.164 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T21:38:59.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:38:59.164 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:38:59Z" 2024-04-13T21:38:59.164 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:38:59.165 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:09.538757Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:31.543192Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T21:38:59Z" } 2024-04-13T21:38:59.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:09.538757Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:31.543192Z'}}] 2024-04-13T21:38:59.165 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003747'} 2024-04-13T21:38:59.165 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003606'} 2024-04-13T21:38:59.165 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000003747 target=00000000000000000000:00000000000000003606 2024-04-13T21:39:04.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:04.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:04.167 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:04.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:04.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:04.270 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:39:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:39:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:39:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:39:04.270 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:09.538757Z" 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:04.272 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:04.273 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003606", 2024-04-13T21:39:04.274 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:31.543192Z" 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T21:39:04.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:04.276 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:39:04Z" 2024-04-13T21:39:04.276 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:39:04.277 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:09.538757Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:31.543192Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T21:39:04Z" } 2024-04-13T21:39:04.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:09.538757Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:31.543192Z'}}] 2024-04-13T21:39:04.277 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003747'} 2024-04-13T21:39:04.277 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003606'} 2024-04-13T21:39:04.277 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000003747 target=00000000000000000000:00000000000000003606 2024-04-13T21:39:09.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.279 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:09.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:09.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:09.379 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.380 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:09.538757Z" 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.381 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.383 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003747", 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:53.544885Z" 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:39:09.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.385 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:39:09.385 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T21:39:09.385 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T21:39:09.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.385 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:39:09Z" 2024-04-13T21:39:09.385 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:39:09.386 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:09.538757Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003747", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:53.544885Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T21:39:09Z" } 2024-04-13T21:39:09.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:09.538757Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003747', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:53.544885Z'}}] 2024-04-13T21:39:09.386 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003747'} 2024-04-13T21:39:09.387 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000003747'} 2024-04-13T21:39:09.387 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T21:39:09.387 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-13T21:39:09.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.387 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:09.435 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:09.435 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:09.481 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:39:09.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.481 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-13T21:39:09.481 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:37:47.514527Z" 2024-04-13T21:39:09.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:30:27.502412Z" 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:30:27.503831Z" 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004170", 2024-04-13T21:39:09.483 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:38:31.518230Z" 2024-04-13T21:39:09.483 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.483 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:39:09.484 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004037', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000001918', 3: '00000000000000000000:00000000000000004170'} 2024-04-13T21:39:09.484 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T21:39:09.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.484 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:09.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:09.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.575 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:37:47.514527Z" 2024-04-13T21:39:09.576 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.502412Z" 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.503831Z" 2024-04-13T21:39:09.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004170", 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.579 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:38:31.518230Z" 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:39:09Z" 2024-04-13T21:39:09.580 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:39:09.581 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004037", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:37:47.514527Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.502412Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.503831Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:38:31.518230Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T21:39:09Z" } 2024-04-13T21:39:09.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:37:47.514527Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.502412Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.503831Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:38:31.518230Z'}}] 2024-04-13T21:39:09.582 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004037', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000001918', 3: '00000000000000000000:00000000000000004170'} 2024-04-13T21:39:09.582 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004037', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000001918', 3: '00000000000000000000:00000000000000004170'} 2024-04-13T21:39:09.582 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T21:39:09.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-04-13T21:39:09.584 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:09.585 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:09.585 DEBUG:boto:Method: PUT 2024-04-13T21:39:09.585 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:39:09.585 DEBUG:boto:Data: 2024-04-13T21:39:09.585 DEBUG:boto:Headers: {} 2024-04-13T21:39:09.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:09.585 DEBUG:boto:Port: 8000 2024-04-13T21:39:09.585 DEBUG:boto:Params: {} 2024-04-13T21:39:09.586 DEBUG:boto:Token: None 2024-04-13T21:39:09.586 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:39:09 GMT /vwnfns-16/ 2024-04-13T21:39:09.586 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9+Tn/D5emnJt7bI+yLuFRKHjL84= 2024-04-13T21:39:09.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9+Tn/D5emnJt7bI+yLuFRKHjL84=', 'Content-Length': '0'} 2024-04-13T21:39:09.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000902b7e956e5b7723-00661afb7d-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:09.599 DEBUG:rgw_multi.tests:created bucket=vwnfns-16 2024-04-13T21:39:09.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.599 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:09.646 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:09.646 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:09.675 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:39:09.686 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:39:09.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.687 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044349.606175_197.1", 2024-04-13T21:39:09.687 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:39:09.606175Z" 2024-04-13T21:39:09.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.688 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:09.689 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:39:09.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.689 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:39:09.690 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044349.606175_197.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:09.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:39:09.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:09.691 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:09.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:09.776+0000 7f8ae267a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:39:09.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:09.777+0000 7f8ae267a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:09.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044349.600529_194.1", 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:39:09.600529Z", 2024-04-13T21:39:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.782 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.783 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:09.784 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:39:09.785 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:39:09.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:09.786 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:39:09Z" 2024-04-13T21:39:09.786 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:39:09.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044349.600529_194.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:39:09.600529Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:39:09Z" } 2024-04-13T21:39:09.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044349.600529_194.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:39:09.600529Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:39:09.787 DEBUG:rgw_multi.tests:log_status={0: '1_1713044349.606175_197.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:09.787 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044349.600529_194.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:09.787 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044349.606175_197.1 target=1_1713044349.600529_194.1 2024-04-13T21:39:14.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:14.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:14.789 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:14.844 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:14.844 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:14.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:14.883+0000 7f7b08397780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:39:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:14.885+0000 7f7b08397780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:39:14.889 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:39:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:39:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:39:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:39:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:39:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044349.606175_197.1", 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:39:09.606175Z", 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:14.892 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:14.893 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:39:14Z" 2024-04-13T21:39:14.895 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:39:14.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044349.606175_197.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:39:09.606175Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:39:14Z" } 2024-04-13T21:39:14.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044349.606175_197.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:39:09.606175Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:39:14.897 DEBUG:rgw_multi.tests:log_status={0: '1_1713044349.606175_197.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:14.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044349.606175_197.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:14.897 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:39:14.897 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:14.897 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:14.897 DEBUG:boto:Method: HEAD 2024-04-13T21:39:14.897 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:39:14.897 DEBUG:boto:Data: 2024-04-13T21:39:14.897 DEBUG:boto:Headers: {} 2024-04-13T21:39:14.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:14.898 DEBUG:boto:Port: 8000 2024-04-13T21:39:14.898 DEBUG:boto:Params: {} 2024-04-13T21:39:14.898 DEBUG:boto:Token: None 2024-04-13T21:39:14.898 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:14 GMT /vwnfns-16/ 2024-04-13T21:39:14.898 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zWjWJAskXSupNOdwMYu6OnnLr+s= 2024-04-13T21:39:14.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zWjWJAskXSupNOdwMYu6OnnLr+s=', 'Content-Length': '0'} 2024-04-13T21:39:14.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ec72b9750e38eb3-00661afb82-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:14.901 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:14.901 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:14.901 DEBUG:boto:Method: PUT 2024-04-13T21:39:14.901 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:39:14.901 DEBUG:boto:Data: 2024-04-13T21:39:14.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:39:14.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:14.902 DEBUG:boto:Port: 8000 2024-04-13T21:39:14.902 DEBUG:boto:Params: {} 2024-04-13T21:39:14.902 DEBUG:boto:Token: None 2024-04-13T21:39:14.902 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:39:14 GMT /vwnfns-16/obj 2024-04-13T21:39:14.902 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zv7Yv4F86Kz8aacvR7VIHzwIXz0= 2024-04-13T21:39:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:39:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zv7Yv4F86Kz8aacvR7VIHzwIXz0='} 2024-04-13T21:39:14.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031a7ce04b8ee9107-00661afb82-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:14.908 DEBUG:rgw_multi.tests:created initial version id=None 2024-04-13T21:39:14.909 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-16 2024-04-13T21:39:14.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:39:14.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:39:14.909 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:39:14.964 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:14.964 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:14.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:15.006+0000 7f98fe61f780 1 waiting to reach incremental sync.. 2024-04-13T21:39:19.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:20.006+0000 7f98fe61f780 1 waiting to reach incremental sync.. 2024-04-13T21:39:24.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:25.006+0000 7f98fe61f780 1 waiting to reach incremental sync.. 2024-04-13T21:39:30.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:30.007+0000 7f98fe61f780 1 bucket sync caught up with source: 2024-04-13T21:39:30.002 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-13T21:39:30.002 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.45.6, , ] 2024-04-13T21:39:30.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:30.007+0000 7f98fe61f780 0 bucket checkpoint complete 2024-04-13T21:39:30.016 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-16 2024-04-13T21:39:30.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:39:30.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:39:30.017 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:39:30.078 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:30.078 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:30.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:30.120+0000 7f0145674780 1 waiting to reach incremental sync.. 2024-04-13T21:39:35.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:35.121+0000 7f0145674780 1 waiting to reach incremental sync.. 2024-04-13T21:39:40.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:40.122+0000 7f0145674780 1 waiting to reach incremental sync.. 2024-04-13T21:39:45.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:45.123+0000 7f0145674780 1 waiting to reach incremental sync.. 2024-04-13T21:39:50.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:50.124+0000 7f0145674780 1 bucket sync caught up with source: 2024-04-13T21:39:50.116 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.37.6, , ] 2024-04-13T21:39:50.117 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.37.6, , ] 2024-04-13T21:39:50.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:50.124+0000 7f0145674780 0 bucket checkpoint complete 2024-04-13T21:39:50.132 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-16 zones={a2, a1} 2024-04-13T21:39:50.133 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.133 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.133 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.133 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:39:50.133 DEBUG:boto:Data: 2024-04-13T21:39:50.133 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.133 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.133 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.133 DEBUG:boto:Params: {} 2024-04-13T21:39:50.134 DEBUG:boto:Token: None 2024-04-13T21:39:50.134 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lQl6ioWpxDW3EpJMbuhszBkGEQ8= 2024-04-13T21:39:50.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lQl6ioWpxDW3EpJMbuhszBkGEQ8=', 'Content-Length': '0'} 2024-04-13T21:39:50.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea4bd34e871002a7-00661afba6-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.139 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.139 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.140 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.140 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:39:50.140 DEBUG:boto:Data: 2024-04-13T21:39:50.140 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.140 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.140 DEBUG:boto:Params: {} 2024-04-13T21:39:50.140 DEBUG:boto:Token: None 2024-04-13T21:39:50.140 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lQl6ioWpxDW3EpJMbuhszBkGEQ8= 2024-04-13T21:39:50.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lQl6ioWpxDW3EpJMbuhszBkGEQ8=', 'Content-Length': '0'} 2024-04-13T21:39:50.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085302484229c6e50-00661afba6-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.143 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:39:50.143 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.143 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.144 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.144 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.144 DEBUG:boto:Method: GET 2024-04-13T21:39:50.144 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.144 DEBUG:boto:Data: 2024-04-13T21:39:50.144 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.144 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.144 DEBUG:boto:Params: {} 2024-04-13T21:39:50.144 DEBUG:boto:Token: None 2024-04-13T21:39:50.145 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/?versions 2024-04-13T21:39:50.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4= 2024-04-13T21:39:50.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4=', 'Content-Length': '0'} 2024-04-13T21:39:50.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000812639cc64a268fb-00661afba6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.148 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:39:50.149 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:39:50.149 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.149 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.149 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.149 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.149 DEBUG:boto:Method: GET 2024-04-13T21:39:50.149 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.149 DEBUG:boto:Data: 2024-04-13T21:39:50.149 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.149 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.150 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.150 DEBUG:boto:Params: {} 2024-04-13T21:39:50.150 DEBUG:boto:Token: None 2024-04-13T21:39:50.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/?versions 2024-04-13T21:39:50.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4= 2024-04-13T21:39:50.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4=', 'Content-Length': '0'} 2024-04-13T21:39:50.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a229f051561b9df0-00661afba6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.154 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:39:50.154 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.154 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.154 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.155 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.155 DEBUG:boto:Method: GET 2024-04-13T21:39:50.155 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.155 DEBUG:boto:Data: 2024-04-13T21:39:50.155 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.155 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.155 DEBUG:boto:Params: {} 2024-04-13T21:39:50.155 DEBUG:boto:Token: None 2024-04-13T21:39:50.155 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/?versions 2024-04-13T21:39:50.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4= 2024-04-13T21:39:50.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4=', 'Content-Length': '0'} 2024-04-13T21:39:50.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54c2131fd018951-00661afba6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.159 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.159 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.159 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.159 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.159 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.159 DEBUG:boto:Method: GET 2024-04-13T21:39:50.159 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:39:50.160 DEBUG:boto:Data: 2024-04-13T21:39:50.160 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.160 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.160 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.160 DEBUG:boto:Params: {} 2024-04-13T21:39:50.160 DEBUG:boto:Token: None 2024-04-13T21:39:50.160 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/?versions 2024-04-13T21:39:50.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4= 2024-04-13T21:39:50.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iQw3FjQWL5+k601LqbXc0Ny/1g4=', 'Content-Length': '0'} 2024-04-13T21:39:50.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002698d0db5b0d1b30-00661afba6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.163 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:39:50.164 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.164 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.164 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.164 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.164 DEBUG:boto:Method: GET 2024-04-13T21:39:50.164 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.164 DEBUG:boto:Data: 2024-04-13T21:39:50.165 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.165 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.165 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.165 DEBUG:boto:Params: {} 2024-04-13T21:39:50.165 DEBUG:boto:Token: None 2024-04-13T21:39:50.165 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0= 2024-04-13T21:39:50.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0=', 'Content-Length': '0'} 2024-04-13T21:39:50.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004fb8f2d6afbb4ddf-00661afba6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.167 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.167 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.167 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.167 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.167 DEBUG:boto:Method: GET 2024-04-13T21:39:50.168 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.168 DEBUG:boto:Data: 2024-04-13T21:39:50.168 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.168 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.168 DEBUG:boto:Params: {} 2024-04-13T21:39:50.168 DEBUG:boto:Token: None 2024-04-13T21:39:50.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0= 2024-04-13T21:39:50.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0=', 'Content-Length': '0'} 2024-04-13T21:39:50.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050d5f612dac63401-00661afba6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.170 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.170 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.170 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.171 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.171 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.171 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.171 DEBUG:boto:Data: 2024-04-13T21:39:50.171 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.171 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.171 DEBUG:boto:Params: {} 2024-04-13T21:39:50.171 DEBUG:boto:Token: None 2024-04-13T21:39:50.171 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:57V0d92VCkNSdUwz/uKBScDvZ9c= 2024-04-13T21:39:50.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:57V0d92VCkNSdUwz/uKBScDvZ9c=', 'Content-Length': '0'} 2024-04-13T21:39:50.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b49a0e1b09d9bf9-00661afba6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.173 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.174 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.174 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.174 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.174 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.174 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.174 DEBUG:boto:Data: 2024-04-13T21:39:50.174 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.174 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.175 DEBUG:boto:Params: {} 2024-04-13T21:39:50.175 DEBUG:boto:Token: None 2024-04-13T21:39:50.175 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:57V0d92VCkNSdUwz/uKBScDvZ9c= 2024-04-13T21:39:50.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:57V0d92VCkNSdUwz/uKBScDvZ9c=', 'Content-Length': '0'} 2024-04-13T21:39:50.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085dad42d2bd6b0d9-00661afba6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:39:50.177 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.177 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.177 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.177 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.177 DEBUG:boto:Method: GET 2024-04-13T21:39:50.177 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.177 DEBUG:boto:Data: 2024-04-13T21:39:50.177 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.178 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.178 DEBUG:boto:Params: {} 2024-04-13T21:39:50.178 DEBUG:boto:Token: None 2024-04-13T21:39:50.178 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0= 2024-04-13T21:39:50.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0=', 'Content-Length': '0'} 2024-04-13T21:39:50.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7ded9870fbeb5ff-00661afba6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.180 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.180 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.180 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.180 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.180 DEBUG:boto:Method: GET 2024-04-13T21:39:50.181 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.181 DEBUG:boto:Data: 2024-04-13T21:39:50.181 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.181 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.181 DEBUG:boto:Params: {} 2024-04-13T21:39:50.181 DEBUG:boto:Token: None 2024-04-13T21:39:50.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:39:50.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0= 2024-04-13T21:39:50.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PS21DaP7fAxUcVJcfKMyiryPtj0=', 'Content-Length': '0'} 2024-04-13T21:39:50.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0b537300fa2ea71-00661afba6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.183 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.183 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.183 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.183 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.184 DEBUG:boto:Method: GET 2024-04-13T21:39:50.184 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:39:50.184 DEBUG:boto:Data: 2024-04-13T21:39:50.184 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.184 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.184 DEBUG:boto:Params: {} 2024-04-13T21:39:50.184 DEBUG:boto:Token: None 2024-04-13T21:39:50.184 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA= 2024-04-13T21:39:50.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA=', 'Content-Length': '0'} 2024-04-13T21:39:50.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7473200892e128a-00661afba6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.230 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.231 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.231 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.231 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.231 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.231 DEBUG:boto:Method: GET 2024-04-13T21:39:50.231 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:39:50.232 DEBUG:boto:Data: 2024-04-13T21:39:50.232 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.232 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.232 DEBUG:boto:Params: {} 2024-04-13T21:39:50.232 DEBUG:boto:Token: None 2024-04-13T21:39:50.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA= 2024-04-13T21:39:50.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA=', 'Content-Length': '0'} 2024-04-13T21:39:50.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9ce9397cb40f240-00661afba6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.235 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.236 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.236 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.236 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.236 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:39:50.236 DEBUG:boto:Data: 2024-04-13T21:39:50.236 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.236 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.236 DEBUG:boto:Params: {} 2024-04-13T21:39:50.237 DEBUG:boto:Token: None 2024-04-13T21:39:50.237 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj 2024-04-13T21:39:50.237 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I5xSBnxK4jXM8slcXxwNKxMeLx4= 2024-04-13T21:39:50.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I5xSBnxK4jXM8slcXxwNKxMeLx4=', 'Content-Length': '0'} 2024-04-13T21:39:50.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f112e5e5db1eefe6-00661afba6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.239 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.239 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.239 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.239 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:39:50.239 DEBUG:boto:Data: 2024-04-13T21:39:50.241 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.241 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.241 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.241 DEBUG:boto:Params: {} 2024-04-13T21:39:50.242 DEBUG:boto:Token: None 2024-04-13T21:39:50.242 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj 2024-04-13T21:39:50.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I5xSBnxK4jXM8slcXxwNKxMeLx4= 2024-04-13T21:39:50.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I5xSBnxK4jXM8slcXxwNKxMeLx4=', 'Content-Length': '0'} 2024-04-13T21:39:50.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006940a919833aa213-00661afba6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:39:50.245 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.245 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.245 DEBUG:boto:Method: GET 2024-04-13T21:39:50.245 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:39:50.245 DEBUG:boto:Data: 2024-04-13T21:39:50.245 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.245 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.245 DEBUG:boto:Params: {} 2024-04-13T21:39:50.246 DEBUG:boto:Token: None 2024-04-13T21:39:50.246 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj 2024-04-13T21:39:50.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ng6ZnREl0zTbep31dpYcUOzLh7s= 2024-04-13T21:39:50.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ng6ZnREl0zTbep31dpYcUOzLh7s=', 'Content-Length': '0'} 2024-04-13T21:39:50.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d007fbb8b82192b4-00661afba6-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.248 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.248 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.248 DEBUG:boto:Method: GET 2024-04-13T21:39:50.248 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:39:50.248 DEBUG:boto:Data: 2024-04-13T21:39:50.248 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.248 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.249 DEBUG:boto:Params: {} 2024-04-13T21:39:50.249 DEBUG:boto:Token: None 2024-04-13T21:39:50.249 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj 2024-04-13T21:39:50.249 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ng6ZnREl0zTbep31dpYcUOzLh7s= 2024-04-13T21:39:50.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ng6ZnREl0zTbep31dpYcUOzLh7s=', 'Content-Length': '0'} 2024-04-13T21:39:50.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022b0cef401035569-00661afba6-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.251 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.251 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.251 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.251 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.251 DEBUG:boto:Method: GET 2024-04-13T21:39:50.251 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:39:50.251 DEBUG:boto:Data: 2024-04-13T21:39:50.251 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:39:50.251 DEBUG:boto:Port: 8001 2024-04-13T21:39:50.251 DEBUG:boto:Params: {} 2024-04-13T21:39:50.252 DEBUG:boto:Token: None 2024-04-13T21:39:50.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA= 2024-04-13T21:39:50.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA=', 'Content-Length': '0'} 2024-04-13T21:39:50.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e189babead8a9b-00661afba6-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.254 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.255 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.255 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.255 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.255 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:39:50.255 DEBUG:boto:Method: GET 2024-04-13T21:39:50.255 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:39:50.255 DEBUG:boto:Data: 2024-04-13T21:39:50.256 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.256 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.256 DEBUG:boto:Params: {} 2024-04-13T21:39:50.256 DEBUG:boto:Token: None 2024-04-13T21:39:50.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA= 2024-04-13T21:39:50.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WeUyb2a/LAcks4KYm8HQCsvSdpA=', 'Content-Length': '0'} 2024-04-13T21:39:50.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099d4e3fb68360288-00661afba6-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.259 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:39:50.259 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-16 zones={a2, a1} 2024-04-13T21:39:50.259 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.259 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.259 DEBUG:boto:path=/vwnfns-16/?versioning 2024-04-13T21:39:50.259 DEBUG:boto:auth_path=/vwnfns-16/?versioning 2024-04-13T21:39:50.259 DEBUG:boto:Method: PUT 2024-04-13T21:39:50.259 DEBUG:boto:Path: /vwnfns-16/?versioning 2024-04-13T21:39:50.260 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:39:50.260 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.260 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.260 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.260 DEBUG:boto:Params: {} 2024-04-13T21:39:50.260 DEBUG:boto:Token: None 2024-04-13T21:39:50.260 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/?versioning 2024-04-13T21:39:50.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8vRkqFVkMjqYdlm+0Tr4pJnOyyg= 2024-04-13T21:39:50.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8vRkqFVkMjqYdlm+0Tr4pJnOyyg=', 'Content-Length': '228'} 2024-04-13T21:39:50.266 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055bcea0f7462f585-00661afba6-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:50.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:50.266 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:50.320 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:50.320 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:50.353 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:39:50.365 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:39:50.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.365 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044390.274385_202.1", 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:39:50.274385Z" 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:39:50.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.367 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:50.367 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:39:50.367 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:50.367 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:39:50.368 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044390.274385_202.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:50.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:39:50.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:50.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:39:50.369 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:39:50.416 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:50.417 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:50.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:50.452+0000 7f2bd8c14780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:39:50.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:50.453+0000 7f2bd8c14780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:39:50.457 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:39:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:39:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:39:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:39:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:39:50.457 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044390.274385_202.1", 2024-04-13T21:39:50.458 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:39:50.274385Z", 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:39:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:50.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:39:50.462 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:39:50.463 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:39:50Z" 2024-04-13T21:39:50.464 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:39:50.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044390.274385_202.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:39:50.274385Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:39:50Z" } 2024-04-13T21:39:50.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044390.274385_202.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:39:50.274385Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:39:50.465 DEBUG:rgw_multi.tests:log_status={0: '1_1713044390.274385_202.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:50.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044390.274385_202.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:39:50.465 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:39:50.465 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:39:50.465 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:39:50.465 DEBUG:boto:Method: HEAD 2024-04-13T21:39:50.465 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:39:50.466 DEBUG:boto:Data: 2024-04-13T21:39:50.466 DEBUG:boto:Headers: {} 2024-04-13T21:39:50.466 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.466 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.466 DEBUG:boto:Params: {} 2024-04-13T21:39:50.466 DEBUG:boto:Token: None 2024-04-13T21:39:50.466 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/ 2024-04-13T21:39:50.466 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lQl6ioWpxDW3EpJMbuhszBkGEQ8= 2024-04-13T21:39:50.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lQl6ioWpxDW3EpJMbuhszBkGEQ8=', 'Content-Length': '0'} 2024-04-13T21:39:50.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c3c01357df76aa2-00661afba6-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.469 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:39:50.469 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:39:50.469 DEBUG:boto:Method: PUT 2024-04-13T21:39:50.469 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:39:50.469 DEBUG:boto:Data: 2024-04-13T21:39:50.469 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:39:50.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:39:50.470 DEBUG:boto:Port: 8000 2024-04-13T21:39:50.470 DEBUG:boto:Params: {} 2024-04-13T21:39:50.470 DEBUG:boto:Token: None 2024-04-13T21:39:50.470 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:39:50 GMT /vwnfns-16/obj 2024-04-13T21:39:50.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:52V2GF++FtQlADPKf1xWHyWJJR8= 2024-04-13T21:39:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:39:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:52V2GF++FtQlADPKf1xWHyWJJR8='} 2024-04-13T21:39:50.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-amz-request-id', 'tx000000b4ef5d32c9e553d-00661afba6-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:39:50.479 DEBUG:rgw_multi.tests:created new version id=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:39:50.479 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-16 2024-04-13T21:39:50.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:39:50.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:39:50.479 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:39:50.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:39:50.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:39:50.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:50.577+0000 7f01b10c1780 1 waiting for incremental sync to catch up: 2024-04-13T21:39:50.568 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-13T21:39:50.568 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:39:55.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:39:55.578+0000 7f01b10c1780 1 waiting for incremental sync to catch up: 2024-04-13T21:39:55.570 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-13T21:39:55.570 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:00.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:00.579+0000 7f01b10c1780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:00.571 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-13T21:40:00.571 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:05.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:05.580+0000 7f01b10c1780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:05.573 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-13T21:40:05.573 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:10.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:10.581+0000 7f01b10c1780 1 bucket sync caught up with source: 2024-04-13T21:40:10.574 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:10.574 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:10.581+0000 7f01b10c1780 0 bucket checkpoint complete 2024-04-13T21:40:10.590 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-16 2024-04-13T21:40:10.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:10.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:10.591 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:40:10.651 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:10.651 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:10.695+0000 7fd050aa9780 1 bucket sync caught up with source: 2024-04-13T21:40:10.688 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-04-13T21:40:10.688 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-04-13T21:40:10.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:10.695+0000 7fd050aa9780 0 bucket checkpoint complete 2024-04-13T21:40:10.701 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-16 zones={a2, a1} 2024-04-13T21:40:10.701 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.701 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.701 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.702 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:40:10.702 DEBUG:boto:Data: 2024-04-13T21:40:10.702 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.702 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.702 DEBUG:boto:Params: {} 2024-04-13T21:40:10.702 DEBUG:boto:Token: None 2024-04-13T21:40:10.702 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.703 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LFm09Bie+0qAIFHAuuX9PUCpQI0= 2024-04-13T21:40:10.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LFm09Bie+0qAIFHAuuX9PUCpQI0=', 'Content-Length': '0'} 2024-04-13T21:40:10.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e342b50b53f0b16c-00661afbba-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.705 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.705 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.705 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.705 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:40:10.706 DEBUG:boto:Data: 2024-04-13T21:40:10.706 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.706 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.706 DEBUG:boto:Params: {} 2024-04-13T21:40:10.706 DEBUG:boto:Token: None 2024-04-13T21:40:10.706 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LFm09Bie+0qAIFHAuuX9PUCpQI0= 2024-04-13T21:40:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LFm09Bie+0qAIFHAuuX9PUCpQI0=', 'Content-Length': '0'} 2024-04-13T21:40:10.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad58a2e276fbbbf8-00661afbba-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.708 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:40:10.709 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.709 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.709 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.709 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.709 DEBUG:boto:Method: GET 2024-04-13T21:40:10.709 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.709 DEBUG:boto:Data: 2024-04-13T21:40:10.709 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.709 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.709 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.709 DEBUG:boto:Params: {} 2024-04-13T21:40:10.710 DEBUG:boto:Token: None 2024-04-13T21:40:10.710 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/?versions 2024-04-13T21:40:10.710 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds= 2024-04-13T21:40:10.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds=', 'Content-Length': '0'} 2024-04-13T21:40:10.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd26164caa6faf0f-00661afbba-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.712 DEBUG:boto:b'vwnfns-16999falseobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rItrue2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:10.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:10.713 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:40:10.713 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.713 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.714 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.714 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.714 DEBUG:boto:Method: GET 2024-04-13T21:40:10.714 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.714 DEBUG:boto:Data: 2024-04-13T21:40:10.714 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.714 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.715 DEBUG:boto:Params: {} 2024-04-13T21:40:10.715 DEBUG:boto:Token: None 2024-04-13T21:40:10.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/?versions 2024-04-13T21:40:10.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds= 2024-04-13T21:40:10.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds=', 'Content-Length': '0'} 2024-04-13T21:40:10.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a126342524fd1b-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.717 DEBUG:boto:b'vwnfns-16999falseobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rItrue2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:10.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:10.718 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.718 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.718 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.718 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.718 DEBUG:boto:Method: GET 2024-04-13T21:40:10.718 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.718 DEBUG:boto:Data: 2024-04-13T21:40:10.718 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.719 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.719 DEBUG:boto:Params: {} 2024-04-13T21:40:10.719 DEBUG:boto:Token: None 2024-04-13T21:40:10.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/?versions 2024-04-13T21:40:10.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds= 2024-04-13T21:40:10.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds=', 'Content-Length': '0'} 2024-04-13T21:40:10.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf62bb48d876499-00661afbba-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.722 DEBUG:boto:b'vwnfns-16999falseobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rItrue2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.722 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.722 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.723 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.723 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.723 DEBUG:boto:Method: GET 2024-04-13T21:40:10.723 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:10.723 DEBUG:boto:Data: 2024-04-13T21:40:10.723 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.723 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.723 DEBUG:boto:Params: {} 2024-04-13T21:40:10.724 DEBUG:boto:Token: None 2024-04-13T21:40:10.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/?versions 2024-04-13T21:40:10.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds= 2024-04-13T21:40:10.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eoVIWBOp/RADW0jgpFgH3fdobds=', 'Content-Length': '0'} 2024-04-13T21:40:10.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13bf94fcbed78a5-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.726 DEBUG:boto:b'vwnfns-16999falseobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rItrue2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-13T21:39:14.913Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:10.727 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.727 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.727 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.727 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.727 DEBUG:boto:Method: GET 2024-04-13T21:40:10.727 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.727 DEBUG:boto:Data: 2024-04-13T21:40:10.728 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.728 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.728 DEBUG:boto:Params: {} 2024-04-13T21:40:10.728 DEBUG:boto:Token: None 2024-04-13T21:40:10.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f76b3b3d64536833-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.731 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.731 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.732 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.732 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.732 DEBUG:boto:Method: GET 2024-04-13T21:40:10.732 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.732 DEBUG:boto:Data: 2024-04-13T21:40:10.732 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.732 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.732 DEBUG:boto:Params: {} 2024-04-13T21:40:10.733 DEBUG:boto:Token: None 2024-04-13T21:40:10.733 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.733 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000842c18ea458144c1-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.735 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.735 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.735 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.735 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.735 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.735 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.735 DEBUG:boto:Data: 2024-04-13T21:40:10.735 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.736 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.736 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.736 DEBUG:boto:Params: {} 2024-04-13T21:40:10.736 DEBUG:boto:Token: None 2024-04-13T21:40:10.736 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pN9kBPsHw/OKxlk+jaRmk0w+YZk= 2024-04-13T21:40:10.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pN9kBPsHw/OKxlk+jaRmk0w+YZk=', 'Content-Length': '0'} 2024-04-13T21:40:10.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054e94bcf6d3c762c-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.738 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.738 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.738 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.738 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.738 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.738 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.738 DEBUG:boto:Data: 2024-04-13T21:40:10.739 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.739 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.739 DEBUG:boto:Params: {} 2024-04-13T21:40:10.739 DEBUG:boto:Token: None 2024-04-13T21:40:10.739 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pN9kBPsHw/OKxlk+jaRmk0w+YZk= 2024-04-13T21:40:10.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pN9kBPsHw/OKxlk+jaRmk0w+YZk=', 'Content-Length': '0'} 2024-04-13T21:40:10.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1540b1aa3fc77d4-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:10.741 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.741 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.741 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.741 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.741 DEBUG:boto:Method: GET 2024-04-13T21:40:10.742 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.742 DEBUG:boto:Data: 2024-04-13T21:40:10.742 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.742 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.742 DEBUG:boto:Params: {} 2024-04-13T21:40:10.742 DEBUG:boto:Token: None 2024-04-13T21:40:10.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057fe8b75b0f312f7-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.744 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.744 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.744 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.744 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.745 DEBUG:boto:Method: GET 2024-04-13T21:40:10.745 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.745 DEBUG:boto:Data: 2024-04-13T21:40:10.745 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.745 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.745 DEBUG:boto:Params: {} 2024-04-13T21:40:10.745 DEBUG:boto:Token: None 2024-04-13T21:40:10.745 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.745 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cfa0886aaa3953be-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.747 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.747 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.747 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.747 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.747 DEBUG:boto:Method: GET 2024-04-13T21:40:10.747 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.748 DEBUG:boto:Data: 2024-04-13T21:40:10.748 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.748 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.748 DEBUG:boto:Params: {} 2024-04-13T21:40:10.748 DEBUG:boto:Token: None 2024-04-13T21:40:10.748 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90e9338f38f847c-00661afbba-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.794 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.795 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.795 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.795 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.795 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.795 DEBUG:boto:Method: GET 2024-04-13T21:40:10.795 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.796 DEBUG:boto:Data: 2024-04-13T21:40:10.796 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.796 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.796 DEBUG:boto:Params: {} 2024-04-13T21:40:10.796 DEBUG:boto:Token: None 2024-04-13T21:40:10.796 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab4cbdc2c94c1e9e-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.799 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.799 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.799 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.799 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.799 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:10.799 DEBUG:boto:Data: 2024-04-13T21:40:10.800 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.800 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.800 DEBUG:boto:Params: {} 2024-04-13T21:40:10.800 DEBUG:boto:Token: None 2024-04-13T21:40:10.800 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj 2024-04-13T21:40:10.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc= 2024-04-13T21:40:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc=', 'Content-Length': '0'} 2024-04-13T21:40:10.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c85775c02a8ca77c-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.802 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.802 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.802 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.802 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:10.802 DEBUG:boto:Data: 2024-04-13T21:40:10.803 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.803 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.803 DEBUG:boto:Params: {} 2024-04-13T21:40:10.803 DEBUG:boto:Token: None 2024-04-13T21:40:10.803 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj 2024-04-13T21:40:10.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc= 2024-04-13T21:40:10.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc=', 'Content-Length': '0'} 2024-04-13T21:40:10.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c5f4653f489bc38-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:10.805 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.805 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.805 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.805 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.805 DEBUG:boto:Method: GET 2024-04-13T21:40:10.805 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.806 DEBUG:boto:Data: 2024-04-13T21:40:10.806 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.806 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.806 DEBUG:boto:Params: {} 2024-04-13T21:40:10.806 DEBUG:boto:Token: None 2024-04-13T21:40:10.806 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.806 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3d6a0e8506548f5-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.808 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.808 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.808 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.808 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.808 DEBUG:boto:Method: GET 2024-04-13T21:40:10.808 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.808 DEBUG:boto:Data: 2024-04-13T21:40:10.809 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.809 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.809 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.809 DEBUG:boto:Params: {} 2024-04-13T21:40:10.809 DEBUG:boto:Token: None 2024-04-13T21:40:10.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f49ce3b4fe07ea1-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.811 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.811 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.811 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.811 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.812 DEBUG:boto:Method: GET 2024-04-13T21:40:10.812 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.812 DEBUG:boto:Data: 2024-04-13T21:40:10.812 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.812 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.812 DEBUG:boto:Params: {} 2024-04-13T21:40:10.812 DEBUG:boto:Token: None 2024-04-13T21:40:10.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672fc557c5cc83e0-00661afbba-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.814 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.815 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.815 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.815 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.815 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.815 DEBUG:boto:Method: GET 2024-04-13T21:40:10.815 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.815 DEBUG:boto:Data: 2024-04-13T21:40:10.816 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.816 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.816 DEBUG:boto:Params: {} 2024-04-13T21:40:10.816 DEBUG:boto:Token: None 2024-04-13T21:40:10.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1cf8d57db564b23-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.866 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:10.867 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.867 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.867 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.867 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.867 DEBUG:boto:Method: GET 2024-04-13T21:40:10.868 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.868 DEBUG:boto:Data: 2024-04-13T21:40:10.868 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.868 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.868 DEBUG:boto:Params: {} 2024-04-13T21:40:10.868 DEBUG:boto:Token: None 2024-04-13T21:40:10.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU= 2024-04-13T21:40:10.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU=', 'Content-Length': '0'} 2024-04-13T21:40:10.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001eb5bd206050ae0b-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.870 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.871 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.871 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.871 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.871 DEBUG:boto:Method: GET 2024-04-13T21:40:10.871 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.871 DEBUG:boto:Data: 2024-04-13T21:40:10.871 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.871 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.871 DEBUG:boto:Params: {} 2024-04-13T21:40:10.872 DEBUG:boto:Token: None 2024-04-13T21:40:10.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU= 2024-04-13T21:40:10.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU=', 'Content-Length': '0'} 2024-04-13T21:40:10.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fd07f38c35f4359-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.873 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.873 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.874 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.874 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.874 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.874 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.874 DEBUG:boto:Data: 2024-04-13T21:40:10.874 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.874 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.874 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.874 DEBUG:boto:Params: {} 2024-04-13T21:40:10.874 DEBUG:boto:Token: None 2024-04-13T21:40:10.875 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:glVThELyg1ihzKKYLnk29qwYXZE= 2024-04-13T21:40:10.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:glVThELyg1ihzKKYLnk29qwYXZE=', 'Content-Length': '0'} 2024-04-13T21:40:10.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061971467051e471e-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.876 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.876 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.877 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.877 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.877 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.877 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.877 DEBUG:boto:Data: 2024-04-13T21:40:10.877 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.877 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.877 DEBUG:boto:Params: {} 2024-04-13T21:40:10.877 DEBUG:boto:Token: None 2024-04-13T21:40:10.877 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:glVThELyg1ihzKKYLnk29qwYXZE= 2024-04-13T21:40:10.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:glVThELyg1ihzKKYLnk29qwYXZE=', 'Content-Length': '0'} 2024-04-13T21:40:10.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1a6861290c0fc42-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:10.879 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.879 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.879 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.880 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.880 DEBUG:boto:Method: GET 2024-04-13T21:40:10.880 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.880 DEBUG:boto:Data: 2024-04-13T21:40:10.880 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.880 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.880 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.880 DEBUG:boto:Params: {} 2024-04-13T21:40:10.880 DEBUG:boto:Token: None 2024-04-13T21:40:10.880 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU= 2024-04-13T21:40:10.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU=', 'Content-Length': '0'} 2024-04-13T21:40:10.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2dc3b0d85e452c4-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.882 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.882 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.882 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.882 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.883 DEBUG:boto:Method: GET 2024-04-13T21:40:10.883 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.883 DEBUG:boto:Data: 2024-04-13T21:40:10.883 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.883 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.883 DEBUG:boto:Params: {} 2024-04-13T21:40:10.883 DEBUG:boto:Token: None 2024-04-13T21:40:10.883 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:10.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU= 2024-04-13T21:40:10.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T1uEZfFGvLvBI7blFDv9XCxqwSU=', 'Content-Length': '0'} 2024-04-13T21:40:10.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c559062619e493e6-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.885 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.885 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.886 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.886 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.886 DEBUG:boto:Method: GET 2024-04-13T21:40:10.886 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.886 DEBUG:boto:Data: 2024-04-13T21:40:10.886 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.886 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.886 DEBUG:boto:Params: {} 2024-04-13T21:40:10.886 DEBUG:boto:Token: None 2024-04-13T21:40:10.886 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000597dabfc2d574ee1-00661afbba-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.889 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.889 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.889 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.889 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.889 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.890 DEBUG:boto:Method: GET 2024-04-13T21:40:10.890 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.890 DEBUG:boto:Data: 2024-04-13T21:40:10.890 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.890 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.890 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.890 DEBUG:boto:Params: {} 2024-04-13T21:40:10.890 DEBUG:boto:Token: None 2024-04-13T21:40:10.890 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.890 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b20626b11b9685f5-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.893 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.893 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.893 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.893 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.893 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:10.893 DEBUG:boto:Data: 2024-04-13T21:40:10.893 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.894 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.894 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.894 DEBUG:boto:Params: {} 2024-04-13T21:40:10.894 DEBUG:boto:Token: None 2024-04-13T21:40:10.894 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj 2024-04-13T21:40:10.894 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc= 2024-04-13T21:40:10.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc=', 'Content-Length': '0'} 2024-04-13T21:40:10.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a6d49615025e081-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.896 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.896 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.896 DEBUG:boto:Method: HEAD 2024-04-13T21:40:10.896 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:10.896 DEBUG:boto:Data: 2024-04-13T21:40:10.896 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.897 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.897 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.897 DEBUG:boto:Params: {} 2024-04-13T21:40:10.897 DEBUG:boto:Token: None 2024-04-13T21:40:10.897 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj 2024-04-13T21:40:10.897 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc= 2024-04-13T21:40:10.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P8nbe+ohtUQ6FdeG2RJKfqKSYCc=', 'Content-Length': '0'} 2024-04-13T21:40:10.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089b66400a3e78271-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:10.899 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.899 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.899 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.899 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.900 DEBUG:boto:Method: GET 2024-04-13T21:40:10.900 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.900 DEBUG:boto:Data: 2024-04-13T21:40:10.900 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.900 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.900 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.900 DEBUG:boto:Params: {} 2024-04-13T21:40:10.900 DEBUG:boto:Token: None 2024-04-13T21:40:10.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000251c5bc33d27f8ba-00661afbba-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.902 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:10.902 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:10.902 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.902 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.902 DEBUG:boto:Method: GET 2024-04-13T21:40:10.903 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.903 DEBUG:boto:Data: 2024-04-13T21:40:10.903 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.903 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.903 DEBUG:boto:Params: {} 2024-04-13T21:40:10.903 DEBUG:boto:Token: None 2024-04-13T21:40:10.903 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:10.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM= 2024-04-13T21:40:10.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHHYOiQmIFjXLYocBoBEn2Sm/RM=', 'Content-Length': '0'} 2024-04-13T21:40:10.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001eac686d9746dae9-00661afbba-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.905 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.905 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.906 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.906 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.906 DEBUG:boto:Method: GET 2024-04-13T21:40:10.906 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.906 DEBUG:boto:Data: 2024-04-13T21:40:10.906 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.906 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:10.906 DEBUG:boto:Port: 8001 2024-04-13T21:40:10.906 DEBUG:boto:Params: {} 2024-04-13T21:40:10.906 DEBUG:boto:Token: None 2024-04-13T21:40:10.907 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0620953c755de4-00661afbba-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.909 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.909 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.909 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.910 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.910 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:10.910 DEBUG:boto:Method: GET 2024-04-13T21:40:10.910 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:10.910 DEBUG:boto:Data: 2024-04-13T21:40:10.910 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.910 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.910 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.910 DEBUG:boto:Params: {} 2024-04-13T21:40:10.910 DEBUG:boto:Token: None 2024-04-13T21:40:10.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/ 2024-04-13T21:40:10.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs= 2024-04-13T21:40:10.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i1op6gecLwdHAfhZOo8CRouggUs=', 'Content-Length': '0'} 2024-04-13T21:40:10.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb6aa6f64082b29-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.954 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:10.955 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-16 zones={a2, a1} 2024-04-13T21:40:10.955 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:10.955 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:10.956 DEBUG:boto:path=/vwnfns-16/?versioning 2024-04-13T21:40:10.956 DEBUG:boto:auth_path=/vwnfns-16/?versioning 2024-04-13T21:40:10.956 DEBUG:boto:Method: PUT 2024-04-13T21:40:10.956 DEBUG:boto:Path: /vwnfns-16/?versioning 2024-04-13T21:40:10.956 DEBUG:boto:Data: Suspended Disabled 2024-04-13T21:40:10.956 DEBUG:boto:Headers: {} 2024-04-13T21:40:10.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:10.956 DEBUG:boto:Port: 8000 2024-04-13T21:40:10.956 DEBUG:boto:Params: {} 2024-04-13T21:40:10.957 DEBUG:boto:Token: None 2024-04-13T21:40:10.957 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:40:10 GMT /vwnfns-16/?versioning 2024-04-13T21:40:10.957 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N4UCiHkjxr4dCgn+kKJYqg6FwL0= 2024-04-13T21:40:10.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N4UCiHkjxr4dCgn+kKJYqg6FwL0=', 'Content-Length': '230'} 2024-04-13T21:40:10.962 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d12e65f7b59a8718-00661afbba-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:10.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:10.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:10.962 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:40:11.009 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:11.009 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:11.038 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:40:11.050 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:40:11.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.050 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044410.970424_204.1", 2024-04-13T21:40:11.050 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:40:10.970424Z" 2024-04-13T21:40:11.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:11.051 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:40:11.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:11.052 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:40:11.053 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044410.970424_204.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:11.053 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:40:11.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:11.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:11.053 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:40:11.102 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:11.102 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:11.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:11.137+0000 7f75fabc3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:40:11.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:11.137+0000 7f75fabc3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:40:11.143 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:40:11.143 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:40:11.143 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:40:11.143 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:40:11.143 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044410.967855_203.1", 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:11.144 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:40:10.967855Z", 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:40:11.145 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.146 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:11.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:11.148 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:40:11Z" 2024-04-13T21:40:11.149 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:40:11.150 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044410.967855_203.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:40:10.967855Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:40:11Z" } 2024-04-13T21:40:11.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044410.967855_203.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:40:10.967855Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:40:11.151 DEBUG:rgw_multi.tests:log_status={0: '1_1713044410.970424_204.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:11.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044410.967855_203.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:11.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044410.970424_204.1 target=1_1713044410.967855_203.1 2024-04-13T21:40:16.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:16.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:16.152 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:40:16.208 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:16.209 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:16.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:16.246+0000 7fec72b30780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:40:16.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:16.247+0000 7fec72b30780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:40:16.252 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:40:16.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044410.970424_204.1", 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:40:10.970424Z", 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:16.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:16.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:16.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:40:16Z" 2024-04-13T21:40:16.258 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:40:16.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044410.970424_204.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:40:10.970424Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:40:16Z" } 2024-04-13T21:40:16.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044410.970424_204.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:40:10.970424Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:40:16.260 DEBUG:rgw_multi.tests:log_status={0: '1_1713044410.970424_204.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:16.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044410.970424_204.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:16.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:40:16.260 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:16.260 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:16.260 DEBUG:boto:Method: HEAD 2024-04-13T21:40:16.260 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:40:16.260 DEBUG:boto:Data: 2024-04-13T21:40:16.260 DEBUG:boto:Headers: {} 2024-04-13T21:40:16.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:16.261 DEBUG:boto:Port: 8000 2024-04-13T21:40:16.261 DEBUG:boto:Params: {} 2024-04-13T21:40:16.261 DEBUG:boto:Token: None 2024-04-13T21:40:16.261 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:16 GMT /vwnfns-16/ 2024-04-13T21:40:16.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HNbxUjwHB56WXm8gDU4tsdApSDI= 2024-04-13T21:40:16.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HNbxUjwHB56WXm8gDU4tsdApSDI=', 'Content-Length': '0'} 2024-04-13T21:40:16.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039d961bb75d9f155-00661afbc0-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:16.264 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:16.264 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:16.264 DEBUG:boto:Method: PUT 2024-04-13T21:40:16.264 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:16.264 DEBUG:boto:Data: 2024-04-13T21:40:16.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:40:16.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:16.265 DEBUG:boto:Port: 8000 2024-04-13T21:40:16.265 DEBUG:boto:Params: {} 2024-04-13T21:40:16.265 DEBUG:boto:Token: None 2024-04-13T21:40:16.265 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:40:16 GMT /vwnfns-16/obj 2024-04-13T21:40:16.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I4OCcvgzAEVdivELZ0XexUie2r0= 2024-04-13T21:40:16.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:40:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I4OCcvgzAEVdivELZ0XexUie2r0='} 2024-04-13T21:40:16.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006142c9544a55da49-00661afbc0-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:16.279 DEBUG:rgw_multi.tests:created null version id=None 2024-04-13T21:40:16.279 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-16 2024-04-13T21:40:16.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:16.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:16.280 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:40:16.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:16.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:16.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:16.377+0000 7f81dbf13780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:16.369 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:16.370 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.52.18, , ] 2024-04-13T21:40:21.370 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:21.378+0000 7f81dbf13780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:21.371 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:21.371 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.52.18, , ] 2024-04-13T21:40:26.374 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:26.380+0000 7f81dbf13780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:26.374 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.48.13, , ] 2024-04-13T21:40:26.374 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.52.18, , ] 2024-04-13T21:40:31.373 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:31.381+0000 7f81dbf13780 1 bucket sync caught up with source: 2024-04-13T21:40:31.373 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000005.52.18, , ] 2024-04-13T21:40:31.373 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.52.18, , ] 2024-04-13T21:40:31.373 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:31.381+0000 7f81dbf13780 0 bucket checkpoint complete 2024-04-13T21:40:31.390 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-16 2024-04-13T21:40:31.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:31.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:31.391 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:40:31.451 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:31.452 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:31.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:31.495+0000 7f3a1b1f2780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:31.487 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-04-13T21:40:31.487 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.46.18, , ] 2024-04-13T21:40:36.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:36.497+0000 7f3a1b1f2780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:36.488 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-04-13T21:40:36.488 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.46.18, , ] 2024-04-13T21:40:41.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:41.497+0000 7f3a1b1f2780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:41.490 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-04-13T21:40:41.490 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.46.18, , ] 2024-04-13T21:40:46.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:46.498+0000 7f3a1b1f2780 1 waiting for incremental sync to catch up: 2024-04-13T21:40:46.491 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-04-13T21:40:46.491 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.46.18, , ] 2024-04-13T21:40:51.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:51.499+0000 7f3a1b1f2780 1 bucket sync caught up with source: 2024-04-13T21:40:51.492 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000005.46.18, , ] 2024-04-13T21:40:51.492 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000005.46.18, , ] 2024-04-13T21:40:51.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:51.499+0000 7f3a1b1f2780 0 bucket checkpoint complete 2024-04-13T21:40:51.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-16 zones={a2, a1} 2024-04-13T21:40:51.508 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.508 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.508 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.508 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:40:51.508 DEBUG:boto:Data: 2024-04-13T21:40:51.508 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.509 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.509 DEBUG:boto:Params: {} 2024-04-13T21:40:51.509 DEBUG:boto:Token: None 2024-04-13T21:40:51.509 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W93fjYj0XQ7a5vmHkLBlnKKq7qM= 2024-04-13T21:40:51.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W93fjYj0XQ7a5vmHkLBlnKKq7qM=', 'Content-Length': '0'} 2024-04-13T21:40:51.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002af5d133351c411a-00661afbe3-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.513 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.513 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.513 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.513 DEBUG:boto:Path: /vwnfns-16/ 2024-04-13T21:40:51.513 DEBUG:boto:Data: 2024-04-13T21:40:51.513 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.513 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.513 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.513 DEBUG:boto:Params: {} 2024-04-13T21:40:51.513 DEBUG:boto:Token: None 2024-04-13T21:40:51.514 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.514 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W93fjYj0XQ7a5vmHkLBlnKKq7qM= 2024-04-13T21:40:51.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W93fjYj0XQ7a5vmHkLBlnKKq7qM=', 'Content-Length': '0'} 2024-04-13T21:40:51.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f898ca1a0b998aa5-00661afbe3-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:40:51.516 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.516 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.516 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.517 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.517 DEBUG:boto:Method: GET 2024-04-13T21:40:51.517 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.517 DEBUG:boto:Data: 2024-04-13T21:40:51.517 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.517 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.517 DEBUG:boto:Params: {} 2024-04-13T21:40:51.517 DEBUG:boto:Token: None 2024-04-13T21:40:51.517 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/?versions 2024-04-13T21:40:51.517 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E= 2024-04-13T21:40:51.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E=', 'Content-Length': '0'} 2024-04-13T21:40:51.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002035661d22b0ffb7-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.520 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rIfalse2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:51.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:51.521 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:40:51.521 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.521 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.521 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.521 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.521 DEBUG:boto:Method: GET 2024-04-13T21:40:51.521 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.522 DEBUG:boto:Data: 2024-04-13T21:40:51.522 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.522 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.522 DEBUG:boto:Params: {} 2024-04-13T21:40:51.522 DEBUG:boto:Token: None 2024-04-13T21:40:51.522 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/?versions 2024-04-13T21:40:51.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E= 2024-04-13T21:40:51.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E=', 'Content-Length': '0'} 2024-04-13T21:40:51.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ed48f12eebc558-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.525 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rIfalse2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:51.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:40:51.525 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.525 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.526 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.526 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.526 DEBUG:boto:Method: GET 2024-04-13T21:40:51.526 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.526 DEBUG:boto:Data: 2024-04-13T21:40:51.526 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.526 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.526 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.526 DEBUG:boto:Params: {} 2024-04-13T21:40:51.526 DEBUG:boto:Token: None 2024-04-13T21:40:51.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/?versions 2024-04-13T21:40:51.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E= 2024-04-13T21:40:51.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E=', 'Content-Length': '0'} 2024-04-13T21:40:51.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e79bcfd1b1f255-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.570 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rIfalse2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.571 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.571 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.571 DEBUG:boto:path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.571 DEBUG:boto:auth_path=/vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.571 DEBUG:boto:Method: GET 2024-04-13T21:40:51.571 DEBUG:boto:Path: /vwnfns-16/?versions&max-keys=999 2024-04-13T21:40:51.571 DEBUG:boto:Data: 2024-04-13T21:40:51.571 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.572 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.572 DEBUG:boto:Params: {} 2024-04-13T21:40:51.572 DEBUG:boto:Token: None 2024-04-13T21:40:51.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/?versions 2024-04-13T21:40:51.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E= 2024-04-13T21:40:51.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qj3GCQM0e5QlEIcl5SNkYnxdC9E=', 'Content-Length': '0'} 2024-04-13T21:40:51.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000841abeb7df555a96-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.575 DEBUG:boto:b'vwnfns-16999falseobjnulltrue2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyU-zxesXBLg9CfcJ-FyzgcuetUnb-rIfalse2024-04-13T21:39:50.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:51.576 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.576 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.576 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.576 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.576 DEBUG:boto:Method: GET 2024-04-13T21:40:51.576 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.576 DEBUG:boto:Data: 2024-04-13T21:40:51.576 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.576 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.576 DEBUG:boto:Params: {} 2024-04-13T21:40:51.577 DEBUG:boto:Token: None 2024-04-13T21:40:51.577 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.577 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0= 2024-04-13T21:40:51.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0=', 'Content-Length': '0'} 2024-04-13T21:40:51.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000804b52be211f740f-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.579 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.579 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.579 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.579 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.579 DEBUG:boto:Method: GET 2024-04-13T21:40:51.579 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.579 DEBUG:boto:Data: 2024-04-13T21:40:51.579 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.580 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.580 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.580 DEBUG:boto:Params: {} 2024-04-13T21:40:51.580 DEBUG:boto:Token: None 2024-04-13T21:40:51.580 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0= 2024-04-13T21:40:51.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0=', 'Content-Length': '0'} 2024-04-13T21:40:51.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000252adea2ae522be4-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.582 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.583 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.583 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.583 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.583 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.583 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.583 DEBUG:boto:Data: 2024-04-13T21:40:51.583 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.583 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.583 DEBUG:boto:Params: {} 2024-04-13T21:40:51.583 DEBUG:boto:Token: None 2024-04-13T21:40:51.584 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YhL2p6A1L94GNfB2oyIQfXCvwdE= 2024-04-13T21:40:51.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YhL2p6A1L94GNfB2oyIQfXCvwdE=', 'Content-Length': '0'} 2024-04-13T21:40:51.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ccf5864362b0bcbc-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.586 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.586 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.586 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.586 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.586 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.586 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.587 DEBUG:boto:Data: 2024-04-13T21:40:51.587 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.587 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.587 DEBUG:boto:Params: {} 2024-04-13T21:40:51.587 DEBUG:boto:Token: None 2024-04-13T21:40:51.587 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YhL2p6A1L94GNfB2oyIQfXCvwdE= 2024-04-13T21:40:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YhL2p6A1L94GNfB2oyIQfXCvwdE=', 'Content-Length': '0'} 2024-04-13T21:40:51.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c56616d8cd36bf0c-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:51.589 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.589 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.589 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.590 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.590 DEBUG:boto:Method: GET 2024-04-13T21:40:51.590 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.590 DEBUG:boto:Data: 2024-04-13T21:40:51.590 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.590 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.590 DEBUG:boto:Params: {} 2024-04-13T21:40:51.590 DEBUG:boto:Token: None 2024-04-13T21:40:51.590 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.591 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0= 2024-04-13T21:40:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0=', 'Content-Length': '0'} 2024-04-13T21:40:51.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008346b87322d67c70-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.592 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.593 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.593 DEBUG:boto:path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.593 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.593 DEBUG:boto:Method: GET 2024-04-13T21:40:51.593 DEBUG:boto:Path: /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.593 DEBUG:boto:Data: 2024-04-13T21:40:51.593 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.593 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.593 DEBUG:boto:Params: {} 2024-04-13T21:40:51.593 DEBUG:boto:Token: None 2024-04-13T21:40:51.594 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=null 2024-04-13T21:40:51.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0= 2024-04-13T21:40:51.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mSl9kaJZ0WEZK4woV9HKHtDaIE0=', 'Content-Length': '0'} 2024-04-13T21:40:51.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009237b672c679f7c4-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.597 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.597 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.597 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.597 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.598 DEBUG:boto:Method: GET 2024-04-13T21:40:51.598 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.598 DEBUG:boto:Data: 2024-04-13T21:40:51.598 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.598 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.598 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.598 DEBUG:boto:Params: {} 2024-04-13T21:40:51.598 DEBUG:boto:Token: None 2024-04-13T21:40:51.598 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.599 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ee884c28b269b22-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.602 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.603 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.603 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.603 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.603 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.603 DEBUG:boto:Method: GET 2024-04-13T21:40:51.603 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.603 DEBUG:boto:Data: 2024-04-13T21:40:51.603 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.603 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.604 DEBUG:boto:Params: {} 2024-04-13T21:40:51.604 DEBUG:boto:Token: None 2024-04-13T21:40:51.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000485338577de7b60-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.650 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.651 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.651 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.651 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.651 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.651 DEBUG:boto:Data: 2024-04-13T21:40:51.651 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.652 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.652 DEBUG:boto:Params: {} 2024-04-13T21:40:51.652 DEBUG:boto:Token: None 2024-04-13T21:40:51.652 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs= 2024-04-13T21:40:51.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs=', 'Content-Length': '0'} 2024-04-13T21:40:51.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007654f2882cabb694-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.654 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.654 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.654 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.655 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.655 DEBUG:boto:Data: 2024-04-13T21:40:51.655 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.655 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.655 DEBUG:boto:Params: {} 2024-04-13T21:40:51.655 DEBUG:boto:Token: None 2024-04-13T21:40:51.655 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs= 2024-04-13T21:40:51.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs=', 'Content-Length': '0'} 2024-04-13T21:40:51.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9d70e5c8486ec2c-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:51.657 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.657 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.658 DEBUG:boto:Method: GET 2024-04-13T21:40:51.658 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.658 DEBUG:boto:Data: 2024-04-13T21:40:51.658 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.658 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.658 DEBUG:boto:Params: {} 2024-04-13T21:40:51.658 DEBUG:boto:Token: None 2024-04-13T21:40:51.658 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.658 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E= 2024-04-13T21:40:51.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E=', 'Content-Length': '0'} 2024-04-13T21:40:51.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d3fbf31f34429b1-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.660 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.660 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.661 DEBUG:boto:Method: GET 2024-04-13T21:40:51.661 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.661 DEBUG:boto:Data: 2024-04-13T21:40:51.661 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.661 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.661 DEBUG:boto:Params: {} 2024-04-13T21:40:51.661 DEBUG:boto:Token: None 2024-04-13T21:40:51.661 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.661 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E= 2024-04-13T21:40:51.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E=', 'Content-Length': '0'} 2024-04-13T21:40:51.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b6592f84592b79e-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.663 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.663 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.664 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.664 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.664 DEBUG:boto:Method: GET 2024-04-13T21:40:51.664 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.664 DEBUG:boto:Data: 2024-04-13T21:40:51.664 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.664 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.664 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.664 DEBUG:boto:Params: {} 2024-04-13T21:40:51.664 DEBUG:boto:Token: None 2024-04-13T21:40:51.665 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.665 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000689da54fe92b6cad-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.667 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.667 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.668 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.668 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.668 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.668 DEBUG:boto:Method: GET 2024-04-13T21:40:51.668 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.668 DEBUG:boto:Data: 2024-04-13T21:40:51.668 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.668 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.668 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.668 DEBUG:boto:Params: {} 2024-04-13T21:40:51.668 DEBUG:boto:Token: None 2024-04-13T21:40:51.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021da88d21d2fa8b9-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.671 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:51.671 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.672 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.672 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.672 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.672 DEBUG:boto:Method: GET 2024-04-13T21:40:51.672 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.672 DEBUG:boto:Data: 2024-04-13T21:40:51.672 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.672 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.672 DEBUG:boto:Params: {} 2024-04-13T21:40:51.673 DEBUG:boto:Token: None 2024-04-13T21:40:51.673 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.673 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8= 2024-04-13T21:40:51.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8=', 'Content-Length': '0'} 2024-04-13T21:40:51.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a4d5d67312a59a4-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.675 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.675 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.675 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.675 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.675 DEBUG:boto:Method: GET 2024-04-13T21:40:51.676 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.676 DEBUG:boto:Data: 2024-04-13T21:40:51.676 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.676 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.676 DEBUG:boto:Params: {} 2024-04-13T21:40:51.676 DEBUG:boto:Token: None 2024-04-13T21:40:51.676 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8= 2024-04-13T21:40:51.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8=', 'Content-Length': '0'} 2024-04-13T21:40:51.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e62ecd6a7cb3e70-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.678 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.678 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.679 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.679 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.679 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.679 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.679 DEBUG:boto:Data: 2024-04-13T21:40:51.679 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.679 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.679 DEBUG:boto:Params: {} 2024-04-13T21:40:51.679 DEBUG:boto:Token: None 2024-04-13T21:40:51.679 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qMg4qwYl/PZb6OFjYUCl1rwrSMk= 2024-04-13T21:40:51.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qMg4qwYl/PZb6OFjYUCl1rwrSMk=', 'Content-Length': '0'} 2024-04-13T21:40:51.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbf969e9774eee97-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.681 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.682 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.682 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.682 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.682 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.682 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.682 DEBUG:boto:Data: 2024-04-13T21:40:51.682 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.682 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.682 DEBUG:boto:Params: {} 2024-04-13T21:40:51.683 DEBUG:boto:Token: None 2024-04-13T21:40:51.683 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.683 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qMg4qwYl/PZb6OFjYUCl1rwrSMk= 2024-04-13T21:40:51.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qMg4qwYl/PZb6OFjYUCl1rwrSMk=', 'Content-Length': '0'} 2024-04-13T21:40:51.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a71c8082e402e778-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:51.685 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.685 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.685 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.685 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.685 DEBUG:boto:Method: GET 2024-04-13T21:40:51.685 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.686 DEBUG:boto:Data: 2024-04-13T21:40:51.686 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.686 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.686 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.686 DEBUG:boto:Params: {} 2024-04-13T21:40:51.686 DEBUG:boto:Token: None 2024-04-13T21:40:51.686 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.686 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8= 2024-04-13T21:40:51.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8=', 'Content-Length': '0'} 2024-04-13T21:40:51.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001befebb958fa6e6d-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.688 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.688 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.689 DEBUG:boto:path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.689 DEBUG:boto:auth_path=/vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.689 DEBUG:boto:Method: GET 2024-04-13T21:40:51.689 DEBUG:boto:Path: /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.689 DEBUG:boto:Data: 2024-04-13T21:40:51.689 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.689 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.689 DEBUG:boto:Params: {} 2024-04-13T21:40:51.689 DEBUG:boto:Token: None 2024-04-13T21:40:51.689 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj?versionId=yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI 2024-04-13T21:40:51.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8= 2024-04-13T21:40:51.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y+frN+97WjxGmUmbQyF+SjB+wh8=', 'Content-Length': '0'} 2024-04-13T21:40:51.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:39:50 GMT'), ('x-amz-version-id', 'yU-zxesXBLg9CfcJ-FyzgcuetUnb-rI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b56c8322787b56c1-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.692 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.692 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.692 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.692 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.692 DEBUG:boto:Method: GET 2024-04-13T21:40:51.692 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.692 DEBUG:boto:Data: 2024-04-13T21:40:51.692 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.692 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.692 DEBUG:boto:Params: {} 2024-04-13T21:40:51.693 DEBUG:boto:Token: None 2024-04-13T21:40:51.693 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.693 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000799bf57175afcc4a-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.738 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.739 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.739 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.739 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.739 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.739 DEBUG:boto:Method: GET 2024-04-13T21:40:51.739 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.739 DEBUG:boto:Data: 2024-04-13T21:40:51.740 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.740 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.740 DEBUG:boto:Params: {} 2024-04-13T21:40:51.740 DEBUG:boto:Token: None 2024-04-13T21:40:51.740 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce4e037b8e639f2-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.743 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.743 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.743 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.743 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.744 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.744 DEBUG:boto:Data: 2024-04-13T21:40:51.744 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.744 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.744 DEBUG:boto:Params: {} 2024-04-13T21:40:51.744 DEBUG:boto:Token: None 2024-04-13T21:40:51.744 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs= 2024-04-13T21:40:51.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs=', 'Content-Length': '0'} 2024-04-13T21:40:51.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c565e0c2b179dab0-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.746 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.746 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.747 DEBUG:boto:Method: HEAD 2024-04-13T21:40:51.747 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.747 DEBUG:boto:Data: 2024-04-13T21:40:51.747 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.747 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.747 DEBUG:boto:Params: {} 2024-04-13T21:40:51.747 DEBUG:boto:Token: None 2024-04-13T21:40:51.747 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs= 2024-04-13T21:40:51.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YLboq3U2GU0LWARCcovf4ay9sZs=', 'Content-Length': '0'} 2024-04-13T21:40:51.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000305d625fc45644a8-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:40:51.749 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.750 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.750 DEBUG:boto:Method: GET 2024-04-13T21:40:51.750 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.750 DEBUG:boto:Data: 2024-04-13T21:40:51.750 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.750 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.750 DEBUG:boto:Params: {} 2024-04-13T21:40:51.750 DEBUG:boto:Token: None 2024-04-13T21:40:51.751 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E= 2024-04-13T21:40:51.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E=', 'Content-Length': '0'} 2024-04-13T21:40:51.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b4a863f863fb7af-00661afbe3-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.753 DEBUG:boto:path=/vwnfns-16/obj 2024-04-13T21:40:51.753 DEBUG:boto:auth_path=/vwnfns-16/obj 2024-04-13T21:40:51.753 DEBUG:boto:Method: GET 2024-04-13T21:40:51.753 DEBUG:boto:Path: /vwnfns-16/obj 2024-04-13T21:40:51.753 DEBUG:boto:Data: 2024-04-13T21:40:51.753 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.753 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.753 DEBUG:boto:Params: {} 2024-04-13T21:40:51.753 DEBUG:boto:Token: None 2024-04-13T21:40:51.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/obj 2024-04-13T21:40:51.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E= 2024-04-13T21:40:51.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kRHnHp10nkr+y3/UaQ5lHdB/09E=', 'Content-Length': '0'} 2024-04-13T21:40:51.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006355852e79e0f913-00661afbe3-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.756 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.756 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.756 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.756 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.756 DEBUG:boto:Method: GET 2024-04-13T21:40:51.756 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.757 DEBUG:boto:Data: 2024-04-13T21:40:51.757 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.757 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.757 DEBUG:boto:Params: {} 2024-04-13T21:40:51.757 DEBUG:boto:Token: None 2024-04-13T21:40:51.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.757 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c523d3e251dc2cd-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.760 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.760 DEBUG:boto:path=/vwnfns-16/ 2024-04-13T21:40:51.760 DEBUG:boto:auth_path=/vwnfns-16/ 2024-04-13T21:40:51.760 DEBUG:boto:path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.760 DEBUG:boto:auth_path=/vwnfns-16/?prefix=obj 2024-04-13T21:40:51.760 DEBUG:boto:Method: GET 2024-04-13T21:40:51.760 DEBUG:boto:Path: /vwnfns-16/?prefix=obj 2024-04-13T21:40:51.761 DEBUG:boto:Data: 2024-04-13T21:40:51.761 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.761 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.761 DEBUG:boto:Params: {} 2024-04-13T21:40:51.761 DEBUG:boto:Token: None 2024-04-13T21:40:51.761 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-16/ 2024-04-13T21:40:51.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI= 2024-04-13T21:40:51.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gQONace7XFQXhtH+3Xg3AvVEtpI=', 'Content-Length': '0'} 2024-04-13T21:40:51.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f26afafad2aadf-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.764 DEBUG:boto:b'vwnfns-16obj1000falseobj2024-04-13T21:40:16.276Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:40:51.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-16 zones={a2, a1} 2024-04-13T21:40:51.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-04-13T21:40:51.766 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-17 2024-04-13T21:40:51.766 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:40:51.766 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:40:51.766 DEBUG:boto:Method: PUT 2024-04-13T21:40:51.766 DEBUG:boto:Path: /vwnfns-17/ 2024-04-13T21:40:51.766 DEBUG:boto:Data: 2024-04-13T21:40:51.766 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.766 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.767 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.767 DEBUG:boto:Params: {} 2024-04-13T21:40:51.767 DEBUG:boto:Token: None 2024-04-13T21:40:51.767 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-17/ 2024-04-13T21:40:51.767 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5RAd3PkwnRBCNmjy3lHp6vk/4EI= 2024-04-13T21:40:51.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5RAd3PkwnRBCNmjy3lHp6vk/4EI=', 'Content-Length': '0'} 2024-04-13T21:40:51.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d67ec66c60324d5e-00661afbe3-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.781 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-18 2024-04-13T21:40:51.782 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:40:51.782 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:40:51.782 DEBUG:boto:Method: PUT 2024-04-13T21:40:51.782 DEBUG:boto:Path: /vwnfns-18/ 2024-04-13T21:40:51.782 DEBUG:boto:Data: 2024-04-13T21:40:51.782 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.782 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.782 DEBUG:boto:Params: {} 2024-04-13T21:40:51.782 DEBUG:boto:Token: None 2024-04-13T21:40:51.783 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-18/ 2024-04-13T21:40:51.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fiyGJzmkxcIioC3gR0E7SsAylOg= 2024-04-13T21:40:51.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fiyGJzmkxcIioC3gR0E7SsAylOg=', 'Content-Length': '0'} 2024-04-13T21:40:51.812 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004383cdb0765c125e-00661afbe3-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.812 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:40:51.812 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:40:51.812 DEBUG:boto:path=/vwnfns-17/?versioning 2024-04-13T21:40:51.812 DEBUG:boto:auth_path=/vwnfns-17/?versioning 2024-04-13T21:40:51.812 DEBUG:boto:Method: PUT 2024-04-13T21:40:51.812 DEBUG:boto:Path: /vwnfns-17/?versioning 2024-04-13T21:40:51.812 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:40:51.813 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.813 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:51.813 DEBUG:boto:Port: 8000 2024-04-13T21:40:51.813 DEBUG:boto:Params: {} 2024-04-13T21:40:51.813 DEBUG:boto:Token: None 2024-04-13T21:40:51.813 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-17/?versioning 2024-04-13T21:40:51.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FPoX/KJQgDGzWnjrbunrmYyz05A= 2024-04-13T21:40:51.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FPoX/KJQgDGzWnjrbunrmYyz05A=', 'Content-Length': '228'} 2024-04-13T21:40:51.818 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000059487035605d3224-00661afbe3-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.818 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:40:51.818 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:40:51.818 DEBUG:boto:path=/vwnfns-18/?versioning 2024-04-13T21:40:51.818 DEBUG:boto:auth_path=/vwnfns-18/?versioning 2024-04-13T21:40:51.819 DEBUG:boto:Method: PUT 2024-04-13T21:40:51.819 DEBUG:boto:Path: /vwnfns-18/?versioning 2024-04-13T21:40:51.819 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:40:51.819 DEBUG:boto:Headers: {} 2024-04-13T21:40:51.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:51.819 DEBUG:boto:Port: 8001 2024-04-13T21:40:51.819 DEBUG:boto:Params: {} 2024-04-13T21:40:51.819 DEBUG:boto:Token: None 2024-04-13T21:40:51.819 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:40:51 GMT /vwnfns-18/?versioning 2024-04-13T21:40:51.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XGI7MKGbwcV0Fu75AvF9gP06hh8= 2024-04-13T21:40:51.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XGI7MKGbwcV0Fu75AvF9gP06hh8=', 'Content-Length': '228'} 2024-04-13T21:40:51.827 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000caa7e4aafe8efaca-00661afbe3-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:51.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:51.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:51.827 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:40:51.876 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:51.877 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:51.907 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:40:51.919 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:40:51.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:51.919 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044451.833599_218.1", 2024-04-13T21:40:51.919 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:40:51.833599Z" 2024-04-13T21:40:51.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:51.920 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:40:51.921 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:51.921 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:40:51.922 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044451.833599_218.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:51.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:40:51.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:51.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:40:51.923 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:40:51.976 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:51.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:52.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:52.012+0000 7fc4d83bb780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:40:52.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:52.013+0000 7fc4d83bb780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:40:52.017 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:40:52.017 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:40:52.017 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:40:52.017 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:52.018 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044451.833599_218.1", 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:40:51.833599Z", 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:52.019 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:52.020 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:40:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:40:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:52.023 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:40:52.023 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:40:52.023 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:40:52.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:40:52.023 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:40:52Z" 2024-04-13T21:40:52.023 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:40:52.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044451.833599_218.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:40:51.833599Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:40:52Z" } 2024-04-13T21:40:52.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044451.833599_218.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:40:51.833599Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:40:52.025 DEBUG:rgw_multi.tests:log_status={0: '1_1713044451.833599_218.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:52.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044451.833599_218.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:40:52.025 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:40:52.025 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:40:52.025 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:40:52.025 DEBUG:boto:Method: HEAD 2024-04-13T21:40:52.025 DEBUG:boto:Path: /vwnfns-17/ 2024-04-13T21:40:52.025 DEBUG:boto:Data: 2024-04-13T21:40:52.025 DEBUG:boto:Headers: {} 2024-04-13T21:40:52.025 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:52.026 DEBUG:boto:Port: 8000 2024-04-13T21:40:52.026 DEBUG:boto:Params: {} 2024-04-13T21:40:52.026 DEBUG:boto:Token: None 2024-04-13T21:40:52.026 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-17/ 2024-04-13T21:40:52.026 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8y16/soI4xq7Y213ZUMTmy9bOD0= 2024-04-13T21:40:52.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8y16/soI4xq7Y213ZUMTmy9bOD0=', 'Content-Length': '0'} 2024-04-13T21:40:52.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ccd87ca5fe62ed1-00661afbe4-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.029 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:40:52.029 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:40:52.029 DEBUG:boto:Method: PUT 2024-04-13T21:40:52.029 DEBUG:boto:Path: /vwnfns-17/obj 2024-04-13T21:40:52.029 DEBUG:boto:Data: 2024-04-13T21:40:52.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:40:52.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:52.029 DEBUG:boto:Port: 8000 2024-04-13T21:40:52.030 DEBUG:boto:Params: {} 2024-04-13T21:40:52.030 DEBUG:boto:Token: None 2024-04-13T21:40:52.030 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-17/obj 2024-04-13T21:40:52.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bUTi6j7+omwfEXGMziF10AhqI/I= 2024-04-13T21:40:52.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bUTi6j7+omwfEXGMziF10AhqI/I='} 2024-04-13T21:40:52.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-amz-request-id', 'tx000008ccb2b6729f29cbb-00661afbe4-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.040 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:40:52.040 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:40:52.040 DEBUG:boto:Method: HEAD 2024-04-13T21:40:52.040 DEBUG:boto:Path: /vwnfns-17/ 2024-04-13T21:40:52.040 DEBUG:boto:Data: 2024-04-13T21:40:52.040 DEBUG:boto:Headers: {} 2024-04-13T21:40:52.040 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:52.040 DEBUG:boto:Port: 8000 2024-04-13T21:40:52.041 DEBUG:boto:Params: {} 2024-04-13T21:40:52.041 DEBUG:boto:Token: None 2024-04-13T21:40:52.041 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-17/ 2024-04-13T21:40:52.041 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8y16/soI4xq7Y213ZUMTmy9bOD0= 2024-04-13T21:40:52.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8y16/soI4xq7Y213ZUMTmy9bOD0=', 'Content-Length': '0'} 2024-04-13T21:40:52.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8c7088242e607f2-00661afbe4-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.043 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:40:52.043 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:40:52.043 DEBUG:boto:Method: DELETE 2024-04-13T21:40:52.043 DEBUG:boto:Path: /vwnfns-17/obj 2024-04-13T21:40:52.043 DEBUG:boto:Data: 2024-04-13T21:40:52.043 DEBUG:boto:Headers: {} 2024-04-13T21:40:52.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:40:52.044 DEBUG:boto:Port: 8000 2024-04-13T21:40:52.044 DEBUG:boto:Params: {} 2024-04-13T21:40:52.044 DEBUG:boto:Token: None 2024-04-13T21:40:52.044 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-17/obj 2024-04-13T21:40:52.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tjYxOS2/Mst3ptxkI3outruBVC0= 2024-04-13T21:40:52.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tjYxOS2/Mst3ptxkI3outruBVC0=', 'Content-Length': '0'} 2024-04-13T21:40:52.050 DEBUG:boto:Response headers: [('x-amz-version-id', '-Bv0b6vQEzaOn7JQp4ooa8whgpRqpwE'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000064bdb0152ff154a5-00661afbe4-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.050 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:40:52.050 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:40:52.050 DEBUG:boto:Method: HEAD 2024-04-13T21:40:52.050 DEBUG:boto:Path: /vwnfns-18/ 2024-04-13T21:40:52.050 DEBUG:boto:Data: 2024-04-13T21:40:52.051 DEBUG:boto:Headers: {} 2024-04-13T21:40:52.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:52.051 DEBUG:boto:Port: 8001 2024-04-13T21:40:52.051 DEBUG:boto:Params: {} 2024-04-13T21:40:52.051 DEBUG:boto:Token: None 2024-04-13T21:40:52.051 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-18/ 2024-04-13T21:40:52.051 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L7spAiEAnmzj5zVyJTYEokfa4p0= 2024-04-13T21:40:52.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L7spAiEAnmzj5zVyJTYEokfa4p0=', 'Content-Length': '0'} 2024-04-13T21:40:52.053 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027b8f98da32c3771-00661afbe4-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.053 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:40:52.053 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:40:52.054 DEBUG:boto:Method: PUT 2024-04-13T21:40:52.054 DEBUG:boto:Path: /vwnfns-18/obj 2024-04-13T21:40:52.054 DEBUG:boto:Data: 2024-04-13T21:40:52.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:40:52.054 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:52.054 DEBUG:boto:Port: 8001 2024-04-13T21:40:52.054 DEBUG:boto:Params: {} 2024-04-13T21:40:52.054 DEBUG:boto:Token: None 2024-04-13T21:40:52.054 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-18/obj 2024-04-13T21:40:52.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UwxxJ7jhBWcNmioF99YL2paPQaE= 2024-04-13T21:40:52.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UwxxJ7jhBWcNmioF99YL2paPQaE='} 2024-04-13T21:40:52.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-amz-request-id', 'tx0000040ee125b98376a25-00661afbe4-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.064 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:40:52.064 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:40:52.064 DEBUG:boto:Method: HEAD 2024-04-13T21:40:52.064 DEBUG:boto:Path: /vwnfns-18/ 2024-04-13T21:40:52.065 DEBUG:boto:Data: 2024-04-13T21:40:52.065 DEBUG:boto:Headers: {} 2024-04-13T21:40:52.065 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:52.065 DEBUG:boto:Port: 8001 2024-04-13T21:40:52.065 DEBUG:boto:Params: {} 2024-04-13T21:40:52.065 DEBUG:boto:Token: None 2024-04-13T21:40:52.065 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-18/ 2024-04-13T21:40:52.065 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L7spAiEAnmzj5zVyJTYEokfa4p0= 2024-04-13T21:40:52.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L7spAiEAnmzj5zVyJTYEokfa4p0=', 'Content-Length': '0'} 2024-04-13T21:40:52.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f537c1766a29922-00661afbe4-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.068 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:40:52.068 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:40:52.068 DEBUG:boto:Method: DELETE 2024-04-13T21:40:52.068 DEBUG:boto:Path: /vwnfns-18/obj 2024-04-13T21:40:52.068 DEBUG:boto:Data: 2024-04-13T21:40:52.068 DEBUG:boto:Headers: {} 2024-04-13T21:40:52.068 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:40:52.068 DEBUG:boto:Port: 8001 2024-04-13T21:40:52.068 DEBUG:boto:Params: {} 2024-04-13T21:40:52.069 DEBUG:boto:Token: None 2024-04-13T21:40:52.069 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:40:52 GMT /vwnfns-18/obj 2024-04-13T21:40:52.069 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vYAM+BZqqdmd/k8zrNt8MEvfpes= 2024-04-13T21:40:52.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:40:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vYAM+BZqqdmd/k8zrNt8MEvfpes=', 'Content-Length': '0'} 2024-04-13T21:40:52.075 DEBUG:boto:Response headers: [('x-amz-version-id', 'Hu8T.a0H-nxIqbRJVRatdOyQsPmCFtQ'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000026cd7c7cde1a5c95-00661afbe4-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:40:52.075 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-17 2024-04-13T21:40:52.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:52.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:40:52.076 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:40:52.130 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:40:52.131 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:40:52.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:52.171+0000 7ff6fcc8e780 1 waiting to reach incremental sync.. 2024-04-13T21:40:57.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:40:57.172+0000 7ff6fcc8e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:02.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:02.173+0000 7ff6fcc8e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:07.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:07.174+0000 7ff6fcc8e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:12.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:12.175+0000 7ff6fcc8e780 1 bucket sync caught up with source: 2024-04-13T21:41:12.167 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.6.12, , ] 2024-04-13T21:41:12.167 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.6.12, , ] 2024-04-13T21:41:12.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:12.175+0000 7ff6fcc8e780 0 bucket checkpoint complete 2024-04-13T21:41:12.184 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-17 2024-04-13T21:41:12.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:41:12.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:41:12.184 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:41:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:41:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:41:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:12.288+0000 7f108dd2e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:17.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:17.289+0000 7f108dd2e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:22.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:22.288+0000 7f108dd2e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:27.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:27.289+0000 7f108dd2e780 1 waiting to reach incremental sync.. 2024-04-13T21:41:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:32.291+0000 7f108dd2e780 1 bucket sync caught up with source: 2024-04-13T21:41:32.284 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000002.8.6, , ] 2024-04-13T21:41:32.284 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000002.8.6, , ] 2024-04-13T21:41:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:32.291+0000 7f108dd2e780 0 bucket checkpoint complete 2024-04-13T21:41:32.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-17 zones={a2, a1} 2024-04-13T21:41:32.299 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.300 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.300 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.300 DEBUG:boto:Path: /vwnfns-17/ 2024-04-13T21:41:32.300 DEBUG:boto:Data: 2024-04-13T21:41:32.300 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.300 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.300 DEBUG:boto:Params: {} 2024-04-13T21:41:32.301 DEBUG:boto:Token: None 2024-04-13T21:41:32.301 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/ 2024-04-13T21:41:32.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ISNuCiPiwEr6YlkfiHe0f48yCl8= 2024-04-13T21:41:32.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ISNuCiPiwEr6YlkfiHe0f48yCl8=', 'Content-Length': '0'} 2024-04-13T21:41:32.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efe39fae52863c24-00661afc0c-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.304 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.305 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.305 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.305 DEBUG:boto:Path: /vwnfns-17/ 2024-04-13T21:41:32.305 DEBUG:boto:Data: 2024-04-13T21:41:32.305 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.305 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.305 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.305 DEBUG:boto:Params: {} 2024-04-13T21:41:32.305 DEBUG:boto:Token: None 2024-04-13T21:41:32.305 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/ 2024-04-13T21:41:32.306 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ISNuCiPiwEr6YlkfiHe0f48yCl8= 2024-04-13T21:41:32.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ISNuCiPiwEr6YlkfiHe0f48yCl8=', 'Content-Length': '0'} 2024-04-13T21:41:32.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000307f3b6ca03067e7-00661afc0c-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:41:32.308 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.308 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.309 DEBUG:boto:path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.309 DEBUG:boto:auth_path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.309 DEBUG:boto:Method: GET 2024-04-13T21:41:32.309 DEBUG:boto:Path: /vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.309 DEBUG:boto:Data: 2024-04-13T21:41:32.309 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.309 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.309 DEBUG:boto:Params: {} 2024-04-13T21:41:32.309 DEBUG:boto:Token: None 2024-04-13T21:41:32.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/?versions 2024-04-13T21:41:32.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE= 2024-04-13T21:41:32.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE=', 'Content-Length': '0'} 2024-04-13T21:41:32.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94c68216962ad90-00661afc0c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.358 DEBUG:boto:b'vwnfns-17999falseobj-Bv0b6vQEzaOn7JQp4ooa8whgpRqpwEtrue2024-04-13T21:40:52.055Zrgw-multisite-test-userMultisite Test UserNormalobjkeFf3uL6NbYooyB1XG3.9ovq2kVonl1false2024-04-13T21:40:52.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:41:32.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:41:32.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:41:32.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:41:32.360 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.360 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.360 DEBUG:boto:path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.360 DEBUG:boto:auth_path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.360 DEBUG:boto:Method: GET 2024-04-13T21:41:32.360 DEBUG:boto:Path: /vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.360 DEBUG:boto:Data: 2024-04-13T21:41:32.360 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.360 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.360 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.361 DEBUG:boto:Params: {} 2024-04-13T21:41:32.361 DEBUG:boto:Token: None 2024-04-13T21:41:32.361 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/?versions 2024-04-13T21:41:32.361 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE= 2024-04-13T21:41:32.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE=', 'Content-Length': '0'} 2024-04-13T21:41:32.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dcde9b8bc2103c3-00661afc0c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.364 DEBUG:boto:b'vwnfns-17999falseobj-Bv0b6vQEzaOn7JQp4ooa8whgpRqpwEtrue2024-04-13T21:40:52.055Zrgw-multisite-test-userMultisite Test UserNormalobjkeFf3uL6NbYooyB1XG3.9ovq2kVonl1false2024-04-13T21:40:52.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:41:32.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:41:32.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:41:32.364 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.364 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.364 DEBUG:boto:path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.364 DEBUG:boto:auth_path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.365 DEBUG:boto:Method: GET 2024-04-13T21:41:32.365 DEBUG:boto:Path: /vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.365 DEBUG:boto:Data: 2024-04-13T21:41:32.365 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.365 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.365 DEBUG:boto:Params: {} 2024-04-13T21:41:32.366 DEBUG:boto:Token: None 2024-04-13T21:41:32.366 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/?versions 2024-04-13T21:41:32.366 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE= 2024-04-13T21:41:32.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE=', 'Content-Length': '0'} 2024-04-13T21:41:32.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa4f2d82df68262-00661afc0c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.368 DEBUG:boto:b'vwnfns-17999falseobj-Bv0b6vQEzaOn7JQp4ooa8whgpRqpwEtrue2024-04-13T21:40:52.055Zrgw-multisite-test-userMultisite Test UserNormalobjkeFf3uL6NbYooyB1XG3.9ovq2kVonl1false2024-04-13T21:40:52.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:41:32.369 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.369 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.369 DEBUG:boto:path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.369 DEBUG:boto:auth_path=/vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.369 DEBUG:boto:Method: GET 2024-04-13T21:41:32.369 DEBUG:boto:Path: /vwnfns-17/?versions&max-keys=999 2024-04-13T21:41:32.370 DEBUG:boto:Data: 2024-04-13T21:41:32.370 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.370 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.370 DEBUG:boto:Params: {} 2024-04-13T21:41:32.370 DEBUG:boto:Token: None 2024-04-13T21:41:32.370 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/?versions 2024-04-13T21:41:32.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE= 2024-04-13T21:41:32.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bQa30J6v62gabEqZVQ8VAsCznwE=', 'Content-Length': '0'} 2024-04-13T21:41:32.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033bcf9292820b937-00661afc0c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.414 DEBUG:boto:b'vwnfns-17999falseobj-Bv0b6vQEzaOn7JQp4ooa8whgpRqpwEtrue2024-04-13T21:40:52.055Zrgw-multisite-test-userMultisite Test UserNormalobjkeFf3uL6NbYooyB1XG3.9ovq2kVonl1false2024-04-13T21:40:52.041Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:41:32.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:41:32.415 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.415 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.415 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.416 DEBUG:boto:Path: /vwnfns-17/obj 2024-04-13T21:41:32.416 DEBUG:boto:Data: 2024-04-13T21:41:32.416 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.416 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.416 DEBUG:boto:Params: {} 2024-04-13T21:41:32.416 DEBUG:boto:Token: None 2024-04-13T21:41:32.416 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj 2024-04-13T21:41:32.417 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE= 2024-04-13T21:41:32.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE=', 'Content-Length': '0'} 2024-04-13T21:41:32.419 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000247346280639683f-00661afc0c-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.419 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.420 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.420 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.420 DEBUG:boto:Path: /vwnfns-17/obj 2024-04-13T21:41:32.420 DEBUG:boto:Data: 2024-04-13T21:41:32.420 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.420 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.420 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.420 DEBUG:boto:Params: {} 2024-04-13T21:41:32.420 DEBUG:boto:Token: None 2024-04-13T21:41:32.420 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj 2024-04-13T21:41:32.421 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE= 2024-04-13T21:41:32.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE=', 'Content-Length': '0'} 2024-04-13T21:41:32.423 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fb3cd646413fd7b8-00661afc0c-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:41:32.423 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.423 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.423 DEBUG:boto:path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.423 DEBUG:boto:auth_path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.423 DEBUG:boto:Method: GET 2024-04-13T21:41:32.424 DEBUG:boto:Path: /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.424 DEBUG:boto:Data: 2024-04-13T21:41:32.424 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.424 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.424 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.424 DEBUG:boto:Params: {} 2024-04-13T21:41:32.424 DEBUG:boto:Token: None 2024-04-13T21:41:32.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM= 2024-04-13T21:41:32.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM=', 'Content-Length': '0'} 2024-04-13T21:41:32.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea13f5d260938896-00661afc0c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.427 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.427 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.427 DEBUG:boto:path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.427 DEBUG:boto:auth_path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.427 DEBUG:boto:Method: GET 2024-04-13T21:41:32.427 DEBUG:boto:Path: /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.427 DEBUG:boto:Data: 2024-04-13T21:41:32.427 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.428 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.428 DEBUG:boto:Params: {} 2024-04-13T21:41:32.428 DEBUG:boto:Token: None 2024-04-13T21:41:32.428 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.428 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM= 2024-04-13T21:41:32.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM=', 'Content-Length': '0'} 2024-04-13T21:41:32.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f04b0a2e5bea858e-00661afc0c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.430 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.430 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.430 DEBUG:boto:path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.430 DEBUG:boto:auth_path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.430 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.431 DEBUG:boto:Path: /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.431 DEBUG:boto:Data: 2024-04-13T21:41:32.431 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.431 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.431 DEBUG:boto:Params: {} 2024-04-13T21:41:32.431 DEBUG:boto:Token: None 2024-04-13T21:41:32.431 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.431 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oJjV9HRsPASRl4/iK1RKiRHcm0Q= 2024-04-13T21:41:32.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oJjV9HRsPASRl4/iK1RKiRHcm0Q=', 'Content-Length': '0'} 2024-04-13T21:41:32.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f906f4878b27d73d-00661afc0c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.433 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.433 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.433 DEBUG:boto:path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.433 DEBUG:boto:auth_path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.434 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.434 DEBUG:boto:Path: /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.434 DEBUG:boto:Data: 2024-04-13T21:41:32.434 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.434 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.434 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.434 DEBUG:boto:Params: {} 2024-04-13T21:41:32.434 DEBUG:boto:Token: None 2024-04-13T21:41:32.434 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.435 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oJjV9HRsPASRl4/iK1RKiRHcm0Q= 2024-04-13T21:41:32.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oJjV9HRsPASRl4/iK1RKiRHcm0Q=', 'Content-Length': '0'} 2024-04-13T21:41:32.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bde226b29d2f98b4-00661afc0c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:41:32.437 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.437 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.437 DEBUG:boto:path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.437 DEBUG:boto:auth_path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.437 DEBUG:boto:Method: GET 2024-04-13T21:41:32.437 DEBUG:boto:Path: /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.437 DEBUG:boto:Data: 2024-04-13T21:41:32.437 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.437 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.438 DEBUG:boto:Params: {} 2024-04-13T21:41:32.438 DEBUG:boto:Token: None 2024-04-13T21:41:32.438 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM= 2024-04-13T21:41:32.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM=', 'Content-Length': '0'} 2024-04-13T21:41:32.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035b010909d5f75a2-00661afc0c-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.440 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.440 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.440 DEBUG:boto:path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.440 DEBUG:boto:auth_path=/vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.440 DEBUG:boto:Method: GET 2024-04-13T21:41:32.440 DEBUG:boto:Path: /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.441 DEBUG:boto:Data: 2024-04-13T21:41:32.441 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.441 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.441 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.441 DEBUG:boto:Params: {} 2024-04-13T21:41:32.441 DEBUG:boto:Token: None 2024-04-13T21:41:32.441 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj?versionId=keFf3uL6NbYooyB1XG3.9ovq2kVonl1 2024-04-13T21:41:32.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM= 2024-04-13T21:41:32.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nnbK3ddItyHv7PsljX40YNHCaZM=', 'Content-Length': '0'} 2024-04-13T21:41:32.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'keFf3uL6NbYooyB1XG3.9ovq2kVonl1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000498cca954e5e201b-00661afc0c-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.443 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.444 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.444 DEBUG:boto:path=/vwnfns-17/?prefix=obj 2024-04-13T21:41:32.444 DEBUG:boto:auth_path=/vwnfns-17/?prefix=obj 2024-04-13T21:41:32.444 DEBUG:boto:Method: GET 2024-04-13T21:41:32.444 DEBUG:boto:Path: /vwnfns-17/?prefix=obj 2024-04-13T21:41:32.444 DEBUG:boto:Data: 2024-04-13T21:41:32.444 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.444 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.444 DEBUG:boto:Params: {} 2024-04-13T21:41:32.444 DEBUG:boto:Token: None 2024-04-13T21:41:32.445 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/ 2024-04-13T21:41:32.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QHp8YIJw/Z7f36AkxD1ks4ssLuA= 2024-04-13T21:41:32.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QHp8YIJw/Z7f36AkxD1ks4ssLuA=', 'Content-Length': '0'} 2024-04-13T21:41:32.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3028ed74ddac54-00661afc0c-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.494 DEBUG:boto:b'vwnfns-17obj1000false' 2024-04-13T21:41:32.495 DEBUG:boto:path=/vwnfns-17/ 2024-04-13T21:41:32.495 DEBUG:boto:auth_path=/vwnfns-17/ 2024-04-13T21:41:32.495 DEBUG:boto:path=/vwnfns-17/?prefix=obj 2024-04-13T21:41:32.495 DEBUG:boto:auth_path=/vwnfns-17/?prefix=obj 2024-04-13T21:41:32.495 DEBUG:boto:Method: GET 2024-04-13T21:41:32.495 DEBUG:boto:Path: /vwnfns-17/?prefix=obj 2024-04-13T21:41:32.495 DEBUG:boto:Data: 2024-04-13T21:41:32.496 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.496 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.496 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.496 DEBUG:boto:Params: {} 2024-04-13T21:41:32.496 DEBUG:boto:Token: None 2024-04-13T21:41:32.497 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/ 2024-04-13T21:41:32.497 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QHp8YIJw/Z7f36AkxD1ks4ssLuA= 2024-04-13T21:41:32.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QHp8YIJw/Z7f36AkxD1ks4ssLuA=', 'Content-Length': '0'} 2024-04-13T21:41:32.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eab78ace45194c70-00661afc0c-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.499 DEBUG:boto:b'vwnfns-17obj1000false' 2024-04-13T21:41:32.500 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.500 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.500 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.500 DEBUG:boto:Path: /vwnfns-17/obj 2024-04-13T21:41:32.500 DEBUG:boto:Data: 2024-04-13T21:41:32.500 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:41:32.500 DEBUG:boto:Port: 8001 2024-04-13T21:41:32.501 DEBUG:boto:Params: {} 2024-04-13T21:41:32.501 DEBUG:boto:Token: None 2024-04-13T21:41:32.501 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj 2024-04-13T21:41:32.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE= 2024-04-13T21:41:32.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE=', 'Content-Length': '0'} 2024-04-13T21:41:32.503 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003adea25a716ffd57-00661afc0c-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.503 DEBUG:boto:path=/vwnfns-17/obj 2024-04-13T21:41:32.503 DEBUG:boto:auth_path=/vwnfns-17/obj 2024-04-13T21:41:32.503 DEBUG:boto:Method: HEAD 2024-04-13T21:41:32.503 DEBUG:boto:Path: /vwnfns-17/obj 2024-04-13T21:41:32.503 DEBUG:boto:Data: 2024-04-13T21:41:32.503 DEBUG:boto:Headers: {} 2024-04-13T21:41:32.504 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:41:32.504 DEBUG:boto:Port: 8000 2024-04-13T21:41:32.504 DEBUG:boto:Params: {} 2024-04-13T21:41:32.504 DEBUG:boto:Token: None 2024-04-13T21:41:32.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:41:32 GMT /vwnfns-17/obj 2024-04-13T21:41:32.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE= 2024-04-13T21:41:32.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:41:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:up6CrjoD0LfWakghtmK2B6+OQcE=', 'Content-Length': '0'} 2024-04-13T21:41:32.506 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ee75c257b080ba86-00661afc0c-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:41:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:41:32.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-17 zones={a2, a1} 2024-04-13T21:41:32.506 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-18 2024-04-13T21:41:32.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:41:32.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:41:32.507 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:41:32.569 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:41:32.570 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:41:32.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:32.613+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:41:37.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:37.614+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:41:42.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:42.615+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:41:47.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:47.616+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:41:52.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:52.616+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:41:57.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:41:57.616+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:42:02.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:02.616+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:42:07.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:07.617+0000 7f2e49f15780 1 waiting to reach incremental sync.. 2024-04-13T21:42:12.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:12.619+0000 7f2e49f15780 1 bucket sync caught up with source: 2024-04-13T21:42:12.611 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000002.23.6, , ] 2024-04-13T21:42:12.612 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000002.23.6, , ] 2024-04-13T21:42:12.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:12.619+0000 7f2e49f15780 0 bucket checkpoint complete 2024-04-13T21:42:12.627 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-18 2024-04-13T21:42:12.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:12.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:12.628 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:42:12.689 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:12.689 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:12.734+0000 7f352fcca780 1 bucket sync caught up with source: 2024-04-13T21:42:12.727 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.51.12, , ] 2024-04-13T21:42:12.727 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.51.12, , ] 2024-04-13T21:42:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:12.734+0000 7f352fcca780 0 bucket checkpoint complete 2024-04-13T21:42:12.739 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-18 zones={a2, a1} 2024-04-13T21:42:12.740 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.740 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.740 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.740 DEBUG:boto:Path: /vwnfns-18/ 2024-04-13T21:42:12.740 DEBUG:boto:Data: 2024-04-13T21:42:12.740 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.741 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.741 DEBUG:boto:Params: {} 2024-04-13T21:42:12.741 DEBUG:boto:Token: None 2024-04-13T21:42:12.741 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/ 2024-04-13T21:42:12.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i3S3yTHDJkg3w0CNGbTD1B7aGck= 2024-04-13T21:42:12.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i3S3yTHDJkg3w0CNGbTD1B7aGck=', 'Content-Length': '0'} 2024-04-13T21:42:12.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2da9f9ed2e3e9bb-00661afc34-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.747 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.748 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.748 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.748 DEBUG:boto:Path: /vwnfns-18/ 2024-04-13T21:42:12.748 DEBUG:boto:Data: 2024-04-13T21:42:12.748 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.748 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.748 DEBUG:boto:Params: {} 2024-04-13T21:42:12.748 DEBUG:boto:Token: None 2024-04-13T21:42:12.749 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/ 2024-04-13T21:42:12.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i3S3yTHDJkg3w0CNGbTD1B7aGck= 2024-04-13T21:42:12.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i3S3yTHDJkg3w0CNGbTD1B7aGck=', 'Content-Length': '0'} 2024-04-13T21:42:12.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016099f0acb2af26f-00661afc34-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.751 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:42:12.751 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.751 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.751 DEBUG:boto:path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.751 DEBUG:boto:auth_path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.751 DEBUG:boto:Method: GET 2024-04-13T21:42:12.752 DEBUG:boto:Path: /vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.752 DEBUG:boto:Data: 2024-04-13T21:42:12.752 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.752 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.752 DEBUG:boto:Params: {} 2024-04-13T21:42:12.752 DEBUG:boto:Token: None 2024-04-13T21:42:12.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/?versions 2024-04-13T21:42:12.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU= 2024-04-13T21:42:12.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU=', 'Content-Length': '0'} 2024-04-13T21:42:12.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000897723683390cc9b-00661afc34-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.755 DEBUG:boto:b'vwnfns-18999falseobjHu8T.a0H-nxIqbRJVRatdOyQsPmCFtQtrue2024-04-13T21:40:52.080Zrgw-multisite-test-userMultisite Test UserNormalobjeLLbN.jIFk5P-5s0LAdpc4OR3KF.ebXfalse2024-04-13T21:40:52.065Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:12.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:12.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:12.756 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:42:12.757 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.757 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.757 DEBUG:boto:path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.757 DEBUG:boto:auth_path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.757 DEBUG:boto:Method: GET 2024-04-13T21:42:12.757 DEBUG:boto:Path: /vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.757 DEBUG:boto:Data: 2024-04-13T21:42:12.757 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.757 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.758 DEBUG:boto:Params: {} 2024-04-13T21:42:12.758 DEBUG:boto:Token: None 2024-04-13T21:42:12.758 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/?versions 2024-04-13T21:42:12.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU= 2024-04-13T21:42:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU=', 'Content-Length': '0'} 2024-04-13T21:42:12.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1c1258bd83a6de-00661afc34-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.760 DEBUG:boto:b'vwnfns-18999falseobjHu8T.a0H-nxIqbRJVRatdOyQsPmCFtQtrue2024-04-13T21:40:52.080Zrgw-multisite-test-userMultisite Test UserNormalobjeLLbN.jIFk5P-5s0LAdpc4OR3KF.ebXfalse2024-04-13T21:40:52.065Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:12.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:12.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:12.761 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.761 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.761 DEBUG:boto:path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.761 DEBUG:boto:auth_path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.761 DEBUG:boto:Method: GET 2024-04-13T21:42:12.761 DEBUG:boto:Path: /vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.761 DEBUG:boto:Data: 2024-04-13T21:42:12.761 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.761 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.762 DEBUG:boto:Params: {} 2024-04-13T21:42:12.762 DEBUG:boto:Token: None 2024-04-13T21:42:12.762 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/?versions 2024-04-13T21:42:12.762 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU= 2024-04-13T21:42:12.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU=', 'Content-Length': '0'} 2024-04-13T21:42:12.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067b90f9f2e7e9398-00661afc34-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.806 DEBUG:boto:b'vwnfns-18999falseobjHu8T.a0H-nxIqbRJVRatdOyQsPmCFtQtrue2024-04-13T21:40:52.080Zrgw-multisite-test-userMultisite Test UserNormalobjeLLbN.jIFk5P-5s0LAdpc4OR3KF.ebXfalse2024-04-13T21:40:52.065Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:12.807 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.807 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.807 DEBUG:boto:path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.807 DEBUG:boto:auth_path=/vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.807 DEBUG:boto:Method: GET 2024-04-13T21:42:12.807 DEBUG:boto:Path: /vwnfns-18/?versions&max-keys=999 2024-04-13T21:42:12.807 DEBUG:boto:Data: 2024-04-13T21:42:12.808 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.808 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.808 DEBUG:boto:Params: {} 2024-04-13T21:42:12.808 DEBUG:boto:Token: None 2024-04-13T21:42:12.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/?versions 2024-04-13T21:42:12.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU= 2024-04-13T21:42:12.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlI8BaVnJ8diJwOa/4PkW1T/VAU=', 'Content-Length': '0'} 2024-04-13T21:42:12.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042823a205d647810-00661afc34-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.810 DEBUG:boto:b'vwnfns-18999falseobjHu8T.a0H-nxIqbRJVRatdOyQsPmCFtQtrue2024-04-13T21:40:52.080Zrgw-multisite-test-userMultisite Test UserNormalobjeLLbN.jIFk5P-5s0LAdpc4OR3KF.ebXfalse2024-04-13T21:40:52.065Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:12.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:42:12.811 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.811 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.811 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.811 DEBUG:boto:Path: /vwnfns-18/obj 2024-04-13T21:42:12.811 DEBUG:boto:Data: 2024-04-13T21:42:12.811 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.811 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.812 DEBUG:boto:Params: {} 2024-04-13T21:42:12.812 DEBUG:boto:Token: None 2024-04-13T21:42:12.812 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj 2024-04-13T21:42:12.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM= 2024-04-13T21:42:12.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM=', 'Content-Length': '0'} 2024-04-13T21:42:12.814 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fddbdc206336196c-00661afc34-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.814 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.814 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.814 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.814 DEBUG:boto:Path: /vwnfns-18/obj 2024-04-13T21:42:12.814 DEBUG:boto:Data: 2024-04-13T21:42:12.814 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.814 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.814 DEBUG:boto:Params: {} 2024-04-13T21:42:12.815 DEBUG:boto:Token: None 2024-04-13T21:42:12.815 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj 2024-04-13T21:42:12.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM= 2024-04-13T21:42:12.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM=', 'Content-Length': '0'} 2024-04-13T21:42:12.816 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d74714d41a37ee96-00661afc34-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:42:12.817 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.817 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.817 DEBUG:boto:path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.817 DEBUG:boto:auth_path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.817 DEBUG:boto:Method: GET 2024-04-13T21:42:12.817 DEBUG:boto:Path: /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.817 DEBUG:boto:Data: 2024-04-13T21:42:12.817 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.817 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.817 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.817 DEBUG:boto:Params: {} 2024-04-13T21:42:12.818 DEBUG:boto:Token: None 2024-04-13T21:42:12.818 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.818 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto= 2024-04-13T21:42:12.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto=', 'Content-Length': '0'} 2024-04-13T21:42:12.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d59da47f6ae5dc1-00661afc34-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.820 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.820 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.820 DEBUG:boto:path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.820 DEBUG:boto:auth_path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.820 DEBUG:boto:Method: GET 2024-04-13T21:42:12.820 DEBUG:boto:Path: /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.820 DEBUG:boto:Data: 2024-04-13T21:42:12.820 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.820 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.820 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.821 DEBUG:boto:Params: {} 2024-04-13T21:42:12.821 DEBUG:boto:Token: None 2024-04-13T21:42:12.821 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.821 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto= 2024-04-13T21:42:12.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto=', 'Content-Length': '0'} 2024-04-13T21:42:12.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006407a1ca5200d13c-00661afc34-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.823 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.823 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.823 DEBUG:boto:path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.823 DEBUG:boto:auth_path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.823 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.823 DEBUG:boto:Path: /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.823 DEBUG:boto:Data: 2024-04-13T21:42:12.823 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.823 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.823 DEBUG:boto:Params: {} 2024-04-13T21:42:12.824 DEBUG:boto:Token: None 2024-04-13T21:42:12.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zNSpwkTt4Ji/+ZE5eiamKphU6pw= 2024-04-13T21:42:12.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zNSpwkTt4Ji/+ZE5eiamKphU6pw=', 'Content-Length': '0'} 2024-04-13T21:42:12.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009904c8b1ae18ca3-00661afc34-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.825 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.826 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.826 DEBUG:boto:path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.826 DEBUG:boto:auth_path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.826 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.826 DEBUG:boto:Path: /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.826 DEBUG:boto:Data: 2024-04-13T21:42:12.826 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.826 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.826 DEBUG:boto:Params: {} 2024-04-13T21:42:12.827 DEBUG:boto:Token: None 2024-04-13T21:42:12.827 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zNSpwkTt4Ji/+ZE5eiamKphU6pw= 2024-04-13T21:42:12.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zNSpwkTt4Ji/+ZE5eiamKphU6pw=', 'Content-Length': '0'} 2024-04-13T21:42:12.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa5da114a880c95d-00661afc34-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:42:12.829 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.829 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.829 DEBUG:boto:path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.829 DEBUG:boto:auth_path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.829 DEBUG:boto:Method: GET 2024-04-13T21:42:12.829 DEBUG:boto:Path: /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.829 DEBUG:boto:Data: 2024-04-13T21:42:12.829 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.829 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.829 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.829 DEBUG:boto:Params: {} 2024-04-13T21:42:12.830 DEBUG:boto:Token: None 2024-04-13T21:42:12.830 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.830 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto= 2024-04-13T21:42:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto=', 'Content-Length': '0'} 2024-04-13T21:42:12.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbbd7ef3bcc05c4b-00661afc34-1110-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.832 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.832 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.832 DEBUG:boto:path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.832 DEBUG:boto:auth_path=/vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.832 DEBUG:boto:Method: GET 2024-04-13T21:42:12.832 DEBUG:boto:Path: /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.832 DEBUG:boto:Data: 2024-04-13T21:42:12.832 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.832 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.833 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.833 DEBUG:boto:Params: {} 2024-04-13T21:42:12.833 DEBUG:boto:Token: None 2024-04-13T21:42:12.833 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj?versionId=eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX 2024-04-13T21:42:12.833 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto= 2024-04-13T21:42:12.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hRvqIPXH6lyXhRdangeas3p5Qto=', 'Content-Length': '0'} 2024-04-13T21:42:12.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:40:52 GMT'), ('x-amz-version-id', 'eLLbN.jIFk5P-5s0LAdpc4OR3KF.ebX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7a574072036dec4-00661afc34-1114-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.835 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.835 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.835 DEBUG:boto:path=/vwnfns-18/?prefix=obj 2024-04-13T21:42:12.835 DEBUG:boto:auth_path=/vwnfns-18/?prefix=obj 2024-04-13T21:42:12.835 DEBUG:boto:Method: GET 2024-04-13T21:42:12.835 DEBUG:boto:Path: /vwnfns-18/?prefix=obj 2024-04-13T21:42:12.835 DEBUG:boto:Data: 2024-04-13T21:42:12.835 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.835 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.836 DEBUG:boto:Params: {} 2024-04-13T21:42:12.836 DEBUG:boto:Token: None 2024-04-13T21:42:12.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/ 2024-04-13T21:42:12.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ErUMVOrmzwlcKfNR/dFGv+8UK10= 2024-04-13T21:42:12.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ErUMVOrmzwlcKfNR/dFGv+8UK10=', 'Content-Length': '0'} 2024-04-13T21:42:12.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000452850409eb981ad-00661afc34-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.838 DEBUG:boto:b'vwnfns-18obj1000false' 2024-04-13T21:42:12.838 DEBUG:boto:path=/vwnfns-18/ 2024-04-13T21:42:12.838 DEBUG:boto:auth_path=/vwnfns-18/ 2024-04-13T21:42:12.839 DEBUG:boto:path=/vwnfns-18/?prefix=obj 2024-04-13T21:42:12.839 DEBUG:boto:auth_path=/vwnfns-18/?prefix=obj 2024-04-13T21:42:12.839 DEBUG:boto:Method: GET 2024-04-13T21:42:12.839 DEBUG:boto:Path: /vwnfns-18/?prefix=obj 2024-04-13T21:42:12.839 DEBUG:boto:Data: 2024-04-13T21:42:12.839 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.839 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.839 DEBUG:boto:Params: {} 2024-04-13T21:42:12.839 DEBUG:boto:Token: None 2024-04-13T21:42:12.839 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/ 2024-04-13T21:42:12.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ErUMVOrmzwlcKfNR/dFGv+8UK10= 2024-04-13T21:42:12.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ErUMVOrmzwlcKfNR/dFGv+8UK10=', 'Content-Length': '0'} 2024-04-13T21:42:12.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0302766b2182cc0-00661afc34-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.882 DEBUG:boto:b'vwnfns-18obj1000false' 2024-04-13T21:42:12.882 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.883 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.883 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.883 DEBUG:boto:Path: /vwnfns-18/obj 2024-04-13T21:42:12.883 DEBUG:boto:Data: 2024-04-13T21:42:12.883 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.883 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.883 DEBUG:boto:Params: {} 2024-04-13T21:42:12.883 DEBUG:boto:Token: None 2024-04-13T21:42:12.883 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj 2024-04-13T21:42:12.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM= 2024-04-13T21:42:12.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM=', 'Content-Length': '0'} 2024-04-13T21:42:12.885 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f89abe290c0dc3fd-00661afc34-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.886 DEBUG:boto:path=/vwnfns-18/obj 2024-04-13T21:42:12.886 DEBUG:boto:auth_path=/vwnfns-18/obj 2024-04-13T21:42:12.886 DEBUG:boto:Method: HEAD 2024-04-13T21:42:12.886 DEBUG:boto:Path: /vwnfns-18/obj 2024-04-13T21:42:12.886 DEBUG:boto:Data: 2024-04-13T21:42:12.886 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.886 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.886 DEBUG:boto:Params: {} 2024-04-13T21:42:12.886 DEBUG:boto:Token: None 2024-04-13T21:42:12.886 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-18/obj 2024-04-13T21:42:12.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM= 2024-04-13T21:42:12.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ERCR+IemeDL17MNuHMVxE6n6rxM=', 'Content-Length': '0'} 2024-04-13T21:42:12.888 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f1439e4fd7927cbc-00661afc34-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-18 zones={a2, a1} 2024-04-13T21:42:12.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-04-13T21:42:12.890 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-19 2024-04-13T21:42:12.890 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:12.890 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:12.891 DEBUG:boto:Method: PUT 2024-04-13T21:42:12.891 DEBUG:boto:Path: /vwnfns-19/ 2024-04-13T21:42:12.891 DEBUG:boto:Data: 2024-04-13T21:42:12.891 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.891 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.891 DEBUG:boto:Params: {} 2024-04-13T21:42:12.891 DEBUG:boto:Token: None 2024-04-13T21:42:12.891 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-19/ 2024-04-13T21:42:12.891 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EjANK22wBlo1xz4a2eZwzz1GAaA= 2024-04-13T21:42:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EjANK22wBlo1xz4a2eZwzz1GAaA=', 'Content-Length': '0'} 2024-04-13T21:42:12.905 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005515b6ad34c2f486-00661afc34-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.906 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-20 2024-04-13T21:42:12.906 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:12.906 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:12.906 DEBUG:boto:Method: PUT 2024-04-13T21:42:12.906 DEBUG:boto:Path: /vwnfns-20/ 2024-04-13T21:42:12.906 DEBUG:boto:Data: 2024-04-13T21:42:12.906 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.907 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.907 DEBUG:boto:Params: {} 2024-04-13T21:42:12.907 DEBUG:boto:Token: None 2024-04-13T21:42:12.907 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-20/ 2024-04-13T21:42:12.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LagOq5DlNCYC7gjnXxhE7aoZH0M= 2024-04-13T21:42:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LagOq5DlNCYC7gjnXxhE7aoZH0M=', 'Content-Length': '0'} 2024-04-13T21:42:12.927 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000143f8c955f4f7a22-00661afc34-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.927 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:12.927 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:12.927 DEBUG:boto:path=/vwnfns-19/?versioning 2024-04-13T21:42:12.927 DEBUG:boto:auth_path=/vwnfns-19/?versioning 2024-04-13T21:42:12.928 DEBUG:boto:Method: PUT 2024-04-13T21:42:12.928 DEBUG:boto:Path: /vwnfns-19/?versioning 2024-04-13T21:42:12.928 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:42:12.928 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.928 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.928 DEBUG:boto:Params: {} 2024-04-13T21:42:12.928 DEBUG:boto:Token: None 2024-04-13T21:42:12.928 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-19/?versioning 2024-04-13T21:42:12.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5VaUGJB6eBUaKWoqdmiZwIh1ork= 2024-04-13T21:42:12.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5VaUGJB6eBUaKWoqdmiZwIh1ork=', 'Content-Length': '228'} 2024-04-13T21:42:12.933 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000f69c4bc943fa572-00661afc34-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.933 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:12.933 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:12.933 DEBUG:boto:path=/vwnfns-19/?versioning 2024-04-13T21:42:12.933 DEBUG:boto:auth_path=/vwnfns-19/?versioning 2024-04-13T21:42:12.933 DEBUG:boto:Method: PUT 2024-04-13T21:42:12.934 DEBUG:boto:Path: /vwnfns-19/?versioning 2024-04-13T21:42:12.934 DEBUG:boto:Data: Suspended Disabled 2024-04-13T21:42:12.934 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:12.934 DEBUG:boto:Port: 8000 2024-04-13T21:42:12.934 DEBUG:boto:Params: {} 2024-04-13T21:42:12.934 DEBUG:boto:Token: None 2024-04-13T21:42:12.934 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-19/?versioning 2024-04-13T21:42:12.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5VaUGJB6eBUaKWoqdmiZwIh1ork= 2024-04-13T21:42:12.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5VaUGJB6eBUaKWoqdmiZwIh1ork=', 'Content-Length': '230'} 2024-04-13T21:42:12.938 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bba9e14d2d689696-00661afc34-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.938 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:12.939 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:12.939 DEBUG:boto:path=/vwnfns-20/?versioning 2024-04-13T21:42:12.940 DEBUG:boto:auth_path=/vwnfns-20/?versioning 2024-04-13T21:42:12.940 DEBUG:boto:Method: PUT 2024-04-13T21:42:12.940 DEBUG:boto:Path: /vwnfns-20/?versioning 2024-04-13T21:42:12.940 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:42:12.941 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.941 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.941 DEBUG:boto:Params: {} 2024-04-13T21:42:12.941 DEBUG:boto:Token: None 2024-04-13T21:42:12.941 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-20/?versioning 2024-04-13T21:42:12.941 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XoW9ceVNxXL8PVo3BJb0J7Paoqk= 2024-04-13T21:42:12.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XoW9ceVNxXL8PVo3BJb0J7Paoqk=', 'Content-Length': '228'} 2024-04-13T21:42:12.949 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa0de01661b21ca1-00661afc34-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.950 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:12.950 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:12.950 DEBUG:boto:path=/vwnfns-20/?versioning 2024-04-13T21:42:12.950 DEBUG:boto:auth_path=/vwnfns-20/?versioning 2024-04-13T21:42:12.950 DEBUG:boto:Method: PUT 2024-04-13T21:42:12.950 DEBUG:boto:Path: /vwnfns-20/?versioning 2024-04-13T21:42:12.950 DEBUG:boto:Data: Suspended Disabled 2024-04-13T21:42:12.950 DEBUG:boto:Headers: {} 2024-04-13T21:42:12.950 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:12.951 DEBUG:boto:Port: 8001 2024-04-13T21:42:12.951 DEBUG:boto:Params: {} 2024-04-13T21:42:12.951 DEBUG:boto:Token: None 2024-04-13T21:42:12.951 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:12 GMT /vwnfns-20/?versioning 2024-04-13T21:42:12.951 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XoW9ceVNxXL8PVo3BJb0J7Paoqk= 2024-04-13T21:42:12.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XoW9ceVNxXL8PVo3BJb0J7Paoqk=', 'Content-Length': '230'} 2024-04-13T21:42:12.957 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a697379b092d1bdf-00661afc34-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:12.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:12.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:12.958 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:42:13.010 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:13.011 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:13.039 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:42:13.050 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:42:13.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.050 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044532.964622_239.1", 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:42:12.964622Z" 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.052 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:13.052 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:42:13.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:13.052 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:42:13.053 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044532.964622_239.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:13.053 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:42:13.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:13.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:13.053 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:42:13.101 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:13.101 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:13.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:13.136+0000 7f2188909780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:13.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:13.137+0000 7f2188909780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:42:13.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044451.833599_218.1", 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:40:51.833599Z", 2024-04-13T21:42:13.142 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:13.143 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:13.144 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:13.145 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:42:13.146 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:42:13.147 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:42:13.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:13.147 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:42:13Z" 2024-04-13T21:42:13.147 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:42:13.148 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044451.833599_218.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:40:51.833599Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:42:13Z" } 2024-04-13T21:42:13.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044451.833599_218.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:40:51.833599Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:42:13.148 DEBUG:rgw_multi.tests:log_status={0: '1_1713044532.964622_239.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:13.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044451.833599_218.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:13.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044532.964622_239.1 target=1_1713044451.833599_218.1 2024-04-13T21:42:18.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:18.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:18.150 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:42:18.205 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:18.206 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:18.245+0000 7f7dfc569780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:18.246+0000 7f7dfc569780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:18.249 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:42:18.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044532.964622_239.1", 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:12.964622Z", 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:18.252 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:18.253 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:18.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:18.255 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:42:18Z" 2024-04-13T21:42:18.256 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:42:18.256 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044532.964622_239.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:12.964622Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:42:18Z" } 2024-04-13T21:42:18.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044532.964622_239.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:12.964622Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:42:18.257 DEBUG:rgw_multi.tests:log_status={0: '1_1713044532.964622_239.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:18.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044532.964622_239.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:18.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:42:18.257 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:18.257 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:18.258 DEBUG:boto:Method: HEAD 2024-04-13T21:42:18.258 DEBUG:boto:Path: /vwnfns-19/ 2024-04-13T21:42:18.258 DEBUG:boto:Data: 2024-04-13T21:42:18.258 DEBUG:boto:Headers: {} 2024-04-13T21:42:18.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:18.258 DEBUG:boto:Port: 8000 2024-04-13T21:42:18.258 DEBUG:boto:Params: {} 2024-04-13T21:42:18.258 DEBUG:boto:Token: None 2024-04-13T21:42:18.258 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-19/ 2024-04-13T21:42:18.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mvJzysW0Q+AsGj4IpOS9oP1n0Sk= 2024-04-13T21:42:18.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mvJzysW0Q+AsGj4IpOS9oP1n0Sk=', 'Content-Length': '0'} 2024-04-13T21:42:18.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040e1f91663c98b81-00661afc3a-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.261 DEBUG:boto:path=/vwnfns-19/obj 2024-04-13T21:42:18.261 DEBUG:boto:auth_path=/vwnfns-19/obj 2024-04-13T21:42:18.262 DEBUG:boto:Method: PUT 2024-04-13T21:42:18.262 DEBUG:boto:Path: /vwnfns-19/obj 2024-04-13T21:42:18.262 DEBUG:boto:Data: 2024-04-13T21:42:18.262 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:42:18.262 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:18.262 DEBUG:boto:Port: 8000 2024-04-13T21:42:18.262 DEBUG:boto:Params: {} 2024-04-13T21:42:18.262 DEBUG:boto:Token: None 2024-04-13T21:42:18.262 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-19/obj 2024-04-13T21:42:18.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4QMczrEsSPQSTHvZsHA4nctKquc= 2024-04-13T21:42:18.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4QMczrEsSPQSTHvZsHA4nctKquc='} 2024-04-13T21:42:18.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000330b507d8b76f96-00661afc3a-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.268 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:18.269 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:18.269 DEBUG:boto:Method: HEAD 2024-04-13T21:42:18.269 DEBUG:boto:Path: /vwnfns-19/ 2024-04-13T21:42:18.269 DEBUG:boto:Data: 2024-04-13T21:42:18.269 DEBUG:boto:Headers: {} 2024-04-13T21:42:18.269 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:18.269 DEBUG:boto:Port: 8000 2024-04-13T21:42:18.269 DEBUG:boto:Params: {} 2024-04-13T21:42:18.270 DEBUG:boto:Token: None 2024-04-13T21:42:18.270 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-19/ 2024-04-13T21:42:18.270 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mvJzysW0Q+AsGj4IpOS9oP1n0Sk= 2024-04-13T21:42:18.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mvJzysW0Q+AsGj4IpOS9oP1n0Sk=', 'Content-Length': '0'} 2024-04-13T21:42:18.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cefe03145103c63c-00661afc3a-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.272 DEBUG:boto:path=/vwnfns-19/obj 2024-04-13T21:42:18.272 DEBUG:boto:auth_path=/vwnfns-19/obj 2024-04-13T21:42:18.272 DEBUG:boto:Method: DELETE 2024-04-13T21:42:18.272 DEBUG:boto:Path: /vwnfns-19/obj 2024-04-13T21:42:18.272 DEBUG:boto:Data: 2024-04-13T21:42:18.273 DEBUG:boto:Headers: {} 2024-04-13T21:42:18.273 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:18.273 DEBUG:boto:Port: 8000 2024-04-13T21:42:18.273 DEBUG:boto:Params: {} 2024-04-13T21:42:18.273 DEBUG:boto:Token: None 2024-04-13T21:42:18.273 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-19/obj 2024-04-13T21:42:18.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WXKho7tpBJk1cohhd4l5+L5EmDc= 2024-04-13T21:42:18.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WXKho7tpBJk1cohhd4l5+L5EmDc=', 'Content-Length': '0'} 2024-04-13T21:42:18.282 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d6879a60e735af94-00661afc3a-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.282 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:18.283 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:18.283 DEBUG:boto:Method: HEAD 2024-04-13T21:42:18.283 DEBUG:boto:Path: /vwnfns-20/ 2024-04-13T21:42:18.283 DEBUG:boto:Data: 2024-04-13T21:42:18.283 DEBUG:boto:Headers: {} 2024-04-13T21:42:18.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:18.283 DEBUG:boto:Port: 8001 2024-04-13T21:42:18.283 DEBUG:boto:Params: {} 2024-04-13T21:42:18.283 DEBUG:boto:Token: None 2024-04-13T21:42:18.283 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-20/ 2024-04-13T21:42:18.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A1/veC1C7qWEkavRlsq7RLzhDBM= 2024-04-13T21:42:18.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A1/veC1C7qWEkavRlsq7RLzhDBM=', 'Content-Length': '0'} 2024-04-13T21:42:18.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081a8e35de72ce210-00661afc3a-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.286 DEBUG:boto:path=/vwnfns-20/obj 2024-04-13T21:42:18.286 DEBUG:boto:auth_path=/vwnfns-20/obj 2024-04-13T21:42:18.286 DEBUG:boto:Method: PUT 2024-04-13T21:42:18.286 DEBUG:boto:Path: /vwnfns-20/obj 2024-04-13T21:42:18.287 DEBUG:boto:Data: 2024-04-13T21:42:18.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-13T21:42:18.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:18.287 DEBUG:boto:Port: 8001 2024-04-13T21:42:18.287 DEBUG:boto:Params: {} 2024-04-13T21:42:18.287 DEBUG:boto:Token: None 2024-04-13T21:42:18.287 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-20/obj 2024-04-13T21:42:18.287 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oySZuvNiwColYrvVqpZwN9lWI3k= 2024-04-13T21:42:18.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oySZuvNiwColYrvVqpZwN9lWI3k='} 2024-04-13T21:42:18.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000614bb0aadb49fcd3-00661afc3a-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.294 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:18.294 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:18.294 DEBUG:boto:Method: HEAD 2024-04-13T21:42:18.294 DEBUG:boto:Path: /vwnfns-20/ 2024-04-13T21:42:18.295 DEBUG:boto:Data: 2024-04-13T21:42:18.295 DEBUG:boto:Headers: {} 2024-04-13T21:42:18.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:18.295 DEBUG:boto:Port: 8001 2024-04-13T21:42:18.295 DEBUG:boto:Params: {} 2024-04-13T21:42:18.295 DEBUG:boto:Token: None 2024-04-13T21:42:18.295 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-20/ 2024-04-13T21:42:18.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A1/veC1C7qWEkavRlsq7RLzhDBM= 2024-04-13T21:42:18.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A1/veC1C7qWEkavRlsq7RLzhDBM=', 'Content-Length': '0'} 2024-04-13T21:42:18.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b33e7ef6cb1fde0-00661afc3a-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.298 DEBUG:boto:path=/vwnfns-20/obj 2024-04-13T21:42:18.298 DEBUG:boto:auth_path=/vwnfns-20/obj 2024-04-13T21:42:18.298 DEBUG:boto:Method: DELETE 2024-04-13T21:42:18.298 DEBUG:boto:Path: /vwnfns-20/obj 2024-04-13T21:42:18.298 DEBUG:boto:Data: 2024-04-13T21:42:18.298 DEBUG:boto:Headers: {} 2024-04-13T21:42:18.298 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:18.298 DEBUG:boto:Port: 8001 2024-04-13T21:42:18.298 DEBUG:boto:Params: {} 2024-04-13T21:42:18.299 DEBUG:boto:Token: None 2024-04-13T21:42:18.299 DEBUG:boto:StringToSign: DELETE Sat, 13 Apr 2024 21:42:18 GMT /vwnfns-20/obj 2024-04-13T21:42:18.299 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVE8L03MZkwfW9PEF+xVh53a9FI= 2024-04-13T21:42:18.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVE8L03MZkwfW9PEF+xVh53a9FI=', 'Content-Length': '0'} 2024-04-13T21:42:18.309 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000dba813a000a3fe0c-00661afc3a-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:42:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:18.309 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-19 2024-04-13T21:42:18.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:18.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:18.310 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-19 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:42:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:18.365 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:18.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:18.406+0000 7f2642412780 1 waiting to reach incremental sync.. 2024-04-13T21:42:23.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:23.406+0000 7f2642412780 1 waiting to reach incremental sync.. 2024-04-13T21:42:28.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:28.408+0000 7f2642412780 1 bucket sync caught up with source: 2024-04-13T21:42:28.400 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.26.7, , ] 2024-04-13T21:42:28.400 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.26.7, , ] 2024-04-13T21:42:28.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:28.408+0000 7f2642412780 0 bucket checkpoint complete 2024-04-13T21:42:28.413 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-19 2024-04-13T21:42:28.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:28.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:28.414 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-19 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:42:28.471 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:28.471 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:28.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:28.513+0000 7f2c10cbd780 1 waiting to reach incremental sync.. 2024-04-13T21:42:33.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:33.514+0000 7f2c10cbd780 1 waiting to reach incremental sync.. 2024-04-13T21:42:38.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:38.513+0000 7f2c10cbd780 1 waiting to reach incremental sync.. 2024-04-13T21:42:43.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:43.514+0000 7f2c10cbd780 1 waiting to reach incremental sync.. 2024-04-13T21:42:48.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:48.515+0000 7f2c10cbd780 1 bucket sync caught up with source: 2024-04-13T21:42:48.508 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.29.14, , ] 2024-04-13T21:42:48.508 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.29.14, , ] 2024-04-13T21:42:48.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:48.515+0000 7f2c10cbd780 0 bucket checkpoint complete 2024-04-13T21:42:48.523 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-19 zones={a2, a1} 2024-04-13T21:42:48.523 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:48.524 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:48.524 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.524 DEBUG:boto:Path: /vwnfns-19/ 2024-04-13T21:42:48.524 DEBUG:boto:Data: 2024-04-13T21:42:48.524 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.524 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.524 DEBUG:boto:Params: {} 2024-04-13T21:42:48.524 DEBUG:boto:Token: None 2024-04-13T21:42:48.525 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/ 2024-04-13T21:42:48.525 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LpS0s/Q4IH3l3bHQBH63RxZXAg4= 2024-04-13T21:42:48.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LpS0s/Q4IH3l3bHQBH63RxZXAg4=', 'Content-Length': '0'} 2024-04-13T21:42:48.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007789c93c8fcfe3d8-00661afc58-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.528 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:48.528 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:48.528 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.528 DEBUG:boto:Path: /vwnfns-19/ 2024-04-13T21:42:48.528 DEBUG:boto:Data: 2024-04-13T21:42:48.528 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.528 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.528 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.528 DEBUG:boto:Params: {} 2024-04-13T21:42:48.528 DEBUG:boto:Token: None 2024-04-13T21:42:48.528 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/ 2024-04-13T21:42:48.529 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LpS0s/Q4IH3l3bHQBH63RxZXAg4= 2024-04-13T21:42:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LpS0s/Q4IH3l3bHQBH63RxZXAg4=', 'Content-Length': '0'} 2024-04-13T21:42:48.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8532707b505df1b-00661afc58-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:42:48.531 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:48.531 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:48.531 DEBUG:boto:path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.531 DEBUG:boto:auth_path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.532 DEBUG:boto:Method: GET 2024-04-13T21:42:48.532 DEBUG:boto:Path: /vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.532 DEBUG:boto:Data: 2024-04-13T21:42:48.532 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.532 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.532 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.532 DEBUG:boto:Params: {} 2024-04-13T21:42:48.532 DEBUG:boto:Token: None 2024-04-13T21:42:48.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/?versions 2024-04-13T21:42:48.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc= 2024-04-13T21:42:48.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc=', 'Content-Length': '0'} 2024-04-13T21:42:48.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b80afff8d42d32f2-00661afc58-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.535 DEBUG:boto:b'vwnfns-19999falseobjnulltrue2024-04-13T21:42:18.284Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:48.536 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:42:48.536 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:48.536 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:48.536 DEBUG:boto:path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.536 DEBUG:boto:auth_path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.536 DEBUG:boto:Method: GET 2024-04-13T21:42:48.537 DEBUG:boto:Path: /vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.537 DEBUG:boto:Data: 2024-04-13T21:42:48.537 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.537 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.537 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.537 DEBUG:boto:Params: {} 2024-04-13T21:42:48.537 DEBUG:boto:Token: None 2024-04-13T21:42:48.537 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/?versions 2024-04-13T21:42:48.537 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc= 2024-04-13T21:42:48.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc=', 'Content-Length': '0'} 2024-04-13T21:42:48.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b507281efdaf36d-00661afc58-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.540 DEBUG:boto:b'vwnfns-19999falseobjnulltrue2024-04-13T21:42:18.284Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:48.540 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:48.540 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:48.540 DEBUG:boto:path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.541 DEBUG:boto:auth_path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.541 DEBUG:boto:Method: GET 2024-04-13T21:42:48.541 DEBUG:boto:Path: /vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.541 DEBUG:boto:Data: 2024-04-13T21:42:48.541 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.541 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.541 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.541 DEBUG:boto:Params: {} 2024-04-13T21:42:48.541 DEBUG:boto:Token: None 2024-04-13T21:42:48.541 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/?versions 2024-04-13T21:42:48.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc= 2024-04-13T21:42:48.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc=', 'Content-Length': '0'} 2024-04-13T21:42:48.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081d94bfa3ab334ff-00661afc58-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.586 DEBUG:boto:b'vwnfns-19999falseobjnulltrue2024-04-13T21:42:18.284Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.587 DEBUG:boto:path=/vwnfns-19/ 2024-04-13T21:42:48.587 DEBUG:boto:auth_path=/vwnfns-19/ 2024-04-13T21:42:48.587 DEBUG:boto:path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.587 DEBUG:boto:auth_path=/vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.587 DEBUG:boto:Method: GET 2024-04-13T21:42:48.587 DEBUG:boto:Path: /vwnfns-19/?versions&max-keys=999 2024-04-13T21:42:48.588 DEBUG:boto:Data: 2024-04-13T21:42:48.588 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.588 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.588 DEBUG:boto:Params: {} 2024-04-13T21:42:48.588 DEBUG:boto:Token: None 2024-04-13T21:42:48.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/?versions 2024-04-13T21:42:48.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc= 2024-04-13T21:42:48.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OD7Q3hdrmtHHYavKa+Vzc2oASuc=', 'Content-Length': '0'} 2024-04-13T21:42:48.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aecac02d46f59622-00661afc58-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.591 DEBUG:boto:b'vwnfns-19999falseobjnulltrue2024-04-13T21:42:18.284Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:42:48.592 DEBUG:boto:path=/vwnfns-19/obj 2024-04-13T21:42:48.592 DEBUG:boto:auth_path=/vwnfns-19/obj 2024-04-13T21:42:48.592 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.592 DEBUG:boto:Path: /vwnfns-19/obj 2024-04-13T21:42:48.592 DEBUG:boto:Data: 2024-04-13T21:42:48.592 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.592 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.592 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.592 DEBUG:boto:Params: {} 2024-04-13T21:42:48.593 DEBUG:boto:Token: None 2024-04-13T21:42:48.593 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/obj 2024-04-13T21:42:48.593 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YcBODe4g8Y87GyvzBKl6wefyNrY= 2024-04-13T21:42:48.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YcBODe4g8Y87GyvzBKl6wefyNrY=', 'Content-Length': '0'} 2024-04-13T21:42:48.594 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000063fe9d93b23db5f2-00661afc58-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.594 DEBUG:boto:path=/vwnfns-19/obj 2024-04-13T21:42:48.595 DEBUG:boto:auth_path=/vwnfns-19/obj 2024-04-13T21:42:48.595 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.595 DEBUG:boto:Path: /vwnfns-19/obj 2024-04-13T21:42:48.595 DEBUG:boto:Data: 2024-04-13T21:42:48.595 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.595 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.595 DEBUG:boto:Params: {} 2024-04-13T21:42:48.595 DEBUG:boto:Token: None 2024-04-13T21:42:48.595 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-19/obj 2024-04-13T21:42:48.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YcBODe4g8Y87GyvzBKl6wefyNrY= 2024-04-13T21:42:48.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YcBODe4g8Y87GyvzBKl6wefyNrY=', 'Content-Length': '0'} 2024-04-13T21:42:48.597 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c1a3b762c02dc115-00661afc58-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-19 zones={a2, a1} 2024-04-13T21:42:48.598 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-20 2024-04-13T21:42:48.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:48.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:48.598 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-20 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:42:48.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:48.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:48.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:48.702+0000 7fcf14593780 1 bucket sync caught up with source: 2024-04-13T21:42:48.695 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.15.14, , ] 2024-04-13T21:42:48.695 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.15.14, , ] 2024-04-13T21:42:48.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:48.702+0000 7fcf14593780 0 bucket checkpoint complete 2024-04-13T21:42:48.708 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-20 2024-04-13T21:42:48.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:48.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T21:42:48.709 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-20 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T21:42:48.765 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:48.766 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:48.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:48.805+0000 7f1b94350780 1 bucket sync caught up with source: 2024-04-13T21:42:48.797 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000003.39.7, , ] 2024-04-13T21:42:48.797 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000003.39.7, , ] 2024-04-13T21:42:48.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:48.805+0000 7f1b94350780 0 bucket checkpoint complete 2024-04-13T21:42:48.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-20 zones={a2, a1} 2024-04-13T21:42:48.809 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:48.809 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:48.809 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.809 DEBUG:boto:Path: /vwnfns-20/ 2024-04-13T21:42:48.809 DEBUG:boto:Data: 2024-04-13T21:42:48.810 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.810 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.810 DEBUG:boto:Params: {} 2024-04-13T21:42:48.810 DEBUG:boto:Token: None 2024-04-13T21:42:48.810 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/ 2024-04-13T21:42:48.810 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sO7yKyvRIwVJ08tikg6YJj0rsPw= 2024-04-13T21:42:48.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sO7yKyvRIwVJ08tikg6YJj0rsPw=', 'Content-Length': '0'} 2024-04-13T21:42:48.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000777d523f2b5207aa-00661afc58-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.813 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:48.813 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:48.813 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.813 DEBUG:boto:Path: /vwnfns-20/ 2024-04-13T21:42:48.813 DEBUG:boto:Data: 2024-04-13T21:42:48.813 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.813 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.813 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.813 DEBUG:boto:Params: {} 2024-04-13T21:42:48.814 DEBUG:boto:Token: None 2024-04-13T21:42:48.814 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/ 2024-04-13T21:42:48.814 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sO7yKyvRIwVJ08tikg6YJj0rsPw= 2024-04-13T21:42:48.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sO7yKyvRIwVJ08tikg6YJj0rsPw=', 'Content-Length': '0'} 2024-04-13T21:42:48.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e9c6dc9e5a02e5-00661afc58-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T21:42:48.816 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:48.816 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:48.816 DEBUG:boto:path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.816 DEBUG:boto:auth_path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.816 DEBUG:boto:Method: GET 2024-04-13T21:42:48.816 DEBUG:boto:Path: /vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.816 DEBUG:boto:Data: 2024-04-13T21:42:48.817 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.817 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.817 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.817 DEBUG:boto:Params: {} 2024-04-13T21:42:48.817 DEBUG:boto:Token: None 2024-04-13T21:42:48.817 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/?versions 2024-04-13T21:42:48.817 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU= 2024-04-13T21:42:48.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU=', 'Content-Length': '0'} 2024-04-13T21:42:48.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752757b3ff423671-00661afc58-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.819 DEBUG:boto:b'vwnfns-20999falseobjnulltrue2024-04-13T21:42:18.310Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:48.820 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T21:42:48.820 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:48.820 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:48.820 DEBUG:boto:path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.821 DEBUG:boto:auth_path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.821 DEBUG:boto:Method: GET 2024-04-13T21:42:48.821 DEBUG:boto:Path: /vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.821 DEBUG:boto:Data: 2024-04-13T21:42:48.821 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.821 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.821 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.821 DEBUG:boto:Params: {} 2024-04-13T21:42:48.821 DEBUG:boto:Token: None 2024-04-13T21:42:48.821 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/?versions 2024-04-13T21:42:48.822 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU= 2024-04-13T21:42:48.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU=', 'Content-Length': '0'} 2024-04-13T21:42:48.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaac5121035612cb-00661afc58-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.824 DEBUG:boto:b'vwnfns-20999falseobjnulltrue2024-04-13T21:42:18.310Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-13T21:42:48.824 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:48.824 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:48.824 DEBUG:boto:path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.824 DEBUG:boto:auth_path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.825 DEBUG:boto:Method: GET 2024-04-13T21:42:48.825 DEBUG:boto:Path: /vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.825 DEBUG:boto:Data: 2024-04-13T21:42:48.825 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.825 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.825 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.825 DEBUG:boto:Params: {} 2024-04-13T21:42:48.825 DEBUG:boto:Token: None 2024-04-13T21:42:48.825 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/?versions 2024-04-13T21:42:48.825 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU= 2024-04-13T21:42:48.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU=', 'Content-Length': '0'} 2024-04-13T21:42:48.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b05376e9b1c429-00661afc58-1110-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.827 DEBUG:boto:b'vwnfns-20999falseobjnulltrue2024-04-13T21:42:18.310Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.828 DEBUG:boto:path=/vwnfns-20/ 2024-04-13T21:42:48.828 DEBUG:boto:auth_path=/vwnfns-20/ 2024-04-13T21:42:48.828 DEBUG:boto:path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.828 DEBUG:boto:auth_path=/vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.828 DEBUG:boto:Method: GET 2024-04-13T21:42:48.828 DEBUG:boto:Path: /vwnfns-20/?versions&max-keys=999 2024-04-13T21:42:48.828 DEBUG:boto:Data: 2024-04-13T21:42:48.829 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.829 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.829 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.829 DEBUG:boto:Params: {} 2024-04-13T21:42:48.829 DEBUG:boto:Token: None 2024-04-13T21:42:48.829 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/?versions 2024-04-13T21:42:48.829 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU= 2024-04-13T21:42:48.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HVWB4YXAAOk+E1JUH33g4iDmIcU=', 'Content-Length': '0'} 2024-04-13T21:42:48.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007427d0da7363ee0c-00661afc58-1114-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.831 DEBUG:boto:b'vwnfns-20999falseobjnulltrue2024-04-13T21:42:18.310Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T21:42:48.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-13T21:42:48.832 DEBUG:boto:path=/vwnfns-20/obj 2024-04-13T21:42:48.832 DEBUG:boto:auth_path=/vwnfns-20/obj 2024-04-13T21:42:48.832 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.833 DEBUG:boto:Path: /vwnfns-20/obj 2024-04-13T21:42:48.833 DEBUG:boto:Data: 2024-04-13T21:42:48.833 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.833 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.833 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.833 DEBUG:boto:Params: {} 2024-04-13T21:42:48.833 DEBUG:boto:Token: None 2024-04-13T21:42:48.834 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/obj 2024-04-13T21:42:48.834 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uns6w9TuOYEqRuR2lR7oCRMnmIg= 2024-04-13T21:42:48.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uns6w9TuOYEqRuR2lR7oCRMnmIg=', 'Content-Length': '0'} 2024-04-13T21:42:48.836 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007a0b6a8d1ed4fad5-00661afc58-1110-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.836 DEBUG:boto:path=/vwnfns-20/obj 2024-04-13T21:42:48.836 DEBUG:boto:auth_path=/vwnfns-20/obj 2024-04-13T21:42:48.836 DEBUG:boto:Method: HEAD 2024-04-13T21:42:48.836 DEBUG:boto:Path: /vwnfns-20/obj 2024-04-13T21:42:48.836 DEBUG:boto:Data: 2024-04-13T21:42:48.836 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.836 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.837 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.837 DEBUG:boto:Params: {} 2024-04-13T21:42:48.837 DEBUG:boto:Token: None 2024-04-13T21:42:48.837 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-20/obj 2024-04-13T21:42:48.837 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uns6w9TuOYEqRuR2lR7oCRMnmIg= 2024-04-13T21:42:48.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uns6w9TuOYEqRuR2lR7oCRMnmIg=', 'Content-Length': '0'} 2024-04-13T21:42:48.839 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000d7839d7ea5a8fe7-00661afc58-1114-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.839 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-20 zones={a2, a1} 2024-04-13T21:42:48.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-04-13T21:42:48.840 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-21 2024-04-13T21:42:48.840 DEBUG:boto:path=/vwnfns-21/ 2024-04-13T21:42:48.841 DEBUG:boto:auth_path=/vwnfns-21/ 2024-04-13T21:42:48.841 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.841 DEBUG:boto:Path: /vwnfns-21/ 2024-04-13T21:42:48.841 DEBUG:boto:Data: 2024-04-13T21:42:48.841 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.841 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.841 DEBUG:boto:Params: {} 2024-04-13T21:42:48.841 DEBUG:boto:Token: None 2024-04-13T21:42:48.841 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-21/ 2024-04-13T21:42:48.842 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x+nrCKiO9WaOSIsGnDIyAqG2LAo= 2024-04-13T21:42:48.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x+nrCKiO9WaOSIsGnDIyAqG2LAo=', 'Content-Length': '0'} 2024-04-13T21:42:48.854 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001cc7f78f257c5d27-00661afc58-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.855 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-22 2024-04-13T21:42:48.855 DEBUG:boto:path=/vwnfns-22/ 2024-04-13T21:42:48.855 DEBUG:boto:auth_path=/vwnfns-22/ 2024-04-13T21:42:48.855 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.855 DEBUG:boto:Path: /vwnfns-22/ 2024-04-13T21:42:48.855 DEBUG:boto:Data: 2024-04-13T21:42:48.855 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.855 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.856 DEBUG:boto:Params: {} 2024-04-13T21:42:48.856 DEBUG:boto:Token: None 2024-04-13T21:42:48.856 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-22/ 2024-04-13T21:42:48.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u8WVMMtUA7JR7pvyK9FLDkvNi8Y= 2024-04-13T21:42:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u8WVMMtUA7JR7pvyK9FLDkvNi8Y=', 'Content-Length': '0'} 2024-04-13T21:42:48.878 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c332fff985ca10c-00661afc58-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.879 DEBUG:boto:Using access key provided by client. 2024-04-13T21:42:48.880 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:42:48.880 DEBUG:boto:Using access key provided by client. 2024-04-13T21:42:48.881 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:42:48.881 DEBUG:boto:Using access key provided by client. 2024-04-13T21:42:48.881 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:42:48.882 DEBUG:boto:Using access key provided by client. 2024-04-13T21:42:48.882 DEBUG:boto:Using secret key provided by client. 2024-04-13T21:42:48.882 INFO:rgw_multi.tests:create bucket zone=b1 name=vwnfns-23 2024-04-13T21:42:48.882 DEBUG:boto:path=/vwnfns-23/ 2024-04-13T21:42:48.882 DEBUG:boto:auth_path=/vwnfns-23/ 2024-04-13T21:42:48.882 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.882 DEBUG:boto:Path: /vwnfns-23/ 2024-04-13T21:42:48.882 DEBUG:boto:Data: 2024-04-13T21:42:48.883 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.883 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:48.883 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.883 DEBUG:boto:Params: {} 2024-04-13T21:42:48.883 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:42:48.883 DEBUG:boto:Token: None 2024-04-13T21:42:48.884 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-23/ 2024-04-13T21:42:48.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:npDRd7N2zzymg8QRAbGfu2RpcRU= 2024-04-13T21:42:48.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:npDRd7N2zzymg8QRAbGfu2RpcRU=', 'Content-Length': '0'} 2024-04-13T21:42:48.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000062ca1283f5e493bb-00661afc58-10fc-b1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.916 INFO:rgw_multi.tests:create bucket zone=b2 name=vwnfns-24 2024-04-13T21:42:48.916 DEBUG:boto:path=/vwnfns-24/ 2024-04-13T21:42:48.917 DEBUG:boto:auth_path=/vwnfns-24/ 2024-04-13T21:42:48.917 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.917 DEBUG:boto:Path: /vwnfns-24/ 2024-04-13T21:42:48.917 DEBUG:boto:Data: 2024-04-13T21:42:48.917 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.917 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:48.917 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.917 DEBUG:boto:Params: {} 2024-04-13T21:42:48.918 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:42:48.918 DEBUG:boto:Token: None 2024-04-13T21:42:48.918 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-24/ 2024-04-13T21:42:48.918 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7RECAWk8QXnOAwqvJFQ3yy5Xmog= 2024-04-13T21:42:48.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7RECAWk8QXnOAwqvJFQ3yy5Xmog=', 'Content-Length': '0'} 2024-04-13T21:42:48.946 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2dff5c5cf85287f-00661afc58-1102-b2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.946 DEBUG:boto:path=/vwnfns-21/ 2024-04-13T21:42:48.946 DEBUG:boto:auth_path=/vwnfns-21/ 2024-04-13T21:42:48.947 DEBUG:boto:path=/vwnfns-21/?versioning 2024-04-13T21:42:48.947 DEBUG:boto:auth_path=/vwnfns-21/?versioning 2024-04-13T21:42:48.947 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.947 DEBUG:boto:Path: /vwnfns-21/?versioning 2024-04-13T21:42:48.947 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:42:48.947 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.947 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.947 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.948 DEBUG:boto:Params: {} 2024-04-13T21:42:48.948 DEBUG:boto:Token: None 2024-04-13T21:42:48.948 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-21/?versioning 2024-04-13T21:42:48.948 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8C9G1r4LlOOVcv2s7PhRLEi6x88= 2024-04-13T21:42:48.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8C9G1r4LlOOVcv2s7PhRLEi6x88=', 'Content-Length': '228'} 2024-04-13T21:42:48.952 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000810f84cf3f74a0c2-00661afc58-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.952 DEBUG:boto:path=/vwnfns-21/ 2024-04-13T21:42:48.953 DEBUG:boto:auth_path=/vwnfns-21/ 2024-04-13T21:42:48.953 DEBUG:boto:path=/vwnfns-21/?versioning 2024-04-13T21:42:48.953 DEBUG:boto:auth_path=/vwnfns-21/?versioning 2024-04-13T21:42:48.953 DEBUG:boto:Method: GET 2024-04-13T21:42:48.953 DEBUG:boto:Path: /vwnfns-21/?versioning 2024-04-13T21:42:48.953 DEBUG:boto:Data: 2024-04-13T21:42:48.953 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.953 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.953 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.953 DEBUG:boto:Params: {} 2024-04-13T21:42:48.953 DEBUG:boto:Token: None 2024-04-13T21:42:48.954 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-21/?versioning 2024-04-13T21:42:48.954 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IO8rYsGrGOGFMFNX+ZrSXBeaTb4= 2024-04-13T21:42:48.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IO8rYsGrGOGFMFNX+ZrSXBeaTb4=', 'Content-Length': '0'} 2024-04-13T21:42:48.955 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055515077c5b008cc-00661afc58-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.955 DEBUG:boto:EnabledDisabled 2024-04-13T21:42:48.956 DEBUG:boto:path=/vwnfns-22/ 2024-04-13T21:42:48.956 DEBUG:boto:auth_path=/vwnfns-22/ 2024-04-13T21:42:48.956 DEBUG:boto:path=/vwnfns-22/?versioning 2024-04-13T21:42:48.956 DEBUG:boto:auth_path=/vwnfns-22/?versioning 2024-04-13T21:42:48.956 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.956 DEBUG:boto:Path: /vwnfns-22/?versioning 2024-04-13T21:42:48.956 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:42:48.956 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.957 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.957 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.957 DEBUG:boto:Params: {} 2024-04-13T21:42:48.957 DEBUG:boto:Token: None 2024-04-13T21:42:48.957 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-22/?versioning 2024-04-13T21:42:48.958 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hIaTjM3ZaokAX+Ra3wAViPvwLUg= 2024-04-13T21:42:48.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hIaTjM3ZaokAX+Ra3wAViPvwLUg=', 'Content-Length': '228'} 2024-04-13T21:42:48.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd5c2f7108efeccc-00661afc58-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.964 DEBUG:boto:path=/vwnfns-22/ 2024-04-13T21:42:48.964 DEBUG:boto:auth_path=/vwnfns-22/ 2024-04-13T21:42:48.965 DEBUG:boto:path=/vwnfns-22/?versioning 2024-04-13T21:42:48.965 DEBUG:boto:auth_path=/vwnfns-22/?versioning 2024-04-13T21:42:48.965 DEBUG:boto:Method: GET 2024-04-13T21:42:48.965 DEBUG:boto:Path: /vwnfns-22/?versioning 2024-04-13T21:42:48.965 DEBUG:boto:Data: 2024-04-13T21:42:48.965 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:48.965 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.965 DEBUG:boto:Params: {} 2024-04-13T21:42:48.966 DEBUG:boto:Token: None 2024-04-13T21:42:48.966 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-22/?versioning 2024-04-13T21:42:48.966 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/4R8gvc3nxKQxImSfDyXzsmpI6A= 2024-04-13T21:42:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/4R8gvc3nxKQxImSfDyXzsmpI6A=', 'Content-Length': '0'} 2024-04-13T21:42:48.967 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050741c622c17b372-00661afc58-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.969 DEBUG:boto:EnabledDisabled 2024-04-13T21:42:48.969 DEBUG:boto:path=/vwnfns-23/ 2024-04-13T21:42:48.970 DEBUG:boto:auth_path=/vwnfns-23/ 2024-04-13T21:42:48.970 DEBUG:boto:path=/vwnfns-23/?versioning 2024-04-13T21:42:48.970 DEBUG:boto:auth_path=/vwnfns-23/?versioning 2024-04-13T21:42:48.970 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.970 DEBUG:boto:Path: /vwnfns-23/?versioning 2024-04-13T21:42:48.970 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:42:48.971 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:48.971 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.971 DEBUG:boto:Params: {} 2024-04-13T21:42:48.971 DEBUG:boto:Token: None 2024-04-13T21:42:48.971 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-23/?versioning 2024-04-13T21:42:48.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hUmDhPVIJa0GzBVHhFpJ+J/kE1I= 2024-04-13T21:42:48.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hUmDhPVIJa0GzBVHhFpJ+J/kE1I=', 'Content-Length': '228'} 2024-04-13T21:42:48.980 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000216cc2f2d133172d-00661afc58-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.980 DEBUG:boto:path=/vwnfns-23/ 2024-04-13T21:42:48.980 DEBUG:boto:auth_path=/vwnfns-23/ 2024-04-13T21:42:48.980 DEBUG:boto:path=/vwnfns-23/?versioning 2024-04-13T21:42:48.980 DEBUG:boto:auth_path=/vwnfns-23/?versioning 2024-04-13T21:42:48.981 DEBUG:boto:Method: GET 2024-04-13T21:42:48.981 DEBUG:boto:Path: /vwnfns-23/?versioning 2024-04-13T21:42:48.981 DEBUG:boto:Data: 2024-04-13T21:42:48.981 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:48.981 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.981 DEBUG:boto:Params: {} 2024-04-13T21:42:48.981 DEBUG:boto:Token: None 2024-04-13T21:42:48.981 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-23/?versioning 2024-04-13T21:42:48.981 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KzVJxa0figAAgv3v0OhcjTN9Dq4= 2024-04-13T21:42:48.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KzVJxa0figAAgv3v0OhcjTN9Dq4=', 'Content-Length': '0'} 2024-04-13T21:42:48.982 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005deedbec6196fecf-00661afc58-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Sat, 13 Apr 2024 21:42:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.983 DEBUG:boto:EnabledDisabled 2024-04-13T21:42:48.983 DEBUG:boto:path=/vwnfns-24/ 2024-04-13T21:42:48.983 DEBUG:boto:auth_path=/vwnfns-24/ 2024-04-13T21:42:48.983 DEBUG:boto:path=/vwnfns-24/?versioning 2024-04-13T21:42:48.983 DEBUG:boto:auth_path=/vwnfns-24/?versioning 2024-04-13T21:42:48.983 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.983 DEBUG:boto:Path: /vwnfns-24/?versioning 2024-04-13T21:42:48.984 DEBUG:boto:Data: Enabled Disabled 2024-04-13T21:42:48.984 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.984 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:48.984 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.984 DEBUG:boto:Params: {} 2024-04-13T21:42:48.984 DEBUG:boto:Token: None 2024-04-13T21:42:48.984 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-24/?versioning 2024-04-13T21:42:48.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M9TH+ZNmluLKc83doJK8WC177OY= 2024-04-13T21:42:48.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M9TH+ZNmluLKc83doJK8WC177OY=', 'Content-Length': '228'} 2024-04-13T21:42:48.992 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007597c2fb26a68e6a-00661afc58-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.992 DEBUG:boto:path=/vwnfns-24/ 2024-04-13T21:42:48.992 DEBUG:boto:auth_path=/vwnfns-24/ 2024-04-13T21:42:48.992 DEBUG:boto:path=/vwnfns-24/?versioning 2024-04-13T21:42:48.993 DEBUG:boto:auth_path=/vwnfns-24/?versioning 2024-04-13T21:42:48.993 DEBUG:boto:Method: GET 2024-04-13T21:42:48.993 DEBUG:boto:Path: /vwnfns-24/?versioning 2024-04-13T21:42:48.993 DEBUG:boto:Data: 2024-04-13T21:42:48.993 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.993 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:48.993 DEBUG:boto:Port: 8001 2024-04-13T21:42:48.994 DEBUG:boto:Params: {} 2024-04-13T21:42:48.994 DEBUG:boto:Token: None 2024-04-13T21:42:48.994 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-24/?versioning 2024-04-13T21:42:48.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cVLgYsuqPrwmMFSoLpCcCF5rWyk= 2024-04-13T21:42:48.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cVLgYsuqPrwmMFSoLpCcCF5rWyk=', 'Content-Length': '0'} 2024-04-13T21:42:48.995 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd65a9e64556df97-00661afc59-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:48.995 DEBUG:boto:EnabledDisabled 2024-04-13T21:42:48.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-04-13T21:42:48.996 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-25 2024-04-13T21:42:48.996 DEBUG:boto:path=/vwnfns-25/ 2024-04-13T21:42:48.997 DEBUG:boto:auth_path=/vwnfns-25/ 2024-04-13T21:42:48.997 DEBUG:boto:Method: PUT 2024-04-13T21:42:48.997 DEBUG:boto:Path: /vwnfns-25/ 2024-04-13T21:42:48.997 DEBUG:boto:Data: 2024-04-13T21:42:48.997 DEBUG:boto:Headers: {} 2024-04-13T21:42:48.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:48.997 DEBUG:boto:Port: 8000 2024-04-13T21:42:48.998 DEBUG:boto:Params: {} 2024-04-13T21:42:48.998 DEBUG:boto:Token: None 2024-04-13T21:42:48.998 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:48 GMT /vwnfns-25/ 2024-04-13T21:42:48.998 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1dYIem+CYNOtBmSC5s8s7yWDALI= 2024-04-13T21:42:48.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1dYIem+CYNOtBmSC5s8s7yWDALI=', 'Content-Length': '0'} 2024-04-13T21:42:49.010 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000076f5ad149a9747b0-00661afc59-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.010 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-26 2024-04-13T21:42:49.010 DEBUG:boto:path=/vwnfns-26/ 2024-04-13T21:42:49.010 DEBUG:boto:auth_path=/vwnfns-26/ 2024-04-13T21:42:49.010 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.011 DEBUG:boto:Path: /vwnfns-26/ 2024-04-13T21:42:49.011 DEBUG:boto:Data: 2024-04-13T21:42:49.011 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:49.011 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.011 DEBUG:boto:Params: {} 2024-04-13T21:42:49.011 DEBUG:boto:Token: None 2024-04-13T21:42:49.011 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-26/ 2024-04-13T21:42:49.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1SkZNawx9tSHYxhsCZM8XIzOuws= 2024-04-13T21:42:49.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1SkZNawx9tSHYxhsCZM8XIzOuws=', 'Content-Length': '0'} 2024-04-13T21:42:49.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d72a0212b14d4a2-00661afc59-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.044 INFO:rgw_multi.tests:create bucket zone=b1 name=vwnfns-27 2024-04-13T21:42:49.044 DEBUG:boto:path=/vwnfns-27/ 2024-04-13T21:42:49.044 DEBUG:boto:auth_path=/vwnfns-27/ 2024-04-13T21:42:49.045 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.045 DEBUG:boto:Path: /vwnfns-27/ 2024-04-13T21:42:49.045 DEBUG:boto:Data: 2024-04-13T21:42:49.045 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.045 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:49.046 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.046 DEBUG:boto:Params: {} 2024-04-13T21:42:49.046 DEBUG:boto:Token: None 2024-04-13T21:42:49.046 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-27/ 2024-04-13T21:42:49.047 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/m41ksNU2IRJoPWeh6lHldwQZQ= 2024-04-13T21:42:49.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/m41ksNU2IRJoPWeh6lHldwQZQ=', 'Content-Length': '0'} 2024-04-13T21:42:49.077 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000814369745c7527a9-00661afc59-10fc-b1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.077 INFO:rgw_multi.tests:create bucket zone=b2 name=vwnfns-28 2024-04-13T21:42:49.078 DEBUG:boto:path=/vwnfns-28/ 2024-04-13T21:42:49.078 DEBUG:boto:auth_path=/vwnfns-28/ 2024-04-13T21:42:49.078 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.078 DEBUG:boto:Path: /vwnfns-28/ 2024-04-13T21:42:49.078 DEBUG:boto:Data: 2024-04-13T21:42:49.078 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.078 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:49.079 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.079 DEBUG:boto:Params: {} 2024-04-13T21:42:49.079 DEBUG:boto:Token: None 2024-04-13T21:42:49.079 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-28/ 2024-04-13T21:42:49.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4uASgnE1/wM4MYn5gHVu9cFbevk= 2024-04-13T21:42:49.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4uASgnE1/wM4MYn5gHVu9cFbevk=', 'Content-Length': '0'} 2024-04-13T21:42:49.109 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005dec1ab2f7eab7ad-00661afc59-1102-b2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.109 DEBUG:boto:path=/vwnfns-25/ 2024-04-13T21:42:49.109 DEBUG:boto:auth_path=/vwnfns-25/ 2024-04-13T21:42:49.109 DEBUG:boto:path=/vwnfns-25/?acl 2024-04-13T21:42:49.109 DEBUG:boto:auth_path=/vwnfns-25/?acl 2024-04-13T21:42:49.110 DEBUG:boto:Method: GET 2024-04-13T21:42:49.110 DEBUG:boto:Path: /vwnfns-25/?acl 2024-04-13T21:42:49.110 DEBUG:boto:Data: 2024-04-13T21:42:49.110 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:49.110 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.110 DEBUG:boto:Params: {} 2024-04-13T21:42:49.110 DEBUG:boto:Token: None 2024-04-13T21:42:49.110 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-25/?acl 2024-04-13T21:42:49.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qPyxidkErypcXrwt1sUu0Ijy79w= 2024-04-13T21:42:49.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qPyxidkErypcXrwt1sUu0Ijy79w=', 'Content-Length': '0'} 2024-04-13T21:42:49.112 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ec2c5d7a78ac794c-00661afc59-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.112 DEBUG:boto:path=/vwnfns-25/ 2024-04-13T21:42:49.113 DEBUG:boto:auth_path=/vwnfns-25/ 2024-04-13T21:42:49.113 DEBUG:boto:path=/vwnfns-25/?acl 2024-04-13T21:42:49.113 DEBUG:boto:auth_path=/vwnfns-25/?acl 2024-04-13T21:42:49.113 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.113 DEBUG:boto:Path: /vwnfns-25/?acl 2024-04-13T21:42:49.113 DEBUG:boto:Data: 2024-04-13T21:42:49.113 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:42:49.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:49.113 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.113 DEBUG:boto:Params: {} 2024-04-13T21:42:49.113 DEBUG:boto:Token: None 2024-04-13T21:42:49.114 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT x-amz-acl:public-read /vwnfns-25/?acl 2024-04-13T21:42:49.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IJ+yj4rVmGQcJP+Kw/rfHKjHoTE= 2024-04-13T21:42:49.114 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IJ+yj4rVmGQcJP+Kw/rfHKjHoTE=', 'Content-Length': '0'} 2024-04-13T21:42:49.118 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0453f044346fb55-00661afc59-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.119 DEBUG:boto:path=/vwnfns-25/ 2024-04-13T21:42:49.119 DEBUG:boto:auth_path=/vwnfns-25/ 2024-04-13T21:42:49.119 DEBUG:boto:path=/vwnfns-25/?acl 2024-04-13T21:42:49.119 DEBUG:boto:auth_path=/vwnfns-25/?acl 2024-04-13T21:42:49.119 DEBUG:boto:Method: GET 2024-04-13T21:42:49.119 DEBUG:boto:Path: /vwnfns-25/?acl 2024-04-13T21:42:49.119 DEBUG:boto:Data: 2024-04-13T21:42:49.119 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:49.119 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.120 DEBUG:boto:Params: {} 2024-04-13T21:42:49.120 DEBUG:boto:Token: None 2024-04-13T21:42:49.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-25/?acl 2024-04-13T21:42:49.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qPyxidkErypcXrwt1sUu0Ijy79w= 2024-04-13T21:42:49.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qPyxidkErypcXrwt1sUu0Ijy79w=', 'Content-Length': '0'} 2024-04-13T21:42:49.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003f6691ccd042d096-00661afc59-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.122 DEBUG:boto:path=/vwnfns-26/ 2024-04-13T21:42:49.122 DEBUG:boto:auth_path=/vwnfns-26/ 2024-04-13T21:42:49.122 DEBUG:boto:path=/vwnfns-26/?acl 2024-04-13T21:42:49.122 DEBUG:boto:auth_path=/vwnfns-26/?acl 2024-04-13T21:42:49.122 DEBUG:boto:Method: GET 2024-04-13T21:42:49.122 DEBUG:boto:Path: /vwnfns-26/?acl 2024-04-13T21:42:49.122 DEBUG:boto:Data: 2024-04-13T21:42:49.123 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:49.123 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.123 DEBUG:boto:Params: {} 2024-04-13T21:42:49.123 DEBUG:boto:Token: None 2024-04-13T21:42:49.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-26/?acl 2024-04-13T21:42:49.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qWmD4T8rzaZLT+0yQ8X5+c1qu/M= 2024-04-13T21:42:49.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qWmD4T8rzaZLT+0yQ8X5+c1qu/M=', 'Content-Length': '0'} 2024-04-13T21:42:49.124 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004beaea2b27bd05e5-00661afc59-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.125 DEBUG:boto:path=/vwnfns-26/ 2024-04-13T21:42:49.125 DEBUG:boto:auth_path=/vwnfns-26/ 2024-04-13T21:42:49.125 DEBUG:boto:path=/vwnfns-26/?acl 2024-04-13T21:42:49.125 DEBUG:boto:auth_path=/vwnfns-26/?acl 2024-04-13T21:42:49.125 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.125 DEBUG:boto:Path: /vwnfns-26/?acl 2024-04-13T21:42:49.125 DEBUG:boto:Data: 2024-04-13T21:42:49.125 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:42:49.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:49.125 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.125 DEBUG:boto:Params: {} 2024-04-13T21:42:49.126 DEBUG:boto:Token: None 2024-04-13T21:42:49.126 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT x-amz-acl:public-read /vwnfns-26/?acl 2024-04-13T21:42:49.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HvTH8iYK2yUAsnEdIevZn0/kGco= 2024-04-13T21:42:49.126 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HvTH8iYK2yUAsnEdIevZn0/kGco=', 'Content-Length': '0'} 2024-04-13T21:42:49.133 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dea0923805406108-00661afc59-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.133 DEBUG:boto:path=/vwnfns-26/ 2024-04-13T21:42:49.133 DEBUG:boto:auth_path=/vwnfns-26/ 2024-04-13T21:42:49.133 DEBUG:boto:path=/vwnfns-26/?acl 2024-04-13T21:42:49.133 DEBUG:boto:auth_path=/vwnfns-26/?acl 2024-04-13T21:42:49.133 DEBUG:boto:Method: GET 2024-04-13T21:42:49.133 DEBUG:boto:Path: /vwnfns-26/?acl 2024-04-13T21:42:49.133 DEBUG:boto:Data: 2024-04-13T21:42:49.133 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.134 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:49.134 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.134 DEBUG:boto:Params: {} 2024-04-13T21:42:49.134 DEBUG:boto:Token: None 2024-04-13T21:42:49.134 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-26/?acl 2024-04-13T21:42:49.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qWmD4T8rzaZLT+0yQ8X5+c1qu/M= 2024-04-13T21:42:49.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qWmD4T8rzaZLT+0yQ8X5+c1qu/M=', 'Content-Length': '0'} 2024-04-13T21:42:49.135 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae50c1c5b682d394-00661afc59-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.135 DEBUG:boto:path=/vwnfns-27/ 2024-04-13T21:42:49.136 DEBUG:boto:auth_path=/vwnfns-27/ 2024-04-13T21:42:49.136 DEBUG:boto:path=/vwnfns-27/?acl 2024-04-13T21:42:49.136 DEBUG:boto:auth_path=/vwnfns-27/?acl 2024-04-13T21:42:49.136 DEBUG:boto:Method: GET 2024-04-13T21:42:49.136 DEBUG:boto:Path: /vwnfns-27/?acl 2024-04-13T21:42:49.136 DEBUG:boto:Data: 2024-04-13T21:42:49.136 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.136 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:49.136 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.136 DEBUG:boto:Params: {} 2024-04-13T21:42:49.137 DEBUG:boto:Token: None 2024-04-13T21:42:49.137 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-27/?acl 2024-04-13T21:42:49.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5SyiBQ8ONvDv22LZJln+9+n4vBw= 2024-04-13T21:42:49.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5SyiBQ8ONvDv22LZJln+9+n4vBw=', 'Content-Length': '0'} 2024-04-13T21:42:49.138 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af37ed7c44b41c68-00661afc59-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.138 DEBUG:boto:path=/vwnfns-27/ 2024-04-13T21:42:49.138 DEBUG:boto:auth_path=/vwnfns-27/ 2024-04-13T21:42:49.139 DEBUG:boto:path=/vwnfns-27/?acl 2024-04-13T21:42:49.139 DEBUG:boto:auth_path=/vwnfns-27/?acl 2024-04-13T21:42:49.139 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.139 DEBUG:boto:Path: /vwnfns-27/?acl 2024-04-13T21:42:49.139 DEBUG:boto:Data: 2024-04-13T21:42:49.139 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:42:49.139 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:49.139 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.139 DEBUG:boto:Params: {} 2024-04-13T21:42:49.139 DEBUG:boto:Token: None 2024-04-13T21:42:49.140 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT x-amz-acl:public-read /vwnfns-27/?acl 2024-04-13T21:42:49.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Sdv184toLeypu1P6q4sftKXErg= 2024-04-13T21:42:49.140 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Sdv184toLeypu1P6q4sftKXErg=', 'Content-Length': '0'} 2024-04-13T21:42:49.148 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6ca7d6914b857bf-00661afc59-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.148 DEBUG:boto:path=/vwnfns-27/ 2024-04-13T21:42:49.148 DEBUG:boto:auth_path=/vwnfns-27/ 2024-04-13T21:42:49.148 DEBUG:boto:path=/vwnfns-27/?acl 2024-04-13T21:42:49.148 DEBUG:boto:auth_path=/vwnfns-27/?acl 2024-04-13T21:42:49.148 DEBUG:boto:Method: GET 2024-04-13T21:42:49.148 DEBUG:boto:Path: /vwnfns-27/?acl 2024-04-13T21:42:49.148 DEBUG:boto:Data: 2024-04-13T21:42:49.148 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.148 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:49.149 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.149 DEBUG:boto:Params: {} 2024-04-13T21:42:49.149 DEBUG:boto:Token: None 2024-04-13T21:42:49.149 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-27/?acl 2024-04-13T21:42:49.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5SyiBQ8ONvDv22LZJln+9+n4vBw= 2024-04-13T21:42:49.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5SyiBQ8ONvDv22LZJln+9+n4vBw=', 'Content-Length': '0'} 2024-04-13T21:42:49.150 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054834c321df2f9bc-00661afc59-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.151 DEBUG:boto:path=/vwnfns-28/ 2024-04-13T21:42:49.151 DEBUG:boto:auth_path=/vwnfns-28/ 2024-04-13T21:42:49.151 DEBUG:boto:path=/vwnfns-28/?acl 2024-04-13T21:42:49.151 DEBUG:boto:auth_path=/vwnfns-28/?acl 2024-04-13T21:42:49.151 DEBUG:boto:Method: GET 2024-04-13T21:42:49.151 DEBUG:boto:Path: /vwnfns-28/?acl 2024-04-13T21:42:49.151 DEBUG:boto:Data: 2024-04-13T21:42:49.152 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.152 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:49.152 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.152 DEBUG:boto:Params: {} 2024-04-13T21:42:49.152 DEBUG:boto:Token: None 2024-04-13T21:42:49.152 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-28/?acl 2024-04-13T21:42:49.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0JpJNxDoy74IjCDO/oGVEEeSa2E= 2024-04-13T21:42:49.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0JpJNxDoy74IjCDO/oGVEEeSa2E=', 'Content-Length': '0'} 2024-04-13T21:42:49.154 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005bb933b5c900794a-00661afc59-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.154 DEBUG:boto:path=/vwnfns-28/ 2024-04-13T21:42:49.154 DEBUG:boto:auth_path=/vwnfns-28/ 2024-04-13T21:42:49.154 DEBUG:boto:path=/vwnfns-28/?acl 2024-04-13T21:42:49.154 DEBUG:boto:auth_path=/vwnfns-28/?acl 2024-04-13T21:42:49.154 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.154 DEBUG:boto:Path: /vwnfns-28/?acl 2024-04-13T21:42:49.155 DEBUG:boto:Data: 2024-04-13T21:42:49.155 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-13T21:42:49.155 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:49.155 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.155 DEBUG:boto:Params: {} 2024-04-13T21:42:49.155 DEBUG:boto:Token: None 2024-04-13T21:42:49.155 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT x-amz-acl:public-read /vwnfns-28/?acl 2024-04-13T21:42:49.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y0Brn1VEaNXi/6DSVqKhbxkliRM= 2024-04-13T21:42:49.155 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y0Brn1VEaNXi/6DSVqKhbxkliRM=', 'Content-Length': '0'} 2024-04-13T21:42:49.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e72f7bc7a0149d70-00661afc59-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.163 DEBUG:boto:path=/vwnfns-28/ 2024-04-13T21:42:49.163 DEBUG:boto:auth_path=/vwnfns-28/ 2024-04-13T21:42:49.163 DEBUG:boto:path=/vwnfns-28/?acl 2024-04-13T21:42:49.163 DEBUG:boto:auth_path=/vwnfns-28/?acl 2024-04-13T21:42:49.163 DEBUG:boto:Method: GET 2024-04-13T21:42:49.163 DEBUG:boto:Path: /vwnfns-28/?acl 2024-04-13T21:42:49.164 DEBUG:boto:Data: 2024-04-13T21:42:49.164 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:49.164 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.164 DEBUG:boto:Params: {} 2024-04-13T21:42:49.164 DEBUG:boto:Token: None 2024-04-13T21:42:49.165 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-28/?acl 2024-04-13T21:42:49.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0JpJNxDoy74IjCDO/oGVEEeSa2E= 2024-04-13T21:42:49.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0JpJNxDoy74IjCDO/oGVEEeSa2E=', 'Content-Length': '0'} 2024-04-13T21:42:49.166 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e28fd083156ebbb-00661afc59-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-04-13T21:42:49.167 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-29 2024-04-13T21:42:49.167 DEBUG:boto:path=/vwnfns-29/ 2024-04-13T21:42:49.167 DEBUG:boto:auth_path=/vwnfns-29/ 2024-04-13T21:42:49.167 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.167 DEBUG:boto:Path: /vwnfns-29/ 2024-04-13T21:42:49.167 DEBUG:boto:Data: 2024-04-13T21:42:49.168 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:49.168 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.168 DEBUG:boto:Params: {} 2024-04-13T21:42:49.168 DEBUG:boto:Token: None 2024-04-13T21:42:49.168 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-29/ 2024-04-13T21:42:49.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6vi/rbigrWeOJ08J5pfOZ50yjiI= 2024-04-13T21:42:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6vi/rbigrWeOJ08J5pfOZ50yjiI=', 'Content-Length': '0'} 2024-04-13T21:42:49.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d36f7b4aaa542c66-00661afc59-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.183 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-30 2024-04-13T21:42:49.183 DEBUG:boto:path=/vwnfns-30/ 2024-04-13T21:42:49.183 DEBUG:boto:auth_path=/vwnfns-30/ 2024-04-13T21:42:49.183 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.183 DEBUG:boto:Path: /vwnfns-30/ 2024-04-13T21:42:49.184 DEBUG:boto:Data: 2024-04-13T21:42:49.184 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:49.184 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.184 DEBUG:boto:Params: {} 2024-04-13T21:42:49.184 DEBUG:boto:Token: None 2024-04-13T21:42:49.184 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-30/ 2024-04-13T21:42:49.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vVZb9r/8JOgQRfe54QkuUL1n5Og= 2024-04-13T21:42:49.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vVZb9r/8JOgQRfe54QkuUL1n5Og=', 'Content-Length': '0'} 2024-04-13T21:42:49.213 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000561853a6d5d8022e-00661afc59-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.214 INFO:rgw_multi.tests:create bucket zone=b1 name=vwnfns-31 2024-04-13T21:42:49.214 DEBUG:boto:path=/vwnfns-31/ 2024-04-13T21:42:49.214 DEBUG:boto:auth_path=/vwnfns-31/ 2024-04-13T21:42:49.214 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.214 DEBUG:boto:Path: /vwnfns-31/ 2024-04-13T21:42:49.214 DEBUG:boto:Data: 2024-04-13T21:42:49.214 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.214 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:49.215 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.215 DEBUG:boto:Params: {} 2024-04-13T21:42:49.215 DEBUG:boto:Token: None 2024-04-13T21:42:49.215 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-31/ 2024-04-13T21:42:49.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Fo6NNd7KlM9TwlRZmxxWdr9jlCU= 2024-04-13T21:42:49.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Fo6NNd7KlM9TwlRZmxxWdr9jlCU=', 'Content-Length': '0'} 2024-04-13T21:42:49.245 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b826f03d2833b45c-00661afc59-10fc-b1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.245 INFO:rgw_multi.tests:create bucket zone=b2 name=vwnfns-32 2024-04-13T21:42:49.246 DEBUG:boto:path=/vwnfns-32/ 2024-04-13T21:42:49.246 DEBUG:boto:auth_path=/vwnfns-32/ 2024-04-13T21:42:49.246 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.246 DEBUG:boto:Path: /vwnfns-32/ 2024-04-13T21:42:49.246 DEBUG:boto:Data: 2024-04-13T21:42:49.246 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.246 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:49.246 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.246 DEBUG:boto:Params: {} 2024-04-13T21:42:49.247 DEBUG:boto:Token: None 2024-04-13T21:42:49.247 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-32/ 2024-04-13T21:42:49.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B2YcOyyzLziz6J8D8ip0O+QCV4c= 2024-04-13T21:42:49.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B2YcOyyzLziz6J8D8ip0O+QCV4c=', 'Content-Length': '0'} 2024-04-13T21:42:49.274 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000070d7fc13a46e9420-00661afc59-1102-b2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.275 DEBUG:boto:path=/vwnfns-29/ 2024-04-13T21:42:49.275 DEBUG:boto:auth_path=/vwnfns-29/ 2024-04-13T21:42:49.275 DEBUG:boto:path=/vwnfns-29/?cors 2024-04-13T21:42:49.275 DEBUG:boto:auth_path=/vwnfns-29/?cors 2024-04-13T21:42:49.275 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.275 DEBUG:boto:Path: /vwnfns-29/?cors 2024-04-13T21:42:49.275 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-13T21:42:49.275 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-13T21:42:49.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:49.275 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.276 DEBUG:boto:Params: {} 2024-04-13T21:42:49.276 DEBUG:boto:Token: None 2024-04-13T21:42:49.276 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-29/?cors 2024-04-13T21:42:49.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ohAGV5mKqgC6KiynK1MdqR5UmWM= 2024-04-13T21:42:49.276 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ohAGV5mKqgC6KiynK1MdqR5UmWM=', 'Content-Length': '218'} 2024-04-13T21:42:49.282 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003cd2da7c7ac19f2c-00661afc59-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.282 DEBUG:boto:path=/vwnfns-29/ 2024-04-13T21:42:49.282 DEBUG:boto:auth_path=/vwnfns-29/ 2024-04-13T21:42:49.282 DEBUG:boto:path=/vwnfns-29/?cors 2024-04-13T21:42:49.282 DEBUG:boto:auth_path=/vwnfns-29/?cors 2024-04-13T21:42:49.282 DEBUG:boto:Method: GET 2024-04-13T21:42:49.282 DEBUG:boto:Path: /vwnfns-29/?cors 2024-04-13T21:42:49.283 DEBUG:boto:Data: 2024-04-13T21:42:49.283 DEBUG:boto:Headers: {} 2024-04-13T21:42:49.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:49.283 DEBUG:boto:Port: 8000 2024-04-13T21:42:49.283 DEBUG:boto:Params: {} 2024-04-13T21:42:49.283 DEBUG:boto:Token: None 2024-04-13T21:42:49.283 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-29/?cors 2024-04-13T21:42:49.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mcKdsM/xzP4wgwB998Xk9IRhgWQ= 2024-04-13T21:42:49.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mcKdsM/xzP4wgwB998Xk9IRhgWQ=', 'Content-Length': '0'} 2024-04-13T21:42:49.284 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bcc053917a5e6b5b-00661afc59-1114-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Sat, 13 Apr 2024 21:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:49.326 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-13T21:42:49.327 DEBUG:boto:path=/vwnfns-30/ 2024-04-13T21:42:49.327 DEBUG:boto:auth_path=/vwnfns-30/ 2024-04-13T21:42:49.327 DEBUG:boto:path=/vwnfns-30/?cors 2024-04-13T21:42:49.327 DEBUG:boto:auth_path=/vwnfns-30/?cors 2024-04-13T21:42:49.327 DEBUG:boto:Method: PUT 2024-04-13T21:42:49.327 DEBUG:boto:Path: /vwnfns-30/?cors 2024-04-13T21:42:49.328 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-13T21:42:49.328 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-13T21:42:49.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:49.328 DEBUG:boto:Port: 8001 2024-04-13T21:42:49.328 DEBUG:boto:Params: {} 2024-04-13T21:42:49.328 DEBUG:boto:Token: None 2024-04-13T21:42:49.328 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Sat, 13 Apr 2024 21:42:49 GMT /vwnfns-30/?cors 2024-04-13T21:42:49.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JrYZQLrPK6Tdk06ZqAAdsZtWmk4= 2024-04-13T21:42:49.329 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JrYZQLrPK6Tdk06ZqAAdsZtWmk4=', 'Content-Length': '218'} 2024-04-13T21:42:50.337 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a237a9af5ff80bc-00661afc59-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:50.337 DEBUG:boto:path=/vwnfns-30/ 2024-04-13T21:42:50.337 DEBUG:boto:auth_path=/vwnfns-30/ 2024-04-13T21:42:50.338 DEBUG:boto:path=/vwnfns-30/?cors 2024-04-13T21:42:50.338 DEBUG:boto:auth_path=/vwnfns-30/?cors 2024-04-13T21:42:50.338 DEBUG:boto:Method: GET 2024-04-13T21:42:50.338 DEBUG:boto:Path: /vwnfns-30/?cors 2024-04-13T21:42:50.338 DEBUG:boto:Data: 2024-04-13T21:42:50.338 DEBUG:boto:Headers: {} 2024-04-13T21:42:50.338 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:50.338 DEBUG:boto:Port: 8001 2024-04-13T21:42:50.338 DEBUG:boto:Params: {} 2024-04-13T21:42:50.339 DEBUG:boto:Token: None 2024-04-13T21:42:50.339 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:50 GMT /vwnfns-30/?cors 2024-04-13T21:42:50.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zmSGs/XKWftzrZ/jOjvQ6Q3eTtE= 2024-04-13T21:42:50.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zmSGs/XKWftzrZ/jOjvQ6Q3eTtE=', 'Content-Length': '0'} 2024-04-13T21:42:50.340 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da247d4c3aadcddc-00661afc5a-1110-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Sat, 13 Apr 2024 21:42:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:50.382 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-13T21:42:50.383 DEBUG:boto:path=/vwnfns-31/ 2024-04-13T21:42:50.383 DEBUG:boto:auth_path=/vwnfns-31/ 2024-04-13T21:42:50.383 DEBUG:boto:path=/vwnfns-31/?cors 2024-04-13T21:42:50.383 DEBUG:boto:auth_path=/vwnfns-31/?cors 2024-04-13T21:42:50.383 DEBUG:boto:Method: PUT 2024-04-13T21:42:50.383 DEBUG:boto:Path: /vwnfns-31/?cors 2024-04-13T21:42:50.383 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-13T21:42:50.383 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-13T21:42:50.383 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:50.383 DEBUG:boto:Port: 8000 2024-04-13T21:42:50.384 DEBUG:boto:Params: {} 2024-04-13T21:42:50.384 DEBUG:boto:Token: None 2024-04-13T21:42:50.384 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Sat, 13 Apr 2024 21:42:50 GMT /vwnfns-31/?cors 2024-04-13T21:42:50.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ylB4+8yF8E4ryerXl7RUBpx3Ak8= 2024-04-13T21:42:50.384 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ylB4+8yF8E4ryerXl7RUBpx3Ak8=', 'Content-Length': '218'} 2024-04-13T21:42:51.394 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000092a3b08aa6aa497d-00661afc5a-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:51.394 DEBUG:boto:path=/vwnfns-31/ 2024-04-13T21:42:51.394 DEBUG:boto:auth_path=/vwnfns-31/ 2024-04-13T21:42:51.395 DEBUG:boto:path=/vwnfns-31/?cors 2024-04-13T21:42:51.395 DEBUG:boto:auth_path=/vwnfns-31/?cors 2024-04-13T21:42:51.395 DEBUG:boto:Method: GET 2024-04-13T21:42:51.395 DEBUG:boto:Path: /vwnfns-31/?cors 2024-04-13T21:42:51.395 DEBUG:boto:Data: 2024-04-13T21:42:51.395 DEBUG:boto:Headers: {} 2024-04-13T21:42:51.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:42:51.395 DEBUG:boto:Port: 8000 2024-04-13T21:42:51.395 DEBUG:boto:Params: {} 2024-04-13T21:42:51.395 DEBUG:boto:Token: None 2024-04-13T21:42:51.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:51 GMT /vwnfns-31/?cors 2024-04-13T21:42:51.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BuuIi2PLImofsNpFp574fX/axHM= 2024-04-13T21:42:51.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BuuIi2PLImofsNpFp574fX/axHM=', 'Content-Length': '0'} 2024-04-13T21:42:51.397 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008a8e8211f9c328de-00661afc5b-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Sat, 13 Apr 2024 21:42:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:51.397 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-13T21:42:51.398 DEBUG:boto:path=/vwnfns-32/ 2024-04-13T21:42:51.398 DEBUG:boto:auth_path=/vwnfns-32/ 2024-04-13T21:42:51.398 DEBUG:boto:path=/vwnfns-32/?cors 2024-04-13T21:42:51.398 DEBUG:boto:auth_path=/vwnfns-32/?cors 2024-04-13T21:42:51.398 DEBUG:boto:Method: PUT 2024-04-13T21:42:51.398 DEBUG:boto:Path: /vwnfns-32/?cors 2024-04-13T21:42:51.398 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-13T21:42:51.398 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-13T21:42:51.398 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:51.398 DEBUG:boto:Port: 8001 2024-04-13T21:42:51.399 DEBUG:boto:Params: {} 2024-04-13T21:42:51.399 DEBUG:boto:Token: None 2024-04-13T21:42:51.399 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Sat, 13 Apr 2024 21:42:51 GMT /vwnfns-32/?cors 2024-04-13T21:42:51.399 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N7s7cOr4aUKslWQNVu3t35nl8ro= 2024-04-13T21:42:51.399 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N7s7cOr4aUKslWQNVu3t35nl8ro=', 'Content-Length': '218'} 2024-04-13T21:42:52.409 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000660ec4068e95ae4e-00661afc5b-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:52.409 DEBUG:boto:path=/vwnfns-32/ 2024-04-13T21:42:52.409 DEBUG:boto:auth_path=/vwnfns-32/ 2024-04-13T21:42:52.409 DEBUG:boto:path=/vwnfns-32/?cors 2024-04-13T21:42:52.410 DEBUG:boto:auth_path=/vwnfns-32/?cors 2024-04-13T21:42:52.410 DEBUG:boto:Method: GET 2024-04-13T21:42:52.410 DEBUG:boto:Path: /vwnfns-32/?cors 2024-04-13T21:42:52.410 DEBUG:boto:Data: 2024-04-13T21:42:52.410 DEBUG:boto:Headers: {} 2024-04-13T21:42:52.410 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:42:52.410 DEBUG:boto:Port: 8001 2024-04-13T21:42:52.410 DEBUG:boto:Params: {} 2024-04-13T21:42:52.410 DEBUG:boto:Token: None 2024-04-13T21:42:52.411 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:42:52 GMT /vwnfns-32/?cors 2024-04-13T21:42:52.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tWOOW/B6LNw1MZgPI9Me2eMDkH8= 2024-04-13T21:42:52.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tWOOW/B6LNw1MZgPI9Me2eMDkH8=', 'Content-Length': '0'} 2024-04-13T21:42:52.412 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001161dbdf99cd4095-00661afc5c-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Sat, 13 Apr 2024 21:42:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:52.458 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-13T21:42:52.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-04-13T21:42:52.460 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-33 2024-04-13T21:42:52.460 DEBUG:boto:path=/vwnfns-33/ 2024-04-13T21:42:52.460 DEBUG:boto:auth_path=/vwnfns-33/ 2024-04-13T21:42:52.460 DEBUG:boto:Method: PUT 2024-04-13T21:42:52.460 DEBUG:boto:Path: /vwnfns-33/ 2024-04-13T21:42:52.460 DEBUG:boto:Data: 2024-04-13T21:42:52.460 DEBUG:boto:Headers: {} 2024-04-13T21:42:52.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:42:52.460 DEBUG:boto:Port: 8000 2024-04-13T21:42:52.461 DEBUG:boto:Params: {} 2024-04-13T21:42:52.461 DEBUG:boto:Token: None 2024-04-13T21:42:52.461 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:52 GMT /vwnfns-33/ 2024-04-13T21:42:52.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TwFUNM3CUNf6qQA5gl837IDak+g= 2024-04-13T21:42:52.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TwFUNM3CUNf6qQA5gl837IDak+g=', 'Content-Length': '0'} 2024-04-13T21:42:52.478 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002efe663443d6da87-00661afc5c-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:52.478 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-34 2024-04-13T21:42:52.478 DEBUG:boto:path=/vwnfns-34/ 2024-04-13T21:42:52.478 DEBUG:boto:auth_path=/vwnfns-34/ 2024-04-13T21:42:52.478 DEBUG:boto:Method: PUT 2024-04-13T21:42:52.479 DEBUG:boto:Path: /vwnfns-34/ 2024-04-13T21:42:52.479 DEBUG:boto:Data: 2024-04-13T21:42:52.479 DEBUG:boto:Headers: {} 2024-04-13T21:42:52.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:42:52.479 DEBUG:boto:Port: 8001 2024-04-13T21:42:52.479 DEBUG:boto:Params: {} 2024-04-13T21:42:52.479 DEBUG:boto:Token: None 2024-04-13T21:42:52.479 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:42:52 GMT /vwnfns-34/ 2024-04-13T21:42:52.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTdpTtLOBP025EudLB50Sl6ofRk= 2024-04-13T21:42:52.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTdpTtLOBP025EudLB50Sl6ofRk=', 'Content-Length': '0'} 2024-04-13T21:42:52.506 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000998467682132c6b8-00661afc5c-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:42:52.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:52.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:52.507 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:42:52.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:52.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:52.594 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:42:52.605 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:42:52.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.605 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044572.504898_335.1", 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:42:52.504898Z" 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:42:52.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.607 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:52.607 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:42:52.607 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:52.607 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:42:52.608 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:52.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:42:52.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:52.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:52.609 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:42:52.661 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:52.661 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:52.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:52.707+0000 7f623d2a4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:52.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:52.708+0000 7f623d2a4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:52.712 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:42:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:42:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:42:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:42:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:42:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:42:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:42:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:42:52Z" 2024-04-13T21:42:52.719 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:42:52.720 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:42:52Z" } 2024-04-13T21:42:52.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:42:52.720 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:52.721 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:52.721 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:42:57.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:57.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:42:57.722 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:42:57.773 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:42:57.773 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:42:57.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:57.810+0000 7fe97fbcd780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:42:57.811+0000 7fe97fbcd780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:42:57.814 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:42:57.815 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:57.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:57.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:57.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:42:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:42:57Z" 2024-04-13T21:42:57.820 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:42:57.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:42:57Z" } 2024-04-13T21:42:57.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:42:57.822 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:57.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:42:57.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:43:02.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:02.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:02.824 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:02.881 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:02.882 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:02.922+0000 7f43cbece780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:02.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:02.923+0000 7f43cbece780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:02.927 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:02.927 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:02.927 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:02.927 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:02.927 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:02.927 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:02.928 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:02.929 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:02.930 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:02.931 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:02.932 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:02.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:02.933 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:02Z" 2024-04-13T21:43:02.933 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:02.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:02Z" } 2024-04-13T21:43:02.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:02.937 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:02.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:02.938 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:43:07.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:07.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:07.939 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:07.995 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:07.995 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:08.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:08.041+0000 7fb38270b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:08.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:08.042+0000 7fb38270b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:08.045 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:08.045 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:08.045 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:08.045 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:08.045 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:43:08.046 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:08.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:08.049 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:08.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:08.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:08.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:08Z" 2024-04-13T21:43:08.053 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:08.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:08Z" } 2024-04-13T21:43:08.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:08.055 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:08.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:08.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:43:13.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:13.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:13.057 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:13.107 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:13.107 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:13.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:13.143+0000 7f4245489780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:13.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:13.144+0000 7f4245489780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:13.147 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:13.148 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:13.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:13.150 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:13.151 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:13.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:13.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:13.154 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:13.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:13.154 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:13.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:13.154 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:13Z" 2024-04-13T21:43:13.154 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:13.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:13Z" } 2024-04-13T21:43:13.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:13.155 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:13.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:13.156 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:43:18.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:18.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:18.157 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:18.212 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:18.212 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:18.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:18.251+0000 7f16db0db780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:18.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:18.252+0000 7f16db0db780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:18.257 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:18.258 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:18.259 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:18.260 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:18.261 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:18Z" 2024-04-13T21:43:18.263 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:18.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:18Z" } 2024-04-13T21:43:18.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:18.265 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:18.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:18.265 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:43:23.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:23.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:23.267 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:23.323 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:23.323 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:23.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:23.362+0000 7f7d48b4d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:23.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:23.363+0000 7f7d48b4d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:23.367 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:23.367 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:23.368 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:23.369 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:23.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:23.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:23.372 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:23Z" 2024-04-13T21:43:23.373 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:23.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:23Z" } 2024-04-13T21:43:23.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:23.375 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:23.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:23.375 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:43:28.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:28.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:28.376 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:28.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:28.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:28.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:28.488+0000 7fc4b1626780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:28.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:28.489+0000 7fc4b1626780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:28.494 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:28.494 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:28.494 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:28.494 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:28.494 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:28.494 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:28.495 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:28.496 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:28.497 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:28.498 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:28.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:28.500 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:28.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:28.500 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:28Z" 2024-04-13T21:43:28.500 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:28.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:28Z" } 2024-04-13T21:43:28.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:28.501 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:28.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:28.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:43:33.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:33.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:33.503 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:33.557 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:33.557 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:33.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:33.596+0000 7fbf67696780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:33.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:33.597+0000 7fbf67696780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:33.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.914750_265.1", 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:33.602 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.914750Z", 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:33.603 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:33.604 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:33.605 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:33.606 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:33Z" 2024-04-13T21:43:33.607 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:33.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.914750_265.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.914750Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:33Z" } 2024-04-13T21:43:33.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.914750_265.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.914750Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:33.608 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:33.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.914750_265.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:33.609 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.914750_265.1 2024-04-13T21:43:38.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:38.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:38.610 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:38.663 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:38.663 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:38.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:38.720+0000 7f47ddc26780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:38.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:38.721+0000 7f47ddc26780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:38.733 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:38.733 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:38.733 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:38.734 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:38.735 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:38.736 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:38.737 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:38.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:38Z" 2024-04-13T21:43:38.739 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:38.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:38Z" } 2024-04-13T21:43:38.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:38.740 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:38.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:38.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:43:43.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:43.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:43.742 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:43.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:43.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:43.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:43.828+0000 7f1fe2f61780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:43.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:43.829+0000 7f1fe2f61780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:43Z" 2024-04-13T21:43:43.838 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:43.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:43Z" } 2024-04-13T21:43:43.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:43.840 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:43.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:43.840 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:43:48.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:48.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:48.842 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:48.896 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:48.896 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:48.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:48.936+0000 7fab47053780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:48.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:48.937+0000 7fab47053780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:48.941 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:43:48.942 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:48.943 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:48.944 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:48.945 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:48.946 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:48.947 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:48.947 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:48.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:48.947 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:48Z" 2024-04-13T21:43:48.947 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:48.948 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:48Z" } 2024-04-13T21:43:48.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:48.949 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:48.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:48.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:43:53.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:53.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:53.951 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:54.006 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:54.006 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:54.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:54.045+0000 7f698181c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:54.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:54.046+0000 7f698181c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:54.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:54.060 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:54.061 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:54.062 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:54.063 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:54.064 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:54.065 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:54.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:54.065 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:54Z" 2024-04-13T21:43:54.065 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:54.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:54Z" } 2024-04-13T21:43:54.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:54.066 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:54.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:54.066 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:43:59.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:59.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:43:59.068 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:43:59.124 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:43:59.125 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:43:59.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:59.163+0000 7f8f6a246780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:59.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:43:59.164+0000 7f8f6a246780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:43:59.170 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:59.171 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:59.172 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:59.173 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:43:59.174 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:59.175 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:43:59.176 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:43:59.176 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:43:59.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:43:59.176 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:43:59Z" 2024-04-13T21:43:59.176 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:43:59.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:43:59Z" } 2024-04-13T21:43:59.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:43:59.177 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:59.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:43:59.178 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.912549_264.1 2024-04-13T21:44:04.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:04.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:04.180 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:04.235 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:04.235 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:04.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:04.274+0000 7f0e1134a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:04.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:04.276+0000 7f0e1134a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:04.280 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:04.280 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:04.281 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:04.282 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:04.283 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:04.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:04.285 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:04Z" 2024-04-13T21:44:04.286 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:04.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:04Z" } 2024-04-13T21:44:04.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:04.287 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:04.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:04.288 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:09.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:09.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:09.289 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:09.342 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:09.343 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:09.374 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:09.381+0000 7f4c2b1b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:09.374 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:09.382+0000 7f4c2b1b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:09.387 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:09.387 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:09.387 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:09.387 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:09.387 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:09.387 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:09.388 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:09.389 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:09.390 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:09.391 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:09.392 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:09.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:09.393 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:09.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:09.393 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:09Z" 2024-04-13T21:44:09.393 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:09.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:09Z" } 2024-04-13T21:44:09.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:09.394 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:09.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:09.394 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:14.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:14.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:14.396 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:14.452 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:14.452 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:14.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:14.492+0000 7f67e081f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:14.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:14.493+0000 7f67e081f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:14.497 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:14.497 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:14.497 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:14.498 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:14.500 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:14.501 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:14.503 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:14.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:14.503 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:14Z" 2024-04-13T21:44:14.503 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:14.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:14Z" } 2024-04-13T21:44:14.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:14.504 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:14.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:14.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:19.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:19.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:19.505 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:19.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:19.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:19.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:19.600+0000 7f54ae26f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:19.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:19.601+0000 7f54ae26f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:19.604 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:19.605 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:19.606 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:19.607 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:19.608 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:19.609 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:19.610 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:19.611 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:19.611 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:19.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:19.611 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:19Z" 2024-04-13T21:44:19.611 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:19.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:19Z" } 2024-04-13T21:44:19.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:19.612 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:19.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:19.613 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:24.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:24.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:24.615 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:24.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:24.707+0000 7fb16a85d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:24.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:24.708+0000 7fb16a85d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:24Z" 2024-04-13T21:44:24.717 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:24.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:24Z" } 2024-04-13T21:44:24.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:24.718 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:24.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:24.719 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:29.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:29.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:29.721 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:29.776 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:29.777 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:29.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:29.816+0000 7ff15743c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:29.809 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:29.817+0000 7ff15743c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:29.821 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:29.821 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:29.821 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:29.822 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:29.823 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:29.824 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:29.825 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:29.826 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:29Z" 2024-04-13T21:44:29.827 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:29.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:29Z" } 2024-04-13T21:44:29.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:29.829 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:29.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:29.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.912549_264.1 2024-04-13T21:44:34.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:34.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:34.831 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:34.886 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:34.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:34.925+0000 7f00da560780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:34.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:34.926+0000 7f00da560780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:34.931 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:34.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:34.933 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:34.934 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:34.935 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:34.936 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:34Z" 2024-04-13T21:44:34.937 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:34.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:34Z" } 2024-04-13T21:44:34.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:34.938 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:34.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:34.938 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:39.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:39.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:39.940 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:39.998 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:39.998 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:40.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:40.093+0000 7f8f01924780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:40.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:40.094+0000 7f8f01924780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:40.098 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:40.098 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:40.099 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:40.100 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:40.101 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:40.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:40Z" 2024-04-13T21:44:40.104 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:40.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:40Z" } 2024-04-13T21:44:40.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:40.106 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:40.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:40.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:45.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:45.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:45.108 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:45.166 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:45.166 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:45.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:45.204+0000 7f2064dd8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:45.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:45.205+0000 7f2064dd8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:45.210 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:45.210 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:45.211 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:45.212 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:45.213 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:45.213 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:45.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:45.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:45.213 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:45.213 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:45.214 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:45.215 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:45.216 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:45.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:45.218 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:45.218 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:45.218 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:45.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:45.218 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:45Z" 2024-04-13T21:44:45.218 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:45.219 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:45Z" } 2024-04-13T21:44:45.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:45.219 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:45.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:45.220 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:50.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:50.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:50.221 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:50.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:50.318+0000 7f0bbd0fa780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:50.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:50.319+0000 7f0bbd0fa780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:50.322 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:50.322 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:50.322 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:50.322 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:50.322 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:50.322 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:50.323 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:50.324 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:50.325 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:50.326 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:50.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:50.328 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:50.328 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:50.328 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:50.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:50.328 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:50Z" 2024-04-13T21:44:50.328 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:50.329 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:50Z" } 2024-04-13T21:44:50.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:50.329 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:50.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:50.330 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:44:55.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:55.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:44:55.332 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:44:55.387 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:44:55.387 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:44:55.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:55.431+0000 7f124ec27780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:55.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:44:55.432+0000 7f124ec27780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:44:55.435 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:44:55.435 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:44:55.435 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:44:55.435 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:44:55.435 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:44:55.436 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:55.437 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:55.438 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:55.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:55.440 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:44:55Z" 2024-04-13T21:44:55.441 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:44:55.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:44:55Z" } 2024-04-13T21:44:55.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:44:55.442 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:55.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:44:55.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:45:00.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:00.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:00.445 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:00.505 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:00.505 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:00.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:00.544+0000 7f0d12fe5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:00.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:00.545+0000 7f0d12fe5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:00.548 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:00Z" 2024-04-13T21:45:00.554 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:00.555 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:00Z" } 2024-04-13T21:45:00.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:00.556 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:00.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:00.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:45:05.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:05.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:05.558 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:05.611 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:05.611 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:05.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:05.649+0000 7fb29c11c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:05.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:05.650+0000 7fb29c11c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:05.653 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:05Z" 2024-04-13T21:45:05.660 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:05.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:05Z" } 2024-04-13T21:45:05.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:05.662 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:05.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:05.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:45:10.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:10.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:10.663 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:10.718 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:10.718 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:10.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:10.757+0000 7f14ed4ae780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:10.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:10.758+0000 7f14ed4ae780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:10.763 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:10.763 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:10.763 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:10.763 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:10.763 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:10.763 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:10.764 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:10.765 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:10.766 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:10.767 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:10.768 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:10.769 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:10.769 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:10.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:10.769 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:10Z" 2024-04-13T21:45:10.769 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:10.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:10Z" } 2024-04-13T21:45:10.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:10.770 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:10.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:10.771 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:45:15.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:15.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:15.772 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:15.829 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:15.829 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:15.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:15.867+0000 7f638f372780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:15.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:15.867+0000 7f638f372780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:15.871 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:15.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:15.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:15.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:15.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:15.876 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:15Z" 2024-04-13T21:45:15.877 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:15.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:15Z" } 2024-04-13T21:45:15.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:15.879 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:15.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:15.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:45:20.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:20.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:20.880 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:20.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:20.936 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:20.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:20.975+0000 7fb978b0c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:20.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:20.976+0000 7fb978b0c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:20.981 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:20.981 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:20.981 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:20.981 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:20.981 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:20.981 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.959035_270.1", 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:20.982 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.959035Z", 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:20.983 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:20.984 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:20.985 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:20.986 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:20.987 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:20.987 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:20.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:20.987 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:20Z" 2024-04-13T21:45:20.987 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:20.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.959035_270.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.959035Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:20Z" } 2024-04-13T21:45:20.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.959035_270.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.959035Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:20.988 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:20.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.959035_270.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:20.988 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.959035_270.1 2024-04-13T21:45:25.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:25.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:25.990 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:26.045 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:26.046 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:26.085+0000 7fb29b872780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:26.086+0000 7fb29b872780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:26.090 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:26.090 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:26.091 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.996146_276.1", 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.996146Z", 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:26.092 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:26.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:26.094 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:26.095 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:26Z" 2024-04-13T21:45:26.096 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:26.097 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.996146_276.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.996146Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:26Z" } 2024-04-13T21:45:26.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.996146_276.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.996146Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:26.098 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:26.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.996146_276.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:26.098 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.996146_276.1 2024-04-13T21:45:31.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:31.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:31.099 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:31.153 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:31.153 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:31.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:31.193+0000 7f53f1352780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:31.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:31.194+0000 7f53f1352780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:31.197 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:31Z" 2024-04-13T21:45:31.203 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:31.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:31Z" } 2024-04-13T21:45:31.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:31.205 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:31.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:31.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:45:36.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:36.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:36.207 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:36.262 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:36.262 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:36.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:36.324+0000 7f64c70c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:36.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:36.326+0000 7f64c70c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:36.335 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:36.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:36.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:36.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:36.339 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:36.340 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:36.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:36.342 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:36Z" 2024-04-13T21:45:36.342 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:36.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:36Z" } 2024-04-13T21:45:36.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:36.343 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:36.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:36.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:45:41.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:41.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:41.345 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:41.402 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:41.402 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:41.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:41.442+0000 7f73ae408780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:41.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:41.443+0000 7f73ae408780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:41.448 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:41.448 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:41.448 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:41.448 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:45:41.449 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:41.450 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:41.451 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:41.452 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:41.453 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:41.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:41.455 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:41.455 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:41.455 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:41.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:41.455 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:41Z" 2024-04-13T21:45:41.455 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:41.456 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:41Z" } 2024-04-13T21:45:41.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:41.456 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:41.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:41.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:45:46.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:46.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:46.458 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:46.514 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:46.514 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:46.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:46.553+0000 7fc234291780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:46.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:46.554+0000 7fc234291780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:46.559 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:46.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:46.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:46.562 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:46.563 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:46.564 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:46Z" 2024-04-13T21:45:46.565 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:46.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:46Z" } 2024-04-13T21:45:46.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:46.567 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:46.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:46.567 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:45:51.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:51.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:51.569 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:51.624 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:51.624 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:51.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:51.663+0000 7f04ae962780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:51.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:51.664+0000 7f04ae962780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:51.669 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:51.671 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:51.672 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:51.673 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:51.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:51.675 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:51Z" 2024-04-13T21:45:51.676 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:51.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:51Z" } 2024-04-13T21:45:51.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:51.678 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:51.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:51.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:45:56.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:56.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:45:56.679 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:45:56.734 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:45:56.734 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:45:56.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:56.775+0000 7ff41c41f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:56.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:45:56.776+0000 7ff41c41f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:45:56.780 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:56.781 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:56.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:56.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:56.784 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:45:56.785 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:45:56Z" 2024-04-13T21:45:56.786 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:45:56.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:45:56Z" } 2024-04-13T21:45:56.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:45:56.788 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:56.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:45:56.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:01.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:01.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:01.790 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:01.845 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:01.846 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:01.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:01.885+0000 7f83041c9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:01.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:01.886+0000 7f83041c9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:01.890 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:01.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:01.894 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:01.895 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:01Z" 2024-04-13T21:46:01.896 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:01.897 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:01Z" } 2024-04-13T21:46:01.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:01.898 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:01.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:01.898 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:06.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:06.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:06.899 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:06.954 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:06.954 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:06.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:06.994+0000 7fc2220c5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:06.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:06.995+0000 7fc2220c5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:06.999 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:07.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:07.001 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:07.002 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:07.003 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:06Z" 2024-04-13T21:46:07.005 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:07.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:06Z" } 2024-04-13T21:46:07.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:07.006 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:07.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:07.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:46:12.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:12.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:12.007 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:12.063 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:12.063 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:12.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:12.103+0000 7f4ac4065780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:12.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:12.104+0000 7f4ac4065780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:12.107 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:12.108 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:12.109 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:12.110 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:12.111 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:12.112 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:12.113 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:12.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:12.114 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:12Z" 2024-04-13T21:46:12.114 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:12.115 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:12Z" } 2024-04-13T21:46:12.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:12.115 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:12.116 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:12.116 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:17.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:17.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:17.118 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:17.176 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:17.177 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:17.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:17.216+0000 7f7e74fd6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:17.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:17.217+0000 7f7e74fd6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:17.220 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:17.220 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:17.221 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:17.222 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:17.223 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:17.225 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:17Z" 2024-04-13T21:46:17.226 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:17.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:17Z" } 2024-04-13T21:46:17.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:17.227 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:17.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:17.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:46:22.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:22.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:22.229 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:22.284 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:22.284 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:22.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:22.324+0000 7ff0587f9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:22.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:22.325+0000 7ff0587f9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:22.329 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:22.329 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:22.329 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:22.329 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:22.329 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:46:22.330 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:22.331 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:22.332 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:22.333 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:22.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:22.335 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:22.335 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:22.335 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:22.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:22.335 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:22Z" 2024-04-13T21:46:22.335 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:22.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:22Z" } 2024-04-13T21:46:22.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:22.336 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:22.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:22.336 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:46:27.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:27.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:27.338 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:27.394 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:27.394 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:27.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:27.434+0000 7f6e8af4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:27.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:27.435+0000 7f6e8af4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:27.441 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:27.442 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:27.443 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:27.444 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:27.445 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:27.446 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:27.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:27.447 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:27Z" 2024-04-13T21:46:27.447 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:27.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:27Z" } 2024-04-13T21:46:27.448 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:27.448 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:27.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:27.448 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:32.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:32.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:32.450 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:32.504 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:32.505 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:32.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:32.543+0000 7fbfccf86780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:32.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:32.544+0000 7fbfccf86780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:32.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:32.550 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:32.551 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:32.552 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:32.553 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:32.554 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:32.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:32.555 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:32Z" 2024-04-13T21:46:32.555 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:32.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:32Z" } 2024-04-13T21:46:32.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:32.556 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:32.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:32.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:37.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:37.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:37.558 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:37.612 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:37.612 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:37.650+0000 7fda5cb2f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:37.651+0000 7fda5cb2f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:37.655 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:37.659 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:37.661 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:37.661 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:37.662 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:37.663 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:37Z" 2024-04-13T21:46:37.664 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:37.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:37Z" } 2024-04-13T21:46:37.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:37.666 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:37.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:37.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:42.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:42.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:42.668 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:42.723 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:42.724 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:42.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:42.763+0000 7f144b33a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:42.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:42.764+0000 7f144b33a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:42.769 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:42.777 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:42Z" 2024-04-13T21:46:42.777 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:42.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:42Z" } 2024-04-13T21:46:42.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:42.779 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:42.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:42.779 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:46:47.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:47.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:47.781 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:47.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:47.875+0000 7f1d42a4e780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:47.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:47.876+0000 7f1d42a4e780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:47.880 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:47.881 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:47.882 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:47.883 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:47.884 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:47Z" 2024-04-13T21:46:47.885 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:47.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:47Z" } 2024-04-13T21:46:47.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:47.887 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:47.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:47.887 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:52.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:52.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:52.888 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:52.943 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:52.944 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:52.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:52.985+0000 7f23cd769780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:52.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:52.986+0000 7f23cd769780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:52.990 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:52.991 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:52.992 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:52.993 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:52.994 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:52.996 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:52.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:52.996 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:52Z" 2024-04-13T21:46:52.996 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:52.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:52Z" } 2024-04-13T21:46:52.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:52.997 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:52.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:52.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:46:57.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:57.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:46:57.999 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:46:58.054 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:46:58.054 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:46:58.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:58.095+0000 7fbe33ac1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:58.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:46:58.096+0000 7fbe33ac1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:46:58.099 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:46:58.100 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:46:58.101 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:58.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:46:58.104 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:46:58Z" 2024-04-13T21:46:58.105 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:46:58.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:46:58Z" } 2024-04-13T21:46:58.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:46:58.106 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:58.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:46:58.107 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.912549_264.1 2024-04-13T21:47:03.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:03.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:03.109 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:03.164 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:03.164 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:03.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:03.203+0000 7f6e4ae8d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:03.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:03.204+0000 7f6e4ae8d780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:03.209 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:03.209 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:03.209 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:03.209 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:03.210 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:03.211 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:03.212 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:03.213 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:03.214 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:03Z" 2024-04-13T21:47:03.215 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:03.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:03Z" } 2024-04-13T21:47:03.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:03.216 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:03.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:03.217 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:08.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:08.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:08.218 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:08.271 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:08.271 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:08.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:08.310+0000 7f2690ed2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:08.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:08.311+0000 7f2690ed2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:08.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:08.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:08.318 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:08.319 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:08Z" 2024-04-13T21:47:08.321 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:08.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:08Z" } 2024-04-13T21:47:08.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:08.323 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:08.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:08.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:47:13.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:13.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:13.325 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:13.381 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:13.381 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:13.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:13.420+0000 7f4b02ec9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:13.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:13.421+0000 7f4b02ec9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:13.426 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:13.426 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:13.426 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:13.426 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:13.426 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:13.427 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:13.428 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:13.429 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:13.430 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:13.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:13.432 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:13.432 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:13.432 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:13.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:13.432 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:13Z" 2024-04-13T21:47:13.432 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:13.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:13Z" } 2024-04-13T21:47:13.433 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:13.433 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:13.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:13.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:18.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:18.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:18.435 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:18.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:18.530+0000 7fe1619f8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:18.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:18.531+0000 7fe1619f8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:18.535 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:18.535 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:18.536 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:18.537 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:18.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:18.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:18.540 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:18Z" 2024-04-13T21:47:18.541 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:18.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:18Z" } 2024-04-13T21:47:18.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:18.542 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:18.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:18.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:23.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:23.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:23.545 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:23.600 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:23.601 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:23.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:23.642+0000 7f465de99780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:23.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:23.643+0000 7f465de99780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:23.647 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:23.647 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:23.647 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:23.647 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:23.648 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:23.649 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:23.650 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:23.651 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:23.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:23.653 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:23.653 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:23.653 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:23.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:23.653 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:23Z" 2024-04-13T21:47:23.653 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:23.654 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:23Z" } 2024-04-13T21:47:23.654 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:23.654 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:23.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:23.654 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:28.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:28.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:28.656 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:28.710 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:28.710 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:28.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:28.751+0000 7feff146c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:28.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:28.752+0000 7feff146c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:28.756 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:28.756 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:28.756 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:28.756 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:28.756 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:28.757 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:28.758 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:28.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:28.760 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:28.761 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:28.762 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:28.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:28.763 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:28Z" 2024-04-13T21:47:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:28.764 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:28Z" } 2024-04-13T21:47:28.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:28.764 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:28.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:28.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T21:47:33.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:33.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:33.766 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:33.821 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:33.822 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:33.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:33.862+0000 7fa9d23d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:33.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:33.863+0000 7fa9d23d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:33.867 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:33.867 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:33.868 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:33.870 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:33.871 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:33.872 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:33.873 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:33.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:33.874 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:33Z" 2024-04-13T21:47:33.874 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:33.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:33Z" } 2024-04-13T21:47:33.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:33.875 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:33.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:33.875 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:38.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:38.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:38.877 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:38.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:38.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:38.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:38.974+0000 7f54b12e6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:38.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:38.975+0000 7f54b12e6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:38.979 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:38.982 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:38.984 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:38.985 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:38Z" 2024-04-13T21:47:38.986 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:38.987 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:38Z" } 2024-04-13T21:47:38.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:38.987 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:38.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:38.987 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:43.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:43.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:43.989 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:44.042 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:44.042 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:44.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:44.081+0000 7f34ba7d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:44.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:44.082+0000 7f34ba7d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:44.086 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:44.086 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:44.086 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:44.086 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:44.087 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:44.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:44.089 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:44.090 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:44.091 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:44.092 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:44.093 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:44.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:44.093 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:44Z" 2024-04-13T21:47:44.093 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:44.094 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:44Z" } 2024-04-13T21:47:44.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:44.094 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:44.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:44.095 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:49.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:49.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:49.096 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:49.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:49.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:49.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:49.185+0000 7f1fd9925780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:49.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:49.186+0000 7f1fd9925780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:49.190 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:49.191 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:49.192 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:49.193 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:49.194 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:49.195 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:49Z" 2024-04-13T21:47:49.196 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:49.196 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:49Z" } 2024-04-13T21:47:49.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:49.197 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:49.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:49.197 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:54.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:54.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:54.199 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:54.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:54.258 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:54.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:54.301+0000 7f32f1452780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:54.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:54.302+0000 7f32f1452780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:54.306 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:54.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:54.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:54.309 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:54.310 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:54.311 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:54.312 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:54Z" 2024-04-13T21:47:54.313 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:54.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:54Z" } 2024-04-13T21:47:54.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:54.314 DEBUG:rgw_multi.tests:log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:54.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:54.314 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T21:47:59.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-04-13T21:47:59.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2024-04-13T21:47:59.321 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-35 2024-04-13T21:47:59.321 DEBUG:boto:path=/vwnfns-35/ 2024-04-13T21:47:59.321 DEBUG:boto:auth_path=/vwnfns-35/ 2024-04-13T21:47:59.321 DEBUG:boto:Method: PUT 2024-04-13T21:47:59.321 DEBUG:boto:Path: /vwnfns-35/ 2024-04-13T21:47:59.322 DEBUG:boto:Data: 2024-04-13T21:47:59.322 DEBUG:boto:Headers: {} 2024-04-13T21:47:59.322 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:47:59.322 DEBUG:boto:Port: 8000 2024-04-13T21:47:59.322 DEBUG:boto:Params: {} 2024-04-13T21:47:59.323 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:47:59.323 DEBUG:boto:Token: None 2024-04-13T21:47:59.323 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:47:59 GMT /vwnfns-35/ 2024-04-13T21:47:59.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YkP6OQzkGnFOwzIAr2QN+4tlpb0= 2024-04-13T21:47:59.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YkP6OQzkGnFOwzIAr2QN+4tlpb0=', 'Content-Length': '0'} 2024-04-13T21:47:59.343 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6c2de908c3b0fec-00661afd8f-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:47:59.343 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-36 2024-04-13T21:47:59.343 DEBUG:boto:path=/vwnfns-36/ 2024-04-13T21:47:59.343 DEBUG:boto:auth_path=/vwnfns-36/ 2024-04-13T21:47:59.344 DEBUG:boto:Method: PUT 2024-04-13T21:47:59.344 DEBUG:boto:Path: /vwnfns-36/ 2024-04-13T21:47:59.344 DEBUG:boto:Data: 2024-04-13T21:47:59.344 DEBUG:boto:Headers: {} 2024-04-13T21:47:59.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:47:59.344 DEBUG:boto:Port: 8001 2024-04-13T21:47:59.344 DEBUG:boto:Params: {} 2024-04-13T21:47:59.344 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:47:59.344 DEBUG:boto:Token: None 2024-04-13T21:47:59.345 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:47:59 GMT /vwnfns-36/ 2024-04-13T21:47:59.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LcncnaVOnKQBeFrru1rzIhQull0= 2024-04-13T21:47:59.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LcncnaVOnKQBeFrru1rzIhQull0=', 'Content-Length': '0'} 2024-04-13T21:47:59.373 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7e587c5fcd64aa4-00661afd8f-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:47:59.374 DEBUG:boto:path=/vwnfns-35/ 2024-04-13T21:47:59.374 DEBUG:boto:auth_path=/vwnfns-35/ 2024-04-13T21:47:59.374 DEBUG:boto:Method: HEAD 2024-04-13T21:47:59.374 DEBUG:boto:Path: /vwnfns-35/ 2024-04-13T21:47:59.374 DEBUG:boto:Data: 2024-04-13T21:47:59.374 DEBUG:boto:Headers: {} 2024-04-13T21:47:59.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:47:59.374 DEBUG:boto:Port: 8000 2024-04-13T21:47:59.374 DEBUG:boto:Params: {} 2024-04-13T21:47:59.375 DEBUG:boto:Token: None 2024-04-13T21:47:59.375 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:47:59 GMT /vwnfns-35/ 2024-04-13T21:47:59.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uD6aFgnFJsP9CJrm07jKmvHOvDs= 2024-04-13T21:47:59.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uD6aFgnFJsP9CJrm07jKmvHOvDs=', 'Content-Length': '0'} 2024-04-13T21:47:59.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004564892b0141f534-00661afd8f-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:47:59.378 DEBUG:boto:path=/vwnfns-35/key 2024-04-13T21:47:59.378 DEBUG:boto:auth_path=/vwnfns-35/key 2024-04-13T21:47:59.378 DEBUG:boto:Method: PUT 2024-04-13T21:47:59.378 DEBUG:boto:Path: /vwnfns-35/key 2024-04-13T21:47:59.378 DEBUG:boto:Data: 2024-04-13T21:47:59.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-13T21:47:59.378 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:47:59.378 DEBUG:boto:Port: 8000 2024-04-13T21:47:59.378 DEBUG:boto:Params: {} 2024-04-13T21:47:59.379 DEBUG:boto:Token: None 2024-04-13T21:47:59.379 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Sat, 13 Apr 2024 21:47:59 GMT /vwnfns-35/key 2024-04-13T21:47:59.379 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoJI+MPOzNdO1cx9XMFpcI8J6QM= 2024-04-13T21:47:59.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoJI+MPOzNdO1cx9XMFpcI8J6QM='} 2024-04-13T21:47:59.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b14c72695ea7d20c-00661afd8f-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:47:59.386 DEBUG:boto:path=/vwnfns-36/ 2024-04-13T21:47:59.386 DEBUG:boto:auth_path=/vwnfns-36/ 2024-04-13T21:47:59.386 DEBUG:boto:Method: HEAD 2024-04-13T21:47:59.386 DEBUG:boto:Path: /vwnfns-36/ 2024-04-13T21:47:59.386 DEBUG:boto:Data: 2024-04-13T21:47:59.386 DEBUG:boto:Headers: {} 2024-04-13T21:47:59.386 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:47:59.386 DEBUG:boto:Port: 8001 2024-04-13T21:47:59.387 DEBUG:boto:Params: {} 2024-04-13T21:47:59.387 DEBUG:boto:Token: None 2024-04-13T21:47:59.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:47:59 GMT /vwnfns-36/ 2024-04-13T21:47:59.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qJz86Cs+LDqbscXoua5RLtEkIaY= 2024-04-13T21:47:59.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qJz86Cs+LDqbscXoua5RLtEkIaY=', 'Content-Length': '0'} 2024-04-13T21:47:59.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003887dbd4dcc447da-00661afd8f-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:47:59.389 DEBUG:boto:path=/vwnfns-36/key 2024-04-13T21:47:59.389 DEBUG:boto:auth_path=/vwnfns-36/key 2024-04-13T21:47:59.389 DEBUG:boto:Method: PUT 2024-04-13T21:47:59.389 DEBUG:boto:Path: /vwnfns-36/key 2024-04-13T21:47:59.390 DEBUG:boto:Data: 2024-04-13T21:47:59.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-13T21:47:59.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:47:59.390 DEBUG:boto:Port: 8001 2024-04-13T21:47:59.390 DEBUG:boto:Params: {} 2024-04-13T21:47:59.390 DEBUG:boto:Token: None 2024-04-13T21:47:59.390 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Sat, 13 Apr 2024 21:47:59 GMT /vwnfns-36/key 2024-04-13T21:47:59.390 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KiwxsnnBbuu6ywOY8b/81L8hPz4= 2024-04-13T21:47:59.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KiwxsnnBbuu6ywOY8b/81L8hPz4='} 2024-04-13T21:47:59.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2121689dac637cc-00661afd8f-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:47:59.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:59.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:59.398 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:59.448 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:59.448 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:59.478 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:47:59.490 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:47:59.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.490 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044879.371767_361.1", 2024-04-13T21:47:59.490 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:47:59.371767Z" 2024-04-13T21:47:59.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:47:59.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.492 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:59.492 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:47:59.492 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:59.492 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:47:59.493 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:59.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:47:59.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:59.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:47:59.494 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:47:59.550 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:47:59.550 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:47:59.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:59.622+0000 7f8fd204f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:59.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:47:59.623+0000 7f8fd204f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:47:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:59.631 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:47:59.632 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:47:59.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:47:59.634 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:47:59Z" 2024-04-13T21:47:59.634 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:47:59.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:47:59Z" } 2024-04-13T21:47:59.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:47:59.635 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:59.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:47:59.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:04.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:04.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:04.637 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:04.693 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:04.694 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:04.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:04.735+0000 7f7efc3b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:04.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:04.748+0000 7f7efc3b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:04.760 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:04.760 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:04.760 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:04.760 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:04.760 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:04.761 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:04.762 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:04.763 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:04.764 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:04Z" 2024-04-13T21:48:04.765 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:04.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:04Z" } 2024-04-13T21:48:04.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:04.767 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:04.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:04.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:08.249 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T21:48:08.258+0000 7f5ea15cd700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T21:48:09.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:09.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:09.768 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:09.819 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:09.819 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:09.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:09.858+0000 7f4da19cc780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:09.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:09.859+0000 7f4da19cc780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:09.862 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:09.862 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:09.863 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:09.864 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:09.865 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:09.866 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:09Z" 2024-04-13T21:48:09.867 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:09.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:09Z" } 2024-04-13T21:48:09.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:09.869 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:09.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:09.869 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:14.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:14.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:14.871 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:14.925 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:14.925 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:14.966+0000 7f6b83602780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:14.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:14.967+0000 7f6b83602780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:14.971 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:14.971 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:14.971 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:14.971 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:14.971 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:14.971 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:14.972 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:14.973 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:14.974 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:14.975 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:14.976 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:14Z" 2024-04-13T21:48:14.977 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:14.977 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:14Z" } 2024-04-13T21:48:14.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:14.978 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:14.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:14.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:19.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:19.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:19.979 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:20.036 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:20.036 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:20.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:20.075+0000 7f9e3ffe4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:20.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:20.076+0000 7f9e3ffe4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:20.083 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:48:20.084 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:20.085 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:20.086 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:20.087 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:20.088 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:20.089 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:20.089 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:20.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:20.089 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:20Z" 2024-04-13T21:48:20.089 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:20.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:20Z" } 2024-04-13T21:48:20.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:20.090 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:20.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:20.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:25.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:25.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:25.091 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:25.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:25.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:25.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:25.187+0000 7fde632c6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:25.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:25.188+0000 7fde632c6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:25.192 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:25.192 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:25.192 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:25.193 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:25.194 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:25.195 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:25.196 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:25Z" 2024-04-13T21:48:25.197 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:25.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:25Z" } 2024-04-13T21:48:25.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:25.199 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:25.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:25.199 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:30.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:30.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:30.200 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:30.256 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:30.256 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:30.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:30.295+0000 7f07af70c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:30.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:30.296+0000 7f07af70c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:30.301 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:30.301 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:30.301 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:30.301 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:30.301 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:30.301 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:30.302 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:30.303 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:30.304 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:30.305 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:30.306 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:30.307 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:30.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:30.307 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:30Z" 2024-04-13T21:48:30.307 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:30.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:30Z" } 2024-04-13T21:48:30.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:30.308 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:30.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:30.308 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:48:35.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:35.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:35.309 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:35.363 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:35.363 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:35.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:35.407+0000 7fb05ca10780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:35.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:35.408+0000 7fb05ca10780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:35.413 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:35Z" 2024-04-13T21:48:35.419 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:35.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:35Z" } 2024-04-13T21:48:35.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:35.420 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:35.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:35.420 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:48:40.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:40.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:40.422 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:40.477 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:40.477 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:40.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:40.517+0000 7fa530f32780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:40.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:40.518+0000 7fa530f32780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:40.522 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:40.523 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:40.524 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:40.525 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:40.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:40.528 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:40Z" 2024-04-13T21:48:40.528 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:40.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:40Z" } 2024-04-13T21:48:40.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:40.529 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:40.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:40.529 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:48:45.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:45.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:45.530 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:45.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:45.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:45.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:45.629+0000 7fb0199ba780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:45.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:45.630+0000 7fb0199ba780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:45.633 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:45.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:45.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:45.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:45.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:45.638 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:45Z" 2024-04-13T21:48:45.639 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:45.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:45Z" } 2024-04-13T21:48:45.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:45.640 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:45.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:45.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:48:50.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:50.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:50.642 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:50.698 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:50.698 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:50.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:50.739+0000 7f3741b1b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:50.740+0000 7f3741b1b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:50.744 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:50.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:50.750 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:50Z" 2024-04-13T21:48:50.750 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:50.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:50Z" } 2024-04-13T21:48:50.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:50.751 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:50.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:50.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:48:55.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:55.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:48:55.753 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:48:55.809 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:48:55.810 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:48:55.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:55.850+0000 7f3926700780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:55.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:48:55.851+0000 7f3926700780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:48:55.855 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:48:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:48:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:48:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:48:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:48:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:48:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:48:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:48:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:48:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:48:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:48:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:48:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:48:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:48:55Z" 2024-04-13T21:48:55.862 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:48:55.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:48:55Z" } 2024-04-13T21:48:55.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:48:55.863 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:55.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:48:55.864 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:49:00.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:00.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:00.865 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:00.918 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:00.919 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:00.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:00.957+0000 7f4564909780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:00.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:00.958+0000 7f4564909780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:00.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:49:00.964 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:00.965 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:00.966 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:00.967 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:00Z" 2024-04-13T21:49:00.968 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:00.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:00Z" } 2024-04-13T21:49:00.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:00.970 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:00.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:00.970 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:49:05.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:05.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:05.972 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:06.026 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:06.027 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:06.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:06.066+0000 7f04201a2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:06.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:06.067+0000 7f04201a2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:06.071 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:06.071 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:06.071 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:06.072 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:06.073 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:06.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:06.075 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:06.076 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:06.077 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:06Z" 2024-04-13T21:49:06.078 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:06.079 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:06Z" } 2024-04-13T21:49:06.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:06.079 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:06.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:06.079 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:49:11.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:11.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:11.081 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:11.137 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:11.137 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:11.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:11.177+0000 7ff70f6e5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:11.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:11.179+0000 7ff70f6e5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:11.193 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:11.193 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:11.193 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:11.193 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:11.193 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:49:11.194 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:11.195 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:11.196 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:11.197 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:11.198 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:11.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:11.199 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:11Z" 2024-04-13T21:49:11.199 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:11.200 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:11Z" } 2024-04-13T21:49:11.200 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:11.200 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:11.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:11.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T21:49:16.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:16.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:16.201 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:16.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:16.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:16.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:16.297+0000 7f123768c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:16.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:16.298+0000 7f123768c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:16.303 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:49:16.304 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:16.305 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:16.307 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:16Z" 2024-04-13T21:49:16.308 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:16.309 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:16Z" } 2024-04-13T21:49:16.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:16.310 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:16.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:16.310 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T21:49:21.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:21.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:21.312 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:21.365 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:21.365 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:21.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:21.405+0000 7fe3c5f68780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:21.406+0000 7fe3c5f68780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:21.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:21.411 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:21.412 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:21.413 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:21.414 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:21Z" 2024-04-13T21:49:21.415 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:21.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:21Z" } 2024-04-13T21:49:21.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:21.416 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:21.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:21.417 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:49:26.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:26.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:26.418 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:26.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:26.474 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:26.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:26.514+0000 7f07f407c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:26.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:26.515+0000 7f07f407c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:26.519 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:26.519 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:26.519 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:26.519 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:26.519 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:26.520 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:26.521 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:26.522 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:26.523 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:26.524 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:26.525 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:26.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:26.525 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:26Z" 2024-04-13T21:49:26.525 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:26.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:26Z" } 2024-04-13T21:49:26.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:26.526 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:26.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:26.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:49:31.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:31.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:31.528 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:31.583 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:31.584 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:31.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:31.624+0000 7ff730ae2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:31.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:31.626+0000 7ff730ae2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:31Z" 2024-04-13T21:49:31.636 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:31.637 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:31Z" } 2024-04-13T21:49:31.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:31.637 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:31.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:31.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:49:36.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:36.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:36.639 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:36.696 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:36.696 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:36.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:36.737+0000 7f2485157780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:36.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:36.738+0000 7f2485157780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:36.743 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:36.743 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.872147_258.1", 2024-04-13T21:49:36.744 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.872147Z", 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:36.745 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:36.746 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:36.747 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:36.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:36.749 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:36Z" 2024-04-13T21:49:36.749 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:36.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.872147_258.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.872147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:36Z" } 2024-04-13T21:49:36.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.872147_258.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.872147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:36.750 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:36.750 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.872147_258.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:36.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.872147_258.1 2024-04-13T21:49:41.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:41.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:41.752 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:41.807 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:41.808 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:41.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:41.848+0000 7f3118dd4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:41.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:41.849+0000 7f3118dd4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:41.853 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:41.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:49:41.855 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:41.856 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:41.857 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:41Z" 2024-04-13T21:49:41.859 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:41.862 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:41Z" } 2024-04-13T21:49:41.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:41.862 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:41.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:41.863 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:49:46.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:46.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:46.865 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:46.920 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:46.921 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:46.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:46.960+0000 7fe9b9a53780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:46.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:46.962+0000 7fe9b9a53780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:46.967 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:46.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:46.969 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:46.970 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:46.973 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:46.973 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:46.973 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:46.973 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:46Z" 2024-04-13T21:49:46.974 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:46.975 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:46Z" } 2024-04-13T21:49:46.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:46.976 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:46.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:46.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:49:51.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:51.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:51.977 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:52.033 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:52.033 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:52.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:52.072+0000 7f3df91ec780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:52.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:52.074+0000 7f3df91ec780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:52.079 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:52.079 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:52.079 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:52.080 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:52.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:52.082 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:52.083 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:52.084 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:52.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:52.086 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:52.086 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:52.086 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:52.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:52.086 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:52Z" 2024-04-13T21:49:52.086 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:52.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:52Z" } 2024-04-13T21:49:52.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:52.088 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:52.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:52.088 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:49:57.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:57.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:49:57.090 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:49:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:49:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:49:57.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:57.187+0000 7fca0e321780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:57.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:49:57.188+0000 7fca0e321780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:49:57.192 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:49:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:49:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:49:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:49:57.193 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:57.194 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:49:57.195 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:49:57.196 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:57.197 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:49:57.198 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:49:57.198 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:49:57.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:49:57.198 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:49:57Z" 2024-04-13T21:49:57.198 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:49:57.199 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:49:57Z" } 2024-04-13T21:49:57.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:49:57.199 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:57.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:49:57.199 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:50:02.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:02.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:02.201 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:02.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:02.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:02.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:02.298+0000 7fb5e4b95780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:02.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:02.299+0000 7fb5e4b95780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:02.302 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:02.303 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:02.304 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:02.305 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:02.306 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:02Z" 2024-04-13T21:50:02.308 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:02.309 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:02Z" } 2024-04-13T21:50:02.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:02.309 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:02.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:02.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:50:07.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:07.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:07.311 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:07.367 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:07.368 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:07.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:07.410+0000 7f1d587b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:07.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:07.412+0000 7f1d587b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:07.417 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:07.418 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:07.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:07.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:07.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:07.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:07Z" 2024-04-13T21:50:07.423 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:07.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:07Z" } 2024-04-13T21:50:07.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:07.424 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:07.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:07.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:50:12.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:12.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:12.425 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:12.524+0000 7f8a078b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:12.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:12.525+0000 7f8a078b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:12.529 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:12.530 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:12.531 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:12.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:12.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:12.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:12Z" 2024-04-13T21:50:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:12.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:12Z" } 2024-04-13T21:50:12.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:12.536 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:12.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:12.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:50:17.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:17.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:17.538 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:17.595 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:17.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:17.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:17.638+0000 7f0bfb063780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:17.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:17.639+0000 7f0bfb063780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:17.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:17.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:17.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:17.649 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:17Z" 2024-04-13T21:50:17.650 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:17.651 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:17Z" } 2024-04-13T21:50:17.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:17.652 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:17.652 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:17.652 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T21:50:22.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:22.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:22.653 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:22.704 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:22.704 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:22.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:22.744+0000 7f8d53a79780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:22.745+0000 7f8d53a79780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:22.749 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:22.750 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:22.751 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:22.752 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:22.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:22.754 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:22.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:22.755 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:22Z" 2024-04-13T21:50:22.755 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:22.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:22Z" } 2024-04-13T21:50:22.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:22.756 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:22.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:22.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:50:27.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:27.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:27.758 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:27.813 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:27.813 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:27.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:27.854+0000 7f16c5b97780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:27.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:27.854+0000 7f16c5b97780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:27.859 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:27.860 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:27.861 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:27.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:27.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:27.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:27.865 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:27.865 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:27.865 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:27.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:27.865 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:27Z" 2024-04-13T21:50:27.865 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:27.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:27Z" } 2024-04-13T21:50:27.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:27.866 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:27.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:27.866 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:50:32.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:32.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:32.868 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:32.924 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:32.924 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:32.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:32.965+0000 7ff4df078780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:32.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:32.966+0000 7ff4df078780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:32.971 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:32.972 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:32.973 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:32.974 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:32.975 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:32Z" 2024-04-13T21:50:32.976 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:32.977 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:32Z" } 2024-04-13T21:50:32.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:32.977 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:32.977 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:32.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:50:37.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:37.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:37.979 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:38.034 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:38.034 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:38.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:38.075+0000 7f7f5c1f4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:38.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:38.076+0000 7f7f5c1f4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:38.080 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:38.080 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:38.081 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:38.082 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:38.083 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:38.084 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:38.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:38.086 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:38.086 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:38.086 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:38.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:38.086 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:38Z" 2024-04-13T21:50:38.086 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:38.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:38Z" } 2024-04-13T21:50:38.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:38.087 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:38.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:38.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:50:43.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:43.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:43.089 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:43.149 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:43.149 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:43.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:43.191+0000 7fa0337a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:43.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:43.192+0000 7fa0337a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:43.196 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:43.196 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:43.196 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:50:43.197 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:43.198 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:43.199 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:43.200 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:43.201 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:43.202 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:43.202 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:43Z" 2024-04-13T21:50:43.202 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:43.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:43Z" } 2024-04-13T21:50:43.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:43.203 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:43.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:43.203 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:50:48.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:48.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:48.206 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:48.261 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:48.261 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:48.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:48.302+0000 7f87e0aab780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:48.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:48.303+0000 7f87e0aab780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:48.312 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:48Z" 2024-04-13T21:50:48.314 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:48.315 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:48Z" } 2024-04-13T21:50:48.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:48.315 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:48.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:48.315 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:50:53.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:53.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:53.317 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:53.373 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:53.373 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:53.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:53.416+0000 7fa302afc780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:53.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:53.417+0000 7fa302afc780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:53.422 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:53.423 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:53.424 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:53.425 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:53.426 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:53.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:53.428 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:53Z" 2024-04-13T21:50:53.428 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:53.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:53Z" } 2024-04-13T21:50:53.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:53.429 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:53.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:53.430 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:50:58.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:58.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:50:58.431 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:50:58.482 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:50:58.482 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:50:58.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:58.523+0000 7f9a9af69780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:58.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:50:58.524+0000 7f9a9af69780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:58.528 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.139905_297.1", 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.139905Z", 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:58.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:50:58.530 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:58.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:50:58.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:50:58Z" 2024-04-13T21:50:58.533 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:50:58.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.139905_297.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.139905Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:50:58Z" } 2024-04-13T21:50:58.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.139905_297.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.139905Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:50:58.535 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:58.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.139905_297.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:50:58.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044569.139905_297.1 2024-04-13T21:51:03.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:03.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:03.536 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:03.591 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:03.591 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:03.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:03.633+0000 7f4c50d6e780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:03.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:03.634+0000 7f4c50d6e780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:03.639 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:03.639 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:03.639 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:03.639 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:03.639 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:51:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:03.641 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:03.642 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:03.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:03.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:03Z" 2024-04-13T21:51:03.645 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:03.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:03Z" } 2024-04-13T21:51:03.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:03.646 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:03.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:03.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:51:08.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:08.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:08.647 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:08.703 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:08.703 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:08.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:08.746+0000 7f23caa34780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:08.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:08.747+0000 7f23caa34780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:08.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:08.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:08Z" 2024-04-13T21:51:08.757 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:08.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:08Z" } 2024-04-13T21:51:08.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:08.758 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:08.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:08.759 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:51:13.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:13.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:13.761 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:13.811 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:13.812 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:13.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:13.850+0000 7f96cc461780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:13.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:13.851+0000 7f96cc461780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:13.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:13.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:13.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:13.858 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:13.859 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:13.860 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:13Z" 2024-04-13T21:51:13.861 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:13.861 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:13Z" } 2024-04-13T21:51:13.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:13.862 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:13.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:13.862 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:51:18.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:18.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:18.863 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:18.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:18.966+0000 7ff439983780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:18.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:18.968+0000 7ff439983780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:18.971 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:18Z" 2024-04-13T21:51:18.977 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:18.978 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:18Z" } 2024-04-13T21:51:18.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:18.978 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:18.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:18.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:51:23.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:23.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:23.979 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:24.034 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:24.034 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:24.076+0000 7f1b9e521780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:24.078+0000 7f1b9e521780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:24.082 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:24.083 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:24.084 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:24.085 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:24.086 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:24Z" 2024-04-13T21:51:24.087 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:24.088 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:24Z" } 2024-04-13T21:51:24.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:24.088 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:24.089 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:24.089 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:51:29.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:29.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:29.090 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:29.146 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:29.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:29.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:29.187+0000 7fae5576a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:29.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:29.188+0000 7fae5576a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:29.192 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:29.192 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:29.192 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:29.192 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:29.192 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:29.193 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:29.194 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:29.195 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:29.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:29.198 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:29Z" 2024-04-13T21:51:29.198 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:29.199 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:29Z" } 2024-04-13T21:51:29.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:29.199 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:29.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:29.199 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:51:34.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:34.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:34.201 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:34.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:34.257 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:34.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:34.297+0000 7f1aadfab780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:34.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:34.298+0000 7f1aadfab780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:34.303 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:34.304 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:34.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:34.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:34.306 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:34.306 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:34.306 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:34.306 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:34.306 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:34.307 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:34.308 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:34.309 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:34.310 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:34.310 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:34.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:34.310 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:34Z" 2024-04-13T21:51:34.310 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:34.311 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:34Z" } 2024-04-13T21:51:34.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:34.311 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:34.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:34.312 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:51:39.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:39.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:39.313 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:39.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:39.410+0000 7fcabd680780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:39.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:39.412+0000 7fcabd680780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:39.426 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:39.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.912549_264.1", 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:39.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:39.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:39.430 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:39.431 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:39Z" 2024-04-13T21:51:39.432 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:39.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.912549_264.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.912549Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:39Z" } 2024-04-13T21:51:39.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:39.434 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:39.434 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:39.434 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T21:51:44.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:44.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:44.435 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:44.490 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:44.490 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:44.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:44.530+0000 7fc6c387c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:44.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:44.531+0000 7fc6c387c780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:44.536 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:44.537 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:44.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:44.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:44.540 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:44.541 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:44.542 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:44.543 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:44.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:44.543 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:44Z" 2024-04-13T21:51:44.543 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:44.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:44Z" } 2024-04-13T21:51:44.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:44.544 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:44.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:44.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:51:49.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:49.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:49.547 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:49.597 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:49.597 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:49.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:49.634+0000 7f5d4ecf4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:49.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:49.635+0000 7f5d4ecf4780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:49.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:51:49.641 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:49.642 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:49.643 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:49.644 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:49Z" 2024-04-13T21:51:49.645 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:49.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:49Z" } 2024-04-13T21:51:49.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:49.647 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:49.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:49.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:51:54.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:54.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:54.649 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:54.704 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:54.705 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:54.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:54.745+0000 7fc57b3d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:54.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:54.746+0000 7fc57b3d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:54.751 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:54.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:54.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:54.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:54.755 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:54Z" 2024-04-13T21:51:54.756 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:54.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:54Z" } 2024-04-13T21:51:54.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:54.758 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:54.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:54.758 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:51:59.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:59.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:51:59.759 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:51:59.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:51:59.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:51:59.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:59.854+0000 7f4725335780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:59.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:51:59.856+0000 7f4725335780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:59.861 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:51:59.862 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:59.863 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:59.864 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:51:59.865 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:51:59.866 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:51:59Z" 2024-04-13T21:51:59.867 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:51:59.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:51:59Z" } 2024-04-13T21:51:59.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:51:59.868 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:59.868 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:51:59.868 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:52:04.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:04.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:04.869 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:04.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:04.966+0000 7f1437587780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:04.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:04.967+0000 7f1437587780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:04.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:04.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:04.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:04.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:04.976 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:04Z" 2024-04-13T21:52:04.977 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:04.978 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:04Z" } 2024-04-13T21:52:04.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:04.978 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:04.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:04.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:09.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:09.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:09.981 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:10.037 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:10.037 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:10.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:10.077+0000 7f2c40115780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:10.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:10.078+0000 7f2c40115780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:10.084 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:10.084 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:10.084 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:10.085 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:10.086 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:10.087 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:10.088 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:10.089 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:10.090 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:10.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:10.090 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:10Z" 2024-04-13T21:52:10.090 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:10.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:10Z" } 2024-04-13T21:52:10.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:10.091 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:10.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:10.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:15.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:15.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:15.093 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:15.148 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:15.149 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:15.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:15.187+0000 7ff6e8f8f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:15.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:15.188+0000 7ff6e8f8f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:15.194 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:15.195 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.137566_296.1", 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.137566Z", 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:15.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:15.198 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:15.199 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:15Z" 2024-04-13T21:52:15.200 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:15.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.137566_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.137566Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:15Z" } 2024-04-13T21:52:15.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.137566_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.137566Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:15.202 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:15.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.137566_296.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:15.202 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044569.137566_296.1 2024-04-13T21:52:20.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:20.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:20.204 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:20.260 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:20.260 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:20.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:20.301+0000 7fb7f7bcc780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:20.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:20.302+0000 7fb7f7bcc780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:20.309 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:20.309 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:20.309 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:20.309 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:20.309 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:20.310 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:20.311 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:20.312 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:20.313 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:20.314 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:20.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:20.315 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:20Z" 2024-04-13T21:52:20.315 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:20.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:20Z" } 2024-04-13T21:52:20.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:20.316 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:20.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:20.316 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:25.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:25.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:25.318 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:25.372 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:25.373 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:25.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:25.409+0000 7fe09a1d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:25.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:25.410+0000 7fe09a1d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:25.416 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:25.417 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:25Z" 2024-04-13T21:52:25.422 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:25.422 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:25Z" } 2024-04-13T21:52:25.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:25.423 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:25.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:25.423 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:52:30.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:30.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:30.425 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:30.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:30.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:30.521+0000 7fc9aaf36780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:30.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:30.522+0000 7fc9aaf36780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:30.527 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:30.527 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:30.527 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:30.527 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:30.527 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:30.527 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:30.528 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:30.529 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:30.530 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:30.531 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:30.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:30.533 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:30.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:30.533 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:30Z" 2024-04-13T21:52:30.533 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:30.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:30Z" } 2024-04-13T21:52:30.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:30.534 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:30.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:30.534 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:35.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:35.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:35.535 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:35.590 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:35.590 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:35.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:35.630+0000 7ff1271b5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:35.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:35.631+0000 7ff1271b5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:35.637 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:35.637 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:35.637 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:35.637 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:35.637 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:35.638 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:35.639 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:35.640 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:35.641 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:35.642 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:35.643 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:35.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:35.643 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:35Z" 2024-04-13T21:52:35.643 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:35.644 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:35Z" } 2024-04-13T21:52:35.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:35.644 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:35.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:35.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:40.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:40.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:40.646 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:40.701 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:40.702 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:40.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:40.743+0000 7f3c24a06780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:40.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:40.744+0000 7f3c24a06780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:40.749 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:40.750 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:40.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:40.753 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:40.754 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:40.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:40.756 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:40.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:40.756 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:40Z" 2024-04-13T21:52:40.756 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:40.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:40Z" } 2024-04-13T21:52:40.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:40.757 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:40.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:40.757 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:45.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:45.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:45.759 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:45.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:45.862+0000 7fd2835df780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:45.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:45.863+0000 7fd2835df780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:45.886 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:45.887 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:45.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:45.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:45Z" 2024-04-13T21:52:45.891 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:45.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:45Z" } 2024-04-13T21:52:45.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:45.893 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:45.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:45.893 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:50.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:50.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:50.894 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:50.950 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:50.951 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:50.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:50.989+0000 7f22e740f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:50.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:50.990+0000 7f22e740f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:50.998 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.907592_262.1", 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:50.999 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:51.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:51.001 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:51.002 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:51.004 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:51.004 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:51.004 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:51.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:51.004 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:50Z" 2024-04-13T21:52:51.004 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:51.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.907592_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.907592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:50Z" } 2024-04-13T21:52:51.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:51.005 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:51.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:51.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T21:52:56.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:56.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:52:56.006 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:52:56.062 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:52:56.062 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:52:56.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:56.101+0000 7f2501127780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:56.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:52:56.102+0000 7f2501127780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:52:56.110 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:52:56.110 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:52:56.111 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:52:56.112 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:52:56.113 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:52:56.114 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:52:56.115 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:52:56Z" 2024-04-13T21:52:56.116 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:52:56.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:52:56Z" } 2024-04-13T21:52:56.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:52:56.117 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:56.117 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:52:56.117 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:53:01.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:53:01.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:53:01.119 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:53:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:53:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:53:01.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:53:01.207+0000 7f6dfc5e7780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:53:01.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:53:01.208+0000 7f6dfc5e7780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:53:01.214 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044568.938952_266.1", 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:53:01.215 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:53:01.216 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:53:01.217 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:53:01.218 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:01.219 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:53:01Z" 2024-04-13T21:53:01.220 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:53:01.220 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044568.938952_266.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:48.938952Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:53:01Z" } 2024-04-13T21:53:01.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:53:01.221 DEBUG:rgw_multi.tests:log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:53:01.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:53:01.221 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T21:53:06.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-04-13T21:53:06.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:53:06.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:53:06.227 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:53:06.281 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:53:06.282 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:53:06.304 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:53:06.304 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:53:06.304 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:53:06.304 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-13T21:53:06.304 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a2.rgw.control", 2024-04-13T21:53:06.304 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a2.rgw.log", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-13T21:53:06.305 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a2.rgw.otp", 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:53:06.306 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.308 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.308 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:53:06.308 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-13T21:53:06.308 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:53:06.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:53:06.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:53:06.309 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:53:06.359 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:53:06.359 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:53:06.478 INFO:teuthology.orchestra.run.smithi132.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:53:06.502 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:53:06.502 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:53:06.502 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5, 2024-04-13T21:53:06.502 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:53:06.503 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.504 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": false, 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:53:06.505 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.506 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:53:06.507 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.508 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:53:06.510 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:53:06.511 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:53:06.512 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:53:06.514 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:53:06.514 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.514 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:53:06.514 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:53:06.515 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:53:06.516 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:53:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:53:06.519 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:53:06.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:53:06.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.522 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:53:06.523 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:53:06.525 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2024-04-13T21:54:06.525 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-37 2024-04-13T21:54:06.526 DEBUG:boto:path=/vwnfns-37/ 2024-04-13T21:54:06.526 DEBUG:boto:auth_path=/vwnfns-37/ 2024-04-13T21:54:06.527 DEBUG:boto:Method: PUT 2024-04-13T21:54:06.527 DEBUG:boto:Path: /vwnfns-37/ 2024-04-13T21:54:06.527 DEBUG:boto:Data: 2024-04-13T21:54:06.527 DEBUG:boto:Headers: {} 2024-04-13T21:54:06.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:54:06.527 DEBUG:boto:Port: 8000 2024-04-13T21:54:06.527 DEBUG:boto:Params: {} 2024-04-13T21:54:06.528 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:54:06.528 DEBUG:boto:Token: None 2024-04-13T21:54:06.528 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:54:06 GMT /vwnfns-37/ 2024-04-13T21:54:06.529 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PBEn9igpPOWhyfw7PceId0gnwdU= 2024-04-13T21:54:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:54:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PBEn9igpPOWhyfw7PceId0gnwdU=', 'Content-Length': '0'} 2024-04-13T21:54:06.550 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f7be69cc8d320ade-00661afefe-1859-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:54:06.551 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:54:06.551 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:54:06.551 DEBUG:boto:Method: PUT 2024-04-13T21:54:06.551 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:54:06.551 DEBUG:boto:Data: 2024-04-13T21:54:06.552 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T21:54:06.552 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:54:06.552 DEBUG:boto:Port: 8000 2024-04-13T21:54:06.552 DEBUG:boto:Params: {} 2024-04-13T21:54:06.552 DEBUG:boto:Token: None 2024-04-13T21:54:06.552 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 13 Apr 2024 21:54:06 GMT /vwnfns-37/testredirect 2024-04-13T21:54:06.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yDHXIn3UmiOTbfYcXvuyVD5zo10= 2024-04-13T21:54:06.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:54:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yDHXIn3UmiOTbfYcXvuyVD5zo10='} 2024-04-13T21:54:06.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047d25663e503752c-00661afefe-1859-a1'), ('Date', 'Sat, 13 Apr 2024 21:54:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:54:06.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:06.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:06.562 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:06.614 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:06.615 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:06.646 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:54:06.658 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:54:06.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.658 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:54:06.555013Z" 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.660 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:06.660 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:54:06.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:06.660 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:54:06.661 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:06.661 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:54:06.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:06.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:06.662 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:06.710 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:06.710 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:06.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:06.745+0000 7f94f2b2f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:06.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:06.746+0000 7f94f2b2f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:06.753 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:06.753 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:06.753 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:06.753 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:06.753 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:06.753 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:06.754 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:06.755 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.756 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:06.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:06Z" 2024-04-13T21:54:06.759 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:06.760 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:06Z" } 2024-04-13T21:54:06.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:06.761 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:06.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:06.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:11.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:11.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:11.763 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:11.819 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:11.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:11.856+0000 7f99c9ea8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:11.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:11.857+0000 7f99c9ea8780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:11.865 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:11.865 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:11.866 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:11.867 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:11.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:11.869 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:11.870 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:11.871 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:11.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:11.872 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:11Z" 2024-04-13T21:54:11.872 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:11.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:11Z" } 2024-04-13T21:54:11.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:11.873 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:11.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:11.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:16.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:16.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:16.875 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:16.930 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:16.930 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:16.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:16.968+0000 7f6f4b6d1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:16.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:16.969+0000 7f6f4b6d1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:16.977 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:16.977 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:16.977 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:16.977 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:16.977 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:16.978 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:16.978 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:16.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:16.978 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:16.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:16.978 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:16.979 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:16.979 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:16.979 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:16.979 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:16.980 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:16.980 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:16.980 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:16.980 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:16.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:16.981 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:16.982 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:16.982 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:16.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:16.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:16.982 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:16.983 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:16.983 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:16.983 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:16.983 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:16.983 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:16.984 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:16.985 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:16.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:16.986 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:16Z" 2024-04-13T21:54:16.986 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:16.987 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:16Z" } 2024-04-13T21:54:16.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:16.987 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:16.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:16.987 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:21.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:21.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:21.988 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:22.046 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:22.046 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:22.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:22.083+0000 7f39052d1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:22.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:22.084+0000 7f39052d1780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:22.092 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:22.093 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:22.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:22.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:22.096 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:22.097 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:22Z" 2024-04-13T21:54:22.098 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:22.099 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:22Z" } 2024-04-13T21:54:22.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:22.100 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:22.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:22.100 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:27.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:27.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:27.101 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:27.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:27.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:27.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:27.195+0000 7ff7a73c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:27.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:27.196+0000 7ff7a73c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:27.204 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:27.205 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:27.207 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:27.208 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:27Z" 2024-04-13T21:54:27.209 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:27.210 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:27Z" } 2024-04-13T21:54:27.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:27.210 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:27.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:27.211 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:32.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:32.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:32.212 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:32.268 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:32.268 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:32.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:32.307+0000 7fe6ed4cf780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:32.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:32.307+0000 7fe6ed4cf780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:32.314 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:32.315 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:32.316 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:32.317 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:32.318 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:32Z" 2024-04-13T21:54:32.320 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:32.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:32Z" } 2024-04-13T21:54:32.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:32.321 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:32.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:32.321 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:37.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:37.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:37.323 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:37.378 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:37.379 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:37.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:37.416+0000 7f14bba37780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:37.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:37.417+0000 7f14bba37780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:37.425 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:37.425 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:37.426 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:37.427 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:37.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:37.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:37.430 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:37Z" 2024-04-13T21:54:37.431 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:37.432 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:37Z" } 2024-04-13T21:54:37.433 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:37.433 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:37.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:37.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:42.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:42.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:42.434 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:42.489 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:42.489 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:42.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:42.528+0000 7f1ed78b3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:42.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:42.529+0000 7f1ed78b3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:42.536 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:42.536 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:42.536 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:42.537 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:42.538 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:42.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:42.541 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:42.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:42.543 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:42Z" 2024-04-13T21:54:42.543 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:42.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:42Z" } 2024-04-13T21:54:42.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:42.544 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:42.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:42.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:47.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:47.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:47.546 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:47.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:47.638+0000 7f8c6fd99780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:47.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:47.639+0000 7f8c6fd99780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:47.648 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:47.649 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:47.650 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:47.652 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:47.653 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:47Z" 2024-04-13T21:54:47.654 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:47.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:47Z" } 2024-04-13T21:54:47.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:47.655 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:47.655 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:47.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:52.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:52.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:52.657 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:52.712 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:52.713 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:52.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:52.749+0000 7fa9c70af780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:52.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:52.750+0000 7fa9c70af780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:52.759 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:52.759 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:52.759 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:52.759 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:52.759 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:52.760 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:52.761 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:52.762 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:52.763 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:52.764 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:52Z" 2024-04-13T21:54:52.765 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:52.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:52Z" } 2024-04-13T21:54:52.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:52.766 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:52.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:52.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:54:57.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:57.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:54:57.768 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:54:57.823 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:54:57.823 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:54:57.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:57.864+0000 7fec2c2f5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:57.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:54:57.865+0000 7fec2c2f5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:54:57.873 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:57.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:57.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:54:57.878 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:54:57Z" 2024-04-13T21:54:57.879 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:54:57.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:54:57Z" } 2024-04-13T21:54:57.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:54:57.881 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:57.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:54:57.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:55:02.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:02.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:02.882 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:02.937 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:02.937 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:02.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:02.973+0000 7f2fa0a03780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:02.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:02.974+0000 7f2fa0a03780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.127419_295.1", 2024-04-13T21:55:02.984 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.127419Z", 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:02.985 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:02.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:02.987 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:02Z" 2024-04-13T21:55:02.990 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:02.991 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.127419_295.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.127419Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:02Z" } 2024-04-13T21:55:02.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.127419_295.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.127419Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:02.992 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:02.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.127419_295.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:02.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.127419_295.1 2024-04-13T21:55:07.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:07.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:07.994 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:08.043 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:08.043 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:08.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:08.077+0000 7f13feaae780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:08.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:08.079+0000 7f13feaae780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:08.086 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:08.086 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:08.086 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.167019_300.1", 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.167019Z", 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:08Z" 2024-04-13T21:55:08.092 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:08.093 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.167019_300.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.167019Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:08Z" } 2024-04-13T21:55:08.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.167019_300.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.167019Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:08.094 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:08.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.167019_300.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:08.094 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.167019_300.1 2024-04-13T21:55:13.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:13.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:13.096 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:13.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:13.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:13.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:13.198+0000 7f5d8c55f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:13.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:13.199+0000 7f5d8c55f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:13.209 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:13.210 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:13.211 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:13.212 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:13.213 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:13.214 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:13.214 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:13.214 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:13.214 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:13Z" 2024-04-13T21:55:13.214 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:13.215 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:13Z" } 2024-04-13T21:55:13.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:13.215 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:13.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:13.216 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:18.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:18.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:18.217 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:18.273 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:18.273 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:18.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:18.310+0000 7f3adff4b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:18.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:18.311+0000 7f3adff4b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:18.320 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:18Z" 2024-04-13T21:55:18.326 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:18.327 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:18Z" } 2024-04-13T21:55:18.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:18.327 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:18.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:18.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:23.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:23.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:23.329 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:23.385 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:23.385 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:23.423+0000 7f76448d3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:23.424+0000 7f76448d3780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:23.434 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:23.435 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:23.436 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:23.437 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:23.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:23.439 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:23.440 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:23Z" 2024-04-13T21:55:23.441 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:23.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:23Z" } 2024-04-13T21:55:23.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:23.442 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:23.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:23.442 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:28.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:28.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:28.444 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:28.502 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:28.502 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:28.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:28.539+0000 7f2a01b1a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:28.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:28.540+0000 7f2a01b1a780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:28.550 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:28.550 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:28.550 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:28.551 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:28.552 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:28.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:28.554 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:28.555 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:28Z" 2024-04-13T21:55:28.556 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:28.557 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:28Z" } 2024-04-13T21:55:28.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:28.558 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:28.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:28.558 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:33.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:33.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:33.560 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:33.628 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:33.628 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:33.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:33.665+0000 7f44c474b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:33.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:33.666+0000 7f44c474b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:33.677 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:33.677 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:33.677 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:33.677 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:33.678 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:33.679 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:33.680 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:33.682 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:33Z" 2024-04-13T21:55:33.683 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:33.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:33Z" } 2024-04-13T21:55:33.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:33.685 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:33.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:33.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:38.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:38.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:38.686 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:38.741 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:38.741 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:38.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:38.778+0000 7f694bb6f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:38.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:38.779+0000 7f694bb6f780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:38.788 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:38.788 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:38.790 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:38.791 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:38.792 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:38.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:38.794 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:38.794 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:38.794 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:38.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:38.794 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:38Z" 2024-04-13T21:55:38.794 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:38.795 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:38Z" } 2024-04-13T21:55:38.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:38.795 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:38.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:38.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:43.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:43.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:43.797 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:43.853 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:43.853 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:43.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:43.891+0000 7f1d73bce780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:43.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:43.892+0000 7f1d73bce780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:43.900 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:43.900 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:43.901 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:43.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:43.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:43.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:43.905 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:43Z" 2024-04-13T21:55:43.906 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:43.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:43Z" } 2024-04-13T21:55:43.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:43.908 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:43.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:43.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:48.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:48.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:48.910 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:48.965 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:48.965 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:48.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:49.002+0000 7f6315535780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:48.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:49.003+0000 7f6315535780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:49.012 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:49.012 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:49.012 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:49.012 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:49.012 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:49.013 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:49.014 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:49.015 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:49.016 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:49.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:49.018 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:49.018 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:49.018 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:49.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:49.018 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:49Z" 2024-04-13T21:55:49.018 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:49.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:49Z" } 2024-04-13T21:55:49.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:49.019 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:49.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:49.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:55:54.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:54.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:54.020 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:54.071 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:54.071 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:54.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:54.106+0000 7f87fb1a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:54.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:54.107+0000 7f87fb1a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:54.115 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.183932_302.1", 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.183932Z", 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:54.117 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:54.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:54.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:54.120 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:54Z" 2024-04-13T21:55:54.121 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:54.122 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.183932_302.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.183932Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:54Z" } 2024-04-13T21:55:54.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.183932_302.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.183932Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:54.122 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:54.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.183932_302.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:54.123 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.183932_302.1 2024-04-13T21:55:59.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:59.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:55:59.124 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:55:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:55:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:55:59.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:59.216+0000 7fed9b3ba780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:59.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:55:59.218+0000 7fed9b3ba780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:55:59.239 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:55:59.239 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:55:59.239 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:55:59.239 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:55:59.239 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:55:59.239 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.183932_302.1", 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:59.240 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.183932Z", 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:59.241 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:59.242 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:55:59.243 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:55:59.244 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:55:59Z" 2024-04-13T21:55:59.245 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:55:59.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.183932_302.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.183932Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:55:59Z" } 2024-04-13T21:55:59.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.183932_302.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.183932Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:55:59.246 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:59.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.183932_302.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:55:59.246 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.183932_302.1 2024-04-13T21:56:04.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:04.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:04.248 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:56:04.303 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:04.304 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:04.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:04.340+0000 7f8f89be9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:04.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:04.341+0000 7f8f89be9780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:04.350 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:56:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:04.352 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:56:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:56:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:04.354 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:04.354 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:56:04.354 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:04.354 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:04.354 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:04.354 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:56:04.355 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:04.355 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:04.356 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:56:04.358 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:56:04.359 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:56:04.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:04.359 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:56:04Z" 2024-04-13T21:56:04.359 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:04.360 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:56:04Z" } 2024-04-13T21:56:04.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:56:04.360 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:04.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:04.361 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:56:09.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:09.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:09.362 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:56:09.420 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:09.420 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:09.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:09.459+0000 7fe201095780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:09.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:09.460+0000 7fe201095780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:09.470 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:56:09Z" 2024-04-13T21:56:09.476 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:09.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:56:09Z" } 2024-04-13T21:56:09.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:56:09.478 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:09.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:09.478 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:56:14.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:14.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:14.480 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:56:14.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:14.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:14.572+0000 7f558e74e780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:14.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:14.573+0000 7f558e74e780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:14.583 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:56:14.584 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:14.585 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:14.586 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:14.587 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:56:14.588 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:56:14.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:14.589 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:56:14Z" 2024-04-13T21:56:14.589 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:14.590 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:56:14Z" } 2024-04-13T21:56:14.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:56:14.590 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:14.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:14.590 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:56:19.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:19.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:19.592 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:56:19.647 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:19.647 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:19.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:19.682+0000 7f7ae8f90780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:19.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:19.683+0000 7f7ae8f90780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:19.692 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:19.692 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:56:19.692 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:19.693 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:19.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:56:19.695 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:19.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:19.697 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:56:19Z" 2024-04-13T21:56:19.698 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:19.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:56:19Z" } 2024-04-13T21:56:19.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:56:19.700 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:19.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:19.700 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:56:24.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:24.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:24.702 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:56:24.756 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:24.756 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:24.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:24.793+0000 7f07c9305780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:24.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:24.794+0000 7f07c9305780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:24.804 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:24.804 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:56:24.804 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:56:24.804 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:56:24.804 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713044569.207788_306.1", 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:24.805 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:42:49.207788Z", 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:24.806 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:24.807 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:24.808 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:56:24.809 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:56:24.810 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:56:24.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:24.810 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:56:24Z" 2024-04-13T21:56:24.810 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:24.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713044569.207788_306.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:42:49.207788Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:56:24Z" } 2024-04-13T21:56:24.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.207788_306.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.207788Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:56:24.811 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:24.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1713044569.207788_306.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:24.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045246.555013_414.1 target=1_1713044569.207788_306.1 2024-04-13T21:56:29.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:29.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:56:29.812 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:56:29.868 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:29.868 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:29.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:29.905+0000 7f713d69b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:29.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:56:29.905+0000 7f713d69b780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:56:29.915 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:29.915 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:56:29.915 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:56:29.915 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:56:29.916 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:29.917 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:56:29.918 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:29.919 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:29.920 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:56:29Z" 2024-04-13T21:56:29.921 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:29.922 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:56:29Z" } 2024-04-13T21:56:29.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:56:29.922 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:29.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:56:29.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:56:29.923 DEBUG:boto:path=/vwnfns-37/ 2024-04-13T21:56:29.923 DEBUG:boto:auth_path=/vwnfns-37/ 2024-04-13T21:56:29.923 DEBUG:boto:Method: HEAD 2024-04-13T21:56:29.923 DEBUG:boto:Path: /vwnfns-37/ 2024-04-13T21:56:29.923 DEBUG:boto:Data: 2024-04-13T21:56:29.924 DEBUG:boto:Headers: {} 2024-04-13T21:56:29.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:56:29.924 DEBUG:boto:Port: 8001 2024-04-13T21:56:29.924 DEBUG:boto:Params: {} 2024-04-13T21:56:29.924 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:56:29.925 DEBUG:boto:Token: None 2024-04-13T21:56:29.925 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:56:29 GMT /vwnfns-37/ 2024-04-13T21:56:29.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5uQOFYFcSTHVcsNelxL7ZIoX7HE= 2024-04-13T21:56:29.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:56:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5uQOFYFcSTHVcsNelxL7ZIoX7HE=', 'Content-Length': '0'} 2024-04-13T21:56:29.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b90e7a01e6c8102-00661aff8d-1856-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:56:29.931 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:56:29.931 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:56:29.931 DEBUG:boto:Method: HEAD 2024-04-13T21:56:29.931 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:56:29.931 DEBUG:boto:Data: 2024-04-13T21:56:29.932 DEBUG:boto:Headers: {} 2024-04-13T21:56:29.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:56:29.932 DEBUG:boto:Port: 8001 2024-04-13T21:56:29.932 DEBUG:boto:Params: {} 2024-04-13T21:56:29.932 DEBUG:boto:Token: None 2024-04-13T21:56:29.932 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:56:29 GMT /vwnfns-37/testredirect 2024-04-13T21:56:29.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KmiXTdSev9vj5RJhgVfW260hFKc= 2024-04-13T21:56:29.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:56:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KmiXTdSev9vj5RJhgVfW260hFKc=', 'Content-Length': '0'} 2024-04-13T21:56:29.933 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002688d04c2623c4ca-00661aff8d-1856-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:56:29.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=f6f208d6-f791-479b-bc3d-112512eec2d4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:56:29.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=f6f208d6-f791-479b-bc3d-112512eec2d4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:56:29.934 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=f6f208d6-f791-479b-bc3d-112512eec2d4 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:56:29.984 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:29.984 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a2.rgw.control", 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-13T21:56:30.004 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a2.rgw.log", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a2.rgw.otp", 2024-04-13T21:56:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:56:30.006 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.007 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:56:30.008 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-13T21:56:30.008 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:30.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:56:30.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:56:30.009 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:56:30.056 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:56:30.056 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:56:30.097 INFO:teuthology.orchestra.run.smithi132.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:56:30.121 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:56:30.121 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:30.121 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6, 2024-04-13T21:56:30.121 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:56:30.122 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:56:30.123 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": false, 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:56:30.124 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:56:30.125 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.127 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:56:30.128 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.129 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:56:30.130 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:56:30.131 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:56:30.131 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:56:30.133 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:56:30.133 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:56:30.133 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:56:30.134 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.135 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:56:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.137 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:56:30.139 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:56:30.140 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:56:30.141 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:56:30.142 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:56:30.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:56:30.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:56:30.143 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:56:30.143 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:56:30.143 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:56:30.143 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:56:30.144 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T21:57:30.146 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.147 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.147 DEBUG:boto:Method: HEAD 2024-04-13T21:57:30.147 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.147 DEBUG:boto:Data: 2024-04-13T21:57:30.147 DEBUG:boto:Headers: {} 2024-04-13T21:57:30.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:57:30.147 DEBUG:boto:Port: 8001 2024-04-13T21:57:30.147 DEBUG:boto:Params: {} 2024-04-13T21:57:30.148 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:57:30.148 DEBUG:boto:Token: None 2024-04-13T21:57:30.148 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aaQn3/6GirmJKN31iq/BfMx0HSA= 2024-04-13T21:57:30.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aaQn3/6GirmJKN31iq/BfMx0HSA=', 'Content-Length': '0'} 2024-04-13T21:57:30.156 DEBUG:boto:Response headers: [('Location', 'http://smithi132.front.sepia.ceph.com:8000/vwnfns-37/testredirect?'), ('x-amz-request-id', 'tx000000d30e44354f0d531-00661affca-195e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.156 DEBUG:boto:Redirecting: http://smithi132.front.sepia.ceph.com/vwnfns-37/testredirect 2024-04-13T21:57:30.156 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2024-04-13T21:57:30.156 DEBUG:boto:Token: None 2024-04-13T21:57:30.157 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.157 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aaQn3/6GirmJKN31iq/BfMx0HSA= 2024-04-13T21:57:30.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aaQn3/6GirmJKN31iq/BfMx0HSA=', 'Content-Length': '0'} 2024-04-13T21:57:30.161 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000009d839a02dfdc7afa-00661affca-1962-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.162 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.162 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.162 DEBUG:boto:Method: GET 2024-04-13T21:57:30.162 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.162 DEBUG:boto:Data: 2024-04-13T21:57:30.162 DEBUG:boto:Headers: {} 2024-04-13T21:57:30.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:57:30.163 DEBUG:boto:Port: 8001 2024-04-13T21:57:30.163 DEBUG:boto:Params: {} 2024-04-13T21:57:30.163 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:57:30.163 DEBUG:boto:Token: None 2024-04-13T21:57:30.163 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.163 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.166 DEBUG:boto:Response headers: [('Location', 'http://smithi132.front.sepia.ceph.com:8000/vwnfns-37/testredirect?'), ('x-amz-request-id', 'tx000007ab3a8e3675d025d-00661affca-195e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.166 DEBUG:boto:Redirecting: http://smithi132.front.sepia.ceph.com/vwnfns-37/testredirect 2024-04-13T21:57:30.167 DEBUG:boto:Token: None 2024-04-13T21:57:30.167 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.167 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.169 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000182ce22df4b745be-00661affca-1962-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.169 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.169 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.170 DEBUG:boto:Method: PUT 2024-04-13T21:57:30.170 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.170 DEBUG:boto:Data: 2024-04-13T21:57:30.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T21:57:30.170 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:57:30.170 DEBUG:boto:Port: 8000 2024-04-13T21:57:30.170 DEBUG:boto:Params: {} 2024-04-13T21:57:30.170 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:57:30.170 DEBUG:boto:Token: None 2024-04-13T21:57:30.171 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:83zFDWL+SUws9hS4PwMNJUSOgAA= 2024-04-13T21:57:30.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:83zFDWL+SUws9hS4PwMNJUSOgAA='} 2024-04-13T21:57:30.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c78ba21022615fc-00661affca-1962-a1'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.183 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.183 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.184 DEBUG:boto:Method: GET 2024-04-13T21:57:30.184 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.184 DEBUG:boto:Data: 2024-04-13T21:57:30.184 DEBUG:boto:Headers: {} 2024-04-13T21:57:30.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:57:30.184 DEBUG:boto:Port: 8001 2024-04-13T21:57:30.184 DEBUG:boto:Params: {} 2024-04-13T21:57:30.184 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:57:30.184 DEBUG:boto:Token: None 2024-04-13T21:57:30.184 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.188 DEBUG:boto:Response headers: [('Location', 'http://smithi132.front.sepia.ceph.com:8000/vwnfns-37/testredirect?'), ('x-amz-request-id', 'tx0000013c9aca3eba4295a-00661affca-195e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.188 DEBUG:boto:Redirecting: http://smithi132.front.sepia.ceph.com/vwnfns-37/testredirect 2024-04-13T21:57:30.188 DEBUG:boto:Token: None 2024-04-13T21:57:30.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.190 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000db2cf44edc067d24-00661affca-1962-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.191 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.191 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.191 DEBUG:boto:Method: PUT 2024-04-13T21:57:30.191 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.191 DEBUG:boto:Data: 2024-04-13T21:57:30.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T21:57:30.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:57:30.192 DEBUG:boto:Port: 8000 2024-04-13T21:57:30.192 DEBUG:boto:Params: {} 2024-04-13T21:57:30.192 DEBUG:boto:Token: None 2024-04-13T21:57:30.192 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MqzduZKkG7fX4ZQIdPhZnJgK2NI= 2024-04-13T21:57:30.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MqzduZKkG7fX4ZQIdPhZnJgK2NI='} 2024-04-13T21:57:30.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000090312092d1e1c7e-00661affca-1962-a1'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.198 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.198 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.198 DEBUG:boto:Method: GET 2024-04-13T21:57:30.198 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.198 DEBUG:boto:Data: 2024-04-13T21:57:30.199 DEBUG:boto:Headers: {} 2024-04-13T21:57:30.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:57:30.199 DEBUG:boto:Port: 8001 2024-04-13T21:57:30.199 DEBUG:boto:Params: {} 2024-04-13T21:57:30.199 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:57:30.199 DEBUG:boto:Token: None 2024-04-13T21:57:30.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.203 DEBUG:boto:Response headers: [('Location', 'http://smithi132.front.sepia.ceph.com:8000/vwnfns-37/testredirect?'), ('x-amz-request-id', 'tx0000095f897130f2cc9dd-00661affca-195e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.203 DEBUG:boto:Redirecting: http://smithi132.front.sepia.ceph.com/vwnfns-37/testredirect 2024-04-13T21:57:30.203 DEBUG:boto:Token: None 2024-04-13T21:57:30.203 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.203 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.205 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000a61767d67ce97f0f-00661affca-1962-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.206 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.206 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.206 DEBUG:boto:Method: PUT 2024-04-13T21:57:30.206 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.206 DEBUG:boto:Data: 2024-04-13T21:57:30.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T21:57:30.206 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:57:30.206 DEBUG:boto:Port: 8000 2024-04-13T21:57:30.206 DEBUG:boto:Params: {} 2024-04-13T21:57:30.207 DEBUG:boto:Token: None 2024-04-13T21:57:30.207 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m6OUqrrGuPCuUGOX8zTyfl2Fivc= 2024-04-13T21:57:30.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m6OUqrrGuPCuUGOX8zTyfl2Fivc='} 2024-04-13T21:57:30.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ec096e470e3f356-00661affca-1962-a1'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.213 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.213 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.213 DEBUG:boto:Method: GET 2024-04-13T21:57:30.213 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.213 DEBUG:boto:Data: 2024-04-13T21:57:30.213 DEBUG:boto:Headers: {} 2024-04-13T21:57:30.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:57:30.213 DEBUG:boto:Port: 8001 2024-04-13T21:57:30.213 DEBUG:boto:Params: {} 2024-04-13T21:57:30.213 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:57:30.214 DEBUG:boto:Token: None 2024-04-13T21:57:30.214 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.214 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.217 DEBUG:boto:Response headers: [('Location', 'http://smithi132.front.sepia.ceph.com:8000/vwnfns-37/testredirect?'), ('x-amz-request-id', 'tx00000ceffe06145cc2580-00661affca-195e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.217 DEBUG:boto:Redirecting: http://smithi132.front.sepia.ceph.com/vwnfns-37/testredirect 2024-04-13T21:57:30.217 DEBUG:boto:Token: None 2024-04-13T21:57:30.217 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.219 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000fd40771bb1999800-00661affca-1962-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.220 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.220 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.220 DEBUG:boto:Method: PUT 2024-04-13T21:57:30.220 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.220 DEBUG:boto:Data: 2024-04-13T21:57:30.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T21:57:30.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:57:30.220 DEBUG:boto:Port: 8000 2024-04-13T21:57:30.220 DEBUG:boto:Params: {} 2024-04-13T21:57:30.221 DEBUG:boto:Token: None 2024-04-13T21:57:30.221 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F1aduFsaDkBs3XooXIa/vq2AFGA= 2024-04-13T21:57:30.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F1aduFsaDkBs3XooXIa/vq2AFGA='} 2024-04-13T21:57:30.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068e7ea5f7ffa51f4-00661affca-1962-a1'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.263 DEBUG:boto:path=/vwnfns-37/testredirect 2024-04-13T21:57:30.263 DEBUG:boto:auth_path=/vwnfns-37/testredirect 2024-04-13T21:57:30.263 DEBUG:boto:Method: GET 2024-04-13T21:57:30.263 DEBUG:boto:Path: /vwnfns-37/testredirect 2024-04-13T21:57:30.263 DEBUG:boto:Data: 2024-04-13T21:57:30.263 DEBUG:boto:Headers: {} 2024-04-13T21:57:30.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:57:30.264 DEBUG:boto:Port: 8001 2024-04-13T21:57:30.264 DEBUG:boto:Params: {} 2024-04-13T21:57:30.264 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:57:30.264 DEBUG:boto:Token: None 2024-04-13T21:57:30.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.268 DEBUG:boto:Response headers: [('Location', 'http://smithi132.front.sepia.ceph.com:8000/vwnfns-37/testredirect?'), ('x-amz-request-id', 'tx0000068fa2ca05d28bbd1-00661affca-195e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.268 DEBUG:boto:Redirecting: http://smithi132.front.sepia.ceph.com/vwnfns-37/testredirect 2024-04-13T21:57:30.268 DEBUG:boto:Token: None 2024-04-13T21:57:30.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:57:30 GMT /vwnfns-37/testredirect 2024-04-13T21:57:30.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY= 2024-04-13T21:57:30.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:57:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n58IESzv1xB382WN8PNUiPys6LY=', 'Content-Length': '0'} 2024-04-13T21:57:30.271 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx00000acd4f740f0187a42-00661affca-1962-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 21:57:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:57:30.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:57:30.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:57:30.272 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:57:30.324 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:57:30.324 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:57:30.345 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:57:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:57:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:57:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-13T21:57:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a2.rgw.control", 2024-04-13T21:57:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a2.rgw.log", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-13T21:57:30.346 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a2.rgw.otp", 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:57:30.347 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.348 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.349 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:57:30.349 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-13T21:57:30.349 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:57:30.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:57:30.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:57:30.350 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:57:30.397 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:57:30.397 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:57:30.440 INFO:teuthology.orchestra.run.smithi132.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:57:30.463 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:57:30.463 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7, 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:57:30.464 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:57:30.465 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:57:30.466 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.467 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:57:30.468 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.469 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:57:30.471 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:57:30.472 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:57:30.473 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:57:30.473 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:57:30.473 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:57:30.473 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:57:30.473 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:57:30.475 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:57:30.475 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.476 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:57:30.477 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:57:30.478 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.479 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.481 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:57:30.482 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:57:30.483 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:57:30.484 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:57:30.485 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:57:30.487 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2024-04-13T21:57:30.750 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T21:57:30.753+0000 7f5e93db2700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T21:57:45.738 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T21:57:45.741+0000 7f5b022dd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T21:58:30.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:58:30.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:58:30.489 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:58:30.545 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:58:30.545 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "a2.rgw.control", 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-13T21:58:30.568 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "a2.rgw.log", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-13T21:58:30.569 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "a2.rgw.otp", 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:58:30.570 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: "compression_type": "random" 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.571 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.572 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:58:30.572 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-13T21:58:30.572 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:58:30.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:58:30.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:58:30.573 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:58:30.624 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:58:30.624 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:58:30.668 INFO:teuthology.orchestra.run.smithi132.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:58:30.691 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:58:30.691 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8, 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:58:30.692 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:58:30.693 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:58:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:58:30.695 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:58:30.696 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:58:30.697 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:58:30.698 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.699 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:58:30.700 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:58:30.701 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:58:30.705 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:58:30.706 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:58:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:58:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:58:30.709 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:58:30.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:58:30.711 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:58:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.713 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:58:30.714 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:58:30.715 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:58:30.716 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2024-04-13T21:59:30.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2024-04-13T21:59:30.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:30.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:30.721 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:30.775 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:30.775 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:30.808 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:59:30.820 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:54:06.555013Z" 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:30.822 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:59:30.824 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:30.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:59:30.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:30.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:30.825 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:30.875 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:30.875 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:30.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:30.906+0000 7f8de98f5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:30.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:30.907+0000 7f8de98f5780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:30.919 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:30.919 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:59:30.919 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:59:30.919 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:59:30.919 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:59:30.919 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:30.920 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:30.921 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:30.923 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:59:30.924 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:59:30.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:30.925 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:59:30Z" 2024-04-13T21:59:30.925 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:30.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:59:30Z" } 2024-04-13T21:59:30.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:59:30.927 DEBUG:rgw_multi.tests:log_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:30.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:30.927 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T21:59:30.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:30.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'FOIIC5FWUHTZ0J8O', '--secret', 'ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:30.927 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key FOIIC5FWUHTZ0J8O --secret ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:30.978 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:30.978 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:30.992 INFO:teuthology.orchestra.run.smithi132.stderr:NOTICE: set zone's realm_id=645d07fe-13a2-4d07-bad9-a662f2f6c3cf 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "9546248a-9355-4ec6-839b-d8605d6185b6", 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "remove", 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "remove.rgw.control", 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-13T21:59:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "remove.rgw.log", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "remove.rgw.otp", 2024-04-13T21:59:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "FOIIC5FWUHTZ0J8O", 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "ZCTk64sUXdKA1I6OLtVD5EXlPWVu2rL7" 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:59:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-13T21:59:31.009 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:31.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:31.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:31.011 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:31.059 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:31.059 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:31.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:31.074+0000 7f41379c7780 0 period (5039ce81-80d5-4d03-a26f-34b9088aaab4 does not have zone 9546248a-9355-4ec6-839b-d8605d6185b6 configured 2024-04-13T21:59:37.607 INFO:teuthology.orchestra.run.smithi132.stderr:Sending period to new master zone f6f208d6-f791-479b-bc3d-112512eec2d4 2024-04-13T21:59:37.634 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9, 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:37.635 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:59:37.636 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.637 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "9546248a-9355-4ec6-839b-d8605d6185b6", 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "remove", 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [], 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.638 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:37.639 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.640 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.649 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:59:37.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "9546248a-9355-4ec6-839b-d8605d6185b6", 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2730909249 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.651 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:37.654 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.656 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:59:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:37.657 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:37.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.658 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:37.711 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:37.711 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:37.734 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:37.741 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:37.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.742 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:37.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:37.797 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:37.823 INFO:teuthology.orchestra.run.smithi132.stderr:zone name remove not found in zonegroup a 2024-04-13T21:59:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-13T21:59:37.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.831 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:37.891 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:37.891 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:37.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:37.943 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.010 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.010 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:38.037+0000 7fb1dafcb780 0 WARNING: can't generate connection for zone 9546248a-9355-4ec6-839b-d8605d6185b6 id remove: no endpoints defined 2024-04-13T21:59:38.098 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10, 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:59:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.101 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.102 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.103 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.104 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.105 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.106 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.107 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.108 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.109 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.109 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.118 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.118 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.118 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.119 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.120 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.120 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.120 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.121 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.121 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.121 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.121 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.121 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.122 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.122 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.122 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.122 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.122 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.123 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.123 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.123 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.123 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.123 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:59:38.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:59:38.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.126 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.126 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:59:38.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.126 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.127 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:38.128 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.130 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:38.131 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:38.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-04-13T21:59:38.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.133 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.191 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.191 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.224 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:38.224 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "d08d8e25-33b2-4d1b-a123-0ca81fb397db", 2024-04-13T21:59:38.224 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "remove_zg", 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "remove_zg", 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [], 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "", 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [], 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.225 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [] 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.226 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.227 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.227 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:38.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.228 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.280 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.280 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:38.324+0000 7f5e9ffdc780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f3e9966c-8869-4d06-8592-b9048a95275f", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "remove", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "control_pool": "remove.rgw.control", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "log_pool": "remove.rgw.log", 2024-04-13T21:59:38.329 INFO:teuthology.orchestra.run.smithi132.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "otp_pool": "remove.rgw.otp", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "system_key": { 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "access_key": "", 2024-04-13T21:59:38.330 INFO:teuthology.orchestra.run.smithi132.stdout: "secret_key": "" 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_pools": [ 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "default-placement", 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": { 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD": { 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.331 INFO:teuthology.orchestra.run.smithi132.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: "index_type": 0, 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: "inline_data": true 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-13T21:59:38.332 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:38.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.334 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.398 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.399 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.489 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11, 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:59:38.490 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.491 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.492 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.492 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:38.492 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:59:38.492 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.492 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.493 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.495 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.496 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.497 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.498 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.499 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.500 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.501 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.502 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.503 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.504 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "d08d8e25-33b2-4d1b-a123-0ca81fb397db", 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "remove_zg", 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "remove_zg", 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [], 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f3e9966c-8869-4d06-8592-b9048a95275f", 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.505 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f3e9966c-8869-4d06-8592-b9048a95275f", 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "remove", 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [], 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": false, 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.507 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.508 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f3e9966c-8869-4d06-8592-b9048a95275f", 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2757326607 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.510 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:59:38.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.511 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.511 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:59:38.511 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.511 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:59:38.512 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:59:38.512 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:38.512 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:38.512 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:38.512 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:59:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.514 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.515 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:59:38.516 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:38.516 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:38.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.516 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.573 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.573 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:38.618+0000 7f9bbd1d1780 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-04-13T21:59:38.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:38.620+0000 7f9bbd1d1780 0 Removed zone from zonegroup remove_zg 2024-04-13T21:59:38.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.631 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.694 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.694 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:38.732+0000 7fb281bf4780 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-04-13T21:59:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-13T21:59:38.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.735 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T21:59:38.819 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T21:59:38.876 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:38.877 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:38.971 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:38.971 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12, 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:38.972 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.973 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.974 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T21:59:38.975 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.976 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.978 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.979 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.980 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.981 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.990 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.990 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T21:59:38.991 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T21:59:38.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T21:59:38.993 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T21:59:38.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:38.996 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T21:59:38.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.998 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:38.999 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T21:59:39.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T21:59:39.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:39.000 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:39.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-04-13T21:59:39.002 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-38 2024-04-13T21:59:39.002 DEBUG:boto:path=/vwnfns-38/ 2024-04-13T21:59:39.002 DEBUG:boto:auth_path=/vwnfns-38/ 2024-04-13T21:59:39.002 DEBUG:boto:Method: PUT 2024-04-13T21:59:39.002 DEBUG:boto:Path: /vwnfns-38/ 2024-04-13T21:59:39.003 DEBUG:boto:Data: 2024-04-13T21:59:39.003 DEBUG:boto:Headers: {} 2024-04-13T21:59:39.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:59:39.003 DEBUG:boto:Port: 8000 2024-04-13T21:59:39.003 DEBUG:boto:Params: {} 2024-04-13T21:59:39.003 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:59:39.004 DEBUG:boto:Token: None 2024-04-13T21:59:39.004 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:39 GMT /vwnfns-38/ 2024-04-13T21:59:39.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WUpBhuYDIqh41rc9cAPUCR0V5ew= 2024-04-13T21:59:39.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WUpBhuYDIqh41rc9cAPUCR0V5ew=', 'Content-Length': '0'} 2024-04-13T21:59:39.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000934d9e63cc7f578b-00661b004b-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:39.322 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-39 2024-04-13T21:59:39.322 DEBUG:boto:path=/vwnfns-39/ 2024-04-13T21:59:39.322 DEBUG:boto:auth_path=/vwnfns-39/ 2024-04-13T21:59:39.322 DEBUG:boto:Method: PUT 2024-04-13T21:59:39.322 DEBUG:boto:Path: /vwnfns-39/ 2024-04-13T21:59:39.322 DEBUG:boto:Data: 2024-04-13T21:59:39.322 DEBUG:boto:Headers: {} 2024-04-13T21:59:39.322 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:59:39.323 DEBUG:boto:Port: 8001 2024-04-13T21:59:39.323 DEBUG:boto:Params: {} 2024-04-13T21:59:39.323 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:59:39.323 DEBUG:boto:Token: None 2024-04-13T21:59:39.323 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:39 GMT /vwnfns-39/ 2024-04-13T21:59:39.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bUCN7yDQiuNalBqlLTVM25rYszc= 2024-04-13T21:59:39.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bUCN7yDQiuNalBqlLTVM25rYszc=', 'Content-Length': '0'} 2024-04-13T21:59:39.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d4be1d91c819ace9-00661b004b-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:39.376 INFO:rgw_multi.tests:create bucket zone=b1 name=vwnfns-40 2024-04-13T21:59:39.376 DEBUG:boto:path=/vwnfns-40/ 2024-04-13T21:59:39.376 DEBUG:boto:auth_path=/vwnfns-40/ 2024-04-13T21:59:39.376 DEBUG:boto:Method: PUT 2024-04-13T21:59:39.377 DEBUG:boto:Path: /vwnfns-40/ 2024-04-13T21:59:39.377 DEBUG:boto:Data: 2024-04-13T21:59:39.377 DEBUG:boto:Headers: {} 2024-04-13T21:59:39.377 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:59:39.377 DEBUG:boto:Port: 8000 2024-04-13T21:59:39.377 DEBUG:boto:Params: {} 2024-04-13T21:59:39.377 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T21:59:39.377 DEBUG:boto:Token: None 2024-04-13T21:59:39.378 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:39 GMT /vwnfns-40/ 2024-04-13T21:59:39.378 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:umQGUgxVroqmwUwPkh9rY6Zha3c= 2024-04-13T21:59:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:umQGUgxVroqmwUwPkh9rY6Zha3c=', 'Content-Length': '0'} 2024-04-13T21:59:40.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000060d9196f96c71450-00661b004c-115c-b1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:40.163 INFO:rgw_multi.tests:create bucket zone=b2 name=vwnfns-41 2024-04-13T21:59:40.163 DEBUG:boto:path=/vwnfns-41/ 2024-04-13T21:59:40.163 DEBUG:boto:auth_path=/vwnfns-41/ 2024-04-13T21:59:40.163 DEBUG:boto:Method: PUT 2024-04-13T21:59:40.163 DEBUG:boto:Path: /vwnfns-41/ 2024-04-13T21:59:40.163 DEBUG:boto:Data: 2024-04-13T21:59:40.163 DEBUG:boto:Headers: {} 2024-04-13T21:59:40.163 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:59:40.163 DEBUG:boto:Port: 8001 2024-04-13T21:59:40.163 DEBUG:boto:Params: {} 2024-04-13T21:59:40.164 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T21:59:40.164 DEBUG:boto:Token: None 2024-04-13T21:59:40.164 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:40 GMT /vwnfns-41/ 2024-04-13T21:59:40.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aaZGnfSJeYszG7sI5f0jKDHO0aI= 2024-04-13T21:59:40.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aaZGnfSJeYszG7sI5f0jKDHO0aI=', 'Content-Length': '0'} 2024-04-13T21:59:40.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002eae19ec1597f565-00661b004c-1160-b2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:40.942 DEBUG:boto:path=/vwnfns-38/ 2024-04-13T21:59:40.942 DEBUG:boto:auth_path=/vwnfns-38/ 2024-04-13T21:59:40.942 DEBUG:boto:path=/vwnfns-38/?website 2024-04-13T21:59:40.942 DEBUG:boto:auth_path=/vwnfns-38/?website 2024-04-13T21:59:40.942 DEBUG:boto:Method: PUT 2024-04-13T21:59:40.942 DEBUG:boto:Path: /vwnfns-38/?website 2024-04-13T21:59:40.942 DEBUG:boto:Data: index.htmlerror.html 2024-04-13T21:59:40.942 DEBUG:boto:Headers: {} 2024-04-13T21:59:40.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:59:40.942 DEBUG:boto:Port: 8000 2024-04-13T21:59:40.943 DEBUG:boto:Params: {} 2024-04-13T21:59:40.943 DEBUG:boto:Token: None 2024-04-13T21:59:40.943 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:40 GMT /vwnfns-38/?website 2024-04-13T21:59:40.943 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XGuldiXWVTfxJ14lNDC1YpJxMFE= 2024-04-13T21:59:40.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XGuldiXWVTfxJ14lNDC1YpJxMFE=', 'Content-Length': '241'} 2024-04-13T21:59:40.944 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx000000c0d2f763b19681e-00661b004c-1a22-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 21:59:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:40.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-04-13T21:59:40.946 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-42 2024-04-13T21:59:40.946 DEBUG:boto:path=/vwnfns-42/ 2024-04-13T21:59:40.946 DEBUG:boto:auth_path=/vwnfns-42/ 2024-04-13T21:59:40.946 DEBUG:boto:Method: PUT 2024-04-13T21:59:40.946 DEBUG:boto:Path: /vwnfns-42/ 2024-04-13T21:59:40.946 DEBUG:boto:Data: 2024-04-13T21:59:40.947 DEBUG:boto:Headers: {} 2024-04-13T21:59:40.947 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:59:40.947 DEBUG:boto:Port: 8000 2024-04-13T21:59:40.947 DEBUG:boto:Params: {} 2024-04-13T21:59:40.947 DEBUG:boto:Token: None 2024-04-13T21:59:40.947 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:40 GMT /vwnfns-42/ 2024-04-13T21:59:40.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:52jP7v01hQ8htX7tgaHrAzMEhuc= 2024-04-13T21:59:40.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:52jP7v01hQ8htX7tgaHrAzMEhuc=', 'Content-Length': '0'} 2024-04-13T21:59:40.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000544007491e3dd31d-00661b004c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:40.964 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-43 2024-04-13T21:59:40.964 DEBUG:boto:path=/vwnfns-43/ 2024-04-13T21:59:40.965 DEBUG:boto:auth_path=/vwnfns-43/ 2024-04-13T21:59:40.965 DEBUG:boto:Method: PUT 2024-04-13T21:59:40.965 DEBUG:boto:Path: /vwnfns-43/ 2024-04-13T21:59:40.965 DEBUG:boto:Data: 2024-04-13T21:59:40.965 DEBUG:boto:Headers: {} 2024-04-13T21:59:40.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:59:40.965 DEBUG:boto:Port: 8001 2024-04-13T21:59:40.965 DEBUG:boto:Params: {} 2024-04-13T21:59:40.965 DEBUG:boto:Token: None 2024-04-13T21:59:40.965 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:40 GMT /vwnfns-43/ 2024-04-13T21:59:40.966 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kl29K+DjMCvymrIXe4mPQVAfgM8= 2024-04-13T21:59:40.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kl29K+DjMCvymrIXe4mPQVAfgM8=', 'Content-Length': '0'} 2024-04-13T21:59:40.995 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a12ca75d708faa64-00661b004c-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:40.995 INFO:rgw_multi.tests:create bucket zone=b1 name=vwnfns-44 2024-04-13T21:59:40.995 DEBUG:boto:path=/vwnfns-44/ 2024-04-13T21:59:40.995 DEBUG:boto:auth_path=/vwnfns-44/ 2024-04-13T21:59:40.996 DEBUG:boto:Method: PUT 2024-04-13T21:59:40.996 DEBUG:boto:Path: /vwnfns-44/ 2024-04-13T21:59:40.996 DEBUG:boto:Data: 2024-04-13T21:59:40.996 DEBUG:boto:Headers: {} 2024-04-13T21:59:40.996 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:59:40.996 DEBUG:boto:Port: 8000 2024-04-13T21:59:40.996 DEBUG:boto:Params: {} 2024-04-13T21:59:40.996 DEBUG:boto:Token: None 2024-04-13T21:59:40.996 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:40 GMT /vwnfns-44/ 2024-04-13T21:59:40.997 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HVkKLASG8xU+YVpacISeWV/JAVQ= 2024-04-13T21:59:40.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HVkKLASG8xU+YVpacISeWV/JAVQ=', 'Content-Length': '0'} 2024-04-13T21:59:41.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cb1681e738fe2293-00661b004d-1164-b1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:41.138 INFO:rgw_multi.tests:create bucket zone=b2 name=vwnfns-45 2024-04-13T21:59:41.138 DEBUG:boto:path=/vwnfns-45/ 2024-04-13T21:59:41.138 DEBUG:boto:auth_path=/vwnfns-45/ 2024-04-13T21:59:41.138 DEBUG:boto:Method: PUT 2024-04-13T21:59:41.138 DEBUG:boto:Path: /vwnfns-45/ 2024-04-13T21:59:41.138 DEBUG:boto:Data: 2024-04-13T21:59:41.138 DEBUG:boto:Headers: {} 2024-04-13T21:59:41.139 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:59:41.139 DEBUG:boto:Port: 8001 2024-04-13T21:59:41.139 DEBUG:boto:Params: {} 2024-04-13T21:59:41.139 DEBUG:boto:Token: None 2024-04-13T21:59:41.139 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:41 GMT /vwnfns-45/ 2024-04-13T21:59:41.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WwT7/fWXG8XlLKYgtQSCuuL/13o= 2024-04-13T21:59:41.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WwT7/fWXG8XlLKYgtQSCuuL/13o=', 'Content-Length': '0'} 2024-04-13T21:59:41.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009a323052413e990e-00661b004d-1160-b2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:41.182 DEBUG:boto:path=/vwnfns-42/ 2024-04-13T21:59:41.182 DEBUG:boto:auth_path=/vwnfns-42/ 2024-04-13T21:59:41.182 DEBUG:boto:path=/vwnfns-42/?policy 2024-04-13T21:59:41.182 DEBUG:boto:auth_path=/vwnfns-42/?policy 2024-04-13T21:59:41.182 DEBUG:boto:Method: PUT 2024-04-13T21:59:41.182 DEBUG:boto:Path: /vwnfns-42/?policy 2024-04-13T21:59:41.182 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-13T21:59:41.182 DEBUG:boto:Headers: {} 2024-04-13T21:59:41.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:59:41.183 DEBUG:boto:Port: 8000 2024-04-13T21:59:41.183 DEBUG:boto:Params: {} 2024-04-13T21:59:41.183 DEBUG:boto:Token: None 2024-04-13T21:59:41.183 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:41 GMT /vwnfns-42/?policy 2024-04-13T21:59:41.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wSFfRT++WmcaV09/IqUsaL5XxqA= 2024-04-13T21:59:41.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wSFfRT++WmcaV09/IqUsaL5XxqA=', 'Content-Length': '97'} 2024-04-13T21:59:41.189 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a2a6098dfafec5e-00661b004d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:41.189 DEBUG:boto:path=/vwnfns-42/ 2024-04-13T21:59:41.189 DEBUG:boto:auth_path=/vwnfns-42/ 2024-04-13T21:59:41.190 DEBUG:boto:path=/vwnfns-42/?policy 2024-04-13T21:59:41.190 DEBUG:boto:auth_path=/vwnfns-42/?policy 2024-04-13T21:59:41.190 DEBUG:boto:Method: GET 2024-04-13T21:59:41.190 DEBUG:boto:Path: /vwnfns-42/?policy 2024-04-13T21:59:41.190 DEBUG:boto:Data: 2024-04-13T21:59:41.190 DEBUG:boto:Headers: {} 2024-04-13T21:59:41.190 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:59:41.190 DEBUG:boto:Port: 8000 2024-04-13T21:59:41.190 DEBUG:boto:Params: {} 2024-04-13T21:59:41.191 DEBUG:boto:Token: None 2024-04-13T21:59:41.191 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:59:41 GMT /vwnfns-42/?policy 2024-04-13T21:59:41.191 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gEAgoKmTG2oarOchtv5XwQL/Bd4= 2024-04-13T21:59:41.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gEAgoKmTG2oarOchtv5XwQL/Bd4=', 'Content-Length': '0'} 2024-04-13T21:59:41.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a354c5cd17bbc8fb-00661b004d-1a22-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 13 Apr 2024 21:59:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:41.192 DEBUG:boto:path=/vwnfns-43/ 2024-04-13T21:59:41.193 DEBUG:boto:auth_path=/vwnfns-43/ 2024-04-13T21:59:41.193 DEBUG:boto:path=/vwnfns-43/?policy 2024-04-13T21:59:41.193 DEBUG:boto:auth_path=/vwnfns-43/?policy 2024-04-13T21:59:41.193 DEBUG:boto:Method: PUT 2024-04-13T21:59:41.193 DEBUG:boto:Path: /vwnfns-43/?policy 2024-04-13T21:59:41.193 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-13T21:59:41.193 DEBUG:boto:Headers: {} 2024-04-13T21:59:41.193 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:59:41.193 DEBUG:boto:Port: 8001 2024-04-13T21:59:41.193 DEBUG:boto:Params: {} 2024-04-13T21:59:41.193 DEBUG:boto:Token: None 2024-04-13T21:59:41.194 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:41 GMT /vwnfns-43/?policy 2024-04-13T21:59:41.194 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B6Fba3hJR2NnL7zq64m9Lu6M0oQ= 2024-04-13T21:59:41.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B6Fba3hJR2NnL7zq64m9Lu6M0oQ=', 'Content-Length': '97'} 2024-04-13T21:59:42.204 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001fed9e4d52b1cd4d-00661b004d-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 21:59:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:42.204 DEBUG:boto:path=/vwnfns-43/ 2024-04-13T21:59:42.205 DEBUG:boto:auth_path=/vwnfns-43/ 2024-04-13T21:59:42.205 DEBUG:boto:path=/vwnfns-43/?policy 2024-04-13T21:59:42.205 DEBUG:boto:auth_path=/vwnfns-43/?policy 2024-04-13T21:59:42.205 DEBUG:boto:Method: GET 2024-04-13T21:59:42.205 DEBUG:boto:Path: /vwnfns-43/?policy 2024-04-13T21:59:42.205 DEBUG:boto:Data: 2024-04-13T21:59:42.205 DEBUG:boto:Headers: {} 2024-04-13T21:59:42.205 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:59:42.205 DEBUG:boto:Port: 8001 2024-04-13T21:59:42.205 DEBUG:boto:Params: {} 2024-04-13T21:59:42.206 DEBUG:boto:Token: None 2024-04-13T21:59:42.206 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:59:42 GMT /vwnfns-43/?policy 2024-04-13T21:59:42.206 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VXrrDG4L9rrU8SA/1SidlpSYZu4= 2024-04-13T21:59:42.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:42 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VXrrDG4L9rrU8SA/1SidlpSYZu4=', 'Content-Length': '0'} 2024-04-13T21:59:42.207 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000398e3994ea94a267-00661b004e-1a1e-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 13 Apr 2024 21:59:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:42.208 DEBUG:boto:path=/vwnfns-44/ 2024-04-13T21:59:42.208 DEBUG:boto:auth_path=/vwnfns-44/ 2024-04-13T21:59:42.208 DEBUG:boto:path=/vwnfns-44/?policy 2024-04-13T21:59:42.208 DEBUG:boto:auth_path=/vwnfns-44/?policy 2024-04-13T21:59:42.208 DEBUG:boto:Method: PUT 2024-04-13T21:59:42.208 DEBUG:boto:Path: /vwnfns-44/?policy 2024-04-13T21:59:42.208 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-13T21:59:42.208 DEBUG:boto:Headers: {} 2024-04-13T21:59:42.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:59:42.208 DEBUG:boto:Port: 8000 2024-04-13T21:59:42.209 DEBUG:boto:Params: {} 2024-04-13T21:59:42.209 DEBUG:boto:Token: None 2024-04-13T21:59:42.209 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:42 GMT /vwnfns-44/?policy 2024-04-13T21:59:42.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t3T52uVoz7r4ybEI26sEqj3oICg= 2024-04-13T21:59:42.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:42 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t3T52uVoz7r4ybEI26sEqj3oICg=', 'Content-Length': '97'} 2024-04-13T21:59:43.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050d9cf72d41f29c5-00661b004e-1164-b1'), ('Date', 'Sat, 13 Apr 2024 21:59:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:43.222 DEBUG:boto:path=/vwnfns-44/ 2024-04-13T21:59:43.222 DEBUG:boto:auth_path=/vwnfns-44/ 2024-04-13T21:59:43.223 DEBUG:boto:path=/vwnfns-44/?policy 2024-04-13T21:59:43.223 DEBUG:boto:auth_path=/vwnfns-44/?policy 2024-04-13T21:59:43.223 DEBUG:boto:Method: GET 2024-04-13T21:59:43.223 DEBUG:boto:Path: /vwnfns-44/?policy 2024-04-13T21:59:43.223 DEBUG:boto:Data: 2024-04-13T21:59:43.223 DEBUG:boto:Headers: {} 2024-04-13T21:59:43.223 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2024-04-13T21:59:43.223 DEBUG:boto:Port: 8000 2024-04-13T21:59:43.223 DEBUG:boto:Params: {} 2024-04-13T21:59:43.223 DEBUG:boto:Token: None 2024-04-13T21:59:43.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:59:43 GMT /vwnfns-44/?policy 2024-04-13T21:59:43.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r/NdTT5sjdWJQrHEy3XnOjl6p/c= 2024-04-13T21:59:43.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:43 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r/NdTT5sjdWJQrHEy3XnOjl6p/c=', 'Content-Length': '0'} 2024-04-13T21:59:43.225 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f73f27534964e862-00661b004f-1164-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 13 Apr 2024 21:59:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:43.225 DEBUG:boto:path=/vwnfns-45/ 2024-04-13T21:59:43.225 DEBUG:boto:auth_path=/vwnfns-45/ 2024-04-13T21:59:43.225 DEBUG:boto:path=/vwnfns-45/?policy 2024-04-13T21:59:43.226 DEBUG:boto:auth_path=/vwnfns-45/?policy 2024-04-13T21:59:43.226 DEBUG:boto:Method: PUT 2024-04-13T21:59:43.226 DEBUG:boto:Path: /vwnfns-45/?policy 2024-04-13T21:59:43.226 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-13T21:59:43.226 DEBUG:boto:Headers: {} 2024-04-13T21:59:43.226 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:59:43.226 DEBUG:boto:Port: 8001 2024-04-13T21:59:43.226 DEBUG:boto:Params: {} 2024-04-13T21:59:43.226 DEBUG:boto:Token: None 2024-04-13T21:59:43.227 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:43 GMT /vwnfns-45/?policy 2024-04-13T21:59:43.227 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5s4ZUxcdkGxZWaeFVeECqnajkUk= 2024-04-13T21:59:43.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:43 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5s4ZUxcdkGxZWaeFVeECqnajkUk=', 'Content-Length': '97'} 2024-04-13T21:59:44.237 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9406f7205aa5829-00661b004f-1160-b2'), ('Date', 'Sat, 13 Apr 2024 21:59:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:44.238 DEBUG:boto:path=/vwnfns-45/ 2024-04-13T21:59:44.238 DEBUG:boto:auth_path=/vwnfns-45/ 2024-04-13T21:59:44.238 DEBUG:boto:path=/vwnfns-45/?policy 2024-04-13T21:59:44.238 DEBUG:boto:auth_path=/vwnfns-45/?policy 2024-04-13T21:59:44.238 DEBUG:boto:Method: GET 2024-04-13T21:59:44.238 DEBUG:boto:Path: /vwnfns-45/?policy 2024-04-13T21:59:44.238 DEBUG:boto:Data: 2024-04-13T21:59:44.238 DEBUG:boto:Headers: {} 2024-04-13T21:59:44.238 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2024-04-13T21:59:44.238 DEBUG:boto:Port: 8001 2024-04-13T21:59:44.239 DEBUG:boto:Params: {} 2024-04-13T21:59:44.239 DEBUG:boto:Token: None 2024-04-13T21:59:44.239 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 21:59:44 GMT /vwnfns-45/?policy 2024-04-13T21:59:44.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1jwziNQxvUiatE+NuqUdakc5yoQ= 2024-04-13T21:59:44.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1jwziNQxvUiatE+NuqUdakc5yoQ=', 'Content-Length': '0'} 2024-04-13T21:59:44.240 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044d9905fbbc5e752-00661b0050-1160-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 13 Apr 2024 21:59:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:44.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-04-13T21:59:44.241 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-46 2024-04-13T21:59:44.241 DEBUG:boto:path=/vwnfns-46/ 2024-04-13T21:59:44.241 DEBUG:boto:auth_path=/vwnfns-46/ 2024-04-13T21:59:44.241 DEBUG:boto:Method: PUT 2024-04-13T21:59:44.241 DEBUG:boto:Path: /vwnfns-46/ 2024-04-13T21:59:44.242 DEBUG:boto:Data: 2024-04-13T21:59:44.242 DEBUG:boto:Headers: {} 2024-04-13T21:59:44.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T21:59:44.242 DEBUG:boto:Port: 8000 2024-04-13T21:59:44.242 DEBUG:boto:Params: {} 2024-04-13T21:59:44.242 DEBUG:boto:Token: None 2024-04-13T21:59:44.242 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:44 GMT /vwnfns-46/ 2024-04-13T21:59:44.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1jRExDPhIaxb7QjHTTmvya1G/7g= 2024-04-13T21:59:44.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1jRExDPhIaxb7QjHTTmvya1G/7g=', 'Content-Length': '0'} 2024-04-13T21:59:44.260 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa9a065cc1cb4112-00661b0050-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:44.260 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-47 2024-04-13T21:59:44.260 DEBUG:boto:path=/vwnfns-47/ 2024-04-13T21:59:44.261 DEBUG:boto:auth_path=/vwnfns-47/ 2024-04-13T21:59:44.261 DEBUG:boto:Method: PUT 2024-04-13T21:59:44.261 DEBUG:boto:Path: /vwnfns-47/ 2024-04-13T21:59:44.261 DEBUG:boto:Data: 2024-04-13T21:59:44.261 DEBUG:boto:Headers: {} 2024-04-13T21:59:44.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T21:59:44.261 DEBUG:boto:Port: 8001 2024-04-13T21:59:44.261 DEBUG:boto:Params: {} 2024-04-13T21:59:44.261 DEBUG:boto:Token: None 2024-04-13T21:59:44.262 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 21:59:44 GMT /vwnfns-47/ 2024-04-13T21:59:44.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C92ynnMyndqw7tx1M5IpnVABRJ0= 2024-04-13T21:59:44.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:59:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C92ynnMyndqw7tx1M5IpnVABRJ0=', 'Content-Length': '0'} 2024-04-13T21:59:44.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e553a2e616b0ae4-00661b0050-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:59:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T21:59:44.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:44.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:44.294 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:44.343 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:44.343 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:44.374 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T21:59:44.386 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T21:59:44.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045584.281589_602.1", 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:59:44.281589Z" 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:44.387 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:59:44.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.388 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:44.388 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T21:59:44.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:44.388 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T21:59:44.389 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:44.389 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T21:59:44.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:44.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:44.390 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:44.438 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:44.438 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:44.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:44.469+0000 7f5aa2aaa780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:44.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:44.470+0000 7f5aa2aaa780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:44.480 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:59:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:59:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:59:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:59:44Z" 2024-04-13T21:59:44.487 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:44.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:59:44Z" } 2024-04-13T21:59:44.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:59:44.489 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:44.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:44.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T21:59:49.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:49.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:49.491 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:49.547 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:49.547 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:49.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:49.580+0000 7f7271573780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:49.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:49.581+0000 7f7271573780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:49.592 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:49.593 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:49.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:49.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:49.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:49.597 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:59:49Z" 2024-04-13T21:59:49.598 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:49.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:59:49Z" } 2024-04-13T21:59:49.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:59:49.600 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:49.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:49.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T21:59:54.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:54.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:54.601 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:54.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:54.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:54.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:54.690+0000 7fcc19143780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:54.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:54.691+0000 7fcc19143780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:54.703 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:59:54.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:54.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:59:54.706 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:54.707 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:54.708 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:59:54Z" 2024-04-13T21:59:54.709 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:54.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:59:54Z" } 2024-04-13T21:59:54.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:59:54.711 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:54.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:54.711 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T21:59:59.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:59.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T21:59:59.713 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T21:59:59.768 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T21:59:59.768 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T21:59:59.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:59.801+0000 7fd5eac10780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:59.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T21:59:59.802+0000 7fd5eac10780 20 RGW-SYNC:meta: read sync status 2024-04-13T21:59:59.814 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:59.815 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:59.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:59.817 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:59.818 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T21:59:59.819 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T21:59:59.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T21:59:59.821 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T21:59:59Z" 2024-04-13T21:59:59.821 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T21:59:59.822 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T21:59:59Z" } 2024-04-13T21:59:59.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T21:59:59.822 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:59.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T21:59:59.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:04.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:04.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:04.824 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:04.880 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:04.880 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:04.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:04.914+0000 7f41127db780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:04.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:04.915+0000 7f41127db780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:04.927 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:04.927 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:04.927 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:04.927 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:04.927 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:04.928 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:04.929 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:04.930 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:04.931 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:04.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:04.933 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:04.933 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:04.933 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:04.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:04.933 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:04Z" 2024-04-13T22:00:04.933 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:04.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:04Z" } 2024-04-13T22:00:04.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:04.934 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:04.934 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:04.935 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:09.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:09.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:09.936 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:09.992 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:09.992 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:10.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:10.025+0000 7f618a799780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:10.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:10.026+0000 7f618a799780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:10.038 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:10.038 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:10.039 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:10.040 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:10.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:10.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:10.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:10Z" 2024-04-13T22:00:10.044 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:10.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:10Z" } 2024-04-13T22:00:10.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:10.046 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:10.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:10.046 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:15.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:15.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:15.048 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:15.103 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:15.103 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:15.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:15.141+0000 7ff8eac06780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:15.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:15.142+0000 7ff8eac06780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:15.153 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:15.153 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:15.153 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:15.153 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:15.153 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:15.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:15.155 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:15.156 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:15.157 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:15.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:15.159 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:15.159 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:15.159 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:15.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:15.159 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:15Z" 2024-04-13T22:00:15.159 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:15.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:15Z" } 2024-04-13T22:00:15.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:15.160 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:15.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:15.161 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:20.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:20.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:20.162 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:20.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:20.218 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:20.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:20.251+0000 7f39caeca780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:20.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:20.252+0000 7f39caeca780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:20.264 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:20.264 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:20.265 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:20.266 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:20.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:20.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:20.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:20.268 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:20.269 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:20.270 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:20.271 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:20.272 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:20.272 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:20.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:20.272 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:20Z" 2024-04-13T22:00:20.272 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:20.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:20Z" } 2024-04-13T22:00:20.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:20.274 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:20.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:20.274 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:25.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:25.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:25.276 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:25.333 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:25.333 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:25.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:25.369+0000 7fdb9046f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:25.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:25.370+0000 7fdb9046f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:25.382 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:25.382 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:25.382 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:25.382 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:25.383 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:25.384 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:25.385 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:25.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:25.387 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:25.388 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:25.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:25.389 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:25Z" 2024-04-13T22:00:25.389 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:25.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:25Z" } 2024-04-13T22:00:25.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:25.390 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:25.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:25.390 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:30.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:30.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:30.392 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:30.447 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:30.448 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:30.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:30.481+0000 7f05919f5780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:30.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:30.482+0000 7f05919f5780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:30.496 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:30.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:30.498 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:30.499 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:30.500 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:30.501 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:30Z" 2024-04-13T22:00:30.502 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:30.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:30Z" } 2024-04-13T22:00:30.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:30.503 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:30.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:30.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:35.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:35.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:35.504 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:35.559 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:35.560 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:35.593+0000 7fa85ccac780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:35.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:35.594+0000 7fa85ccac780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:35.606 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:35.606 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:35.606 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:35.606 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:35.606 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:35.607 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:35.608 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:35.609 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:35.610 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:35.611 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:35.612 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:35.612 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:35.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:35.612 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:35Z" 2024-04-13T22:00:35.612 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:35.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:35Z" } 2024-04-13T22:00:35.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:35.613 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:35.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:35.613 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:40.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:40.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:40.615 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:40.669 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:40.669 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:40.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:40.703+0000 7f7a70fe0780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:40.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:40.704+0000 7f7a70fe0780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:40.716 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:40.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:40.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:40.719 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:40.720 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:40.721 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:40.722 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:40.723 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:40.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:40.723 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:40Z" 2024-04-13T22:00:40.723 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:40.724 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:40Z" } 2024-04-13T22:00:40.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:40.724 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:40.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:40.724 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:45.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:45.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:45.726 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:45.782 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:45.782 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:45.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:45.816+0000 7fec68ffa780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:45.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:45.817+0000 7fec68ffa780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:45.830 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:45.830 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:45.831 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:45.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:45.833 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:45.834 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:45.836 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:45Z" 2024-04-13T22:00:45.837 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:45.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:45Z" } 2024-04-13T22:00:45.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:45.838 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:45.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:45.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:50.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:50.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:50.839 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:50.896 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:50.896 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:50.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:50.931+0000 7f2b55097780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:50.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:50.932+0000 7f2b55097780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:50.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:50.946 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:50.947 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:50.948 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:50Z" 2024-04-13T22:00:50.949 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:50.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:50Z" } 2024-04-13T22:00:50.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:50.951 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:50.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:50.951 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:00:55.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:55.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:00:55.953 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:00:56.008 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:00:56.008 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:00:56.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:56.043+0000 7f1426447780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:56.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:00:56.044+0000 7f1426447780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:00:56.056 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:56.057 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:56.058 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:56.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:56.060 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:00:56.061 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:56.062 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:00:56.063 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:00:56.063 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:00:56.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:00:56.063 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:00:56Z" 2024-04-13T22:00:56.063 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:00:56.064 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:00:56Z" } 2024-04-13T22:00:56.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:00:56.064 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:56.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:00:56.064 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:01.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:01.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:01.067 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:01.123 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:01.123 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:01.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:01.157+0000 7f4e5a480780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:01.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:01.158+0000 7f4e5a480780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:01.171 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:01.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:01.173 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:01.174 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:01.175 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:01.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:01.177 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:01Z" 2024-04-13T22:01:01.177 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:01.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:01Z" } 2024-04-13T22:01:01.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:01.178 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:01.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:01.178 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:06.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:06.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:06.180 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:06.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:06.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:06.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:06.271+0000 7f90bf89b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:06.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:06.271+0000 7f90bf89b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:06.287 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:06.288 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:06.289 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:06.290 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:06.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:06.290 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:06Z" 2024-04-13T22:01:06.290 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:06.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:06Z" } 2024-04-13T22:01:06.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:06.291 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:06.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:06.291 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:11.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:11.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:11.292 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:11.349 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:11.349 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:11.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:11.384+0000 7f37d3cbd780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:11.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:11.385+0000 7f37d3cbd780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:11.398 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:11.399 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:11.400 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:11.401 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:11.402 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:11.403 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:11.404 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:11.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:11.404 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:11Z" 2024-04-13T22:01:11.404 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:11.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:11Z" } 2024-04-13T22:01:11.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:11.405 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:11.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:11.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:16.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:16.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:16.407 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:16.462 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:16.463 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:16.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:16.496+0000 7f578c62b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:16.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:16.497+0000 7f578c62b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:16.509 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:16.514 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:16.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:16.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:16.514 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:16Z" 2024-04-13T22:01:16.516 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:16.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:16Z" } 2024-04-13T22:01:16.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:16.517 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:16.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:16.518 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:21.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:21.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:21.519 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:21.575 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:21.576 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:21.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:21.609+0000 7fa2e401c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:21.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:21.610+0000 7fa2e401c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:21.627 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:21.628 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:21.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:21.629 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:21Z" 2024-04-13T22:01:21.629 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:21.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:21Z" } 2024-04-13T22:01:21.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:21.630 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:21.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:21.630 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:26.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:26.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:26.632 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:26.686 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:26.687 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:26.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:26.719+0000 7f88bdad6780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:26.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:26.720+0000 7f88bdad6780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:26.733 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:26Z" 2024-04-13T22:01:26.740 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:26.741 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:26Z" } 2024-04-13T22:01:26.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:26.742 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:26.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:26.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:31.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:31.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:31.743 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:31.798 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:31.799 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:31.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:31.833+0000 7f678b781780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:31.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:31.834+0000 7f678b781780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:31.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:31.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:31.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:31.850 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:31.851 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:31Z" 2024-04-13T22:01:31.852 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:31.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:31Z" } 2024-04-13T22:01:31.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:31.854 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:31.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:31.854 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:36.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:36.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:36.855 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:36.911 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:36.911 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:36.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:36.946+0000 7fbbf8071780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:36.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:36.947+0000 7fbbf8071780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:36.959 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:36.960 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045246.555013_414.1", 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:54:06.555013Z", 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:36.961 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:36.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:36.963 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:36.964 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:36.965 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:36Z" 2024-04-13T22:01:36.966 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:36.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045246.555013_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:54:06.555013Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:36Z" } 2024-04-13T22:01:36.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045246.555013_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:54:06.555013Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:36.967 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:36.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045246.555013_414.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:36.967 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045246.555013_414.1 2024-04-13T22:01:40.319 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:01:40.320+0000 7f5e93db2700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T22:01:41.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:41.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:41.969 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:42.022 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:42.022 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:42.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:42.054+0000 7f9d7d27c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:42.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:42.055+0000 7f9d7d27c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:42.067 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045579.305762_541.1", 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:59:39.305762Z", 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:42Z" 2024-04-13T22:01:42.073 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:42.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045579.305762_541.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:59:39.305762Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:42Z" } 2024-04-13T22:01:42.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045579.305762_541.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:59:39.305762Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:42.074 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:42.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045579.305762_541.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:42.075 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045579.305762_541.1 2024-04-13T22:01:47.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:47.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:47.076 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:47.126 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:47.126 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:47.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:47.155+0000 7f87fa25d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:47.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:47.156+0000 7f87fa25d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:47.168 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:47.169 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045580.098873_563.1", 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:59:40.098873Z", 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:47.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:47.171 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:47.172 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:47.173 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:47.174 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:47Z" 2024-04-13T22:01:47.175 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:47.175 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045580.098873_563.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:59:40.098873Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:47Z" } 2024-04-13T22:01:47.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045580.098873_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:59:40.098873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:47.176 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:47.176 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045580.098873_563.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:47.176 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045580.098873_563.1 2024-04-13T22:01:52.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:52.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:52.178 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:52.226 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:52.226 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:52.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:52.256+0000 7f2a04820780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:52.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:52.257+0000 7f2a04820780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:52.269 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:52.270 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045580.098873_563.1", 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:59:40.098873Z", 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:52.272 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:52.273 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:52.274 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:52.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:52.276 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:52Z" 2024-04-13T22:01:52.276 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:52.277 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045580.098873_563.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:59:40.098873Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:52Z" } 2024-04-13T22:01:52.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045580.098873_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:59:40.098873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:52.277 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:52.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045580.098873_563.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:52.277 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045580.098873_563.1 2024-04-13T22:01:57.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:57.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:01:57.279 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:01:57.338 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:01:57.338 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:01:57.374 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:57.374+0000 7fb13a406780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:57.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:01:57.375+0000 7fb13a406780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:01:57.388 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:01:57.389 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045580.864236_567.1", 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:59:40.864236Z", 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:57.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:57.391 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:57.392 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:01:57.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:01:57Z" 2024-04-13T22:01:57.394 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:01:57.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045580.864236_567.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:59:40.864236Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:01:57Z" } 2024-04-13T22:01:57.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045580.864236_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:59:40.864236Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:01:57.396 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:57.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045580.864236_567.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:01:57.396 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045580.864236_567.1 2024-04-13T22:02:02.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:02.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:02.398 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:02.449 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:02.449 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:02.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:02.487+0000 7fc47adce780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:02:02.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:02.488+0000 7fc47adce780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:02.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045580.098873_563.1", 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:02.502 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:59:40.098873Z", 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:02.503 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:02.504 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:02.505 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:02.506 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:02.507 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:02:02.507 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:02:02.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:02.507 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:02Z" 2024-04-13T22:02:02.507 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:02.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045580.098873_563.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:59:40.098873Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:02:02Z" } 2024-04-13T22:02:02.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045580.098873_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:59:40.098873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:02:02.508 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:02.508 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045580.098873_563.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:02.508 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045584.281589_602.1 target=1_1713045580.098873_563.1 2024-04-13T22:02:07.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:07.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:07.510 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:07.566 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:07.566 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:07.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:07.601+0000 7ff316706780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:02:07.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:07.602+0000 7ff316706780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:02:07.618 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:07.618 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:07.619 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045584.281589_602.1", 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:59:44.281589Z", 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:07.620 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:07.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:07.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:02:07.623 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:07Z" 2024-04-13T22:02:07.624 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:07.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045584.281589_602.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T21:59:44.281589Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:02:07Z" } 2024-04-13T22:02:07.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045584.281589_602.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:59:44.281589Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:02:07.625 DEBUG:rgw_multi.tests:log_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:07.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045584.281589_602.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:07.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:02:07.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vwnfns-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:02:07.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vwnfns-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:02:07.625 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vwnfns-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:02:07.676 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:07.676 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:07.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vwnfns-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:02:07.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vwnfns-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:02:07.742 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vwnfns-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:02:07.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:07.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:07.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:07.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:07.855 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:07.900 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:07.900 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:07.942 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:02:07.943 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:5", 2024-04-13T22:02:07.943 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.29", 2024-04-13T22:02:07.943 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000078", 2024-04-13T22:02:07.944 INFO:teuthology.orchestra.run.smithi132.stdout: "meta.history", 2024-04-13T22:02:07.944 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:4", 2024-04-13T22:02:07.944 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.23", 2024-04-13T22:02:07.944 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000070", 2024-04-13T22:02:07.944 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:0", 2024-04-13T22:02:07.944 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000104", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:8", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:8", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:9", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000026", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:9", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:7", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.8", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:1", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000028", 2024-04-13T22:02:07.945 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:7", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.30", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:9", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.2", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000040", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.6", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "meta.log.5039ce81-80d5-4d03-a26f-34b9088aaab4.1", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:9", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000015", 2024-04-13T22:02:07.946 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000069", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000095", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000003", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000047", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:1", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:0", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:1", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:0", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000035", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000037", 2024-04-13T22:02:07.947 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:3", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:7", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:0", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:4", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:6", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:8", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000024", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:8", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:1", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000000", 2024-04-13T22:02:07.948 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000031", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.3", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.1.retry", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:2", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:4", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:4", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "data_loggenerations_metadata", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:0", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000076", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:5", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000113", 2024-04-13T22:02:07.949 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000005", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000011", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:1", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:3", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000058", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:7", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000052", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.0", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.21", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:2", 2024-04-13T22:02:07.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:2", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:9", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:5", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000088", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000080", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000090", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:6", 2024-04-13T22:02:07.951 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000110", 2024-04-13T22:02:07.952 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.0", 2024-04-13T22:02:07.952 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:5", 2024-04-13T22:02:07.952 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000096", 2024-04-13T22:02:07.952 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:1", 2024-04-13T22:02:07.952 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000087", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:2", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000008", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000006", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000029", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:3", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:3", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000089", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000022", 2024-04-13T22:02:07.953 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:4", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog.trim", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000016", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:6", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:3", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000041", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:6", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000018", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000092", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:2", 2024-04-13T22:02:07.954 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:7", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:6", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.27", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.4", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:1", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000014", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.3", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000112", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000007", 2024-04-13T22:02:07.955 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000021", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.16", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000064", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000071", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000074", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:8", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:10", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:10", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:4", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:7", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000081", 2024-04-13T22:02:07.956 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000009", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:10", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000121", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.1", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:2", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:4", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.17", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:8", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000125", 2024-04-13T22:02:07.957 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000082", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:7", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:1", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:3", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000105", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000059", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:0", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000077", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000032", 2024-04-13T22:02:07.958 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:2", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.24", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:3", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000053", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:3", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000091", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.2.retry", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.1", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:8", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000065", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000083", 2024-04-13T22:02:07.959 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000010", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:6", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:3", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:5", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.7", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.3", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:8", 2024-04-13T22:02:07.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:4", 2024-04-13T22:02:07.961 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:10", 2024-04-13T22:02:07.961 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:9", 2024-04-13T22:02:07.961 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:1", 2024-04-13T22:02:07.961 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000045", 2024-04-13T22:02:07.962 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:10", 2024-04-13T22:02:07.962 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:3", 2024-04-13T22:02:07.962 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000002", 2024-04-13T22:02:07.962 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.12", 2024-04-13T22:02:07.962 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:8", 2024-04-13T22:02:07.963 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000116", 2024-04-13T22:02:07.963 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.25", 2024-04-13T22:02:07.963 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:7", 2024-04-13T22:02:07.963 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:9", 2024-04-13T22:02:07.963 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17", 2024-04-13T22:02:07.963 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18", 2024-04-13T22:02:07.964 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000034", 2024-04-13T22:02:07.964 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:0", 2024-04-13T22:02:07.964 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000101", 2024-04-13T22:02:07.964 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:1", 2024-04-13T22:02:07.964 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.13", 2024-04-13T22:02:07.965 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:6", 2024-04-13T22:02:07.965 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:3", 2024-04-13T22:02:07.965 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000079", 2024-04-13T22:02:07.965 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000049", 2024-04-13T22:02:07.965 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000117", 2024-04-13T22:02:07.966 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000044", 2024-04-13T22:02:07.966 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:1", 2024-04-13T22:02:07.966 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.14", 2024-04-13T22:02:07.966 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000066", 2024-04-13T22:02:07.966 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:7", 2024-04-13T22:02:07.967 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000068", 2024-04-13T22:02:07.967 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000085", 2024-04-13T22:02:07.967 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36", 2024-04-13T22:02:07.967 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:5", 2024-04-13T22:02:07.967 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:5", 2024-04-13T22:02:07.968 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000073", 2024-04-13T22:02:07.968 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000038", 2024-04-13T22:02:07.968 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:1", 2024-04-13T22:02:07.968 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.10", 2024-04-13T22:02:07.968 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000118", 2024-04-13T22:02:07.969 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:8", 2024-04-13T22:02:07.969 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:2", 2024-04-13T22:02:07.969 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:10", 2024-04-13T22:02:07.969 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000036", 2024-04-13T22:02:07.969 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:6", 2024-04-13T22:02:07.970 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000103", 2024-04-13T22:02:07.970 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:9", 2024-04-13T22:02:07.970 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:6", 2024-04-13T22:02:07.970 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:0", 2024-04-13T22:02:07.971 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:10", 2024-04-13T22:02:07.971 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:9", 2024-04-13T22:02:07.971 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000119", 2024-04-13T22:02:07.971 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:2", 2024-04-13T22:02:07.971 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:6", 2024-04-13T22:02:07.972 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000098", 2024-04-13T22:02:07.972 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:4", 2024-04-13T22:02:07.972 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:2", 2024-04-13T22:02:07.972 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:4", 2024-04-13T22:02:07.972 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000027", 2024-04-13T22:02:07.973 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:0", 2024-04-13T22:02:07.973 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19", 2024-04-13T22:02:07.973 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000019", 2024-04-13T22:02:07.973 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000039", 2024-04-13T22:02:07.973 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:3", 2024-04-13T22:02:07.974 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000100", 2024-04-13T22:02:07.974 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:10", 2024-04-13T22:02:07.974 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:1", 2024-04-13T22:02:07.974 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:8", 2024-04-13T22:02:07.975 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:0", 2024-04-13T22:02:07.975 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000093", 2024-04-13T22:02:07.975 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:10", 2024-04-13T22:02:07.975 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000004", 2024-04-13T22:02:07.975 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000063", 2024-04-13T22:02:07.975 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000122", 2024-04-13T22:02:07.976 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:7", 2024-04-13T22:02:07.976 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000057", 2024-04-13T22:02:07.976 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000054", 2024-04-13T22:02:07.976 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000114", 2024-04-13T22:02:07.976 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.2", 2024-04-13T22:02:07.977 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.0", 2024-04-13T22:02:07.977 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:7", 2024-04-13T22:02:07.977 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000012", 2024-04-13T22:02:07.977 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:5", 2024-04-13T22:02:07.977 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000084", 2024-04-13T22:02:07.978 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000043", 2024-04-13T22:02:07.978 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:3", 2024-04-13T22:02:07.978 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.11", 2024-04-13T22:02:07.978 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13", 2024-04-13T22:02:07.979 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:4", 2024-04-13T22:02:07.979 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.0.u+sFiSCDd8x.0", 2024-04-13T22:02:07.979 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:9", 2024-04-13T22:02:07.979 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:0", 2024-04-13T22:02:07.979 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:6", 2024-04-13T22:02:07.980 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.3.retry", 2024-04-13T22:02:07.980 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:2", 2024-04-13T22:02:07.980 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:6", 2024-04-13T22:02:07.980 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000111", 2024-04-13T22:02:07.980 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000048", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:10", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000020", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:5", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:9", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:6", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:10", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000099", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000056", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.19", 2024-04-13T22:02:07.981 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.28", 2024-04-13T22:02:07.982 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:9", 2024-04-13T22:02:07.982 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000072", 2024-04-13T22:02:07.982 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000062", 2024-04-13T22:02:07.982 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000115", 2024-04-13T22:02:07.982 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.1", 2024-04-13T22:02:07.982 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000033", 2024-04-13T22:02:07.983 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:6", 2024-04-13T22:02:07.983 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000025", 2024-04-13T22:02:07.983 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:4", 2024-04-13T22:02:07.983 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000123", 2024-04-13T22:02:07.983 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.31", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.18", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000108", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:0", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:7", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000094", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:8", 2024-04-13T22:02:07.984 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:5", 2024-04-13T22:02:07.985 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000060", 2024-04-13T22:02:07.985 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:7", 2024-04-13T22:02:07.985 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000109", 2024-04-13T22:02:07.985 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.15", 2024-04-13T22:02:07.985 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:10", 2024-04-13T22:02:07.985 INFO:teuthology.orchestra.run.smithi132.stdout: "meta.log.5039ce81-80d5-4d03-a26f-34b9088aaab4.0", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:4", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:0", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:9", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:9", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000013", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:4", 2024-04-13T22:02:07.986 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:2", 2024-04-13T22:02:07.987 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:10", 2024-04-13T22:02:07.987 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:5", 2024-04-13T22:02:07.987 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9", 2024-04-13T22:02:07.987 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000051", 2024-04-13T22:02:07.987 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000106", 2024-04-13T22:02:07.987 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:7", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:3", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:10", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:6", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000017", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:4", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:6", 2024-04-13T22:02:07.988 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000023", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:8", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:7", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000107", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:8", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:9", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000046", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:0", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.26", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:2", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:0", 2024-04-13T22:02:07.989 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.22", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:4", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:1", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.2", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:5", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000067", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:0", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:9", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:2", 2024-04-13T22:02:07.990 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.9", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:8", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000086", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.5", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:5", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:3", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:2", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:9", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:8", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16", 2024-04-13T22:02:07.991 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:3", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:7", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:10", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.20", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:5", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.5da80fbc-5edc-4ba7-826d-471359a9fc92.0.retry", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:5", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:6", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:1", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000050", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:0", 2024-04-13T22:02:07.992 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000102", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000124", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.2.ka6sVXVoU6V.0", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000001", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:2", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000126", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.1.TRPBC7OSLo+.0", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:5", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000055", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000061", 2024-04-13T22:02:07.993 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000097", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:3", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:2", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:10", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:7", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:4", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000042", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:1", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:1", 2024-04-13T22:02:07.994 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:8", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:5", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000075", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000120", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000030", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.3.7TgMTbhNWvP.0", 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:10" 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:02:07.995 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T22:02:07.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:07.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:07.997 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:08.042 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:08.043 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:08.087 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:02:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.29", 2024-04-13T22:02:08.087 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000078", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "meta.history", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:9", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.23", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000070", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000104", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000026", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:10", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.8", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000028", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:9", 2024-04-13T22:02:08.088 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:3", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.30", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:4", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:8", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000040", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.6", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "meta.log.5039ce81-80d5-4d03-a26f-34b9088aaab4.1", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000015", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:10", 2024-04-13T22:02:08.089 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000069", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000095", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:0", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000003", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000047", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:3", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:0", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:9", 2024-04-13T22:02:08.090 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000035", 2024-04-13T22:02:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:4", 2024-04-13T22:02:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000037", 2024-04-13T22:02:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:2", 2024-04-13T22:02:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:5", 2024-04-13T22:02:08.091 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:7", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:1", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000024", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000000", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000031", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.3", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.1.oBreT3w7a98.0", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:0", 2024-04-13T22:02:08.092 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:0", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:3", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:7", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "data_loggenerations_metadata", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000076", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000113", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000005", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000011", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000058", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:10", 2024-04-13T22:02:08.093 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000052", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.21", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:9", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:10", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:7", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:1", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:10", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000088", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:5", 2024-04-13T22:02:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000080", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000090", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000110", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:6", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.0", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000096", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:6", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000087", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000008", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:4", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:1", 2024-04-13T22:02:08.095 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:7", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:1", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000006", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:8", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:2", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000029", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:6", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:10", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000089", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000022", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog.trim", 2024-04-13T22:02:08.096 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:7", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000016", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:2", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000041", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:2", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:3", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:9", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.2.tuYM6oX/n19.0", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000018", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000092", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:5", 2024-04-13T22:02:08.097 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:1", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.27", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.4", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.3", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:8", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:7", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:10", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:0", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000014", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:0", 2024-04-13T22:02:08.098 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000112", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000007", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000021", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:0", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.16", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:3", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000064", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000071", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.0.wz1HRosBSGH.0", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000074", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000081", 2024-04-13T22:02:08.099 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000009", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:4", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:1", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000121", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:2", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.1", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:5", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.17", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000125", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000082", 2024-04-13T22:02:08.100 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:3", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:8", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:3", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000105", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000059", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:8", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:6", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:2", 2024-04-13T22:02:08.101 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:10", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000077", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000032", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:0", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:8", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:10", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:7", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.24", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:0", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000053", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000091", 2024-04-13T22:02:08.102 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:0", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000065", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000083", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000010", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:1", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.7", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.3", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:8", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:3", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:7", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:5", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "mdlog.sync-status.shard.2", 2024-04-13T22:02:08.103 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:5", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:3", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:7", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:5", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000045", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:10", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:0", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:9", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000002", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.12", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:2", 2024-04-13T22:02:08.104 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000116", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:3", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.25", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.0", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:6", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:2", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000034", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:10", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000101", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.13", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "mdlog.sync-status.shard.1", 2024-04-13T22:02:08.105 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:3", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000079", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000049", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000117", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000044", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:7", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:4", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:0", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:8", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.14", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000066", 2024-04-13T22:02:08.106 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000068", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:5", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000085", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:7", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000073", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:1", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000038", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:3", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:4", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:6", 2024-04-13T22:02:08.107 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:6", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.0.retry", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.10", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000118", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:3", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:5", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000036", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000103", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "mdlog.sync-status.shard.3", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.2.retry", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:1", 2024-04-13T22:02:08.108 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:2", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000119", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000098", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:4", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "mdlog.sync-status.shard.0", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.3.3G1Dr5SL3UG.0", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:10", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:6", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:2", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:3", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7", 2024-04-13T22:02:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:8", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000027", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:10", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:6", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "mdlog.sync-status", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000019", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000039", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:4", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:7", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:6", 2024-04-13T22:02:08.110 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:8", 2024-04-13T22:02:08.111 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36", 2024-04-13T22:02:08.111 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000100", 2024-04-13T22:02:08.111 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:1", 2024-04-13T22:02:08.111 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:10", 2024-04-13T22:02:08.111 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:8", 2024-04-13T22:02:08.111 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:6", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000093", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:2", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:1", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:4", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000004", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000063", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000122", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:5", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:7", 2024-04-13T22:02:08.112 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000057", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:8", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:2", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000054", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000114", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.2", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.0", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:9", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000012", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:4", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:8", 2024-04-13T22:02:08.113 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:6", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:4", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000084", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000043", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.11", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:7", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:2", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:4", 2024-04-13T22:02:08.114 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:2", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000111", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000048", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000020", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:2", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:4", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:7", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:2", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:5", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000099", 2024-04-13T22:02:08.115 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000056", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.19", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:4", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.28", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000072", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000062", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000115", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.1", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000033", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:10", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:1", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000025", 2024-04-13T22:02:08.116 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.2", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:2", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000123", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.31", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:10", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.18", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000108", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35:1", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000094", 2024-04-13T22:02:08.117 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000060", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:8", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000109", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.15", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "meta.log.5039ce81-80d5-4d03-a26f-34b9088aaab4.0", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:9", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:6", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:0", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000013", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:1", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.1", 2024-04-13T22:02:08.118 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:1", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-9:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.9:5", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:0", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000051", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000106", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000017", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:5", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:9", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:0", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.3.retry", 2024-04-13T22:02:08.119 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:6", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000023", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000107", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000046", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:9", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.26", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-35:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.35", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:7", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-10:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.10:4", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.22", 2024-04-13T22:02:08.120 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:10", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:5", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:4", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "data_log.2", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000067", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:9", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.9", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000086", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.5", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-17:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.17:0", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:3", 2024-04-13T22:02:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-18:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.18:9", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "sync.error-log.20", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "datalog.sync-status.shard.f6f208d6-f791-479b-bc3d-112512eec2d4.1.retry", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:4", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000050", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000102", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:8", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:1", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:6", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:5", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:3", 2024-04-13T22:02:08.122 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000124", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:9", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-8:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.8:8", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000001", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:7", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-15:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.15:5", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000126", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000055", 2024-04-13T22:02:08.123 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000061", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-7:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.7:9", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-14:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.14:9", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000097", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-13:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.13:6", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-11:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.11:6", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-20:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.20:1", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-16:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.16:0", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:5", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000042", 2024-04-13T22:02:08.124 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-37:f6f208d6-f791-479b-bc3d-112512eec2d4.6236.1:9", 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-19:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.19:9", 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-12:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.12:3", 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000075", 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000120", 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj_delete_at_hint.0000000030", 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-36:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.36:8" 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:02:08.125 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-13T22:02:08.127 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-13T22:02:08.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:08.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:08.127 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:08.174 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:08.175 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005582", 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:07.839368Z" 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003467", 2024-04-13T22:02:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:07.840426Z" 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002339", 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:07.837357Z" 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006705", 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:07.838390Z" 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:08.221 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:02:08.222 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000003467', 2: '00000000000000000000:00000000000000002339', 3: '00000000000000000000:00000000000000006705'} 2024-04-13T22:02:08.222 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:02:08.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:08.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:08.223 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:08.274 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:08.274 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:08.319 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.320 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004740", 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.627941Z" 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:08.321 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.322 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:49:49.438404Z" 2024-04-13T22:02:08.324 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:08Z" 2024-04-13T22:02:08.325 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:08.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.627941Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:49:49.438404Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:08Z" } 2024-04-13T22:02:08.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.627941Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:49:49.438404Z'}}] 2024-04-13T22:02:08.327 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000003467', 2: '00000000000000000000:00000000000000002339', 3: '00000000000000000000:00000000000000006705'} 2024-04-13T22:02:08.327 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004740', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000005862'} 2024-04-13T22:02:08.327 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000005582 target=00000000000000000000:00000000000000004740, shard=1 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002622, shard=2 master=00000000000000000000:00000000000000002339 target=00000000000000000000:00000000000000001777, shard=3 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862 2024-04-13T22:02:13.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:13.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:13.329 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:13.388 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:13.388 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:13.438 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:13.438 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:13.439 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:13.439 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:13.439 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:13.439 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:02:13.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:13.439 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:13.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:13.440 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:13.440 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:13.440 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:13.440 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004740", 2024-04-13T22:02:13.441 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:13.441 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:13.441 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:13.441 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.627941Z" 2024-04-13T22:02:13.441 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:13.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:13.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:13.442 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:13.442 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:13.442 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:13.442 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T22:02:13.443 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:13.443 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:13.443 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:13.443 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T22:02:13.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:13.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:13.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:13.444 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:13.444 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:13.444 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:13.444 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T22:02:13.445 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:13.445 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:13.445 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:13.445 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T22:02:13.445 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:13.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:13.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:13.446 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:13.446 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:13.446 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:13.446 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-13T22:02:13.447 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:13.447 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:13.447 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:13.447 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:49:49.438404Z" 2024-04-13T22:02:13.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:13.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:13.448 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:13.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:13.448 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:13.448 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:13.448 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:13.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:13.449 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:13Z" 2024-04-13T22:02:13.449 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:13.450 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.627941Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:49:49.438404Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:13Z" } 2024-04-13T22:02:13.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.627941Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:49:49.438404Z'}}] 2024-04-13T22:02:13.451 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000003467', 2: '00000000000000000000:00000000000000002339', 3: '00000000000000000000:00000000000000006705'} 2024-04-13T22:02:13.451 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004740', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000005862'} 2024-04-13T22:02:13.451 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000005582 target=00000000000000000000:00000000000000004740, shard=1 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002622, shard=2 master=00000000000000000000:00000000000000002339 target=00000000000000000000:00000000000000001777, shard=3 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862 2024-04-13T22:02:18.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:18.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:18.453 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:18.510 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:18.510 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:02:18.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004740", 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.627941Z" 2024-04-13T22:02:18.560 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.522018Z" 2024-04-13T22:02:18.561 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001777", 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:05.523190Z" 2024-04-13T22:02:18.562 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:49:49.438404Z" 2024-04-13T22:02:18.563 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:18Z" 2024-04-13T22:02:18.564 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:18.565 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.627941Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.522018Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:05.523190Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:49:49.438404Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:18Z" } 2024-04-13T22:02:18.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.627941Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.522018Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:05.523190Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:49:49.438404Z'}}] 2024-04-13T22:02:18.566 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000003467', 2: '00000000000000000000:00000000000000002339', 3: '00000000000000000000:00000000000000006705'} 2024-04-13T22:02:18.566 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004740', 1: '00000000000000000000:00000000000000002622', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000005862'} 2024-04-13T22:02:18.566 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000005582 target=00000000000000000000:00000000000000004740, shard=1 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002622, shard=2 master=00000000000000000000:00000000000000002339 target=00000000000000000000:00000000000000001777, shard=3 master=00000000000000000000:00000000000000006705 target=00000000000000000000:00000000000000005862 2024-04-13T22:02:23.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:23.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:23.567 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:23.622 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:23.622 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:23.667 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:23.667 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:23.667 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:23.667 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005582", 2024-04-13T22:02:23.668 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:07.839368Z" 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003467", 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.669 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:07.840426Z" 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002339", 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.670 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:07.837357Z" 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006705", 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.671 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:07.838390Z" 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:23Z" 2024-04-13T22:02:23.672 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:23.673 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:07.839368Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003467", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:07.840426Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:07.837357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006705", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:07.838390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:23Z" } 2024-04-13T22:02:23.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:07.839368Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003467', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:07.840426Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:07.837357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006705', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:07.838390Z'}}] 2024-04-13T22:02:23.674 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000003467', 2: '00000000000000000000:00000000000000002339', 3: '00000000000000000000:00000000000000006705'} 2024-04-13T22:02:23.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000003467', 2: '00000000000000000000:00000000000000002339', 3: '00000000000000000000:00000000000000006705'} 2024-04-13T22:02:23.674 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:02:23.674 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-13T22:02:23.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:23.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:23.675 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:23.722 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:23.722 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:23.771 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:02:23.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.771 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005021", 2024-04-13T22:02:23.771 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:23.481807Z" 2024-04-13T22:02:23.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003610", 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:23.482861Z" 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002481", 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:23.479448Z" 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.772 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007411", 2024-04-13T22:02:23.773 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:23.480573Z" 2024-04-13T22:02:23.773 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.773 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:02:23.774 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:23.774 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:02:23.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:23.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:23.775 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:23.824 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:23.824 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:23.865 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:23.865 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:23.865 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:23.865 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:23.865 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.866 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.928950Z" 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.867 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.502412Z" 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.868 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.503831Z" 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:23.869 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:48:08.413843Z" 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:23.870 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:23.871 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:23.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:23.871 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:23Z" 2024-04-13T22:02:23.871 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:23.872 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.928950Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.502412Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.503831Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:48:08.413843Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:23Z" } 2024-04-13T22:02:23.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.928950Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.502412Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.503831Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:48:08.413843Z'}}] 2024-04-13T22:02:23.872 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:23.872 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004178', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000001918', 3: '00000000000000000000:00000000000000006567'} 2024-04-13T22:02:23.872 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000005021 target=00000000000000000000:00000000000000004178, shard=1 master=00000000000000000000:00000000000000003610 target=00000000000000000000:00000000000000002764, shard=2 master=00000000000000000000:00000000000000002481 target=00000000000000000000:00000000000000001918, shard=3 master=00000000000000000000:00000000000000007411 target=00000000000000000000:00000000000000006567 2024-04-13T22:02:28.089 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T22:02:28.088+0000 7fbf88f16700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T22:02:28.089 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T22:02:28.089+0000 7fbf7a6f9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T22:02:28.089 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T22:02:28.089+0000 7fbf796f7700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T22:02:28.129 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T22:02:28.129+0000 7fbf7e701700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T22:02:28.129 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T22:02:28.130+0000 7fbf85f10700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T22:02:28.130 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T22:02:28.130+0000 7fbf83f0c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T22:02:28.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:28.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:28.874 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:28.931 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:28.932 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:28.978 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:28.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.928950Z" 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:28.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.502412Z" 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:28.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002481", 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.479448Z" 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:28.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:48:08.413843Z" 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:28.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:28.984 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:28.984 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:28.984 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:28.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:28.985 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:28Z" 2024-04-13T22:02:28.985 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:28.986 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.928950Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.502412Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.479448Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:48:08.413843Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:28Z" } 2024-04-13T22:02:28.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.928950Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.502412Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.479448Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:48:08.413843Z'}}] 2024-04-13T22:02:28.986 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:28.986 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004178', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000006567'} 2024-04-13T22:02:28.986 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000005021 target=00000000000000000000:00000000000000004178, shard=1 master=00000000000000000000:00000000000000003610 target=00000000000000000000:00000000000000002764, shard=3 master=00000000000000000000:00000000000000007411 target=00000000000000000000:00000000000000006567 2024-04-13T22:02:33.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:33.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:33.988 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:34.045 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:34.046 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:34.091 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:34.091 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:34.091 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:34.091 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:34.091 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:34.092 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:02:34.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:34.092 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:34.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:34.092 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.928950Z" 2024-04-13T22:02:34.093 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:34.094 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.502412Z" 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002481", 2024-04-13T22:02:34.095 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.479448Z" 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:34.096 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:48:08.413843Z" 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:34.097 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:34Z" 2024-04-13T22:02:34.098 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:34.099 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.928950Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.502412Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.479448Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:48:08.413843Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:34Z" } 2024-04-13T22:02:34.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.928950Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.502412Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.479448Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:48:08.413843Z'}}] 2024-04-13T22:02:34.100 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:34.100 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004178', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000006567'} 2024-04-13T22:02:34.100 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000005021 target=00000000000000000000:00000000000000004178, shard=1 master=00000000000000000000:00000000000000003610 target=00000000000000000000:00000000000000002764, shard=3 master=00000000000000000000:00000000000000007411 target=00000000000000000000:00000000000000006567 2024-04-13T22:02:39.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:39.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:39.102 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:39.159 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:39.159 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:02:39.206 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:39.207 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:57:30.928950Z" 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:39.208 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:30:27.502412Z" 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002481", 2024-04-13T22:02:39.209 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.479448Z" 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:39.210 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:48:08.413843Z" 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:39.211 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:39.212 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:39.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:39.212 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:39Z" 2024-04-13T22:02:39.212 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:39.213 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:57:30.928950Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:30:27.502412Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.479448Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:48:08.413843Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:39Z" } 2024-04-13T22:02:39.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:57:30.928950Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:30:27.502412Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.479448Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:48:08.413843Z'}}] 2024-04-13T22:02:39.213 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:39.213 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004178', 1: '00000000000000000000:00000000000000002764', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000006567'} 2024-04-13T22:02:39.214 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000005021 target=00000000000000000000:00000000000000004178, shard=1 master=00000000000000000000:00000000000000003610 target=00000000000000000000:00000000000000002764, shard=3 master=00000000000000000000:00000000000000007411 target=00000000000000000000:00000000000000006567 2024-04-13T22:02:44.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:44.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:44.215 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:44.273 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:44.273 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:44.320 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:44.320 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:44.321 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005021", 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.481807Z" 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.322 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003610", 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.482861Z" 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:44.323 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000002481", 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.479448Z" 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:44.324 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007411", 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.480573Z" 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:44.326 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:02:44.326 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:02:44.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.326 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:44Z" 2024-04-13T22:02:44.326 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:44.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005021", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.481807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003610", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.482861Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.479448Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007411", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.480573Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:02:44Z" } 2024-04-13T22:02:44.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005021', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.481807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003610', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.482861Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.479448Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007411', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.480573Z'}}] 2024-04-13T22:02:44.327 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:44.327 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000003610', 2: '00000000000000000000:00000000000000002481', 3: '00000000000000000000:00000000000000007411'} 2024-04-13T22:02:44.327 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:02:44.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2024-04-13T22:02:44.330 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-48 2024-04-13T22:02:44.330 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:02:44.330 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:02:44.330 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.330 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:02:44.330 DEBUG:boto:Data: 2024-04-13T22:02:44.331 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.331 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.331 DEBUG:boto:Params: {} 2024-04-13T22:02:44.331 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:02:44.331 DEBUG:boto:Token: None 2024-04-13T22:02:44.332 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/ 2024-04-13T22:02:44.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oZv1bKcCYcGQjI3FKNez9PVCycY= 2024-04-13T22:02:44.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oZv1bKcCYcGQjI3FKNez9PVCycY=', 'Content-Length': '0'} 2024-04-13T22:02:44.350 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000243ed1c21f7f908d-00661b0104-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.350 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-49 2024-04-13T22:02:44.350 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:02:44.350 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:02:44.351 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.351 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:02:44.351 DEBUG:boto:Data: 2024-04-13T22:02:44.351 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.351 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.351 DEBUG:boto:Params: {} 2024-04-13T22:02:44.351 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:02:44.351 DEBUG:boto:Token: None 2024-04-13T22:02:44.352 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/ 2024-04-13T22:02:44.352 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V5nEprbL8YTz+deqRZJuUuSUU5c= 2024-04-13T22:02:44.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V5nEprbL8YTz+deqRZJuUuSUU5c=', 'Content-Length': '0'} 2024-04-13T22:02:44.379 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001cd2c70130c3aa26-00661b0104-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.379 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:02:44.379 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:02:44.379 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.379 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:02:44.379 DEBUG:boto:Data: 2024-04-13T22:02:44.379 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.380 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.380 DEBUG:boto:Params: {} 2024-04-13T22:02:44.380 DEBUG:boto:Token: None 2024-04-13T22:02:44.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/ 2024-04-13T22:02:44.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk= 2024-04-13T22:02:44.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk=', 'Content-Length': '0'} 2024-04-13T22:02:44.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e386ff1fd9b8f60-00661b0104-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.383 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:02:44.383 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:02:44.383 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.383 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:02:44.383 DEBUG:boto:Data: 2024-04-13T22:02:44.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.383 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.383 DEBUG:boto:Params: {} 2024-04-13T22:02:44.383 DEBUG:boto:Token: None 2024-04-13T22:02:44.384 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/obj1 2024-04-13T22:02:44.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rFGnp1HSvAnI0KzO6NBl7MabcQw= 2024-04-13T22:02:44.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rFGnp1HSvAnI0KzO6NBl7MabcQw='} 2024-04-13T22:02:44.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007df23ec9de294dd2-00661b0104-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.390 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:02:44.390 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:02:44.390 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.390 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:02:44.390 DEBUG:boto:Data: 2024-04-13T22:02:44.390 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.390 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.390 DEBUG:boto:Params: {} 2024-04-13T22:02:44.391 DEBUG:boto:Token: None 2024-04-13T22:02:44.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/ 2024-04-13T22:02:44.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk= 2024-04-13T22:02:44.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk=', 'Content-Length': '0'} 2024-04-13T22:02:44.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044a44fc2ec50e2b9-00661b0104-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.393 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:02:44.393 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:02:44.393 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.393 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:02:44.393 DEBUG:boto:Data: 2024-04-13T22:02:44.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.394 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.394 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.394 DEBUG:boto:Params: {} 2024-04-13T22:02:44.394 DEBUG:boto:Token: None 2024-04-13T22:02:44.394 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/obj2 2024-04-13T22:02:44.394 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nZH+OVf/IwKEZrmb551LN76g/a8= 2024-04-13T22:02:44.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nZH+OVf/IwKEZrmb551LN76g/a8='} 2024-04-13T22:02:44.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087a429c4b6d5f6f9-00661b0104-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.399 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:02:44.399 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:02:44.399 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.400 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:02:44.400 DEBUG:boto:Data: 2024-04-13T22:02:44.400 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.400 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.400 DEBUG:boto:Params: {} 2024-04-13T22:02:44.400 DEBUG:boto:Token: None 2024-04-13T22:02:44.400 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/ 2024-04-13T22:02:44.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk= 2024-04-13T22:02:44.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk=', 'Content-Length': '0'} 2024-04-13T22:02:44.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb0a9babfebcd0c3-00661b0104-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.403 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:02:44.403 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:02:44.403 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.403 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:02:44.403 DEBUG:boto:Data: 2024-04-13T22:02:44.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.404 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.404 DEBUG:boto:Params: {} 2024-04-13T22:02:44.404 DEBUG:boto:Token: None 2024-04-13T22:02:44.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/obj3 2024-04-13T22:02:44.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A2ourIeGbm8yKvUaZMfEUv2uxnE= 2024-04-13T22:02:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A2ourIeGbm8yKvUaZMfEUv2uxnE='} 2024-04-13T22:02:44.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047f45af2d72db1ff-00661b0104-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.447 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:02:44.447 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:02:44.447 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.447 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:02:44.447 DEBUG:boto:Data: 2024-04-13T22:02:44.447 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.447 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.447 DEBUG:boto:Params: {} 2024-04-13T22:02:44.448 DEBUG:boto:Token: None 2024-04-13T22:02:44.448 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/ 2024-04-13T22:02:44.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk= 2024-04-13T22:02:44.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JEESFc4n6rXIZu9ijCswbuiI1Rk=', 'Content-Length': '0'} 2024-04-13T22:02:44.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a1654f777f0990b-00661b0104-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.450 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:02:44.450 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:02:44.450 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.451 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:02:44.451 DEBUG:boto:Data: 2024-04-13T22:02:44.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:02:44.451 DEBUG:boto:Port: 8000 2024-04-13T22:02:44.451 DEBUG:boto:Params: {} 2024-04-13T22:02:44.451 DEBUG:boto:Token: None 2024-04-13T22:02:44.451 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-48/obj4 2024-04-13T22:02:44.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GzMzCsytIaCjjbqOwbfWyQt4EPw= 2024-04-13T22:02:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GzMzCsytIaCjjbqOwbfWyQt4EPw='} 2024-04-13T22:02:44.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b728aaf8c144308-00661b0104-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.456 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:02:44.456 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:02:44.456 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.457 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:02:44.457 DEBUG:boto:Data: 2024-04-13T22:02:44.457 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.457 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.457 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.457 DEBUG:boto:Params: {} 2024-04-13T22:02:44.457 DEBUG:boto:Token: None 2024-04-13T22:02:44.457 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/ 2024-04-13T22:02:44.457 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ= 2024-04-13T22:02:44.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ=', 'Content-Length': '0'} 2024-04-13T22:02:44.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083f740b32b969fa5-00661b0104-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.460 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:02:44.460 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:02:44.460 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.460 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:02:44.460 DEBUG:boto:Data: 2024-04-13T22:02:44.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.460 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.460 DEBUG:boto:Params: {} 2024-04-13T22:02:44.461 DEBUG:boto:Token: None 2024-04-13T22:02:44.461 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/obj1 2024-04-13T22:02:44.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UdU7Jks2Ix/AOOGCbkggzScnFCw= 2024-04-13T22:02:44.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UdU7Jks2Ix/AOOGCbkggzScnFCw='} 2024-04-13T22:02:44.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fdf699d29a0346a1-00661b0104-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.467 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:02:44.467 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:02:44.467 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.467 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:02:44.467 DEBUG:boto:Data: 2024-04-13T22:02:44.467 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.467 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.467 DEBUG:boto:Params: {} 2024-04-13T22:02:44.467 DEBUG:boto:Token: None 2024-04-13T22:02:44.468 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/ 2024-04-13T22:02:44.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ= 2024-04-13T22:02:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ=', 'Content-Length': '0'} 2024-04-13T22:02:44.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b07f6215b4061cf1-00661b0104-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.470 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:02:44.470 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:02:44.470 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.470 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:02:44.470 DEBUG:boto:Data: 2024-04-13T22:02:44.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.471 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.471 DEBUG:boto:Params: {} 2024-04-13T22:02:44.471 DEBUG:boto:Token: None 2024-04-13T22:02:44.471 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/obj2 2024-04-13T22:02:44.471 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RoiyDBdiDXfbxXn9WFiUGQ7t36w= 2024-04-13T22:02:44.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RoiyDBdiDXfbxXn9WFiUGQ7t36w='} 2024-04-13T22:02:44.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a6cad161ffea3e3-00661b0104-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.476 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:02:44.476 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:02:44.476 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.476 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:02:44.476 DEBUG:boto:Data: 2024-04-13T22:02:44.476 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.477 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.477 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.477 DEBUG:boto:Params: {} 2024-04-13T22:02:44.477 DEBUG:boto:Token: None 2024-04-13T22:02:44.477 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/ 2024-04-13T22:02:44.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ= 2024-04-13T22:02:44.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ=', 'Content-Length': '0'} 2024-04-13T22:02:44.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0fbd09f848eaf55-00661b0104-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.479 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:02:44.479 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:02:44.479 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.480 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:02:44.480 DEBUG:boto:Data: 2024-04-13T22:02:44.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.480 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.480 DEBUG:boto:Params: {} 2024-04-13T22:02:44.480 DEBUG:boto:Token: None 2024-04-13T22:02:44.480 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/obj3 2024-04-13T22:02:44.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rGvbZ0mGd3bEjepqGBWSyKxUkX8= 2024-04-13T22:02:44.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rGvbZ0mGd3bEjepqGBWSyKxUkX8='} 2024-04-13T22:02:44.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001976acbd96b3430b-00661b0104-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.523 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:02:44.523 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:02:44.523 DEBUG:boto:Method: HEAD 2024-04-13T22:02:44.523 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:02:44.523 DEBUG:boto:Data: 2024-04-13T22:02:44.523 DEBUG:boto:Headers: {} 2024-04-13T22:02:44.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.524 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.524 DEBUG:boto:Params: {} 2024-04-13T22:02:44.524 DEBUG:boto:Token: None 2024-04-13T22:02:44.524 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/ 2024-04-13T22:02:44.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ= 2024-04-13T22:02:44.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HFMkrJ0IjnewnSeKDiKEvMLdMjQ=', 'Content-Length': '0'} 2024-04-13T22:02:44.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007201c4d3dc3d1e77-00661b0104-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.527 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:02:44.527 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:02:44.527 DEBUG:boto:Method: PUT 2024-04-13T22:02:44.527 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:02:44.527 DEBUG:boto:Data: 2024-04-13T22:02:44.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:02:44.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:02:44.527 DEBUG:boto:Port: 8001 2024-04-13T22:02:44.527 DEBUG:boto:Params: {} 2024-04-13T22:02:44.527 DEBUG:boto:Token: None 2024-04-13T22:02:44.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:02:44 GMT /vwnfns-49/obj4 2024-04-13T22:02:44.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nM6F+dYR8ikaIwofwjxchC8hK3U= 2024-04-13T22:02:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nM6F+dYR8ikaIwofwjxchC8hK3U='} 2024-04-13T22:02:44.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e7144d722f83b9c-00661b0104-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:02:44.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:44.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:44.533 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:44.580 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:44.580 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:44.609 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045764.369377_620.1", 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:44.369377Z" 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.623 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:02:44.624 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045764.369377_620.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:44.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:02:44.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:44.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:02:44.625 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:02:44.672 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:44.672 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:44.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:44.704+0000 7f639c3a8780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:02:44.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:44.705+0000 7f639c3a8780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:02:44.716 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:02:44.716 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:02:44.716 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:02:44.716 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:02:44.716 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045764.369377_620.1", 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:44.369377Z", 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:44.718 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:02:44.719 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:02:44.720 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.721 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:02:44Z" 2024-04-13T22:02:44.722 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:02:44.723 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045764.369377_620.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:02:44.369377Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:02:44Z" } 2024-04-13T22:02:44.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045764.369377_620.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:02:44.369377Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:02:44.724 DEBUG:rgw_multi.tests:log_status={0: '1_1713045764.369377_620.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:44.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045764.369377_620.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:02:44.724 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:02:44.724 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-48 2024-04-13T22:02:44.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:02:44.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:02:44.725 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:02:44.780 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:02:44.780 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:02:44.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:44.816+0000 7fd7e6647780 1 waiting to reach incremental sync.. 2024-04-13T22:02:49.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:49.816+0000 7fd7e6647780 1 waiting to reach incremental sync.. 2024-04-13T22:02:54.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:54.817+0000 7fd7e6647780 1 waiting to reach incremental sync.. 2024-04-13T22:02:59.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:02:59.818+0000 7fd7e6647780 1 waiting to reach incremental sync.. 2024-04-13T22:03:04.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:04.818+0000 7fd7e6647780 1 bucket sync caught up with source: 2024-04-13T22:03:04.819 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.33.6, , 00000000001.19.6, , 00000000001.67.6, , , 00000000001.22.6, , , ] 2024-04-13T22:03:04.819 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.33.6, , 00000000001.19.6, , 00000000001.67.6, , , 00000000001.22.6, , , ] 2024-04-13T22:03:04.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:04.818+0000 7fd7e6647780 0 bucket checkpoint complete 2024-04-13T22:03:04.836 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-48 2024-04-13T22:03:04.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:04.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:04.837 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:03:04.897 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:04.897 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:04.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:04.934+0000 7f12ff167780 1 waiting to reach incremental sync.. 2024-04-13T22:03:09.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:09.936+0000 7f12ff167780 1 bucket sync caught up with source: 2024-04-13T22:03:09.937 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:09.937 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:09.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:09.936+0000 7f12ff167780 0 bucket checkpoint complete 2024-04-13T22:03:09.954 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-48 zones={a2, a1} 2024-04-13T22:03:09.954 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:09.954 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:09.954 DEBUG:boto:Method: HEAD 2024-04-13T22:03:09.954 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:03:09.954 DEBUG:boto:Data: 2024-04-13T22:03:09.954 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:09.955 DEBUG:boto:Port: 8001 2024-04-13T22:03:09.955 DEBUG:boto:Params: {} 2024-04-13T22:03:09.955 DEBUG:boto:Token: None 2024-04-13T22:03:09.955 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/ 2024-04-13T22:03:09.955 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pJLjY3f38Y/EKYRvkgnwZuPfxl0= 2024-04-13T22:03:09.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pJLjY3f38Y/EKYRvkgnwZuPfxl0=', 'Content-Length': '0'} 2024-04-13T22:03:09.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e3e11f20120669f-00661b011d-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:09.959 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:09.959 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:09.959 DEBUG:boto:Method: HEAD 2024-04-13T22:03:09.959 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:03:09.959 DEBUG:boto:Data: 2024-04-13T22:03:09.959 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:09.959 DEBUG:boto:Port: 8000 2024-04-13T22:03:09.959 DEBUG:boto:Params: {} 2024-04-13T22:03:09.960 DEBUG:boto:Token: None 2024-04-13T22:03:09.960 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/ 2024-04-13T22:03:09.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pJLjY3f38Y/EKYRvkgnwZuPfxl0= 2024-04-13T22:03:09.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pJLjY3f38Y/EKYRvkgnwZuPfxl0=', 'Content-Length': '0'} 2024-04-13T22:03:09.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035d882dcdb8da569-00661b011d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:09.962 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:03:09.963 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:09.963 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:09.963 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.963 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.963 DEBUG:boto:Method: GET 2024-04-13T22:03:09.963 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.963 DEBUG:boto:Data: 2024-04-13T22:03:09.963 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:09.963 DEBUG:boto:Port: 8001 2024-04-13T22:03:09.963 DEBUG:boto:Params: {} 2024-04-13T22:03:09.964 DEBUG:boto:Token: None 2024-04-13T22:03:09.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/?versions 2024-04-13T22:03:09.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4= 2024-04-13T22:03:09.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4=', 'Content-Length': '0'} 2024-04-13T22:03:09.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8b8577bda10da92-00661b011d-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:09.966 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:09.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:03:09.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:03:09.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:03:09.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:03:09.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:03:09.968 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:09.968 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:09.968 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.968 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.968 DEBUG:boto:Method: GET 2024-04-13T22:03:09.968 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.968 DEBUG:boto:Data: 2024-04-13T22:03:09.968 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:09.969 DEBUG:boto:Port: 8000 2024-04-13T22:03:09.969 DEBUG:boto:Params: {} 2024-04-13T22:03:09.969 DEBUG:boto:Token: None 2024-04-13T22:03:09.969 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/?versions 2024-04-13T22:03:09.969 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4= 2024-04-13T22:03:09.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4=', 'Content-Length': '0'} 2024-04-13T22:03:09.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f7f87bbfd6517c-00661b011d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:09.972 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:09.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:03:09.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:03:09.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:03:09.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:03:09.973 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:09.973 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:09.973 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.973 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.973 DEBUG:boto:Method: GET 2024-04-13T22:03:09.973 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.973 DEBUG:boto:Data: 2024-04-13T22:03:09.973 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.974 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:09.974 DEBUG:boto:Port: 8001 2024-04-13T22:03:09.974 DEBUG:boto:Params: {} 2024-04-13T22:03:09.974 DEBUG:boto:Token: None 2024-04-13T22:03:09.974 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/?versions 2024-04-13T22:03:09.974 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4= 2024-04-13T22:03:09.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4=', 'Content-Length': '0'} 2024-04-13T22:03:09.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b870852de4afd3ac-00661b011d-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:09.977 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:09.977 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:09.977 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:09.977 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.977 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.977 DEBUG:boto:Method: GET 2024-04-13T22:03:09.978 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:03:09.978 DEBUG:boto:Data: 2024-04-13T22:03:09.978 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.978 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:09.978 DEBUG:boto:Port: 8000 2024-04-13T22:03:09.978 DEBUG:boto:Params: {} 2024-04-13T22:03:09.978 DEBUG:boto:Token: None 2024-04-13T22:03:09.978 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/?versions 2024-04-13T22:03:09.978 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4= 2024-04-13T22:03:09.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JZtKsXVeAOKJJvA0cMpqPhsJcE4=', 'Content-Length': '0'} 2024-04-13T22:03:09.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026252c338c5dcbcf-00661b011d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:09.981 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:09.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:03:09.982 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:09.982 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:09.982 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:09.982 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:09.982 DEBUG:boto:Method: GET 2024-04-13T22:03:09.982 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:09.982 DEBUG:boto:Data: 2024-04-13T22:03:09.982 DEBUG:boto:Headers: {} 2024-04-13T22:03:09.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:09.983 DEBUG:boto:Port: 8001 2024-04-13T22:03:09.983 DEBUG:boto:Params: {} 2024-04-13T22:03:09.983 DEBUG:boto:Token: None 2024-04-13T22:03:09.983 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:09 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:09.983 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SQJh3AUlkVenFUZ8r9fKjGsDMhc= 2024-04-13T22:03:09.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SQJh3AUlkVenFUZ8r9fKjGsDMhc=', 'Content-Length': '0'} 2024-04-13T22:03:09.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059796e590092d319-00661b011d-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.030 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.031 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.031 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.031 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.031 DEBUG:boto:Method: GET 2024-04-13T22:03:10.031 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.031 DEBUG:boto:Data: 2024-04-13T22:03:10.031 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.031 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.031 DEBUG:boto:Params: {} 2024-04-13T22:03:10.031 DEBUG:boto:Token: None 2024-04-13T22:03:10.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wcGEmzgx91if44cCgmAQ6ucNN3c= 2024-04-13T22:03:10.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wcGEmzgx91if44cCgmAQ6ucNN3c=', 'Content-Length': '0'} 2024-04-13T22:03:10.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a081afcf427670a9-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.034 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.034 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.034 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.034 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.034 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.035 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.035 DEBUG:boto:Data: 2024-04-13T22:03:10.035 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.035 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.035 DEBUG:boto:Params: {} 2024-04-13T22:03:10.035 DEBUG:boto:Token: None 2024-04-13T22:03:10.035 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.035 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+4J+uzJALRFXFIVqmKTUKHKgE6s= 2024-04-13T22:03:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+4J+uzJALRFXFIVqmKTUKHKgE6s=', 'Content-Length': '0'} 2024-04-13T22:03:10.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ee8e5bc1ce44c88-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.038 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.038 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.038 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.038 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.038 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.038 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.038 DEBUG:boto:Data: 2024-04-13T22:03:10.038 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.038 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.038 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.039 DEBUG:boto:Params: {} 2024-04-13T22:03:10.039 DEBUG:boto:Token: None 2024-04-13T22:03:10.039 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+4J+uzJALRFXFIVqmKTUKHKgE6s= 2024-04-13T22:03:10.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+4J+uzJALRFXFIVqmKTUKHKgE6s=', 'Content-Length': '0'} 2024-04-13T22:03:10.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a06fd59e1aa64b6d-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:03:10.041 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.041 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.041 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.041 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.041 DEBUG:boto:Method: GET 2024-04-13T22:03:10.042 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.042 DEBUG:boto:Data: 2024-04-13T22:03:10.042 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.042 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.042 DEBUG:boto:Params: {} 2024-04-13T22:03:10.042 DEBUG:boto:Token: None 2024-04-13T22:03:10.042 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wcGEmzgx91if44cCgmAQ6ucNN3c= 2024-04-13T22:03:10.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wcGEmzgx91if44cCgmAQ6ucNN3c=', 'Content-Length': '0'} 2024-04-13T22:03:10.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000787e8ea517956e88-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.045 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.045 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.045 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.045 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.045 DEBUG:boto:Method: GET 2024-04-13T22:03:10.045 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.045 DEBUG:boto:Data: 2024-04-13T22:03:10.045 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.045 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.045 DEBUG:boto:Params: {} 2024-04-13T22:03:10.046 DEBUG:boto:Token: None 2024-04-13T22:03:10.046 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:03:10.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wcGEmzgx91if44cCgmAQ6ucNN3c= 2024-04-13T22:03:10.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wcGEmzgx91if44cCgmAQ6ucNN3c=', 'Content-Length': '0'} 2024-04-13T22:03:10.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000224068ca5c3e4ed1-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.048 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.048 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.048 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.048 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.048 DEBUG:boto:Method: GET 2024-04-13T22:03:10.048 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.049 DEBUG:boto:Data: 2024-04-13T22:03:10.049 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.049 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.049 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.049 DEBUG:boto:Params: {} 2024-04-13T22:03:10.049 DEBUG:boto:Token: None 2024-04-13T22:03:10.049 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac618f1fc085f3f-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.052 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.052 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.052 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.052 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.052 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.052 DEBUG:boto:Method: GET 2024-04-13T22:03:10.052 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.052 DEBUG:boto:Data: 2024-04-13T22:03:10.053 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.053 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.053 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.053 DEBUG:boto:Params: {} 2024-04-13T22:03:10.053 DEBUG:boto:Token: None 2024-04-13T22:03:10.053 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.053 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a850fb9a182b425a-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.056 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.056 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.056 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.056 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.056 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:03:10.056 DEBUG:boto:Data: 2024-04-13T22:03:10.057 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.057 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.057 DEBUG:boto:Params: {} 2024-04-13T22:03:10.057 DEBUG:boto:Token: None 2024-04-13T22:03:10.057 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1 2024-04-13T22:03:10.057 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVncUdUBZ9syEdQbOjFLZhUp1Bg= 2024-04-13T22:03:10.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVncUdUBZ9syEdQbOjFLZhUp1Bg=', 'Content-Length': '0'} 2024-04-13T22:03:10.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2bb7f1f1c436713-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.059 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.059 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.060 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.060 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:03:10.060 DEBUG:boto:Data: 2024-04-13T22:03:10.060 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.060 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.060 DEBUG:boto:Params: {} 2024-04-13T22:03:10.060 DEBUG:boto:Token: None 2024-04-13T22:03:10.060 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1 2024-04-13T22:03:10.060 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVncUdUBZ9syEdQbOjFLZhUp1Bg= 2024-04-13T22:03:10.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVncUdUBZ9syEdQbOjFLZhUp1Bg=', 'Content-Length': '0'} 2024-04-13T22:03:10.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9fd03431bd0f5eb-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:03:10.063 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.063 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.063 DEBUG:boto:Method: GET 2024-04-13T22:03:10.063 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:03:10.063 DEBUG:boto:Data: 2024-04-13T22:03:10.063 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.063 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.063 DEBUG:boto:Params: {} 2024-04-13T22:03:10.063 DEBUG:boto:Token: None 2024-04-13T22:03:10.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1 2024-04-13T22:03:10.064 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nwkuOJeR/qQJwNHD0Nt+2bzb6j8= 2024-04-13T22:03:10.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nwkuOJeR/qQJwNHD0Nt+2bzb6j8=', 'Content-Length': '0'} 2024-04-13T22:03:10.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1dd38c6fe03f856-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.107 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:03:10.107 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:03:10.107 DEBUG:boto:Method: GET 2024-04-13T22:03:10.107 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:03:10.107 DEBUG:boto:Data: 2024-04-13T22:03:10.107 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.107 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.108 DEBUG:boto:Params: {} 2024-04-13T22:03:10.108 DEBUG:boto:Token: None 2024-04-13T22:03:10.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj1 2024-04-13T22:03:10.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nwkuOJeR/qQJwNHD0Nt+2bzb6j8= 2024-04-13T22:03:10.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nwkuOJeR/qQJwNHD0Nt+2bzb6j8=', 'Content-Length': '0'} 2024-04-13T22:03:10.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046a944608aec7478-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.110 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.111 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.111 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.111 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.111 DEBUG:boto:Method: GET 2024-04-13T22:03:10.111 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.111 DEBUG:boto:Data: 2024-04-13T22:03:10.111 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.111 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.111 DEBUG:boto:Params: {} 2024-04-13T22:03:10.112 DEBUG:boto:Token: None 2024-04-13T22:03:10.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9d010bbeeb87a40-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.114 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.115 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.115 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.115 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.115 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.115 DEBUG:boto:Method: GET 2024-04-13T22:03:10.115 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:03:10.115 DEBUG:boto:Data: 2024-04-13T22:03:10.115 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.116 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.116 DEBUG:boto:Params: {} 2024-04-13T22:03:10.116 DEBUG:boto:Token: None 2024-04-13T22:03:10.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da45c045499094f8-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.119 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:03:10.119 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.119 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.119 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.119 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.119 DEBUG:boto:Method: GET 2024-04-13T22:03:10.120 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.120 DEBUG:boto:Data: 2024-04-13T22:03:10.120 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.120 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.120 DEBUG:boto:Params: {} 2024-04-13T22:03:10.120 DEBUG:boto:Token: None 2024-04-13T22:03:10.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps= 2024-04-13T22:03:10.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps=', 'Content-Length': '0'} 2024-04-13T22:03:10.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3fb002ff7a963ad-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.122 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.122 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.122 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.123 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.123 DEBUG:boto:Method: GET 2024-04-13T22:03:10.123 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.123 DEBUG:boto:Data: 2024-04-13T22:03:10.123 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.123 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.123 DEBUG:boto:Params: {} 2024-04-13T22:03:10.123 DEBUG:boto:Token: None 2024-04-13T22:03:10.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps= 2024-04-13T22:03:10.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps=', 'Content-Length': '0'} 2024-04-13T22:03:10.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b561ba42f092e79b-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.126 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.126 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.126 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.126 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.126 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.126 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.126 DEBUG:boto:Data: 2024-04-13T22:03:10.127 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.127 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.127 DEBUG:boto:Params: {} 2024-04-13T22:03:10.127 DEBUG:boto:Token: None 2024-04-13T22:03:10.127 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.127 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoFK7hEdVaFtfTj8NTDxFPCAn9M= 2024-04-13T22:03:10.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoFK7hEdVaFtfTj8NTDxFPCAn9M=', 'Content-Length': '0'} 2024-04-13T22:03:10.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd963f672fe420ab-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.129 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.129 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.129 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.129 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.130 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.130 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.130 DEBUG:boto:Data: 2024-04-13T22:03:10.130 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.130 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.130 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.130 DEBUG:boto:Params: {} 2024-04-13T22:03:10.130 DEBUG:boto:Token: None 2024-04-13T22:03:10.130 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.131 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoFK7hEdVaFtfTj8NTDxFPCAn9M= 2024-04-13T22:03:10.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoFK7hEdVaFtfTj8NTDxFPCAn9M=', 'Content-Length': '0'} 2024-04-13T22:03:10.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad8b92dd16824a15-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:03:10.133 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.133 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.133 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.133 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.133 DEBUG:boto:Method: GET 2024-04-13T22:03:10.133 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.133 DEBUG:boto:Data: 2024-04-13T22:03:10.133 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.133 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.133 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.133 DEBUG:boto:Params: {} 2024-04-13T22:03:10.133 DEBUG:boto:Token: None 2024-04-13T22:03:10.134 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps= 2024-04-13T22:03:10.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps=', 'Content-Length': '0'} 2024-04-13T22:03:10.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000558d0b01e76892ed-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.136 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.136 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.136 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.136 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.136 DEBUG:boto:Method: GET 2024-04-13T22:03:10.136 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.136 DEBUG:boto:Data: 2024-04-13T22:03:10.137 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.137 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.137 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.137 DEBUG:boto:Params: {} 2024-04-13T22:03:10.137 DEBUG:boto:Token: None 2024-04-13T22:03:10.137 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:03:10.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps= 2024-04-13T22:03:10.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3JxYarSN99Q2/0wHe61Wu0oq8ps=', 'Content-Length': '0'} 2024-04-13T22:03:10.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039cb55f641c4bb55-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.139 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.140 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.140 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.140 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.140 DEBUG:boto:Method: GET 2024-04-13T22:03:10.140 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.140 DEBUG:boto:Data: 2024-04-13T22:03:10.140 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.140 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.140 DEBUG:boto:Params: {} 2024-04-13T22:03:10.140 DEBUG:boto:Token: None 2024-04-13T22:03:10.140 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017eec0af979ddb6d-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.143 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.143 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.143 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.143 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.143 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.144 DEBUG:boto:Method: GET 2024-04-13T22:03:10.144 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.144 DEBUG:boto:Data: 2024-04-13T22:03:10.144 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.144 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.144 DEBUG:boto:Params: {} 2024-04-13T22:03:10.144 DEBUG:boto:Token: None 2024-04-13T22:03:10.144 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3bce9bea03ad21e-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.190 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.191 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.191 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.191 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.191 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:03:10.191 DEBUG:boto:Data: 2024-04-13T22:03:10.191 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.191 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.191 DEBUG:boto:Params: {} 2024-04-13T22:03:10.191 DEBUG:boto:Token: None 2024-04-13T22:03:10.192 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2 2024-04-13T22:03:10.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tYo3mZC7MJy0lU2rudLPuYuXp9A= 2024-04-13T22:03:10.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tYo3mZC7MJy0lU2rudLPuYuXp9A=', 'Content-Length': '0'} 2024-04-13T22:03:10.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a84765d92dcbe6f-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.194 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.194 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.194 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.194 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:03:10.194 DEBUG:boto:Data: 2024-04-13T22:03:10.194 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.194 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.194 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.194 DEBUG:boto:Params: {} 2024-04-13T22:03:10.195 DEBUG:boto:Token: None 2024-04-13T22:03:10.195 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2 2024-04-13T22:03:10.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tYo3mZC7MJy0lU2rudLPuYuXp9A= 2024-04-13T22:03:10.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tYo3mZC7MJy0lU2rudLPuYuXp9A=', 'Content-Length': '0'} 2024-04-13T22:03:10.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d988e640ef694169-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:03:10.197 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.197 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.197 DEBUG:boto:Method: GET 2024-04-13T22:03:10.197 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:03:10.198 DEBUG:boto:Data: 2024-04-13T22:03:10.198 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.198 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.198 DEBUG:boto:Params: {} 2024-04-13T22:03:10.198 DEBUG:boto:Token: None 2024-04-13T22:03:10.198 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2 2024-04-13T22:03:10.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TfbSPT05+A7e4iTA9AJ62BODPbs= 2024-04-13T22:03:10.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TfbSPT05+A7e4iTA9AJ62BODPbs=', 'Content-Length': '0'} 2024-04-13T22:03:10.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eed26075b02fc991-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.200 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:03:10.200 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:03:10.201 DEBUG:boto:Method: GET 2024-04-13T22:03:10.201 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:03:10.201 DEBUG:boto:Data: 2024-04-13T22:03:10.201 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.201 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.201 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.201 DEBUG:boto:Params: {} 2024-04-13T22:03:10.201 DEBUG:boto:Token: None 2024-04-13T22:03:10.201 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj2 2024-04-13T22:03:10.201 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TfbSPT05+A7e4iTA9AJ62BODPbs= 2024-04-13T22:03:10.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TfbSPT05+A7e4iTA9AJ62BODPbs=', 'Content-Length': '0'} 2024-04-13T22:03:10.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c5bceb50d50ada5-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.204 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.204 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.204 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.204 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.204 DEBUG:boto:Method: GET 2024-04-13T22:03:10.204 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.204 DEBUG:boto:Data: 2024-04-13T22:03:10.204 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.204 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.204 DEBUG:boto:Params: {} 2024-04-13T22:03:10.205 DEBUG:boto:Token: None 2024-04-13T22:03:10.205 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.205 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fcf0c2aa515f4e5-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.207 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.208 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.208 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.208 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.208 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.208 DEBUG:boto:Method: GET 2024-04-13T22:03:10.208 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:03:10.208 DEBUG:boto:Data: 2024-04-13T22:03:10.208 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.208 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.208 DEBUG:boto:Params: {} 2024-04-13T22:03:10.208 DEBUG:boto:Token: None 2024-04-13T22:03:10.209 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1baf14069802f94-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.211 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:03:10.212 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.212 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.212 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.212 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.212 DEBUG:boto:Method: GET 2024-04-13T22:03:10.212 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.213 DEBUG:boto:Data: 2024-04-13T22:03:10.213 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.213 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.213 DEBUG:boto:Params: {} 2024-04-13T22:03:10.213 DEBUG:boto:Token: None 2024-04-13T22:03:10.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8= 2024-04-13T22:03:10.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8=', 'Content-Length': '0'} 2024-04-13T22:03:10.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de8bb60ed28fc219-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.262 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.263 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.263 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.263 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.263 DEBUG:boto:Method: GET 2024-04-13T22:03:10.263 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.263 DEBUG:boto:Data: 2024-04-13T22:03:10.263 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.263 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.263 DEBUG:boto:Params: {} 2024-04-13T22:03:10.264 DEBUG:boto:Token: None 2024-04-13T22:03:10.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8= 2024-04-13T22:03:10.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8=', 'Content-Length': '0'} 2024-04-13T22:03:10.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a43a46994afcf2c-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.266 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.266 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.267 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.267 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.267 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.267 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.267 DEBUG:boto:Data: 2024-04-13T22:03:10.267 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.267 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.267 DEBUG:boto:Params: {} 2024-04-13T22:03:10.267 DEBUG:boto:Token: None 2024-04-13T22:03:10.267 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RG936T4qBdScfyYyS0AVspBgB/o= 2024-04-13T22:03:10.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RG936T4qBdScfyYyS0AVspBgB/o=', 'Content-Length': '0'} 2024-04-13T22:03:10.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000829ef591199e8070-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.270 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.270 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.270 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.270 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.270 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.270 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.270 DEBUG:boto:Data: 2024-04-13T22:03:10.270 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.270 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.270 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.270 DEBUG:boto:Params: {} 2024-04-13T22:03:10.271 DEBUG:boto:Token: None 2024-04-13T22:03:10.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RG936T4qBdScfyYyS0AVspBgB/o= 2024-04-13T22:03:10.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RG936T4qBdScfyYyS0AVspBgB/o=', 'Content-Length': '0'} 2024-04-13T22:03:10.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2526a2298260fc1-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:03:10.273 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.273 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.273 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.273 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.273 DEBUG:boto:Method: GET 2024-04-13T22:03:10.273 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.273 DEBUG:boto:Data: 2024-04-13T22:03:10.273 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.274 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.274 DEBUG:boto:Params: {} 2024-04-13T22:03:10.274 DEBUG:boto:Token: None 2024-04-13T22:03:10.274 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8= 2024-04-13T22:03:10.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8=', 'Content-Length': '0'} 2024-04-13T22:03:10.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb220723017fc641-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.276 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.276 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.276 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.276 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.276 DEBUG:boto:Method: GET 2024-04-13T22:03:10.276 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.277 DEBUG:boto:Data: 2024-04-13T22:03:10.277 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.277 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.277 DEBUG:boto:Params: {} 2024-04-13T22:03:10.277 DEBUG:boto:Token: None 2024-04-13T22:03:10.277 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:03:10.277 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8= 2024-04-13T22:03:10.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hrNQObIfa0tq7qHi08LID0DcIm8=', 'Content-Length': '0'} 2024-04-13T22:03:10.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000350796eb366cdc44-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.279 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.280 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.280 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.280 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.280 DEBUG:boto:Method: GET 2024-04-13T22:03:10.280 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.280 DEBUG:boto:Data: 2024-04-13T22:03:10.280 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.280 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.280 DEBUG:boto:Params: {} 2024-04-13T22:03:10.281 DEBUG:boto:Token: None 2024-04-13T22:03:10.281 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60a2ad9b2e066df-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.283 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.283 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.284 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.284 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.284 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.284 DEBUG:boto:Method: GET 2024-04-13T22:03:10.284 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.284 DEBUG:boto:Data: 2024-04-13T22:03:10.284 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.284 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.284 DEBUG:boto:Params: {} 2024-04-13T22:03:10.285 DEBUG:boto:Token: None 2024-04-13T22:03:10.285 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.285 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78085d6afa62c81-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.330 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.331 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.331 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.331 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.331 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:03:10.331 DEBUG:boto:Data: 2024-04-13T22:03:10.331 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.331 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.331 DEBUG:boto:Params: {} 2024-04-13T22:03:10.332 DEBUG:boto:Token: None 2024-04-13T22:03:10.332 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3 2024-04-13T22:03:10.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t12Fv4KTnLHUkBCTlRFRrK9AkwE= 2024-04-13T22:03:10.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t12Fv4KTnLHUkBCTlRFRrK9AkwE=', 'Content-Length': '0'} 2024-04-13T22:03:10.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000351ca63e035a81b2-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.334 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.334 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.334 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.335 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:03:10.335 DEBUG:boto:Data: 2024-04-13T22:03:10.335 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.335 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.335 DEBUG:boto:Params: {} 2024-04-13T22:03:10.335 DEBUG:boto:Token: None 2024-04-13T22:03:10.335 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3 2024-04-13T22:03:10.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t12Fv4KTnLHUkBCTlRFRrK9AkwE= 2024-04-13T22:03:10.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t12Fv4KTnLHUkBCTlRFRrK9AkwE=', 'Content-Length': '0'} 2024-04-13T22:03:10.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7515b353f33d941-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:03:10.338 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.338 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.338 DEBUG:boto:Method: GET 2024-04-13T22:03:10.338 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:03:10.338 DEBUG:boto:Data: 2024-04-13T22:03:10.338 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.338 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.338 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.338 DEBUG:boto:Params: {} 2024-04-13T22:03:10.339 DEBUG:boto:Token: None 2024-04-13T22:03:10.339 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3 2024-04-13T22:03:10.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9XFEF8ToH9ffYT9WE4Me5noeuMs= 2024-04-13T22:03:10.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9XFEF8ToH9ffYT9WE4Me5noeuMs=', 'Content-Length': '0'} 2024-04-13T22:03:10.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ed2f6cf7d4839ce-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.341 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:03:10.341 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:03:10.341 DEBUG:boto:Method: GET 2024-04-13T22:03:10.341 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:03:10.341 DEBUG:boto:Data: 2024-04-13T22:03:10.341 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.341 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.342 DEBUG:boto:Params: {} 2024-04-13T22:03:10.342 DEBUG:boto:Token: None 2024-04-13T22:03:10.342 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj3 2024-04-13T22:03:10.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9XFEF8ToH9ffYT9WE4Me5noeuMs= 2024-04-13T22:03:10.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9XFEF8ToH9ffYT9WE4Me5noeuMs=', 'Content-Length': '0'} 2024-04-13T22:03:10.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9590d9fe5ba3fae-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.344 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.344 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.344 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.344 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.345 DEBUG:boto:Method: GET 2024-04-13T22:03:10.345 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.345 DEBUG:boto:Data: 2024-04-13T22:03:10.345 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.345 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.345 DEBUG:boto:Params: {} 2024-04-13T22:03:10.345 DEBUG:boto:Token: None 2024-04-13T22:03:10.345 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.346 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f744163951174d2-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.348 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.348 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.349 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.349 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.349 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.349 DEBUG:boto:Method: GET 2024-04-13T22:03:10.349 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:03:10.349 DEBUG:boto:Data: 2024-04-13T22:03:10.349 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.349 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.349 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.349 DEBUG:boto:Params: {} 2024-04-13T22:03:10.350 DEBUG:boto:Token: None 2024-04-13T22:03:10.350 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.350 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c4cb8d770fb778-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.352 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:03:10.353 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.353 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.353 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.353 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.353 DEBUG:boto:Method: GET 2024-04-13T22:03:10.353 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.353 DEBUG:boto:Data: 2024-04-13T22:03:10.353 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.354 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.354 DEBUG:boto:Params: {} 2024-04-13T22:03:10.354 DEBUG:boto:Token: None 2024-04-13T22:03:10.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA= 2024-04-13T22:03:10.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA=', 'Content-Length': '0'} 2024-04-13T22:03:10.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064dbcf7322b37ff0-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.357 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.357 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.357 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.357 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.357 DEBUG:boto:Method: GET 2024-04-13T22:03:10.357 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.357 DEBUG:boto:Data: 2024-04-13T22:03:10.357 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.358 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.358 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.358 DEBUG:boto:Params: {} 2024-04-13T22:03:10.358 DEBUG:boto:Token: None 2024-04-13T22:03:10.358 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA= 2024-04-13T22:03:10.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA=', 'Content-Length': '0'} 2024-04-13T22:03:10.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002250d8a84a3d3f0e-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.361 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.361 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.361 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.361 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.361 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.361 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.361 DEBUG:boto:Data: 2024-04-13T22:03:10.361 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.361 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.362 DEBUG:boto:Params: {} 2024-04-13T22:03:10.362 DEBUG:boto:Token: None 2024-04-13T22:03:10.362 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.362 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YtzcKqJZhYt/3sGQd8f+pKUcCIg= 2024-04-13T22:03:10.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YtzcKqJZhYt/3sGQd8f+pKUcCIg=', 'Content-Length': '0'} 2024-04-13T22:03:10.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026e367361b8d22cb-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.364 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.364 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.364 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.364 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.364 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.364 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.364 DEBUG:boto:Data: 2024-04-13T22:03:10.364 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.365 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.365 DEBUG:boto:Params: {} 2024-04-13T22:03:10.365 DEBUG:boto:Token: None 2024-04-13T22:03:10.365 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YtzcKqJZhYt/3sGQd8f+pKUcCIg= 2024-04-13T22:03:10.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YtzcKqJZhYt/3sGQd8f+pKUcCIg=', 'Content-Length': '0'} 2024-04-13T22:03:10.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032d2dd0c1811c0e5-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:03:10.367 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.367 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.367 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.367 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.367 DEBUG:boto:Method: GET 2024-04-13T22:03:10.367 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.367 DEBUG:boto:Data: 2024-04-13T22:03:10.368 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.368 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.368 DEBUG:boto:Params: {} 2024-04-13T22:03:10.368 DEBUG:boto:Token: None 2024-04-13T22:03:10.368 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA= 2024-04-13T22:03:10.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA=', 'Content-Length': '0'} 2024-04-13T22:03:10.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fccc6dc6af4865f7-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.370 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.371 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.371 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.371 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.371 DEBUG:boto:Method: GET 2024-04-13T22:03:10.371 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.371 DEBUG:boto:Data: 2024-04-13T22:03:10.371 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.371 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.371 DEBUG:boto:Params: {} 2024-04-13T22:03:10.372 DEBUG:boto:Token: None 2024-04-13T22:03:10.372 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:03:10.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA= 2024-04-13T22:03:10.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g4doVUB0QuXkWP2IhF0vR07WpNA=', 'Content-Length': '0'} 2024-04-13T22:03:10.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c4e8e42e181c70d-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.374 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.374 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.374 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.374 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.374 DEBUG:boto:Method: GET 2024-04-13T22:03:10.374 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.374 DEBUG:boto:Data: 2024-04-13T22:03:10.374 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.374 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.375 DEBUG:boto:Params: {} 2024-04-13T22:03:10.375 DEBUG:boto:Token: None 2024-04-13T22:03:10.375 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63a149c45598761-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.377 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.378 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.378 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.378 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.378 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.378 DEBUG:boto:Method: GET 2024-04-13T22:03:10.378 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.378 DEBUG:boto:Data: 2024-04-13T22:03:10.379 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.379 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.379 DEBUG:boto:Params: {} 2024-04-13T22:03:10.379 DEBUG:boto:Token: None 2024-04-13T22:03:10.380 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6ed149a12168185-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.382 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.383 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.383 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.383 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.383 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:03:10.383 DEBUG:boto:Data: 2024-04-13T22:03:10.383 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.384 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.384 DEBUG:boto:Params: {} 2024-04-13T22:03:10.384 DEBUG:boto:Token: None 2024-04-13T22:03:10.384 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4 2024-04-13T22:03:10.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XMfNKDj8SewzQY5PLvXKAxSHq00= 2024-04-13T22:03:10.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XMfNKDj8SewzQY5PLvXKAxSHq00=', 'Content-Length': '0'} 2024-04-13T22:03:10.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025c7d07cab72a389-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.386 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.387 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.387 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.387 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:03:10.387 DEBUG:boto:Data: 2024-04-13T22:03:10.387 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.387 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.387 DEBUG:boto:Params: {} 2024-04-13T22:03:10.387 DEBUG:boto:Token: None 2024-04-13T22:03:10.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4 2024-04-13T22:03:10.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XMfNKDj8SewzQY5PLvXKAxSHq00= 2024-04-13T22:03:10.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XMfNKDj8SewzQY5PLvXKAxSHq00=', 'Content-Length': '0'} 2024-04-13T22:03:10.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012ab02d767e36d97-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:03:10.390 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.390 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.390 DEBUG:boto:Method: GET 2024-04-13T22:03:10.390 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:03:10.390 DEBUG:boto:Data: 2024-04-13T22:03:10.390 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.390 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.390 DEBUG:boto:Params: {} 2024-04-13T22:03:10.391 DEBUG:boto:Token: None 2024-04-13T22:03:10.391 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4 2024-04-13T22:03:10.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RKCdCKOyhGMwIaTbhnr3g16yUFQ= 2024-04-13T22:03:10.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RKCdCKOyhGMwIaTbhnr3g16yUFQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007289f6a0b517299f-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.434 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:03:10.435 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:03:10.435 DEBUG:boto:Method: GET 2024-04-13T22:03:10.435 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:03:10.435 DEBUG:boto:Data: 2024-04-13T22:03:10.435 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.436 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.436 DEBUG:boto:Params: {} 2024-04-13T22:03:10.436 DEBUG:boto:Token: None 2024-04-13T22:03:10.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/obj4 2024-04-13T22:03:10.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RKCdCKOyhGMwIaTbhnr3g16yUFQ= 2024-04-13T22:03:10.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RKCdCKOyhGMwIaTbhnr3g16yUFQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be10ca83087ea4a9-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.439 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.439 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.439 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.439 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.439 DEBUG:boto:Method: GET 2024-04-13T22:03:10.439 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.439 DEBUG:boto:Data: 2024-04-13T22:03:10.439 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.440 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.440 DEBUG:boto:Params: {} 2024-04-13T22:03:10.440 DEBUG:boto:Token: None 2024-04-13T22:03:10.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f870916eeaf9a80e-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.442 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.443 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:10.443 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:10.443 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.443 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.443 DEBUG:boto:Method: GET 2024-04-13T22:03:10.443 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:03:10.443 DEBUG:boto:Data: 2024-04-13T22:03:10.445 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.445 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.445 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.445 DEBUG:boto:Params: {} 2024-04-13T22:03:10.446 DEBUG:boto:Token: None 2024-04-13T22:03:10.446 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-48/ 2024-04-13T22:03:10.446 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo= 2024-04-13T22:03:10.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3LI9HkQguydQdPAT1vIZapn1eo=', 'Content-Length': '0'} 2024-04-13T22:03:10.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efe239369aaf3572-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.449 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.449 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-48 zones={a2, a1} 2024-04-13T22:03:10.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-49 2024-04-13T22:03:10.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:10.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:10.450 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:03:10.508 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:10.508 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:10.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:10.544+0000 7f146912c780 1 bucket sync caught up with source: 2024-04-13T22:03:10.545 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.24.6, , 00000000001.24.6, , 00000000001.21.6, , , 00000000001.24.6, , , ] 2024-04-13T22:03:10.545 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.24.6, , 00000000001.24.6, , 00000000001.21.6, , , 00000000001.24.6, , , ] 2024-04-13T22:03:10.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:10.544+0000 7f146912c780 0 bucket checkpoint complete 2024-04-13T22:03:10.558 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-49 2024-04-13T22:03:10.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:10.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:10.559 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:03:10.617 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:10.618 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:10.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:10.653+0000 7f7a0547f780 1 bucket sync caught up with source: 2024-04-13T22:03:10.653 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1447.6, , 00000000001.1721.6, , 00000000001.1702.6, , , 00000000001.592.6, , , ] 2024-04-13T22:03:10.653 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.1447.6, , 00000000001.1721.6, , 00000000001.1702.6, , , 00000000001.592.6, , , ] 2024-04-13T22:03:10.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:10.653+0000 7f7a0547f780 0 bucket checkpoint complete 2024-04-13T22:03:10.666 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-49 zones={a2, a1} 2024-04-13T22:03:10.666 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.666 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.666 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.666 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:03:10.666 DEBUG:boto:Data: 2024-04-13T22:03:10.666 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.666 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.667 DEBUG:boto:Params: {} 2024-04-13T22:03:10.667 DEBUG:boto:Token: None 2024-04-13T22:03:10.667 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q7jfaccK0jFo7TsdM7d9z8MTwMQ= 2024-04-13T22:03:10.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q7jfaccK0jFo7TsdM7d9z8MTwMQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bcc7921ab18ce96-00661b011e-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.670 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.670 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.670 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.670 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:03:10.670 DEBUG:boto:Data: 2024-04-13T22:03:10.670 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.670 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.670 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.670 DEBUG:boto:Params: {} 2024-04-13T22:03:10.670 DEBUG:boto:Token: None 2024-04-13T22:03:10.671 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.671 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q7jfaccK0jFo7TsdM7d9z8MTwMQ= 2024-04-13T22:03:10.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q7jfaccK0jFo7TsdM7d9z8MTwMQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9735674e0984c5b-00661b011e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:03:10.673 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.673 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.673 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.673 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.673 DEBUG:boto:Method: GET 2024-04-13T22:03:10.673 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.673 DEBUG:boto:Data: 2024-04-13T22:03:10.674 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.674 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.674 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.674 DEBUG:boto:Params: {} 2024-04-13T22:03:10.674 DEBUG:boto:Token: None 2024-04-13T22:03:10.674 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/?versions 2024-04-13T22:03:10.674 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c= 2024-04-13T22:03:10.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c=', 'Content-Length': '0'} 2024-04-13T22:03:10.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5aabc2d21a8d7a9-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.676 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:03:10.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:03:10.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:03:10.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:03:10.677 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:03:10.677 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.677 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.678 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.678 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.678 DEBUG:boto:Method: GET 2024-04-13T22:03:10.678 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.678 DEBUG:boto:Data: 2024-04-13T22:03:10.678 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.678 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.678 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.678 DEBUG:boto:Params: {} 2024-04-13T22:03:10.678 DEBUG:boto:Token: None 2024-04-13T22:03:10.679 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/?versions 2024-04-13T22:03:10.679 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c= 2024-04-13T22:03:10.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c=', 'Content-Length': '0'} 2024-04-13T22:03:10.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60ada8c0cfb87f9-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.681 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:03:10.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:03:10.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:03:10.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:03:10.682 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.682 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.682 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.682 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.682 DEBUG:boto:Method: GET 2024-04-13T22:03:10.682 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.682 DEBUG:boto:Data: 2024-04-13T22:03:10.682 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.682 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.682 DEBUG:boto:Params: {} 2024-04-13T22:03:10.683 DEBUG:boto:Token: None 2024-04-13T22:03:10.683 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/?versions 2024-04-13T22:03:10.683 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c= 2024-04-13T22:03:10.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c=', 'Content-Length': '0'} 2024-04-13T22:03:10.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395d7e896cc7ae62-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.685 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.686 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.686 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.686 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.686 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.686 DEBUG:boto:Method: GET 2024-04-13T22:03:10.686 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:03:10.686 DEBUG:boto:Data: 2024-04-13T22:03:10.686 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.686 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.687 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.687 DEBUG:boto:Params: {} 2024-04-13T22:03:10.687 DEBUG:boto:Token: None 2024-04-13T22:03:10.687 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/?versions 2024-04-13T22:03:10.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c= 2024-04-13T22:03:10.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qGjxbiPIrbjdY/wGzMqAYrD5M6c=', 'Content-Length': '0'} 2024-04-13T22:03:10.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002766b7d773c51681-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.689 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:03:10.690 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.690 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.690 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.690 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.690 DEBUG:boto:Method: GET 2024-04-13T22:03:10.691 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.691 DEBUG:boto:Data: 2024-04-13T22:03:10.691 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.691 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.691 DEBUG:boto:Params: {} 2024-04-13T22:03:10.691 DEBUG:boto:Token: None 2024-04-13T22:03:10.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY= 2024-04-13T22:03:10.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY=', 'Content-Length': '0'} 2024-04-13T22:03:10.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd489b569e7ddf9e-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.693 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.693 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.693 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.693 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.693 DEBUG:boto:Method: GET 2024-04-13T22:03:10.693 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.694 DEBUG:boto:Data: 2024-04-13T22:03:10.694 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.694 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.694 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.694 DEBUG:boto:Params: {} 2024-04-13T22:03:10.694 DEBUG:boto:Token: None 2024-04-13T22:03:10.694 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.694 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY= 2024-04-13T22:03:10.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY=', 'Content-Length': '0'} 2024-04-13T22:03:10.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004960bbd83617264c-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.696 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.696 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.696 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.696 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.696 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.697 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.697 DEBUG:boto:Data: 2024-04-13T22:03:10.697 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.697 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.697 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.697 DEBUG:boto:Params: {} 2024-04-13T22:03:10.697 DEBUG:boto:Token: None 2024-04-13T22:03:10.697 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.697 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:twczFV92k9BdSn39sKwU493IMtc= 2024-04-13T22:03:10.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:twczFV92k9BdSn39sKwU493IMtc=', 'Content-Length': '0'} 2024-04-13T22:03:10.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be822304298c3d67-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.699 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.699 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.699 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.700 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.700 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.700 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.700 DEBUG:boto:Data: 2024-04-13T22:03:10.700 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.700 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.700 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.700 DEBUG:boto:Params: {} 2024-04-13T22:03:10.700 DEBUG:boto:Token: None 2024-04-13T22:03:10.700 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:twczFV92k9BdSn39sKwU493IMtc= 2024-04-13T22:03:10.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:twczFV92k9BdSn39sKwU493IMtc=', 'Content-Length': '0'} 2024-04-13T22:03:10.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dcffa6e0b475cdc-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:03:10.702 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.702 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.702 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.703 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.703 DEBUG:boto:Method: GET 2024-04-13T22:03:10.703 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.703 DEBUG:boto:Data: 2024-04-13T22:03:10.703 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.703 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.703 DEBUG:boto:Params: {} 2024-04-13T22:03:10.703 DEBUG:boto:Token: None 2024-04-13T22:03:10.703 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY= 2024-04-13T22:03:10.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY=', 'Content-Length': '0'} 2024-04-13T22:03:10.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009500124a1b64d40-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.705 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.705 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.705 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.706 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.706 DEBUG:boto:Method: GET 2024-04-13T22:03:10.706 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.706 DEBUG:boto:Data: 2024-04-13T22:03:10.706 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.706 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.706 DEBUG:boto:Params: {} 2024-04-13T22:03:10.706 DEBUG:boto:Token: None 2024-04-13T22:03:10.706 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:03:10.707 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY= 2024-04-13T22:03:10.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9JdTm6oXlJbcW5E2u2TAEiDoTOY=', 'Content-Length': '0'} 2024-04-13T22:03:10.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000069f23745924e3f8-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.708 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.708 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.708 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.709 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.709 DEBUG:boto:Method: GET 2024-04-13T22:03:10.709 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.709 DEBUG:boto:Data: 2024-04-13T22:03:10.709 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.709 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.709 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.709 DEBUG:boto:Params: {} 2024-04-13T22:03:10.709 DEBUG:boto:Token: None 2024-04-13T22:03:10.709 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.710 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094dc35d1b73468f0-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.712 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.712 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.712 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.712 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.712 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.712 DEBUG:boto:Method: GET 2024-04-13T22:03:10.712 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.712 DEBUG:boto:Data: 2024-04-13T22:03:10.713 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.713 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.713 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.713 DEBUG:boto:Params: {} 2024-04-13T22:03:10.713 DEBUG:boto:Token: None 2024-04-13T22:03:10.713 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.713 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04cb788f81b8ec0-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.715 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.715 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.715 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.716 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.716 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:03:10.716 DEBUG:boto:Data: 2024-04-13T22:03:10.716 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.716 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.716 DEBUG:boto:Params: {} 2024-04-13T22:03:10.716 DEBUG:boto:Token: None 2024-04-13T22:03:10.716 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1 2024-04-13T22:03:10.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A9TZ0YDYylxzjxNkOUfHdWC70NM= 2024-04-13T22:03:10.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A9TZ0YDYylxzjxNkOUfHdWC70NM=', 'Content-Length': '0'} 2024-04-13T22:03:10.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6df8ac820a634c3-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.718 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.718 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.718 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.719 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:03:10.719 DEBUG:boto:Data: 2024-04-13T22:03:10.719 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.719 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.719 DEBUG:boto:Params: {} 2024-04-13T22:03:10.719 DEBUG:boto:Token: None 2024-04-13T22:03:10.719 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1 2024-04-13T22:03:10.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A9TZ0YDYylxzjxNkOUfHdWC70NM= 2024-04-13T22:03:10.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A9TZ0YDYylxzjxNkOUfHdWC70NM=', 'Content-Length': '0'} 2024-04-13T22:03:10.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb5248d84113ac02-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:03:10.722 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.722 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.722 DEBUG:boto:Method: GET 2024-04-13T22:03:10.722 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:03:10.722 DEBUG:boto:Data: 2024-04-13T22:03:10.722 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.722 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.722 DEBUG:boto:Params: {} 2024-04-13T22:03:10.722 DEBUG:boto:Token: None 2024-04-13T22:03:10.723 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1 2024-04-13T22:03:10.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ijmgloZ7G9kUPsXN9q33TG9CnBM= 2024-04-13T22:03:10.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ijmgloZ7G9kUPsXN9q33TG9CnBM=', 'Content-Length': '0'} 2024-04-13T22:03:10.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003782a5b7e695628-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.725 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:03:10.725 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:03:10.725 DEBUG:boto:Method: GET 2024-04-13T22:03:10.725 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:03:10.725 DEBUG:boto:Data: 2024-04-13T22:03:10.725 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.725 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.725 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.726 DEBUG:boto:Params: {} 2024-04-13T22:03:10.726 DEBUG:boto:Token: None 2024-04-13T22:03:10.726 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj1 2024-04-13T22:03:10.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ijmgloZ7G9kUPsXN9q33TG9CnBM= 2024-04-13T22:03:10.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ijmgloZ7G9kUPsXN9q33TG9CnBM=', 'Content-Length': '0'} 2024-04-13T22:03:10.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a960b24f71591cd-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.728 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.728 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.728 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.728 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.728 DEBUG:boto:Method: GET 2024-04-13T22:03:10.728 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.728 DEBUG:boto:Data: 2024-04-13T22:03:10.728 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.728 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.729 DEBUG:boto:Params: {} 2024-04-13T22:03:10.729 DEBUG:boto:Token: None 2024-04-13T22:03:10.729 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c47bcf642851ec-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.731 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.732 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.732 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.732 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.732 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.732 DEBUG:boto:Method: GET 2024-04-13T22:03:10.732 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:03:10.732 DEBUG:boto:Data: 2024-04-13T22:03:10.732 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.732 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.732 DEBUG:boto:Params: {} 2024-04-13T22:03:10.733 DEBUG:boto:Token: None 2024-04-13T22:03:10.733 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.733 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db92a5f683548e77-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.735 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:03:10.735 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.736 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.736 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.736 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.736 DEBUG:boto:Method: GET 2024-04-13T22:03:10.736 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.736 DEBUG:boto:Data: 2024-04-13T22:03:10.736 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.736 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.736 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.736 DEBUG:boto:Params: {} 2024-04-13T22:03:10.736 DEBUG:boto:Token: None 2024-04-13T22:03:10.737 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k= 2024-04-13T22:03:10.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k=', 'Content-Length': '0'} 2024-04-13T22:03:10.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc2d400a2ce3ada-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.782 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.782 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.783 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.783 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.783 DEBUG:boto:Method: GET 2024-04-13T22:03:10.783 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.783 DEBUG:boto:Data: 2024-04-13T22:03:10.783 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.783 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.783 DEBUG:boto:Params: {} 2024-04-13T22:03:10.783 DEBUG:boto:Token: None 2024-04-13T22:03:10.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k= 2024-04-13T22:03:10.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k=', 'Content-Length': '0'} 2024-04-13T22:03:10.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ed6c2ffcfddbf48-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.785 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.786 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.786 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.786 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.786 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.786 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.786 DEBUG:boto:Data: 2024-04-13T22:03:10.786 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.786 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.787 DEBUG:boto:Params: {} 2024-04-13T22:03:10.787 DEBUG:boto:Token: None 2024-04-13T22:03:10.787 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.787 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iPU4A1tfxUWRjKbg4Z2AiEaCxpc= 2024-04-13T22:03:10.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iPU4A1tfxUWRjKbg4Z2AiEaCxpc=', 'Content-Length': '0'} 2024-04-13T22:03:10.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000701ec956a9ef31c0-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.789 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.789 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.789 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.789 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.789 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.789 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.789 DEBUG:boto:Data: 2024-04-13T22:03:10.789 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.790 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.790 DEBUG:boto:Params: {} 2024-04-13T22:03:10.790 DEBUG:boto:Token: None 2024-04-13T22:03:10.790 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.790 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iPU4A1tfxUWRjKbg4Z2AiEaCxpc= 2024-04-13T22:03:10.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iPU4A1tfxUWRjKbg4Z2AiEaCxpc=', 'Content-Length': '0'} 2024-04-13T22:03:10.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb0743a567ba9f15-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:03:10.792 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.792 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.792 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.793 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.793 DEBUG:boto:Method: GET 2024-04-13T22:03:10.793 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.793 DEBUG:boto:Data: 2024-04-13T22:03:10.793 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.793 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.793 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.793 DEBUG:boto:Params: {} 2024-04-13T22:03:10.793 DEBUG:boto:Token: None 2024-04-13T22:03:10.793 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.794 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k= 2024-04-13T22:03:10.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k=', 'Content-Length': '0'} 2024-04-13T22:03:10.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a3152af4f941eef-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.795 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.796 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.796 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.796 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.796 DEBUG:boto:Method: GET 2024-04-13T22:03:10.796 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.796 DEBUG:boto:Data: 2024-04-13T22:03:10.796 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.796 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.796 DEBUG:boto:Params: {} 2024-04-13T22:03:10.796 DEBUG:boto:Token: None 2024-04-13T22:03:10.797 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:03:10.797 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k= 2024-04-13T22:03:10.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e/TTSBTBNlRR4hfRXiRT2tp7V9k=', 'Content-Length': '0'} 2024-04-13T22:03:10.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd1165eb4e326bd2-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.799 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.799 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.799 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.799 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.799 DEBUG:boto:Method: GET 2024-04-13T22:03:10.799 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.799 DEBUG:boto:Data: 2024-04-13T22:03:10.799 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.800 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.800 DEBUG:boto:Params: {} 2024-04-13T22:03:10.800 DEBUG:boto:Token: None 2024-04-13T22:03:10.800 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1262dbf5e754cd1-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.802 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.803 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.803 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.803 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.803 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.804 DEBUG:boto:Method: GET 2024-04-13T22:03:10.804 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.804 DEBUG:boto:Data: 2024-04-13T22:03:10.804 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.804 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.804 DEBUG:boto:Params: {} 2024-04-13T22:03:10.804 DEBUG:boto:Token: None 2024-04-13T22:03:10.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000898ffcf87214a1fa-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.850 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.851 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.854 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.854 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.854 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:03:10.854 DEBUG:boto:Data: 2024-04-13T22:03:10.854 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.854 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.854 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.854 DEBUG:boto:Params: {} 2024-04-13T22:03:10.854 DEBUG:boto:Token: None 2024-04-13T22:03:10.855 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2 2024-04-13T22:03:10.855 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X3OLBdDGwMRo13IcFdGEx1T5sVQ= 2024-04-13T22:03:10.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X3OLBdDGwMRo13IcFdGEx1T5sVQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dfb63d84ec40d14-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.856 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.856 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.856 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.857 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:03:10.857 DEBUG:boto:Data: 2024-04-13T22:03:10.857 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.857 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.857 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.857 DEBUG:boto:Params: {} 2024-04-13T22:03:10.857 DEBUG:boto:Token: None 2024-04-13T22:03:10.857 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2 2024-04-13T22:03:10.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X3OLBdDGwMRo13IcFdGEx1T5sVQ= 2024-04-13T22:03:10.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X3OLBdDGwMRo13IcFdGEx1T5sVQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef0f18b75406d2e4-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:03:10.860 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.860 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.860 DEBUG:boto:Method: GET 2024-04-13T22:03:10.860 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:03:10.860 DEBUG:boto:Data: 2024-04-13T22:03:10.860 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.860 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.860 DEBUG:boto:Params: {} 2024-04-13T22:03:10.860 DEBUG:boto:Token: None 2024-04-13T22:03:10.861 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2 2024-04-13T22:03:10.861 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WjkdlS2oRWAeIpKaIi+pPg1FZbQ= 2024-04-13T22:03:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WjkdlS2oRWAeIpKaIi+pPg1FZbQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd2e6778ee60a9cf-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.862 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:03:10.863 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:03:10.863 DEBUG:boto:Method: GET 2024-04-13T22:03:10.863 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:03:10.863 DEBUG:boto:Data: 2024-04-13T22:03:10.863 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.863 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.863 DEBUG:boto:Params: {} 2024-04-13T22:03:10.863 DEBUG:boto:Token: None 2024-04-13T22:03:10.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj2 2024-04-13T22:03:10.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WjkdlS2oRWAeIpKaIi+pPg1FZbQ= 2024-04-13T22:03:10.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WjkdlS2oRWAeIpKaIi+pPg1FZbQ=', 'Content-Length': '0'} 2024-04-13T22:03:10.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002401aee8c4fe30ef-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.865 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.866 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.866 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.866 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.866 DEBUG:boto:Method: GET 2024-04-13T22:03:10.866 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.866 DEBUG:boto:Data: 2024-04-13T22:03:10.866 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.866 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.866 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.866 DEBUG:boto:Params: {} 2024-04-13T22:03:10.867 DEBUG:boto:Token: None 2024-04-13T22:03:10.867 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066952ba77c770662-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.869 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.869 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.869 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.869 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.870 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.870 DEBUG:boto:Method: GET 2024-04-13T22:03:10.870 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:03:10.870 DEBUG:boto:Data: 2024-04-13T22:03:10.870 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.870 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.870 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.870 DEBUG:boto:Params: {} 2024-04-13T22:03:10.870 DEBUG:boto:Token: None 2024-04-13T22:03:10.870 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd2c3c5e71b131c2-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.873 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:03:10.873 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.873 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.874 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.874 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.874 DEBUG:boto:Method: GET 2024-04-13T22:03:10.874 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.874 DEBUG:boto:Data: 2024-04-13T22:03:10.874 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.874 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.874 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.874 DEBUG:boto:Params: {} 2024-04-13T22:03:10.874 DEBUG:boto:Token: None 2024-04-13T22:03:10.874 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M= 2024-04-13T22:03:10.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M=', 'Content-Length': '0'} 2024-04-13T22:03:10.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006769f1c41b58e0f9-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.877 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.877 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.877 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.877 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.877 DEBUG:boto:Method: GET 2024-04-13T22:03:10.877 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.877 DEBUG:boto:Data: 2024-04-13T22:03:10.877 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.878 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.878 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.878 DEBUG:boto:Params: {} 2024-04-13T22:03:10.878 DEBUG:boto:Token: None 2024-04-13T22:03:10.878 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.878 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M= 2024-04-13T22:03:10.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M=', 'Content-Length': '0'} 2024-04-13T22:03:10.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091594139b02fa7c0-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.880 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.880 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.880 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.880 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.880 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.880 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.881 DEBUG:boto:Data: 2024-04-13T22:03:10.881 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.881 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.881 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.881 DEBUG:boto:Params: {} 2024-04-13T22:03:10.881 DEBUG:boto:Token: None 2024-04-13T22:03:10.881 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.882 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N5WBP/cIvF6UTdOhyw8BilVCKLs= 2024-04-13T22:03:10.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N5WBP/cIvF6UTdOhyw8BilVCKLs=', 'Content-Length': '0'} 2024-04-13T22:03:10.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc9eadbc58ce72f-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.883 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.883 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.883 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.883 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.884 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.884 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.884 DEBUG:boto:Data: 2024-04-13T22:03:10.884 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.884 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.884 DEBUG:boto:Params: {} 2024-04-13T22:03:10.884 DEBUG:boto:Token: None 2024-04-13T22:03:10.884 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N5WBP/cIvF6UTdOhyw8BilVCKLs= 2024-04-13T22:03:10.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N5WBP/cIvF6UTdOhyw8BilVCKLs=', 'Content-Length': '0'} 2024-04-13T22:03:10.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef7704f09ff246c8-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:03:10.887 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.887 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.887 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.887 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.887 DEBUG:boto:Method: GET 2024-04-13T22:03:10.887 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.887 DEBUG:boto:Data: 2024-04-13T22:03:10.888 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.888 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.888 DEBUG:boto:Params: {} 2024-04-13T22:03:10.888 DEBUG:boto:Token: None 2024-04-13T22:03:10.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M= 2024-04-13T22:03:10.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M=', 'Content-Length': '0'} 2024-04-13T22:03:10.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008004c1caf08cf2e3-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.930 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.931 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.931 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.931 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.931 DEBUG:boto:Method: GET 2024-04-13T22:03:10.931 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.931 DEBUG:boto:Data: 2024-04-13T22:03:10.931 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.931 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.931 DEBUG:boto:Params: {} 2024-04-13T22:03:10.932 DEBUG:boto:Token: None 2024-04-13T22:03:10.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:03:10.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M= 2024-04-13T22:03:10.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:17LS/kK5O+O3P2Ok3qC1ME6s62M=', 'Content-Length': '0'} 2024-04-13T22:03:10.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001790c977e6f2fe82-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.934 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.934 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.934 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.934 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.934 DEBUG:boto:Method: GET 2024-04-13T22:03:10.935 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.935 DEBUG:boto:Data: 2024-04-13T22:03:10.935 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.935 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.935 DEBUG:boto:Params: {} 2024-04-13T22:03:10.935 DEBUG:boto:Token: None 2024-04-13T22:03:10.935 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a769964b6ce3f1-00661b011e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.937 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.938 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.938 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.938 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.938 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.938 DEBUG:boto:Method: GET 2024-04-13T22:03:10.938 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.938 DEBUG:boto:Data: 2024-04-13T22:03:10.938 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.939 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.939 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.939 DEBUG:boto:Params: {} 2024-04-13T22:03:10.939 DEBUG:boto:Token: None 2024-04-13T22:03:10.939 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:10.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:10.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:10.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021603a92e3ca8b60-00661b011e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.941 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:10.941 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.942 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.942 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.942 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:03:10.942 DEBUG:boto:Data: 2024-04-13T22:03:10.942 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.942 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.942 DEBUG:boto:Params: {} 2024-04-13T22:03:10.942 DEBUG:boto:Token: None 2024-04-13T22:03:10.942 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3 2024-04-13T22:03:10.943 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uxjJs6oGkUOmGKKHkuyIphz31vI= 2024-04-13T22:03:10.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uxjJs6oGkUOmGKKHkuyIphz31vI=', 'Content-Length': '0'} 2024-04-13T22:03:10.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5276878c7dd891b-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.944 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.944 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.945 DEBUG:boto:Method: HEAD 2024-04-13T22:03:10.945 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:03:10.945 DEBUG:boto:Data: 2024-04-13T22:03:10.945 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.945 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.945 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.945 DEBUG:boto:Params: {} 2024-04-13T22:03:10.945 DEBUG:boto:Token: None 2024-04-13T22:03:10.945 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3 2024-04-13T22:03:10.945 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uxjJs6oGkUOmGKKHkuyIphz31vI= 2024-04-13T22:03:10.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uxjJs6oGkUOmGKKHkuyIphz31vI=', 'Content-Length': '0'} 2024-04-13T22:03:10.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015c2cd9cf9871e5a-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:03:10.947 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.948 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.948 DEBUG:boto:Method: GET 2024-04-13T22:03:10.948 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:03:10.948 DEBUG:boto:Data: 2024-04-13T22:03:10.948 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.948 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.948 DEBUG:boto:Params: {} 2024-04-13T22:03:10.948 DEBUG:boto:Token: None 2024-04-13T22:03:10.948 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3 2024-04-13T22:03:10.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JsB4AMq+uCFiwsx+LwgLD6gn9g= 2024-04-13T22:03:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JsB4AMq+uCFiwsx+LwgLD6gn9g=', 'Content-Length': '0'} 2024-04-13T22:03:10.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000472c150821827be9-00661b011e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.951 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:03:10.951 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:03:10.951 DEBUG:boto:Method: GET 2024-04-13T22:03:10.951 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:03:10.951 DEBUG:boto:Data: 2024-04-13T22:03:10.951 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:10.951 DEBUG:boto:Port: 8000 2024-04-13T22:03:10.951 DEBUG:boto:Params: {} 2024-04-13T22:03:10.951 DEBUG:boto:Token: None 2024-04-13T22:03:10.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/obj3 2024-04-13T22:03:10.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JsB4AMq+uCFiwsx+LwgLD6gn9g= 2024-04-13T22:03:10.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JsB4AMq+uCFiwsx+LwgLD6gn9g=', 'Content-Length': '0'} 2024-04-13T22:03:10.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4e51746452538be-00661b011e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:10.998 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:10.998 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:10.999 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.999 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.999 DEBUG:boto:Method: GET 2024-04-13T22:03:10.999 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:03:10.999 DEBUG:boto:Data: 2024-04-13T22:03:10.999 DEBUG:boto:Headers: {} 2024-04-13T22:03:10.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:10.999 DEBUG:boto:Port: 8001 2024-04-13T22:03:10.999 DEBUG:boto:Params: {} 2024-04-13T22:03:10.999 DEBUG:boto:Token: None 2024-04-13T22:03:11.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:10 GMT /vwnfns-49/ 2024-04-13T22:03:11.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc= 2024-04-13T22:03:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uqazxz6iebmOmZcUvhkIK1vsRoc=', 'Content-Length': '0'} 2024-04-13T22:03:11.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c39ae110cfa0b9-00661b011f-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.002 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:11.003 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.003 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.003 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:11.003 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:03:11.003 DEBUG:boto:Method: GET 2024-04-13T22:03:11.003 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:03:11.003 DEBUG:boto:Data: 2024-04-13T22:03:11.003 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.003 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.004 DEBUG:boto:Params: {} 2024-04-13T22:03:11.004 DEBUG:boto:Token: None 2024-04-13T22:03:11.004 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los= 2024-04-13T22:03:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los=', 'Content-Length': '0'} 2024-04-13T22:03:11.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f7c57bf1ad86bbe-00661b011f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.006 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:11.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:03:11.007 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.007 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.007 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.007 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.007 DEBUG:boto:Method: GET 2024-04-13T22:03:11.007 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.007 DEBUG:boto:Data: 2024-04-13T22:03:11.007 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.008 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.008 DEBUG:boto:Params: {} 2024-04-13T22:03:11.008 DEBUG:boto:Token: None 2024-04-13T22:03:11.008 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE= 2024-04-13T22:03:11.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE=', 'Content-Length': '0'} 2024-04-13T22:03:11.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed4a5e69a5d30c87-00661b011f-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.010 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.010 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.010 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.010 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.010 DEBUG:boto:Method: GET 2024-04-13T22:03:11.010 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.011 DEBUG:boto:Data: 2024-04-13T22:03:11.011 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.011 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.011 DEBUG:boto:Params: {} 2024-04-13T22:03:11.011 DEBUG:boto:Token: None 2024-04-13T22:03:11.011 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE= 2024-04-13T22:03:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE=', 'Content-Length': '0'} 2024-04-13T22:03:11.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c54e206c0335a25-00661b011f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.013 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.014 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.014 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.014 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.014 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.014 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.014 DEBUG:boto:Data: 2024-04-13T22:03:11.014 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.014 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.014 DEBUG:boto:Params: {} 2024-04-13T22:03:11.014 DEBUG:boto:Token: None 2024-04-13T22:03:11.015 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.015 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:74MKiYTGlePUlHnQJpVWQz4Z7YQ= 2024-04-13T22:03:11.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:74MKiYTGlePUlHnQJpVWQz4Z7YQ=', 'Content-Length': '0'} 2024-04-13T22:03:11.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ef87f529cfdaf01-00661b011f-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.016 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.017 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.017 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.017 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.017 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.017 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.017 DEBUG:boto:Data: 2024-04-13T22:03:11.017 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.017 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.017 DEBUG:boto:Params: {} 2024-04-13T22:03:11.017 DEBUG:boto:Token: None 2024-04-13T22:03:11.018 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.018 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:74MKiYTGlePUlHnQJpVWQz4Z7YQ= 2024-04-13T22:03:11.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:74MKiYTGlePUlHnQJpVWQz4Z7YQ=', 'Content-Length': '0'} 2024-04-13T22:03:11.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4e856214f3f2300-00661b011f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:03:11.020 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.020 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.020 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.020 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.020 DEBUG:boto:Method: GET 2024-04-13T22:03:11.020 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.020 DEBUG:boto:Data: 2024-04-13T22:03:11.020 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.021 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.021 DEBUG:boto:Params: {} 2024-04-13T22:03:11.021 DEBUG:boto:Token: None 2024-04-13T22:03:11.021 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE= 2024-04-13T22:03:11.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE=', 'Content-Length': '0'} 2024-04-13T22:03:11.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094bde782ac5f1755-00661b011f-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.023 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.023 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.024 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.024 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.024 DEBUG:boto:Method: GET 2024-04-13T22:03:11.024 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.024 DEBUG:boto:Data: 2024-04-13T22:03:11.024 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.024 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.025 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.025 DEBUG:boto:Params: {} 2024-04-13T22:03:11.025 DEBUG:boto:Token: None 2024-04-13T22:03:11.025 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:03:11.025 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE= 2024-04-13T22:03:11.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKZZrNiDiJiTT9FRfAsMc0SCWpE=', 'Content-Length': '0'} 2024-04-13T22:03:11.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d034b7f12987df4-00661b011f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.027 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.028 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.028 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.028 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.028 DEBUG:boto:Method: GET 2024-04-13T22:03:11.028 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.028 DEBUG:boto:Data: 2024-04-13T22:03:11.029 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.029 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.029 DEBUG:boto:Params: {} 2024-04-13T22:03:11.029 DEBUG:boto:Token: None 2024-04-13T22:03:11.029 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los= 2024-04-13T22:03:11.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los=', 'Content-Length': '0'} 2024-04-13T22:03:11.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3ba8eba4afa6d2-00661b011f-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.074 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:11.074 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.075 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.075 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.075 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.075 DEBUG:boto:Method: GET 2024-04-13T22:03:11.075 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.075 DEBUG:boto:Data: 2024-04-13T22:03:11.075 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.075 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.075 DEBUG:boto:Params: {} 2024-04-13T22:03:11.075 DEBUG:boto:Token: None 2024-04-13T22:03:11.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los= 2024-04-13T22:03:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los=', 'Content-Length': '0'} 2024-04-13T22:03:11.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b731183293959f31-00661b011f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.078 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:11.079 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.079 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.079 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.079 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:03:11.079 DEBUG:boto:Data: 2024-04-13T22:03:11.079 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.079 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.079 DEBUG:boto:Params: {} 2024-04-13T22:03:11.079 DEBUG:boto:Token: None 2024-04-13T22:03:11.080 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4 2024-04-13T22:03:11.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cLDOBedUcNM6A9PlQI6g0zFFDqI= 2024-04-13T22:03:11.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cLDOBedUcNM6A9PlQI6g0zFFDqI=', 'Content-Length': '0'} 2024-04-13T22:03:11.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f272f5b668fb3514-00661b011f-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.081 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.081 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.081 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.081 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:03:11.081 DEBUG:boto:Data: 2024-04-13T22:03:11.082 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.082 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.082 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.082 DEBUG:boto:Params: {} 2024-04-13T22:03:11.082 DEBUG:boto:Token: None 2024-04-13T22:03:11.082 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4 2024-04-13T22:03:11.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cLDOBedUcNM6A9PlQI6g0zFFDqI= 2024-04-13T22:03:11.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cLDOBedUcNM6A9PlQI6g0zFFDqI=', 'Content-Length': '0'} 2024-04-13T22:03:11.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004e7d65f0db6db24-00661b011f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:03:11.084 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.084 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.084 DEBUG:boto:Method: GET 2024-04-13T22:03:11.084 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:03:11.085 DEBUG:boto:Data: 2024-04-13T22:03:11.085 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.085 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.085 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.085 DEBUG:boto:Params: {} 2024-04-13T22:03:11.085 DEBUG:boto:Token: None 2024-04-13T22:03:11.085 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4 2024-04-13T22:03:11.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qny8SdqZUzt7eJJ93lNx+2x4mGA= 2024-04-13T22:03:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qny8SdqZUzt7eJJ93lNx+2x4mGA=', 'Content-Length': '0'} 2024-04-13T22:03:11.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000954b5507b44c10fe-00661b011f-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.087 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:03:11.087 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:03:11.087 DEBUG:boto:Method: GET 2024-04-13T22:03:11.087 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:03:11.087 DEBUG:boto:Data: 2024-04-13T22:03:11.087 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.088 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.088 DEBUG:boto:Params: {} 2024-04-13T22:03:11.088 DEBUG:boto:Token: None 2024-04-13T22:03:11.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj4 2024-04-13T22:03:11.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Qny8SdqZUzt7eJJ93lNx+2x4mGA= 2024-04-13T22:03:11.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Qny8SdqZUzt7eJJ93lNx+2x4mGA=', 'Content-Length': '0'} 2024-04-13T22:03:11.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b2d59f37024ee98-00661b011f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.090 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.090 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.090 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.090 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.090 DEBUG:boto:Method: GET 2024-04-13T22:03:11.090 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.090 DEBUG:boto:Data: 2024-04-13T22:03:11.091 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.091 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.091 DEBUG:boto:Params: {} 2024-04-13T22:03:11.091 DEBUG:boto:Token: None 2024-04-13T22:03:11.091 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los= 2024-04-13T22:03:11.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los=', 'Content-Length': '0'} 2024-04-13T22:03:11.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c92700f39ac63fb4-00661b011f-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.093 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:11.094 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.094 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.094 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.094 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.094 DEBUG:boto:Method: GET 2024-04-13T22:03:11.094 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:03:11.094 DEBUG:boto:Data: 2024-04-13T22:03:11.094 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.095 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.095 DEBUG:boto:Params: {} 2024-04-13T22:03:11.095 DEBUG:boto:Token: None 2024-04-13T22:03:11.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los= 2024-04-13T22:03:11.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqH/xXjc01WTlIjtlGOwMwD4Los=', 'Content-Length': '0'} 2024-04-13T22:03:11.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f26b877b197d0d3-00661b011f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.097 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:03:11.098 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-49 zones={a2, a1} 2024-04-13T22:03:11.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vwnfns-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vwnfns-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.098 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vwnfns-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:03:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:11.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vwnfns-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vwnfns-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.211 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vwnfns-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:03:11.261 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:11.262 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:11.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vwnfns-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vwnfns-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.327 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vwnfns-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:03:11.376 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:11.376 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:11.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vwnfns-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vwnfns-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:03:11.445 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vwnfns-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:03:11.494 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:11.494 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:11.560 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:11.560 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:11.560 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.560 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:03:11.560 DEBUG:boto:Data: 2024-04-13T22:03:11.560 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.560 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.560 DEBUG:boto:Params: {} 2024-04-13T22:03:11.561 DEBUG:boto:Token: None 2024-04-13T22:03:11.561 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/ 2024-04-13T22:03:11.561 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg= 2024-04-13T22:03:11.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg=', 'Content-Length': '0'} 2024-04-13T22:03:11.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d08852ca09c9361-00661b011f-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.564 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:03:11.564 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:03:11.564 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.564 DEBUG:boto:Path: /vwnfns-48/obj5 2024-04-13T22:03:11.564 DEBUG:boto:Data: 2024-04-13T22:03:11.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.564 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.564 DEBUG:boto:Params: {} 2024-04-13T22:03:11.564 DEBUG:boto:Token: None 2024-04-13T22:03:11.565 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/obj5 2024-04-13T22:03:11.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cYvFVCWJSI+Gk74QA8Z2d+vXBdw= 2024-04-13T22:03:11.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cYvFVCWJSI+Gk74QA8Z2d+vXBdw='} 2024-04-13T22:03:11.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b209a321e54651a3-00661b011f-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.607 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:11.607 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:11.607 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.607 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:03:11.607 DEBUG:boto:Data: 2024-04-13T22:03:11.607 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.607 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.607 DEBUG:boto:Params: {} 2024-04-13T22:03:11.607 DEBUG:boto:Token: None 2024-04-13T22:03:11.608 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/ 2024-04-13T22:03:11.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg= 2024-04-13T22:03:11.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg=', 'Content-Length': '0'} 2024-04-13T22:03:11.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fbb3ce81c4c0de9-00661b011f-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.610 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:03:11.610 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:03:11.610 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.610 DEBUG:boto:Path: /vwnfns-48/obj6 2024-04-13T22:03:11.610 DEBUG:boto:Data: 2024-04-13T22:03:11.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.611 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.611 DEBUG:boto:Params: {} 2024-04-13T22:03:11.611 DEBUG:boto:Token: None 2024-04-13T22:03:11.611 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/obj6 2024-04-13T22:03:11.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MASetVX8/3JoG92d+TPy6Am9baU= 2024-04-13T22:03:11.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MASetVX8/3JoG92d+TPy6Am9baU='} 2024-04-13T22:03:11.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbabc494a542e65b-00661b011f-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.615 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:11.616 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:11.616 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.616 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:03:11.616 DEBUG:boto:Data: 2024-04-13T22:03:11.616 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.616 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.616 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.616 DEBUG:boto:Params: {} 2024-04-13T22:03:11.616 DEBUG:boto:Token: None 2024-04-13T22:03:11.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/ 2024-04-13T22:03:11.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg= 2024-04-13T22:03:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg=', 'Content-Length': '0'} 2024-04-13T22:03:11.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062a6bfd6585fa34a-00661b011f-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.619 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:03:11.619 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:03:11.619 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.619 DEBUG:boto:Path: /vwnfns-48/obj7 2024-04-13T22:03:11.619 DEBUG:boto:Data: 2024-04-13T22:03:11.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.620 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.620 DEBUG:boto:Params: {} 2024-04-13T22:03:11.620 DEBUG:boto:Token: None 2024-04-13T22:03:11.620 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/obj7 2024-04-13T22:03:11.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R1QUgKlpRskb9UR7sXsQtE3mVbI= 2024-04-13T22:03:11.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R1QUgKlpRskb9UR7sXsQtE3mVbI='} 2024-04-13T22:03:11.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d76e2d02a6e4f955-00661b011f-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.663 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:03:11.663 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:03:11.663 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.663 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:03:11.663 DEBUG:boto:Data: 2024-04-13T22:03:11.663 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.663 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.663 DEBUG:boto:Params: {} 2024-04-13T22:03:11.663 DEBUG:boto:Token: None 2024-04-13T22:03:11.664 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/ 2024-04-13T22:03:11.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg= 2024-04-13T22:03:11.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XYrqxyJEcEiHYRrePu7MdgB0TOg=', 'Content-Length': '0'} 2024-04-13T22:03:11.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000426617f96fc8a3ad-00661b011f-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.666 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:03:11.666 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:03:11.666 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.666 DEBUG:boto:Path: /vwnfns-48/obj8 2024-04-13T22:03:11.666 DEBUG:boto:Data: 2024-04-13T22:03:11.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:03:11.667 DEBUG:boto:Port: 8000 2024-04-13T22:03:11.667 DEBUG:boto:Params: {} 2024-04-13T22:03:11.667 DEBUG:boto:Token: None 2024-04-13T22:03:11.667 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-48/obj8 2024-04-13T22:03:11.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:koKXebF7ataG20zfrRQnTtnHeZY= 2024-04-13T22:03:11.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:koKXebF7ataG20zfrRQnTtnHeZY='} 2024-04-13T22:03:11.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050f64f52f1cbc0a2-00661b011f-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.671 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.671 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.671 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.671 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:03:11.672 DEBUG:boto:Data: 2024-04-13T22:03:11.672 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.672 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.672 DEBUG:boto:Params: {} 2024-04-13T22:03:11.672 DEBUG:boto:Token: None 2024-04-13T22:03:11.672 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM= 2024-04-13T22:03:11.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM=', 'Content-Length': '0'} 2024-04-13T22:03:11.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4957a3310c021c0-00661b011f-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.675 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:03:11.675 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:03:11.675 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.675 DEBUG:boto:Path: /vwnfns-49/obj5 2024-04-13T22:03:11.675 DEBUG:boto:Data: 2024-04-13T22:03:11.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.675 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.675 DEBUG:boto:Params: {} 2024-04-13T22:03:11.675 DEBUG:boto:Token: None 2024-04-13T22:03:11.676 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj5 2024-04-13T22:03:11.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G50feP/QWddZaKY4H0yLU1/wqYU= 2024-04-13T22:03:11.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G50feP/QWddZaKY4H0yLU1/wqYU='} 2024-04-13T22:03:11.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021dfa0b939e3c684-00661b011f-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.719 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.719 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.719 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.719 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:03:11.719 DEBUG:boto:Data: 2024-04-13T22:03:11.719 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.719 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.719 DEBUG:boto:Params: {} 2024-04-13T22:03:11.719 DEBUG:boto:Token: None 2024-04-13T22:03:11.720 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM= 2024-04-13T22:03:11.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM=', 'Content-Length': '0'} 2024-04-13T22:03:11.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000887c0e9a6f579fb3-00661b011f-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.722 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:03:11.722 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:03:11.722 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.722 DEBUG:boto:Path: /vwnfns-49/obj6 2024-04-13T22:03:11.723 DEBUG:boto:Data: 2024-04-13T22:03:11.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.723 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.723 DEBUG:boto:Params: {} 2024-04-13T22:03:11.723 DEBUG:boto:Token: None 2024-04-13T22:03:11.723 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj6 2024-04-13T22:03:11.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LoEnkHTa1A1saklu0lNnaGSX7u8= 2024-04-13T22:03:11.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LoEnkHTa1A1saklu0lNnaGSX7u8='} 2024-04-13T22:03:11.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003049cfdabab3997c-00661b011f-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.727 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.727 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.728 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.728 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:03:11.728 DEBUG:boto:Data: 2024-04-13T22:03:11.728 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.728 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.728 DEBUG:boto:Params: {} 2024-04-13T22:03:11.728 DEBUG:boto:Token: None 2024-04-13T22:03:11.728 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM= 2024-04-13T22:03:11.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM=', 'Content-Length': '0'} 2024-04-13T22:03:11.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8b9664244ee6164-00661b011f-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.731 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:03:11.731 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:03:11.731 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.731 DEBUG:boto:Path: /vwnfns-49/obj7 2024-04-13T22:03:11.732 DEBUG:boto:Data: 2024-04-13T22:03:11.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.732 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.732 DEBUG:boto:Params: {} 2024-04-13T22:03:11.732 DEBUG:boto:Token: None 2024-04-13T22:03:11.732 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj7 2024-04-13T22:03:11.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QIIkJQIapbOFM7RCmrhOOsDQosU= 2024-04-13T22:03:11.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QIIkJQIapbOFM7RCmrhOOsDQosU='} 2024-04-13T22:03:11.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d0c33cac4095021-00661b011f-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.775 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:03:11.775 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:03:11.775 DEBUG:boto:Method: HEAD 2024-04-13T22:03:11.775 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:03:11.775 DEBUG:boto:Data: 2024-04-13T22:03:11.775 DEBUG:boto:Headers: {} 2024-04-13T22:03:11.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.775 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.775 DEBUG:boto:Params: {} 2024-04-13T22:03:11.775 DEBUG:boto:Token: None 2024-04-13T22:03:11.776 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/ 2024-04-13T22:03:11.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM= 2024-04-13T22:03:11.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gG/368QWRLKB90t5fmATmEASspM=', 'Content-Length': '0'} 2024-04-13T22:03:11.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079b1ba581fbcec27-00661b011f-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.778 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:03:11.778 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:03:11.778 DEBUG:boto:Method: PUT 2024-04-13T22:03:11.778 DEBUG:boto:Path: /vwnfns-49/obj8 2024-04-13T22:03:11.778 DEBUG:boto:Data: 2024-04-13T22:03:11.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:03:11.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:03:11.779 DEBUG:boto:Port: 8001 2024-04-13T22:03:11.779 DEBUG:boto:Params: {} 2024-04-13T22:03:11.779 DEBUG:boto:Token: None 2024-04-13T22:03:11.779 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:03:11 GMT /vwnfns-49/obj8 2024-04-13T22:03:11.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IPDO0ORWlOZdHSrkir2fRsKF0as= 2024-04-13T22:03:11.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:03:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IPDO0ORWlOZdHSrkir2fRsKF0as='} 2024-04-13T22:03:11.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eca6cd70435b7740-00661b011f-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:03:11.783 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-48 2024-04-13T22:03:11.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:11.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:11.783 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:03:11.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:11.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:11.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:11.874+0000 7f392a0ac780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:11.874 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.33.6, , 00000000001.19.6, , 00000000001.67.6, , , 00000000001.22.6, , , ] 2024-04-13T22:03:11.874 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.39.6, 00000000002.33.1, 00000000004.23.6, 00000000002.48.1, 00000000004.71.6, 00000000002.13.1, 00000000002.41.1, 00000000004.32.6, 00000000002.39.1, 00000000002.37.1, 00000000002.13.1] 2024-04-13T22:03:16.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:16.875+0000 7f392a0ac780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:16.876 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.33.6, , 00000000001.19.6, , 00000000001.67.6, , , 00000000001.22.6, , , ] 2024-04-13T22:03:16.876 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.39.6, 00000000002.33.1, 00000000004.23.6, 00000000002.48.1, 00000000004.71.6, 00000000002.13.1, 00000000002.41.1, 00000000004.32.6, 00000000002.39.1, 00000000002.37.1, 00000000002.13.1] 2024-04-13T22:03:21.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:21.876+0000 7f392a0ac780 1 bucket sync caught up with source: 2024-04-13T22:03:21.877 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.39.6, 00000000002.33.1, 00000000004.23.6, 00000000002.48.1, 00000000004.71.6, 00000000002.13.1, 00000000002.41.1, 00000000004.32.6, 00000000002.39.1, 00000000002.37.1, 00000000002.13.1] 2024-04-13T22:03:21.877 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.39.6, 00000000002.33.1, 00000000004.23.6, 00000000002.48.1, 00000000004.71.6, 00000000002.13.1, 00000000002.41.1, 00000000004.32.6, 00000000002.39.1, 00000000002.37.1, 00000000002.13.1] 2024-04-13T22:03:21.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:21.876+0000 7f392a0ac780 0 bucket checkpoint complete 2024-04-13T22:03:21.892 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-48 2024-04-13T22:03:21.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:21.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:03:21.892 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:03:21.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:03:21.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:03:21.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:21.983+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:21.983 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:21.983 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:26.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:26.986+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:26.985 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:26.985 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:31.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:31.987+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:31.986 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:31.987 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:36.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:36.988+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:36.988 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:36.988 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:41.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:41.988+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:41.989 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000001.1723.6, , , ] 2024-04-13T22:03:41.989 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:46.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:46.990+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:46.990 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:46.991 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:51.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:51.991+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:51.992 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:51.992 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:56.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:03:56.992+0000 7f4437280780 1 waiting for incremental sync to catch up: 2024-04-13T22:03:56.993 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.1096.6, , 00000000001.2610.6, , 00000000001.537.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:03:56.993 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:04:01.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:01.995+0000 7f4437280780 1 bucket sync caught up with source: 2024-04-13T22:04:01.995 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:04:01.995 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1098.6, , 00000000002.2612.6, , 00000000002.539.6, , , 00000000002.1727.6, , , ] 2024-04-13T22:04:01.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:01.995+0000 7f4437280780 0 bucket checkpoint complete 2024-04-13T22:04:02.014 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-48 zones={a2, a1} 2024-04-13T22:04:02.014 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.015 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.015 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.015 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:04:02.015 DEBUG:boto:Data: 2024-04-13T22:04:02.015 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.015 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.015 DEBUG:boto:Params: {} 2024-04-13T22:04:02.016 DEBUG:boto:Token: None 2024-04-13T22:04:02.016 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0hZZ0/rc0b/Qg4I63Zc5ZPcXKY8= 2024-04-13T22:04:02.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0hZZ0/rc0b/Qg4I63Zc5ZPcXKY8=', 'Content-Length': '0'} 2024-04-13T22:04:02.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001289422918886f83-00661b0152-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.019 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.020 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.020 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.020 DEBUG:boto:Path: /vwnfns-48/ 2024-04-13T22:04:02.020 DEBUG:boto:Data: 2024-04-13T22:04:02.020 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.020 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.020 DEBUG:boto:Params: {} 2024-04-13T22:04:02.020 DEBUG:boto:Token: None 2024-04-13T22:04:02.021 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0hZZ0/rc0b/Qg4I63Zc5ZPcXKY8= 2024-04-13T22:04:02.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0hZZ0/rc0b/Qg4I63Zc5ZPcXKY8=', 'Content-Length': '0'} 2024-04-13T22:04:02.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047590e5f321a143d-00661b0152-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.024 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:04:02.024 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.024 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.024 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.024 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.024 DEBUG:boto:Method: GET 2024-04-13T22:04:02.024 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.024 DEBUG:boto:Data: 2024-04-13T22:04:02.024 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.025 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.025 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.025 DEBUG:boto:Params: {} 2024-04-13T22:04:02.025 DEBUG:boto:Token: None 2024-04-13T22:04:02.025 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/?versions 2024-04-13T22:04:02.025 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU= 2024-04-13T22:04:02.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU=', 'Content-Length': '0'} 2024-04-13T22:04:02.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016fd914c66a67ed8-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.028 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:04:02.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:04:02.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:04:02.030 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:04:02.030 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.030 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.030 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.030 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.030 DEBUG:boto:Method: GET 2024-04-13T22:04:02.030 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.030 DEBUG:boto:Data: 2024-04-13T22:04:02.030 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.031 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.031 DEBUG:boto:Params: {} 2024-04-13T22:04:02.031 DEBUG:boto:Token: None 2024-04-13T22:04:02.031 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/?versions 2024-04-13T22:04:02.031 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU= 2024-04-13T22:04:02.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU=', 'Content-Length': '0'} 2024-04-13T22:04:02.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d9f9cd73d65fcd4-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.034 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:02.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:02.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:02.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:02.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:04:02.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:04:02.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:04:02.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:04:02.035 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.035 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.035 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.035 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.035 DEBUG:boto:Method: GET 2024-04-13T22:04:02.035 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.035 DEBUG:boto:Data: 2024-04-13T22:04:02.035 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.036 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.036 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.036 DEBUG:boto:Params: {} 2024-04-13T22:04:02.036 DEBUG:boto:Token: None 2024-04-13T22:04:02.036 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/?versions 2024-04-13T22:04:02.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU= 2024-04-13T22:04:02.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU=', 'Content-Length': '0'} 2024-04-13T22:04:02.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054bf486ce15c1408-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.039 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.039 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.039 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.040 DEBUG:boto:path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.040 DEBUG:boto:auth_path=/vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.040 DEBUG:boto:Method: GET 2024-04-13T22:04:02.040 DEBUG:boto:Path: /vwnfns-48/?versions&max-keys=999 2024-04-13T22:04:02.040 DEBUG:boto:Data: 2024-04-13T22:04:02.040 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.040 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.040 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.040 DEBUG:boto:Params: {} 2024-04-13T22:04:02.040 DEBUG:boto:Token: None 2024-04-13T22:04:02.041 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/?versions 2024-04-13T22:04:02.041 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU= 2024-04-13T22:04:02.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VfPM61BEBomZvkwDRhJbgwD+5aU=', 'Content-Length': '0'} 2024-04-13T22:04:02.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e709f92ce38b724-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.043 DEBUG:boto:b'vwnfns-48999falseobj1nulltrue2024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:02.044 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.044 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.044 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.044 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.044 DEBUG:boto:Method: GET 2024-04-13T22:04:02.044 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.044 DEBUG:boto:Data: 2024-04-13T22:04:02.044 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.044 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.044 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.045 DEBUG:boto:Params: {} 2024-04-13T22:04:02.045 DEBUG:boto:Token: None 2024-04-13T22:04:02.045 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc= 2024-04-13T22:04:02.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc=', 'Content-Length': '0'} 2024-04-13T22:04:02.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034683117773df058-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.090 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.091 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.091 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.091 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.091 DEBUG:boto:Method: GET 2024-04-13T22:04:02.091 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.091 DEBUG:boto:Data: 2024-04-13T22:04:02.091 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.091 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.091 DEBUG:boto:Params: {} 2024-04-13T22:04:02.092 DEBUG:boto:Token: None 2024-04-13T22:04:02.092 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc= 2024-04-13T22:04:02.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc=', 'Content-Length': '0'} 2024-04-13T22:04:02.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085b7c8a8b2d08ee8-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.094 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.094 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.095 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.095 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.095 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.095 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.095 DEBUG:boto:Data: 2024-04-13T22:04:02.095 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.095 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.095 DEBUG:boto:Params: {} 2024-04-13T22:04:02.095 DEBUG:boto:Token: None 2024-04-13T22:04:02.095 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y7gQ6kfaLu58Oz0+C9MKZATDxWI= 2024-04-13T22:04:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y7gQ6kfaLu58Oz0+C9MKZATDxWI=', 'Content-Length': '0'} 2024-04-13T22:04:02.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005157ca116fc8b797-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.098 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.098 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.098 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.098 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.098 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.098 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.098 DEBUG:boto:Data: 2024-04-13T22:04:02.098 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.099 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.099 DEBUG:boto:Params: {} 2024-04-13T22:04:02.099 DEBUG:boto:Token: None 2024-04-13T22:04:02.099 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y7gQ6kfaLu58Oz0+C9MKZATDxWI= 2024-04-13T22:04:02.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y7gQ6kfaLu58Oz0+C9MKZATDxWI=', 'Content-Length': '0'} 2024-04-13T22:04:02.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000978d9e91c716f333-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:02.101 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.101 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.101 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.101 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.101 DEBUG:boto:Method: GET 2024-04-13T22:04:02.102 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.102 DEBUG:boto:Data: 2024-04-13T22:04:02.102 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.102 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.102 DEBUG:boto:Params: {} 2024-04-13T22:04:02.102 DEBUG:boto:Token: None 2024-04-13T22:04:02.102 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc= 2024-04-13T22:04:02.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc=', 'Content-Length': '0'} 2024-04-13T22:04:02.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6985b323eb7fc95-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.104 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.105 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.105 DEBUG:boto:path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.105 DEBUG:boto:auth_path=/vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.105 DEBUG:boto:Method: GET 2024-04-13T22:04:02.105 DEBUG:boto:Path: /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.105 DEBUG:boto:Data: 2024-04-13T22:04:02.105 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.105 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.105 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.105 DEBUG:boto:Params: {} 2024-04-13T22:04:02.105 DEBUG:boto:Token: None 2024-04-13T22:04:02.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1?versionId=null 2024-04-13T22:04:02.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc= 2024-04-13T22:04:02.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tE/hOsBIiKWkEC5JKz2J1TCzELc=', 'Content-Length': '0'} 2024-04-13T22:04:02.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0af53f1eaa62e70-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.108 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.108 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.108 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.108 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.108 DEBUG:boto:Method: GET 2024-04-13T22:04:02.109 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.109 DEBUG:boto:Data: 2024-04-13T22:04:02.109 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.109 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.109 DEBUG:boto:Params: {} 2024-04-13T22:04:02.109 DEBUG:boto:Token: None 2024-04-13T22:04:02.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c67be6b5c6e2ed0b-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.112 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.112 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.112 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.112 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.112 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.113 DEBUG:boto:Method: GET 2024-04-13T22:04:02.113 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.113 DEBUG:boto:Data: 2024-04-13T22:04:02.113 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.113 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.113 DEBUG:boto:Params: {} 2024-04-13T22:04:02.113 DEBUG:boto:Token: None 2024-04-13T22:04:02.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30df618e6fbb9ac-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.116 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.116 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.116 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.116 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.116 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:04:02.117 DEBUG:boto:Data: 2024-04-13T22:04:02.117 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.117 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.117 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.117 DEBUG:boto:Params: {} 2024-04-13T22:04:02.117 DEBUG:boto:Token: None 2024-04-13T22:04:02.117 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1 2024-04-13T22:04:02.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:759WQcUSSs7yl5u4oB/tohPwEdQ= 2024-04-13T22:04:02.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:759WQcUSSs7yl5u4oB/tohPwEdQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c41e52923513b9a-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.119 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.119 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.119 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.119 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:04:02.119 DEBUG:boto:Data: 2024-04-13T22:04:02.119 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.120 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.120 DEBUG:boto:Params: {} 2024-04-13T22:04:02.120 DEBUG:boto:Token: None 2024-04-13T22:04:02.120 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1 2024-04-13T22:04:02.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:759WQcUSSs7yl5u4oB/tohPwEdQ= 2024-04-13T22:04:02.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:759WQcUSSs7yl5u4oB/tohPwEdQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1758da353780ceb-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:02.122 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.122 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.122 DEBUG:boto:Method: GET 2024-04-13T22:04:02.122 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:04:02.122 DEBUG:boto:Data: 2024-04-13T22:04:02.123 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.123 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.123 DEBUG:boto:Params: {} 2024-04-13T22:04:02.123 DEBUG:boto:Token: None 2024-04-13T22:04:02.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1 2024-04-13T22:04:02.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:um6/iMPQhayeG3Qjvbfxs2ZrE+c= 2024-04-13T22:04:02.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:um6/iMPQhayeG3Qjvbfxs2ZrE+c=', 'Content-Length': '0'} 2024-04-13T22:04:02.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa3f0400f546671b-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.166 DEBUG:boto:path=/vwnfns-48/obj1 2024-04-13T22:04:02.167 DEBUG:boto:auth_path=/vwnfns-48/obj1 2024-04-13T22:04:02.167 DEBUG:boto:Method: GET 2024-04-13T22:04:02.167 DEBUG:boto:Path: /vwnfns-48/obj1 2024-04-13T22:04:02.167 DEBUG:boto:Data: 2024-04-13T22:04:02.167 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.167 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.167 DEBUG:boto:Params: {} 2024-04-13T22:04:02.167 DEBUG:boto:Token: None 2024-04-13T22:04:02.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj1 2024-04-13T22:04:02.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:um6/iMPQhayeG3Qjvbfxs2ZrE+c= 2024-04-13T22:04:02.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:um6/iMPQhayeG3Qjvbfxs2ZrE+c=', 'Content-Length': '0'} 2024-04-13T22:04:02.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000708faabe8de89244-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.170 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.170 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.170 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.170 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.170 DEBUG:boto:Method: GET 2024-04-13T22:04:02.170 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.170 DEBUG:boto:Data: 2024-04-13T22:04:02.171 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.171 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.171 DEBUG:boto:Params: {} 2024-04-13T22:04:02.171 DEBUG:boto:Token: None 2024-04-13T22:04:02.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a35a91f694b560-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.174 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.174 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.174 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.174 DEBUG:boto:path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.174 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.175 DEBUG:boto:Method: GET 2024-04-13T22:04:02.175 DEBUG:boto:Path: /vwnfns-48/?prefix=obj1 2024-04-13T22:04:02.175 DEBUG:boto:Data: 2024-04-13T22:04:02.175 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.175 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.175 DEBUG:boto:Params: {} 2024-04-13T22:04:02.175 DEBUG:boto:Token: None 2024-04-13T22:04:02.175 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7bf77d457adceb7-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.178 DEBUG:boto:b'vwnfns-48obj11000falseobj12024-04-13T22:02:44.387Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:02.178 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.179 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.179 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.179 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.179 DEBUG:boto:Method: GET 2024-04-13T22:04:02.179 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.179 DEBUG:boto:Data: 2024-04-13T22:04:02.179 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.179 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.179 DEBUG:boto:Params: {} 2024-04-13T22:04:02.179 DEBUG:boto:Token: None 2024-04-13T22:04:02.180 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.180 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y= 2024-04-13T22:04:02.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y=', 'Content-Length': '0'} 2024-04-13T22:04:02.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088d044a81f582870-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.182 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.182 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.182 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.182 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.182 DEBUG:boto:Method: GET 2024-04-13T22:04:02.182 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.182 DEBUG:boto:Data: 2024-04-13T22:04:02.182 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.183 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.183 DEBUG:boto:Params: {} 2024-04-13T22:04:02.183 DEBUG:boto:Token: None 2024-04-13T22:04:02.183 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y= 2024-04-13T22:04:02.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y=', 'Content-Length': '0'} 2024-04-13T22:04:02.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e127fe448cb04663-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.185 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.185 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.185 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.185 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.185 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.186 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.186 DEBUG:boto:Data: 2024-04-13T22:04:02.186 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.186 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.186 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.186 DEBUG:boto:Params: {} 2024-04-13T22:04:02.186 DEBUG:boto:Token: None 2024-04-13T22:04:02.186 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:age0omtKc/w2TjEDskWZnVzFZeU= 2024-04-13T22:04:02.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:age0omtKc/w2TjEDskWZnVzFZeU=', 'Content-Length': '0'} 2024-04-13T22:04:02.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c9a711b2c87de91-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.188 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.189 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.189 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.189 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.189 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.189 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.189 DEBUG:boto:Data: 2024-04-13T22:04:02.189 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.189 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.189 DEBUG:boto:Params: {} 2024-04-13T22:04:02.189 DEBUG:boto:Token: None 2024-04-13T22:04:02.190 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:age0omtKc/w2TjEDskWZnVzFZeU= 2024-04-13T22:04:02.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:age0omtKc/w2TjEDskWZnVzFZeU=', 'Content-Length': '0'} 2024-04-13T22:04:02.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004513263cc41c156-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:02.192 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.192 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.192 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.192 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.192 DEBUG:boto:Method: GET 2024-04-13T22:04:02.192 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.192 DEBUG:boto:Data: 2024-04-13T22:04:02.193 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.193 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.193 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.193 DEBUG:boto:Params: {} 2024-04-13T22:04:02.193 DEBUG:boto:Token: None 2024-04-13T22:04:02.193 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y= 2024-04-13T22:04:02.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y=', 'Content-Length': '0'} 2024-04-13T22:04:02.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000983049de33c49d77-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.195 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.196 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.196 DEBUG:boto:path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.196 DEBUG:boto:auth_path=/vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.196 DEBUG:boto:Method: GET 2024-04-13T22:04:02.196 DEBUG:boto:Path: /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.196 DEBUG:boto:Data: 2024-04-13T22:04:02.196 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.196 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.196 DEBUG:boto:Params: {} 2024-04-13T22:04:02.196 DEBUG:boto:Token: None 2024-04-13T22:04:02.197 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2?versionId=null 2024-04-13T22:04:02.197 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y= 2024-04-13T22:04:02.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9hlwCIlbX+899CzsHnwPeWQTY6Y=', 'Content-Length': '0'} 2024-04-13T22:04:02.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ff81c133450f26-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.199 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.199 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.199 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.199 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.200 DEBUG:boto:Method: GET 2024-04-13T22:04:02.200 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.200 DEBUG:boto:Data: 2024-04-13T22:04:02.200 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.200 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.200 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.200 DEBUG:boto:Params: {} 2024-04-13T22:04:02.200 DEBUG:boto:Token: None 2024-04-13T22:04:02.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db52696c91cccc5-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.203 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.203 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.203 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.203 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.204 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.204 DEBUG:boto:Method: GET 2024-04-13T22:04:02.204 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.204 DEBUG:boto:Data: 2024-04-13T22:04:02.204 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.204 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.204 DEBUG:boto:Params: {} 2024-04-13T22:04:02.204 DEBUG:boto:Token: None 2024-04-13T22:04:02.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1fb916dae50678-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.207 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.207 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.207 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.207 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.208 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:04:02.208 DEBUG:boto:Data: 2024-04-13T22:04:02.208 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.208 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.208 DEBUG:boto:Params: {} 2024-04-13T22:04:02.208 DEBUG:boto:Token: None 2024-04-13T22:04:02.208 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2 2024-04-13T22:04:02.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qHzUB6DKfjhsnuWOc/xVTPbtQz0= 2024-04-13T22:04:02.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qHzUB6DKfjhsnuWOc/xVTPbtQz0=', 'Content-Length': '0'} 2024-04-13T22:04:02.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a613880c0067020-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.210 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.210 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.210 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.210 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:04:02.211 DEBUG:boto:Data: 2024-04-13T22:04:02.211 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.211 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.211 DEBUG:boto:Params: {} 2024-04-13T22:04:02.211 DEBUG:boto:Token: None 2024-04-13T22:04:02.211 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2 2024-04-13T22:04:02.211 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qHzUB6DKfjhsnuWOc/xVTPbtQz0= 2024-04-13T22:04:02.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qHzUB6DKfjhsnuWOc/xVTPbtQz0=', 'Content-Length': '0'} 2024-04-13T22:04:02.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a3935161ab49b06-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:02.213 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.213 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.214 DEBUG:boto:Method: GET 2024-04-13T22:04:02.214 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:04:02.214 DEBUG:boto:Data: 2024-04-13T22:04:02.214 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.214 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.214 DEBUG:boto:Params: {} 2024-04-13T22:04:02.214 DEBUG:boto:Token: None 2024-04-13T22:04:02.214 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2 2024-04-13T22:04:02.214 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JrpNN2Bx0Vt/RSao+UqbK1JSps8= 2024-04-13T22:04:02.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JrpNN2Bx0Vt/RSao+UqbK1JSps8=', 'Content-Length': '0'} 2024-04-13T22:04:02.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc480fdfecff4a3-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.258 DEBUG:boto:path=/vwnfns-48/obj2 2024-04-13T22:04:02.259 DEBUG:boto:auth_path=/vwnfns-48/obj2 2024-04-13T22:04:02.259 DEBUG:boto:Method: GET 2024-04-13T22:04:02.259 DEBUG:boto:Path: /vwnfns-48/obj2 2024-04-13T22:04:02.259 DEBUG:boto:Data: 2024-04-13T22:04:02.259 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.259 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.259 DEBUG:boto:Params: {} 2024-04-13T22:04:02.259 DEBUG:boto:Token: None 2024-04-13T22:04:02.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj2 2024-04-13T22:04:02.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JrpNN2Bx0Vt/RSao+UqbK1JSps8= 2024-04-13T22:04:02.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JrpNN2Bx0Vt/RSao+UqbK1JSps8=', 'Content-Length': '0'} 2024-04-13T22:04:02.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5fcbddc50392d0a-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.263 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.263 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.263 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.263 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.263 DEBUG:boto:Method: GET 2024-04-13T22:04:02.263 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.263 DEBUG:boto:Data: 2024-04-13T22:04:02.263 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.263 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.263 DEBUG:boto:Params: {} 2024-04-13T22:04:02.264 DEBUG:boto:Token: None 2024-04-13T22:04:02.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f1b8a2b35cdc6e-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.266 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.267 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.267 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.267 DEBUG:boto:path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.267 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.267 DEBUG:boto:Method: GET 2024-04-13T22:04:02.267 DEBUG:boto:Path: /vwnfns-48/?prefix=obj2 2024-04-13T22:04:02.268 DEBUG:boto:Data: 2024-04-13T22:04:02.268 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.268 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.268 DEBUG:boto:Params: {} 2024-04-13T22:04:02.268 DEBUG:boto:Token: None 2024-04-13T22:04:02.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf7832c4dabc4d88-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.314 DEBUG:boto:b'vwnfns-48obj21000falseobj22024-04-13T22:02:44.397Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:02.315 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.315 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.315 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.315 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.315 DEBUG:boto:Method: GET 2024-04-13T22:04:02.316 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.316 DEBUG:boto:Data: 2024-04-13T22:04:02.316 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.316 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.316 DEBUG:boto:Params: {} 2024-04-13T22:04:02.316 DEBUG:boto:Token: None 2024-04-13T22:04:02.316 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw= 2024-04-13T22:04:02.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw=', 'Content-Length': '0'} 2024-04-13T22:04:02.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072aee409a540851f-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.319 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.319 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.319 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.319 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.319 DEBUG:boto:Method: GET 2024-04-13T22:04:02.319 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.319 DEBUG:boto:Data: 2024-04-13T22:04:02.319 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.320 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.320 DEBUG:boto:Params: {} 2024-04-13T22:04:02.320 DEBUG:boto:Token: None 2024-04-13T22:04:02.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw= 2024-04-13T22:04:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw=', 'Content-Length': '0'} 2024-04-13T22:04:02.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a57dbbaf2d658486-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.322 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.322 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.322 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.322 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.323 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.323 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.323 DEBUG:boto:Data: 2024-04-13T22:04:02.323 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.323 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.323 DEBUG:boto:Params: {} 2024-04-13T22:04:02.323 DEBUG:boto:Token: None 2024-04-13T22:04:02.323 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+LcDLOGpBWK2B6CKWlTkuArthgQ= 2024-04-13T22:04:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+LcDLOGpBWK2B6CKWlTkuArthgQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fbe95e673cd0ccf-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.325 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.325 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.325 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.326 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.326 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.326 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.326 DEBUG:boto:Data: 2024-04-13T22:04:02.326 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.326 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.326 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.326 DEBUG:boto:Params: {} 2024-04-13T22:04:02.326 DEBUG:boto:Token: None 2024-04-13T22:04:02.326 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+LcDLOGpBWK2B6CKWlTkuArthgQ= 2024-04-13T22:04:02.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+LcDLOGpBWK2B6CKWlTkuArthgQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096089ec247bdd129-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:02.329 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.329 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.329 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.329 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.329 DEBUG:boto:Method: GET 2024-04-13T22:04:02.329 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.329 DEBUG:boto:Data: 2024-04-13T22:04:02.329 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.329 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.330 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.330 DEBUG:boto:Params: {} 2024-04-13T22:04:02.330 DEBUG:boto:Token: None 2024-04-13T22:04:02.330 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.330 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw= 2024-04-13T22:04:02.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw=', 'Content-Length': '0'} 2024-04-13T22:04:02.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000321ef3ecd3c58037-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.332 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.332 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.332 DEBUG:boto:path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.332 DEBUG:boto:auth_path=/vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.332 DEBUG:boto:Method: GET 2024-04-13T22:04:02.332 DEBUG:boto:Path: /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.332 DEBUG:boto:Data: 2024-04-13T22:04:02.333 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.333 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.333 DEBUG:boto:Params: {} 2024-04-13T22:04:02.333 DEBUG:boto:Token: None 2024-04-13T22:04:02.333 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3?versionId=null 2024-04-13T22:04:02.333 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw= 2024-04-13T22:04:02.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GohfuEXvf+kKGTqLh6xNsZjTbLw=', 'Content-Length': '0'} 2024-04-13T22:04:02.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f882567a87f17ff-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.335 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.335 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.335 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.335 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.336 DEBUG:boto:Method: GET 2024-04-13T22:04:02.336 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.336 DEBUG:boto:Data: 2024-04-13T22:04:02.336 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.336 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.336 DEBUG:boto:Params: {} 2024-04-13T22:04:02.336 DEBUG:boto:Token: None 2024-04-13T22:04:02.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4fb5ce01132d34a-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.339 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.339 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.339 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.339 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.339 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.339 DEBUG:boto:Method: GET 2024-04-13T22:04:02.339 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.339 DEBUG:boto:Data: 2024-04-13T22:04:02.340 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.340 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.340 DEBUG:boto:Params: {} 2024-04-13T22:04:02.340 DEBUG:boto:Token: None 2024-04-13T22:04:02.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0fb3f8b5a27d59d-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.343 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.343 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.343 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.343 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.343 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:04:02.343 DEBUG:boto:Data: 2024-04-13T22:04:02.344 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.344 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.344 DEBUG:boto:Params: {} 2024-04-13T22:04:02.344 DEBUG:boto:Token: None 2024-04-13T22:04:02.344 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3 2024-04-13T22:04:02.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:onbXiMrrSxzTveh6MXIDF13jXJM= 2024-04-13T22:04:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:onbXiMrrSxzTveh6MXIDF13jXJM=', 'Content-Length': '0'} 2024-04-13T22:04:02.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099ab8a413357e49a-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.346 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.346 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.346 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.346 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:04:02.347 DEBUG:boto:Data: 2024-04-13T22:04:02.347 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.347 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.347 DEBUG:boto:Params: {} 2024-04-13T22:04:02.347 DEBUG:boto:Token: None 2024-04-13T22:04:02.347 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3 2024-04-13T22:04:02.347 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:onbXiMrrSxzTveh6MXIDF13jXJM= 2024-04-13T22:04:02.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:onbXiMrrSxzTveh6MXIDF13jXJM=', 'Content-Length': '0'} 2024-04-13T22:04:02.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bddfe4ff74e9c04e-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:02.349 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.349 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.349 DEBUG:boto:Method: GET 2024-04-13T22:04:02.350 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:04:02.350 DEBUG:boto:Data: 2024-04-13T22:04:02.350 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.350 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.350 DEBUG:boto:Params: {} 2024-04-13T22:04:02.350 DEBUG:boto:Token: None 2024-04-13T22:04:02.350 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3 2024-04-13T22:04:02.350 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4lKHwStit43jFYrKp6ls1Wt+iyg= 2024-04-13T22:04:02.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lKHwStit43jFYrKp6ls1Wt+iyg=', 'Content-Length': '0'} 2024-04-13T22:04:02.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b03d0810acc7883b-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.394 DEBUG:boto:path=/vwnfns-48/obj3 2024-04-13T22:04:02.395 DEBUG:boto:auth_path=/vwnfns-48/obj3 2024-04-13T22:04:02.395 DEBUG:boto:Method: GET 2024-04-13T22:04:02.395 DEBUG:boto:Path: /vwnfns-48/obj3 2024-04-13T22:04:02.395 DEBUG:boto:Data: 2024-04-13T22:04:02.395 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.395 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.395 DEBUG:boto:Params: {} 2024-04-13T22:04:02.395 DEBUG:boto:Token: None 2024-04-13T22:04:02.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj3 2024-04-13T22:04:02.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4lKHwStit43jFYrKp6ls1Wt+iyg= 2024-04-13T22:04:02.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lKHwStit43jFYrKp6ls1Wt+iyg=', 'Content-Length': '0'} 2024-04-13T22:04:02.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054f15e4591042e50-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.398 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.398 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.398 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.398 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.398 DEBUG:boto:Method: GET 2024-04-13T22:04:02.398 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.398 DEBUG:boto:Data: 2024-04-13T22:04:02.398 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.398 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.399 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.399 DEBUG:boto:Params: {} 2024-04-13T22:04:02.399 DEBUG:boto:Token: None 2024-04-13T22:04:02.399 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.399 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750d540727db9f54-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.401 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.401 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.402 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.402 DEBUG:boto:path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.402 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.402 DEBUG:boto:Method: GET 2024-04-13T22:04:02.402 DEBUG:boto:Path: /vwnfns-48/?prefix=obj3 2024-04-13T22:04:02.402 DEBUG:boto:Data: 2024-04-13T22:04:02.402 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.402 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.402 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.402 DEBUG:boto:Params: {} 2024-04-13T22:04:02.403 DEBUG:boto:Token: None 2024-04-13T22:04:02.403 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.403 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b275a56fcd19ec-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.405 DEBUG:boto:b'vwnfns-48obj31000falseobj32024-04-13T22:02:44.407Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:02.405 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.406 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.406 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.406 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.406 DEBUG:boto:Method: GET 2024-04-13T22:04:02.406 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.406 DEBUG:boto:Data: 2024-04-13T22:04:02.406 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.406 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.406 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.406 DEBUG:boto:Params: {} 2024-04-13T22:04:02.406 DEBUG:boto:Token: None 2024-04-13T22:04:02.407 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.407 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M= 2024-04-13T22:04:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M=', 'Content-Length': '0'} 2024-04-13T22:04:02.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa7d0378e0edbf0c-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.409 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.409 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.409 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.409 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.409 DEBUG:boto:Method: GET 2024-04-13T22:04:02.409 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.409 DEBUG:boto:Data: 2024-04-13T22:04:02.409 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.410 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.410 DEBUG:boto:Params: {} 2024-04-13T22:04:02.410 DEBUG:boto:Token: None 2024-04-13T22:04:02.410 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M= 2024-04-13T22:04:02.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M=', 'Content-Length': '0'} 2024-04-13T22:04:02.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd61e4eb767411bb-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.412 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.412 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.412 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.412 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.412 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.412 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.412 DEBUG:boto:Data: 2024-04-13T22:04:02.412 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.413 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.413 DEBUG:boto:Params: {} 2024-04-13T22:04:02.413 DEBUG:boto:Token: None 2024-04-13T22:04:02.413 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHgZJGTiaJdu3oZ44cp7RkKeG84= 2024-04-13T22:04:02.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHgZJGTiaJdu3oZ44cp7RkKeG84=', 'Content-Length': '0'} 2024-04-13T22:04:02.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bace008ae9d615fb-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.415 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.415 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.415 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.415 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.415 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.415 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.415 DEBUG:boto:Data: 2024-04-13T22:04:02.415 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.416 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.416 DEBUG:boto:Params: {} 2024-04-13T22:04:02.416 DEBUG:boto:Token: None 2024-04-13T22:04:02.416 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHgZJGTiaJdu3oZ44cp7RkKeG84= 2024-04-13T22:04:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHgZJGTiaJdu3oZ44cp7RkKeG84=', 'Content-Length': '0'} 2024-04-13T22:04:02.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fc800ed684c2c2d-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:02.418 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.418 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.418 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.418 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.418 DEBUG:boto:Method: GET 2024-04-13T22:04:02.418 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.418 DEBUG:boto:Data: 2024-04-13T22:04:02.419 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.419 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.419 DEBUG:boto:Params: {} 2024-04-13T22:04:02.419 DEBUG:boto:Token: None 2024-04-13T22:04:02.419 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.419 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M= 2024-04-13T22:04:02.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M=', 'Content-Length': '0'} 2024-04-13T22:04:02.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063c07b0054963c8f-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.421 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.421 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.421 DEBUG:boto:path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.421 DEBUG:boto:auth_path=/vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.421 DEBUG:boto:Method: GET 2024-04-13T22:04:02.421 DEBUG:boto:Path: /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.422 DEBUG:boto:Data: 2024-04-13T22:04:02.422 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.422 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.422 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.422 DEBUG:boto:Params: {} 2024-04-13T22:04:02.422 DEBUG:boto:Token: None 2024-04-13T22:04:02.422 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4?versionId=null 2024-04-13T22:04:02.422 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M= 2024-04-13T22:04:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rf/Yfg8tS60AEestuUA12n+Im5M=', 'Content-Length': '0'} 2024-04-13T22:04:02.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074a2e9df14c67757-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.424 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.424 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.424 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.424 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.424 DEBUG:boto:Method: GET 2024-04-13T22:04:02.425 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.425 DEBUG:boto:Data: 2024-04-13T22:04:02.425 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.425 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.425 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.425 DEBUG:boto:Params: {} 2024-04-13T22:04:02.425 DEBUG:boto:Token: None 2024-04-13T22:04:02.425 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.425 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001660ae45f1208ad6-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.428 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.428 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.428 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.428 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.428 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.429 DEBUG:boto:Method: GET 2024-04-13T22:04:02.429 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.429 DEBUG:boto:Data: 2024-04-13T22:04:02.429 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.429 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.429 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.429 DEBUG:boto:Params: {} 2024-04-13T22:04:02.429 DEBUG:boto:Token: None 2024-04-13T22:04:02.429 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.429 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6a620d1cef073d-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.432 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.432 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.432 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.432 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.432 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:04:02.432 DEBUG:boto:Data: 2024-04-13T22:04:02.432 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.432 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.432 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.433 DEBUG:boto:Params: {} 2024-04-13T22:04:02.433 DEBUG:boto:Token: None 2024-04-13T22:04:02.433 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4 2024-04-13T22:04:02.433 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:90rMcY4N9He62/rqNl36OJ/qCp0= 2024-04-13T22:04:02.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:90rMcY4N9He62/rqNl36OJ/qCp0=', 'Content-Length': '0'} 2024-04-13T22:04:02.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061521c3e26318ffd-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.434 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.435 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.435 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.435 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:04:02.435 DEBUG:boto:Data: 2024-04-13T22:04:02.435 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.435 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.435 DEBUG:boto:Params: {} 2024-04-13T22:04:02.435 DEBUG:boto:Token: None 2024-04-13T22:04:02.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4 2024-04-13T22:04:02.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:90rMcY4N9He62/rqNl36OJ/qCp0= 2024-04-13T22:04:02.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:90rMcY4N9He62/rqNl36OJ/qCp0=', 'Content-Length': '0'} 2024-04-13T22:04:02.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfb36315b591b82f-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:02.437 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.437 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.438 DEBUG:boto:Method: GET 2024-04-13T22:04:02.438 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:04:02.438 DEBUG:boto:Data: 2024-04-13T22:04:02.438 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.438 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.438 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.438 DEBUG:boto:Params: {} 2024-04-13T22:04:02.438 DEBUG:boto:Token: None 2024-04-13T22:04:02.438 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4 2024-04-13T22:04:02.439 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ICXNBxAySTkkQRwNGX4plm6/rBY= 2024-04-13T22:04:02.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ICXNBxAySTkkQRwNGX4plm6/rBY=', 'Content-Length': '0'} 2024-04-13T22:04:02.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043cfa6dedc6b63f7-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.482 DEBUG:boto:path=/vwnfns-48/obj4 2024-04-13T22:04:02.483 DEBUG:boto:auth_path=/vwnfns-48/obj4 2024-04-13T22:04:02.483 DEBUG:boto:Method: GET 2024-04-13T22:04:02.483 DEBUG:boto:Path: /vwnfns-48/obj4 2024-04-13T22:04:02.483 DEBUG:boto:Data: 2024-04-13T22:04:02.483 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.483 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.483 DEBUG:boto:Params: {} 2024-04-13T22:04:02.483 DEBUG:boto:Token: None 2024-04-13T22:04:02.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj4 2024-04-13T22:04:02.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ICXNBxAySTkkQRwNGX4plm6/rBY= 2024-04-13T22:04:02.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ICXNBxAySTkkQRwNGX4plm6/rBY=', 'Content-Length': '0'} 2024-04-13T22:04:02.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074bac2674f78e865-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.486 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.486 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.486 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.486 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.486 DEBUG:boto:Method: GET 2024-04-13T22:04:02.486 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.486 DEBUG:boto:Data: 2024-04-13T22:04:02.486 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.487 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.487 DEBUG:boto:Params: {} 2024-04-13T22:04:02.487 DEBUG:boto:Token: None 2024-04-13T22:04:02.487 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.487 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f25ff40f29c7a93-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.489 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.490 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.490 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.490 DEBUG:boto:path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.490 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.490 DEBUG:boto:Method: GET 2024-04-13T22:04:02.490 DEBUG:boto:Path: /vwnfns-48/?prefix=obj4 2024-04-13T22:04:02.490 DEBUG:boto:Data: 2024-04-13T22:04:02.491 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.491 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.491 DEBUG:boto:Params: {} 2024-04-13T22:04:02.491 DEBUG:boto:Token: None 2024-04-13T22:04:02.491 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007038cc7e011151c6-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.493 DEBUG:boto:b'vwnfns-48obj41000falseobj42024-04-13T22:02:44.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:04:02.494 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.494 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.494 DEBUG:boto:path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.494 DEBUG:boto:auth_path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.494 DEBUG:boto:Method: GET 2024-04-13T22:04:02.494 DEBUG:boto:Path: /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.494 DEBUG:boto:Data: 2024-04-13T22:04:02.494 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.495 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.495 DEBUG:boto:Params: {} 2024-04-13T22:04:02.495 DEBUG:boto:Token: None 2024-04-13T22:04:02.495 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs= 2024-04-13T22:04:02.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs=', 'Content-Length': '0'} 2024-04-13T22:04:02.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d421ba4e94197045-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.497 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.497 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.498 DEBUG:boto:path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.498 DEBUG:boto:auth_path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.498 DEBUG:boto:Method: GET 2024-04-13T22:04:02.498 DEBUG:boto:Path: /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.498 DEBUG:boto:Data: 2024-04-13T22:04:02.498 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.498 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.498 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.498 DEBUG:boto:Params: {} 2024-04-13T22:04:02.498 DEBUG:boto:Token: None 2024-04-13T22:04:02.499 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.499 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs= 2024-04-13T22:04:02.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs=', 'Content-Length': '0'} 2024-04-13T22:04:02.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7c694493d4c4a09-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.501 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.501 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.501 DEBUG:boto:path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.501 DEBUG:boto:auth_path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.501 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.501 DEBUG:boto:Path: /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.501 DEBUG:boto:Data: 2024-04-13T22:04:02.501 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.501 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.501 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.502 DEBUG:boto:Params: {} 2024-04-13T22:04:02.502 DEBUG:boto:Token: None 2024-04-13T22:04:02.502 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.502 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iX3TqRF3iKSQ2NluOhF/+JQM5KY= 2024-04-13T22:04:02.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iX3TqRF3iKSQ2NluOhF/+JQM5KY=', 'Content-Length': '0'} 2024-04-13T22:04:02.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb2fe02b1b8fdc45-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.504 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.504 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.504 DEBUG:boto:path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.504 DEBUG:boto:auth_path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.504 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.504 DEBUG:boto:Path: /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.504 DEBUG:boto:Data: 2024-04-13T22:04:02.504 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.504 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.504 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.505 DEBUG:boto:Params: {} 2024-04-13T22:04:02.505 DEBUG:boto:Token: None 2024-04-13T22:04:02.505 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.505 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iX3TqRF3iKSQ2NluOhF/+JQM5KY= 2024-04-13T22:04:02.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iX3TqRF3iKSQ2NluOhF/+JQM5KY=', 'Content-Length': '0'} 2024-04-13T22:04:02.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003161976d36b34274-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:04:02.507 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.507 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.507 DEBUG:boto:path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.507 DEBUG:boto:auth_path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.508 DEBUG:boto:Method: GET 2024-04-13T22:04:02.508 DEBUG:boto:Path: /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.508 DEBUG:boto:Data: 2024-04-13T22:04:02.508 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.508 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.508 DEBUG:boto:Params: {} 2024-04-13T22:04:02.508 DEBUG:boto:Token: None 2024-04-13T22:04:02.508 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs= 2024-04-13T22:04:02.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs=', 'Content-Length': '0'} 2024-04-13T22:04:02.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d5bb2aea8f1f5d2-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.510 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.511 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.511 DEBUG:boto:path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.511 DEBUG:boto:auth_path=/vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.511 DEBUG:boto:Method: GET 2024-04-13T22:04:02.511 DEBUG:boto:Path: /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.511 DEBUG:boto:Data: 2024-04-13T22:04:02.511 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.511 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.511 DEBUG:boto:Params: {} 2024-04-13T22:04:02.512 DEBUG:boto:Token: None 2024-04-13T22:04:02.512 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5?versionId=null 2024-04-13T22:04:02.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs= 2024-04-13T22:04:02.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/7OuNGWVrpMjMubTcaogHanjtEs=', 'Content-Length': '0'} 2024-04-13T22:04:02.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f596f797be5a27fc-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.514 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.514 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.514 DEBUG:boto:path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.514 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.514 DEBUG:boto:Method: GET 2024-04-13T22:04:02.514 DEBUG:boto:Path: /vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.514 DEBUG:boto:Data: 2024-04-13T22:04:02.514 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.514 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.515 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.515 DEBUG:boto:Params: {} 2024-04-13T22:04:02.515 DEBUG:boto:Token: None 2024-04-13T22:04:02.515 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b028623d99e42c-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.517 DEBUG:boto:b'vwnfns-48obj51000falseobj52024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.518 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.518 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.518 DEBUG:boto:path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.518 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.518 DEBUG:boto:Method: GET 2024-04-13T22:04:02.518 DEBUG:boto:Path: /vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.518 DEBUG:boto:Data: 2024-04-13T22:04:02.518 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.518 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.519 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.519 DEBUG:boto:Params: {} 2024-04-13T22:04:02.519 DEBUG:boto:Token: None 2024-04-13T22:04:02.519 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.519 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a9d4832ad6a5b49-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.521 DEBUG:boto:b'vwnfns-48obj51000falseobj52024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.522 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.522 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.522 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.522 DEBUG:boto:Path: /vwnfns-48/obj5 2024-04-13T22:04:02.522 DEBUG:boto:Data: 2024-04-13T22:04:02.522 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.522 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.522 DEBUG:boto:Params: {} 2024-04-13T22:04:02.522 DEBUG:boto:Token: None 2024-04-13T22:04:02.523 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5 2024-04-13T22:04:02.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iI0zuQnF8YCjh7dbg23klsiezCk= 2024-04-13T22:04:02.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iI0zuQnF8YCjh7dbg23klsiezCk=', 'Content-Length': '0'} 2024-04-13T22:04:02.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe36b310f9754cce-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.525 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.525 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.525 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.525 DEBUG:boto:Path: /vwnfns-48/obj5 2024-04-13T22:04:02.525 DEBUG:boto:Data: 2024-04-13T22:04:02.525 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.525 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.525 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.525 DEBUG:boto:Params: {} 2024-04-13T22:04:02.525 DEBUG:boto:Token: None 2024-04-13T22:04:02.526 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5 2024-04-13T22:04:02.526 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iI0zuQnF8YCjh7dbg23klsiezCk= 2024-04-13T22:04:02.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iI0zuQnF8YCjh7dbg23klsiezCk=', 'Content-Length': '0'} 2024-04-13T22:04:02.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3d81b96529966ca-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:04:02.527 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.528 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.528 DEBUG:boto:Method: GET 2024-04-13T22:04:02.528 DEBUG:boto:Path: /vwnfns-48/obj5 2024-04-13T22:04:02.528 DEBUG:boto:Data: 2024-04-13T22:04:02.528 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.528 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.528 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.528 DEBUG:boto:Params: {} 2024-04-13T22:04:02.528 DEBUG:boto:Token: None 2024-04-13T22:04:02.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5 2024-04-13T22:04:02.529 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lG8uR1AYR6q/Ct2zSqxHN8DBqnM= 2024-04-13T22:04:02.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lG8uR1AYR6q/Ct2zSqxHN8DBqnM=', 'Content-Length': '0'} 2024-04-13T22:04:02.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006089d545effbe1da-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.530 DEBUG:boto:path=/vwnfns-48/obj5 2024-04-13T22:04:02.530 DEBUG:boto:auth_path=/vwnfns-48/obj5 2024-04-13T22:04:02.531 DEBUG:boto:Method: GET 2024-04-13T22:04:02.531 DEBUG:boto:Path: /vwnfns-48/obj5 2024-04-13T22:04:02.531 DEBUG:boto:Data: 2024-04-13T22:04:02.531 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.531 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.531 DEBUG:boto:Params: {} 2024-04-13T22:04:02.531 DEBUG:boto:Token: None 2024-04-13T22:04:02.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj5 2024-04-13T22:04:02.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lG8uR1AYR6q/Ct2zSqxHN8DBqnM= 2024-04-13T22:04:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lG8uR1AYR6q/Ct2zSqxHN8DBqnM=', 'Content-Length': '0'} 2024-04-13T22:04:02.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997bea87c6d7cc4a-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.533 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.534 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.534 DEBUG:boto:path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.534 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.534 DEBUG:boto:Method: GET 2024-04-13T22:04:02.534 DEBUG:boto:Path: /vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.534 DEBUG:boto:Data: 2024-04-13T22:04:02.534 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.534 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.534 DEBUG:boto:Params: {} 2024-04-13T22:04:02.535 DEBUG:boto:Token: None 2024-04-13T22:04:02.535 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000044bfc1c1ff97a26-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.537 DEBUG:boto:b'vwnfns-48obj51000falseobj52024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.538 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.538 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.538 DEBUG:boto:path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.538 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.538 DEBUG:boto:Method: GET 2024-04-13T22:04:02.538 DEBUG:boto:Path: /vwnfns-48/?prefix=obj5 2024-04-13T22:04:02.538 DEBUG:boto:Data: 2024-04-13T22:04:02.538 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.538 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.538 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.539 DEBUG:boto:Params: {} 2024-04-13T22:04:02.539 DEBUG:boto:Token: None 2024-04-13T22:04:02.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b62222aa1db45c3-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.541 DEBUG:boto:b'vwnfns-48obj51000falseobj52024-04-13T22:03:11.568Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:04:02.542 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.542 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.542 DEBUG:boto:path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.542 DEBUG:boto:auth_path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.542 DEBUG:boto:Method: GET 2024-04-13T22:04:02.542 DEBUG:boto:Path: /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.542 DEBUG:boto:Data: 2024-04-13T22:04:02.542 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.542 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.542 DEBUG:boto:Params: {} 2024-04-13T22:04:02.543 DEBUG:boto:Token: None 2024-04-13T22:04:02.543 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY= 2024-04-13T22:04:02.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY=', 'Content-Length': '0'} 2024-04-13T22:04:02.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055cbce0097552c3e-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.545 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.545 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.545 DEBUG:boto:path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.545 DEBUG:boto:auth_path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.545 DEBUG:boto:Method: GET 2024-04-13T22:04:02.545 DEBUG:boto:Path: /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.546 DEBUG:boto:Data: 2024-04-13T22:04:02.546 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.546 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.546 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.546 DEBUG:boto:Params: {} 2024-04-13T22:04:02.546 DEBUG:boto:Token: None 2024-04-13T22:04:02.546 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY= 2024-04-13T22:04:02.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY=', 'Content-Length': '0'} 2024-04-13T22:04:02.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000024fd2f004040d65-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.548 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.548 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.548 DEBUG:boto:path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.549 DEBUG:boto:auth_path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.549 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.549 DEBUG:boto:Path: /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.549 DEBUG:boto:Data: 2024-04-13T22:04:02.549 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.549 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.549 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.549 DEBUG:boto:Params: {} 2024-04-13T22:04:02.549 DEBUG:boto:Token: None 2024-04-13T22:04:02.549 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.550 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U2IEoCl83u4AOMlOS5xzt+iHe5s= 2024-04-13T22:04:02.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U2IEoCl83u4AOMlOS5xzt+iHe5s=', 'Content-Length': '0'} 2024-04-13T22:04:02.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7b4d23521d9f42f-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.551 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.551 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.551 DEBUG:boto:path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.552 DEBUG:boto:auth_path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.552 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.552 DEBUG:boto:Path: /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.552 DEBUG:boto:Data: 2024-04-13T22:04:02.552 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.552 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.552 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.552 DEBUG:boto:Params: {} 2024-04-13T22:04:02.552 DEBUG:boto:Token: None 2024-04-13T22:04:02.552 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U2IEoCl83u4AOMlOS5xzt+iHe5s= 2024-04-13T22:04:02.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U2IEoCl83u4AOMlOS5xzt+iHe5s=', 'Content-Length': '0'} 2024-04-13T22:04:02.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf4e8aca1dbce185-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:04:02.554 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.554 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.555 DEBUG:boto:path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.555 DEBUG:boto:auth_path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.555 DEBUG:boto:Method: GET 2024-04-13T22:04:02.555 DEBUG:boto:Path: /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.555 DEBUG:boto:Data: 2024-04-13T22:04:02.555 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.555 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.555 DEBUG:boto:Params: {} 2024-04-13T22:04:02.555 DEBUG:boto:Token: None 2024-04-13T22:04:02.556 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY= 2024-04-13T22:04:02.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY=', 'Content-Length': '0'} 2024-04-13T22:04:02.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c98be97cdaf2e82-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.557 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.558 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.558 DEBUG:boto:path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.558 DEBUG:boto:auth_path=/vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.558 DEBUG:boto:Method: GET 2024-04-13T22:04:02.558 DEBUG:boto:Path: /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.558 DEBUG:boto:Data: 2024-04-13T22:04:02.558 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.558 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.558 DEBUG:boto:Params: {} 2024-04-13T22:04:02.559 DEBUG:boto:Token: None 2024-04-13T22:04:02.559 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6?versionId=null 2024-04-13T22:04:02.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY= 2024-04-13T22:04:02.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2sIaEhc0yPvmwgVN5cZfEm3NDwY=', 'Content-Length': '0'} 2024-04-13T22:04:02.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5014bb65c21c1a4-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.561 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.561 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.561 DEBUG:boto:path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.561 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.561 DEBUG:boto:Method: GET 2024-04-13T22:04:02.561 DEBUG:boto:Path: /vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.561 DEBUG:boto:Data: 2024-04-13T22:04:02.561 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.561 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.561 DEBUG:boto:Params: {} 2024-04-13T22:04:02.562 DEBUG:boto:Token: None 2024-04-13T22:04:02.562 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d255347ce6808b5e-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.564 DEBUG:boto:b'vwnfns-48obj61000falseobj62024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.564 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.564 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.565 DEBUG:boto:path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.565 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.565 DEBUG:boto:Method: GET 2024-04-13T22:04:02.565 DEBUG:boto:Path: /vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.565 DEBUG:boto:Data: 2024-04-13T22:04:02.565 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.565 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.565 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.565 DEBUG:boto:Params: {} 2024-04-13T22:04:02.565 DEBUG:boto:Token: None 2024-04-13T22:04:02.565 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.566 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa828b481704d6e6-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.568 DEBUG:boto:b'vwnfns-48obj61000falseobj62024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.568 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.568 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.568 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.569 DEBUG:boto:Path: /vwnfns-48/obj6 2024-04-13T22:04:02.569 DEBUG:boto:Data: 2024-04-13T22:04:02.569 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.569 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.569 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.569 DEBUG:boto:Params: {} 2024-04-13T22:04:02.569 DEBUG:boto:Token: None 2024-04-13T22:04:02.569 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6 2024-04-13T22:04:02.569 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5HoIovwk1c9QB/pMpnXeK5dcpA8= 2024-04-13T22:04:02.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5HoIovwk1c9QB/pMpnXeK5dcpA8=', 'Content-Length': '0'} 2024-04-13T22:04:02.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc335cb8af4f4d4e-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.571 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.571 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.571 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.571 DEBUG:boto:Path: /vwnfns-48/obj6 2024-04-13T22:04:02.571 DEBUG:boto:Data: 2024-04-13T22:04:02.571 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.572 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.572 DEBUG:boto:Params: {} 2024-04-13T22:04:02.572 DEBUG:boto:Token: None 2024-04-13T22:04:02.572 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6 2024-04-13T22:04:02.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5HoIovwk1c9QB/pMpnXeK5dcpA8= 2024-04-13T22:04:02.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5HoIovwk1c9QB/pMpnXeK5dcpA8=', 'Content-Length': '0'} 2024-04-13T22:04:02.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042062c01be4ec740-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:04:02.574 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.574 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.574 DEBUG:boto:Method: GET 2024-04-13T22:04:02.574 DEBUG:boto:Path: /vwnfns-48/obj6 2024-04-13T22:04:02.574 DEBUG:boto:Data: 2024-04-13T22:04:02.574 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.574 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.575 DEBUG:boto:Params: {} 2024-04-13T22:04:02.575 DEBUG:boto:Token: None 2024-04-13T22:04:02.575 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6 2024-04-13T22:04:02.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D+Z7LYHMgOkRDdSFnmoFOxGjVnk= 2024-04-13T22:04:02.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D+Z7LYHMgOkRDdSFnmoFOxGjVnk=', 'Content-Length': '0'} 2024-04-13T22:04:02.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000142a225763c10a2-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.618 DEBUG:boto:path=/vwnfns-48/obj6 2024-04-13T22:04:02.619 DEBUG:boto:auth_path=/vwnfns-48/obj6 2024-04-13T22:04:02.619 DEBUG:boto:Method: GET 2024-04-13T22:04:02.619 DEBUG:boto:Path: /vwnfns-48/obj6 2024-04-13T22:04:02.619 DEBUG:boto:Data: 2024-04-13T22:04:02.619 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.619 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.619 DEBUG:boto:Params: {} 2024-04-13T22:04:02.619 DEBUG:boto:Token: None 2024-04-13T22:04:02.620 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj6 2024-04-13T22:04:02.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D+Z7LYHMgOkRDdSFnmoFOxGjVnk= 2024-04-13T22:04:02.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D+Z7LYHMgOkRDdSFnmoFOxGjVnk=', 'Content-Length': '0'} 2024-04-13T22:04:02.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd52480d94ab718e-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.622 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.622 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.622 DEBUG:boto:path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.622 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.622 DEBUG:boto:Method: GET 2024-04-13T22:04:02.622 DEBUG:boto:Path: /vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.622 DEBUG:boto:Data: 2024-04-13T22:04:02.622 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.622 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.623 DEBUG:boto:Params: {} 2024-04-13T22:04:02.623 DEBUG:boto:Token: None 2024-04-13T22:04:02.623 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b11338bb85991b85-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.625 DEBUG:boto:b'vwnfns-48obj61000falseobj62024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.625 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.625 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.626 DEBUG:boto:path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.626 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.626 DEBUG:boto:Method: GET 2024-04-13T22:04:02.626 DEBUG:boto:Path: /vwnfns-48/?prefix=obj6 2024-04-13T22:04:02.626 DEBUG:boto:Data: 2024-04-13T22:04:02.626 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.626 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.626 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.626 DEBUG:boto:Params: {} 2024-04-13T22:04:02.626 DEBUG:boto:Token: None 2024-04-13T22:04:02.627 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.627 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb35762ef926190-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.629 DEBUG:boto:b'vwnfns-48obj61000falseobj62024-04-13T22:03:11.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:04:02.629 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.629 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.629 DEBUG:boto:path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.629 DEBUG:boto:auth_path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.629 DEBUG:boto:Method: GET 2024-04-13T22:04:02.630 DEBUG:boto:Path: /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.630 DEBUG:boto:Data: 2024-04-13T22:04:02.630 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.630 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.630 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.630 DEBUG:boto:Params: {} 2024-04-13T22:04:02.630 DEBUG:boto:Token: None 2024-04-13T22:04:02.630 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg= 2024-04-13T22:04:02.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg=', 'Content-Length': '0'} 2024-04-13T22:04:02.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1ae5a0174773a51-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.632 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.632 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.633 DEBUG:boto:path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.633 DEBUG:boto:auth_path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.633 DEBUG:boto:Method: GET 2024-04-13T22:04:02.633 DEBUG:boto:Path: /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.633 DEBUG:boto:Data: 2024-04-13T22:04:02.633 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.633 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.633 DEBUG:boto:Params: {} 2024-04-13T22:04:02.633 DEBUG:boto:Token: None 2024-04-13T22:04:02.634 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.634 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg= 2024-04-13T22:04:02.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg=', 'Content-Length': '0'} 2024-04-13T22:04:02.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b624e251868d99d-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.636 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.636 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.636 DEBUG:boto:path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.636 DEBUG:boto:auth_path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.636 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.636 DEBUG:boto:Path: /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.636 DEBUG:boto:Data: 2024-04-13T22:04:02.636 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.636 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.636 DEBUG:boto:Params: {} 2024-04-13T22:04:02.637 DEBUG:boto:Token: None 2024-04-13T22:04:02.637 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.637 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sRQpi2RA8rvEgtENjmUE8CzFV8M= 2024-04-13T22:04:02.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sRQpi2RA8rvEgtENjmUE8CzFV8M=', 'Content-Length': '0'} 2024-04-13T22:04:02.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cd2d56b78297bda-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.638 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.639 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.639 DEBUG:boto:path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.639 DEBUG:boto:auth_path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.639 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.639 DEBUG:boto:Path: /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.639 DEBUG:boto:Data: 2024-04-13T22:04:02.639 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.639 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.639 DEBUG:boto:Params: {} 2024-04-13T22:04:02.639 DEBUG:boto:Token: None 2024-04-13T22:04:02.640 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sRQpi2RA8rvEgtENjmUE8CzFV8M= 2024-04-13T22:04:02.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sRQpi2RA8rvEgtENjmUE8CzFV8M=', 'Content-Length': '0'} 2024-04-13T22:04:02.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b391fcf71ca849c7-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:04:02.641 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.642 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.642 DEBUG:boto:path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.642 DEBUG:boto:auth_path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.642 DEBUG:boto:Method: GET 2024-04-13T22:04:02.642 DEBUG:boto:Path: /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.642 DEBUG:boto:Data: 2024-04-13T22:04:02.642 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.642 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.642 DEBUG:boto:Params: {} 2024-04-13T22:04:02.643 DEBUG:boto:Token: None 2024-04-13T22:04:02.643 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg= 2024-04-13T22:04:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg=', 'Content-Length': '0'} 2024-04-13T22:04:02.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5695349b613669a-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.645 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.645 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.645 DEBUG:boto:path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.645 DEBUG:boto:auth_path=/vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.645 DEBUG:boto:Method: GET 2024-04-13T22:04:02.645 DEBUG:boto:Path: /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.645 DEBUG:boto:Data: 2024-04-13T22:04:02.645 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.645 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.645 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.646 DEBUG:boto:Params: {} 2024-04-13T22:04:02.646 DEBUG:boto:Token: None 2024-04-13T22:04:02.646 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7?versionId=null 2024-04-13T22:04:02.646 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg= 2024-04-13T22:04:02.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vKjgxBHLcuFLF6U6YSJj2xvE+Vg=', 'Content-Length': '0'} 2024-04-13T22:04:02.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4c23b23446b5b6d-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.648 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.648 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.648 DEBUG:boto:path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.648 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.648 DEBUG:boto:Method: GET 2024-04-13T22:04:02.648 DEBUG:boto:Path: /vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.648 DEBUG:boto:Data: 2024-04-13T22:04:02.648 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.648 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.648 DEBUG:boto:Params: {} 2024-04-13T22:04:02.649 DEBUG:boto:Token: None 2024-04-13T22:04:02.649 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.649 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f49a540eada3c65f-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.651 DEBUG:boto:b'vwnfns-48obj71000falseobj72024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.651 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.651 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.652 DEBUG:boto:path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.652 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.652 DEBUG:boto:Method: GET 2024-04-13T22:04:02.652 DEBUG:boto:Path: /vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.652 DEBUG:boto:Data: 2024-04-13T22:04:02.652 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.652 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.652 DEBUG:boto:Params: {} 2024-04-13T22:04:02.652 DEBUG:boto:Token: None 2024-04-13T22:04:02.652 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a076ed263dbdb0c4-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.655 DEBUG:boto:b'vwnfns-48obj71000falseobj72024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.655 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.655 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.655 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.655 DEBUG:boto:Path: /vwnfns-48/obj7 2024-04-13T22:04:02.655 DEBUG:boto:Data: 2024-04-13T22:04:02.656 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.656 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.656 DEBUG:boto:Params: {} 2024-04-13T22:04:02.656 DEBUG:boto:Token: None 2024-04-13T22:04:02.656 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7 2024-04-13T22:04:02.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gvpPiuNAWYa3Mo8cyP2Ukvrh1bE= 2024-04-13T22:04:02.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gvpPiuNAWYa3Mo8cyP2Ukvrh1bE=', 'Content-Length': '0'} 2024-04-13T22:04:02.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f096756a0996f78e-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.658 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.658 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.658 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.658 DEBUG:boto:Path: /vwnfns-48/obj7 2024-04-13T22:04:02.658 DEBUG:boto:Data: 2024-04-13T22:04:02.658 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.659 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.659 DEBUG:boto:Params: {} 2024-04-13T22:04:02.659 DEBUG:boto:Token: None 2024-04-13T22:04:02.659 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7 2024-04-13T22:04:02.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gvpPiuNAWYa3Mo8cyP2Ukvrh1bE= 2024-04-13T22:04:02.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gvpPiuNAWYa3Mo8cyP2Ukvrh1bE=', 'Content-Length': '0'} 2024-04-13T22:04:02.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020fc61eeaf11ff92-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:04:02.661 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.661 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.661 DEBUG:boto:Method: GET 2024-04-13T22:04:02.661 DEBUG:boto:Path: /vwnfns-48/obj7 2024-04-13T22:04:02.661 DEBUG:boto:Data: 2024-04-13T22:04:02.661 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.662 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.662 DEBUG:boto:Params: {} 2024-04-13T22:04:02.662 DEBUG:boto:Token: None 2024-04-13T22:04:02.662 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7 2024-04-13T22:04:02.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+f/bsLi/sECWfdwidEpidUBE/ho= 2024-04-13T22:04:02.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+f/bsLi/sECWfdwidEpidUBE/ho=', 'Content-Length': '0'} 2024-04-13T22:04:02.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038a5568a7b14a0ef-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.706 DEBUG:boto:path=/vwnfns-48/obj7 2024-04-13T22:04:02.706 DEBUG:boto:auth_path=/vwnfns-48/obj7 2024-04-13T22:04:02.707 DEBUG:boto:Method: GET 2024-04-13T22:04:02.707 DEBUG:boto:Path: /vwnfns-48/obj7 2024-04-13T22:04:02.707 DEBUG:boto:Data: 2024-04-13T22:04:02.707 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.707 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.707 DEBUG:boto:Params: {} 2024-04-13T22:04:02.707 DEBUG:boto:Token: None 2024-04-13T22:04:02.707 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj7 2024-04-13T22:04:02.707 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+f/bsLi/sECWfdwidEpidUBE/ho= 2024-04-13T22:04:02.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+f/bsLi/sECWfdwidEpidUBE/ho=', 'Content-Length': '0'} 2024-04-13T22:04:02.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ebb87109853818f-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.710 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.710 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.710 DEBUG:boto:path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.710 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.710 DEBUG:boto:Method: GET 2024-04-13T22:04:02.710 DEBUG:boto:Path: /vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.710 DEBUG:boto:Data: 2024-04-13T22:04:02.710 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.710 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.711 DEBUG:boto:Params: {} 2024-04-13T22:04:02.711 DEBUG:boto:Token: None 2024-04-13T22:04:02.711 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000221a739ee4d02d97-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.713 DEBUG:boto:b'vwnfns-48obj71000falseobj72024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.714 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.714 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.714 DEBUG:boto:path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.714 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.714 DEBUG:boto:Method: GET 2024-04-13T22:04:02.714 DEBUG:boto:Path: /vwnfns-48/?prefix=obj7 2024-04-13T22:04:02.714 DEBUG:boto:Data: 2024-04-13T22:04:02.714 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.715 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.715 DEBUG:boto:Params: {} 2024-04-13T22:04:02.715 DEBUG:boto:Token: None 2024-04-13T22:04:02.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e9d4af076ec9cd-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.717 DEBUG:boto:b'vwnfns-48obj71000falseobj72024-04-13T22:03:11.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:04:02.718 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.718 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.718 DEBUG:boto:path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.718 DEBUG:boto:auth_path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.718 DEBUG:boto:Method: GET 2024-04-13T22:04:02.718 DEBUG:boto:Path: /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.718 DEBUG:boto:Data: 2024-04-13T22:04:02.718 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.719 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.719 DEBUG:boto:Params: {} 2024-04-13T22:04:02.719 DEBUG:boto:Token: None 2024-04-13T22:04:02.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw= 2024-04-13T22:04:02.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw=', 'Content-Length': '0'} 2024-04-13T22:04:02.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024da267fc8c30873-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.721 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.721 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.721 DEBUG:boto:path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.721 DEBUG:boto:auth_path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.722 DEBUG:boto:Method: GET 2024-04-13T22:04:02.722 DEBUG:boto:Path: /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.722 DEBUG:boto:Data: 2024-04-13T22:04:02.722 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.722 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.722 DEBUG:boto:Params: {} 2024-04-13T22:04:02.722 DEBUG:boto:Token: None 2024-04-13T22:04:02.722 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.722 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw= 2024-04-13T22:04:02.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw=', 'Content-Length': '0'} 2024-04-13T22:04:02.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0b5c1b856aecf67-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.724 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.724 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.724 DEBUG:boto:path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.724 DEBUG:boto:auth_path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.725 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.725 DEBUG:boto:Path: /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.725 DEBUG:boto:Data: 2024-04-13T22:04:02.725 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.725 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.725 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.725 DEBUG:boto:Params: {} 2024-04-13T22:04:02.725 DEBUG:boto:Token: None 2024-04-13T22:04:02.725 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.725 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8vgJ/aHwfVG+yHKp7NyJj2PV4BE= 2024-04-13T22:04:02.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8vgJ/aHwfVG+yHKp7NyJj2PV4BE=', 'Content-Length': '0'} 2024-04-13T22:04:02.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063378b2a9a46a7cc-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.727 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.727 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.727 DEBUG:boto:path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.728 DEBUG:boto:auth_path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.728 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.728 DEBUG:boto:Path: /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.728 DEBUG:boto:Data: 2024-04-13T22:04:02.728 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.728 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.728 DEBUG:boto:Params: {} 2024-04-13T22:04:02.728 DEBUG:boto:Token: None 2024-04-13T22:04:02.728 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8vgJ/aHwfVG+yHKp7NyJj2PV4BE= 2024-04-13T22:04:02.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8vgJ/aHwfVG+yHKp7NyJj2PV4BE=', 'Content-Length': '0'} 2024-04-13T22:04:02.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005bde461d4c23038-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:04:02.730 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.730 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.730 DEBUG:boto:path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.730 DEBUG:boto:auth_path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.731 DEBUG:boto:Method: GET 2024-04-13T22:04:02.731 DEBUG:boto:Path: /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.731 DEBUG:boto:Data: 2024-04-13T22:04:02.731 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.731 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.731 DEBUG:boto:Params: {} 2024-04-13T22:04:02.731 DEBUG:boto:Token: None 2024-04-13T22:04:02.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.731 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw= 2024-04-13T22:04:02.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw=', 'Content-Length': '0'} 2024-04-13T22:04:02.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1e76c8a197365af-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.733 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.733 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.734 DEBUG:boto:path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.734 DEBUG:boto:auth_path=/vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.734 DEBUG:boto:Method: GET 2024-04-13T22:04:02.734 DEBUG:boto:Path: /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.734 DEBUG:boto:Data: 2024-04-13T22:04:02.734 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.734 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.734 DEBUG:boto:Params: {} 2024-04-13T22:04:02.734 DEBUG:boto:Token: None 2024-04-13T22:04:02.735 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8?versionId=null 2024-04-13T22:04:02.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw= 2024-04-13T22:04:02.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bxNixMmWJAso1JxOcH0InNTjDqw=', 'Content-Length': '0'} 2024-04-13T22:04:02.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001573ee23865f586a-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.737 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.737 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.737 DEBUG:boto:path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.737 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.737 DEBUG:boto:Method: GET 2024-04-13T22:04:02.737 DEBUG:boto:Path: /vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.737 DEBUG:boto:Data: 2024-04-13T22:04:02.737 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.738 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.738 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.738 DEBUG:boto:Params: {} 2024-04-13T22:04:02.738 DEBUG:boto:Token: None 2024-04-13T22:04:02.738 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.738 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051df85020badd327-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.740 DEBUG:boto:b'vwnfns-48obj81000falseobj82024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.741 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.741 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.741 DEBUG:boto:path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.741 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.741 DEBUG:boto:Method: GET 2024-04-13T22:04:02.741 DEBUG:boto:Path: /vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.741 DEBUG:boto:Data: 2024-04-13T22:04:02.741 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.742 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.742 DEBUG:boto:Params: {} 2024-04-13T22:04:02.742 DEBUG:boto:Token: None 2024-04-13T22:04:02.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f7e9a304381c240-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.744 DEBUG:boto:b'vwnfns-48obj81000falseobj82024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.744 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.745 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.745 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.745 DEBUG:boto:Path: /vwnfns-48/obj8 2024-04-13T22:04:02.745 DEBUG:boto:Data: 2024-04-13T22:04:02.745 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.745 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.745 DEBUG:boto:Params: {} 2024-04-13T22:04:02.745 DEBUG:boto:Token: None 2024-04-13T22:04:02.745 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8 2024-04-13T22:04:02.746 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IEuUcXRTJG8+83+3prSjW5HoW4w= 2024-04-13T22:04:02.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IEuUcXRTJG8+83+3prSjW5HoW4w=', 'Content-Length': '0'} 2024-04-13T22:04:02.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003307027c564d2520-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.747 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.747 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.747 DEBUG:boto:Method: HEAD 2024-04-13T22:04:02.748 DEBUG:boto:Path: /vwnfns-48/obj8 2024-04-13T22:04:02.748 DEBUG:boto:Data: 2024-04-13T22:04:02.748 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.748 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.748 DEBUG:boto:Params: {} 2024-04-13T22:04:02.748 DEBUG:boto:Token: None 2024-04-13T22:04:02.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8 2024-04-13T22:04:02.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IEuUcXRTJG8+83+3prSjW5HoW4w= 2024-04-13T22:04:02.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IEuUcXRTJG8+83+3prSjW5HoW4w=', 'Content-Length': '0'} 2024-04-13T22:04:02.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bafff0bd064a495-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:04:02.750 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.750 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.750 DEBUG:boto:Method: GET 2024-04-13T22:04:02.750 DEBUG:boto:Path: /vwnfns-48/obj8 2024-04-13T22:04:02.750 DEBUG:boto:Data: 2024-04-13T22:04:02.751 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.751 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.751 DEBUG:boto:Params: {} 2024-04-13T22:04:02.751 DEBUG:boto:Token: None 2024-04-13T22:04:02.751 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8 2024-04-13T22:04:02.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ERz+4O4g1hLFshmHWqbxybhbvf8= 2024-04-13T22:04:02.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ERz+4O4g1hLFshmHWqbxybhbvf8=', 'Content-Length': '0'} 2024-04-13T22:04:02.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-48:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf79f043df35a4f6-00661b0152-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.753 DEBUG:boto:path=/vwnfns-48/obj8 2024-04-13T22:04:02.753 DEBUG:boto:auth_path=/vwnfns-48/obj8 2024-04-13T22:04:02.753 DEBUG:boto:Method: GET 2024-04-13T22:04:02.753 DEBUG:boto:Path: /vwnfns-48/obj8 2024-04-13T22:04:02.753 DEBUG:boto:Data: 2024-04-13T22:04:02.753 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.754 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.754 DEBUG:boto:Params: {} 2024-04-13T22:04:02.754 DEBUG:boto:Token: None 2024-04-13T22:04:02.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/obj8 2024-04-13T22:04:02.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ERz+4O4g1hLFshmHWqbxybhbvf8= 2024-04-13T22:04:02.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ERz+4O4g1hLFshmHWqbxybhbvf8=', 'Content-Length': '0'} 2024-04-13T22:04:02.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009de9b5683d65573a-00661b0152-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.756 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.756 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.757 DEBUG:boto:path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.757 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.757 DEBUG:boto:Method: GET 2024-04-13T22:04:02.757 DEBUG:boto:Path: /vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.757 DEBUG:boto:Data: 2024-04-13T22:04:02.757 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:02.757 DEBUG:boto:Port: 8001 2024-04-13T22:04:02.757 DEBUG:boto:Params: {} 2024-04-13T22:04:02.757 DEBUG:boto:Token: None 2024-04-13T22:04:02.758 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3f893757955fc8-00661b0152-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.760 DEBUG:boto:b'vwnfns-48obj81000falseobj82024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.760 DEBUG:boto:path=/vwnfns-48/ 2024-04-13T22:04:02.760 DEBUG:boto:auth_path=/vwnfns-48/ 2024-04-13T22:04:02.761 DEBUG:boto:path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.761 DEBUG:boto:auth_path=/vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.761 DEBUG:boto:Method: GET 2024-04-13T22:04:02.761 DEBUG:boto:Path: /vwnfns-48/?prefix=obj8 2024-04-13T22:04:02.761 DEBUG:boto:Data: 2024-04-13T22:04:02.761 DEBUG:boto:Headers: {} 2024-04-13T22:04:02.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:02.761 DEBUG:boto:Port: 8000 2024-04-13T22:04:02.761 DEBUG:boto:Params: {} 2024-04-13T22:04:02.761 DEBUG:boto:Token: None 2024-04-13T22:04:02.762 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:02 GMT /vwnfns-48/ 2024-04-13T22:04:02.762 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ= 2024-04-13T22:04:02.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iMtQEBPS9LMOiw2T9LKNYcxijGQ=', 'Content-Length': '0'} 2024-04-13T22:04:02.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004907a3b704fd4967-00661b0152-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:02.764 DEBUG:boto:b'vwnfns-48obj81000falseobj82024-04-13T22:03:11.671Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:02.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-48 zones={a2, a1} 2024-04-13T22:04:02.764 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-49 2024-04-13T22:04:02.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:02.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:02.765 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:04:02.819 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:02.820 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:02.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:02.855+0000 7f3dcf4c0780 1 bucket sync caught up with source: 2024-04-13T22:04:02.855 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.30.6, 00000000002.29.1, 00000000004.34.6, 00000000002.28.1, 00000000004.25.6, 00000000002.39.1, 00000000002.50.1, 00000000003.26.1, 00000000002.28.1, 00000000002.15.1, 00000000002.27.1] 2024-04-13T22:04:02.855 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.30.6, 00000000002.29.1, 00000000004.34.6, 00000000002.28.1, 00000000004.25.6, 00000000002.39.1, 00000000002.50.1, 00000000003.26.1, 00000000002.28.1, 00000000002.15.1, 00000000002.27.1] 2024-04-13T22:04:02.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:02.855+0000 7f3dcf4c0780 0 bucket checkpoint complete 2024-04-13T22:04:02.868 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-49 2024-04-13T22:04:02.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:02.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:02.869 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:04:02.922 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:02.922 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:02.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:02.963+0000 7f0f1cec6780 1 waiting for incremental sync to catch up: 2024-04-13T22:04:02.963 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000002.1449.6, , 00000000002.1725.6, , 00000000002.1704.6, , , 00000000001.592.6, , , ] 2024-04-13T22:04:02.963 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1449.6, , 00000000002.1725.6, , 00000000002.1704.6, , , 00000000002.594.6, , , ] 2024-04-13T22:04:07.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:07.965+0000 7f0f1cec6780 1 bucket sync caught up with source: 2024-04-13T22:04:07.965 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000002.1449.6, , 00000000002.1725.6, , 00000000002.1704.6, , , 00000000002.594.6, , , ] 2024-04-13T22:04:07.965 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.1449.6, , 00000000002.1725.6, , 00000000002.1704.6, , , 00000000002.594.6, , , ] 2024-04-13T22:04:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:07.965+0000 7f0f1cec6780 0 bucket checkpoint complete 2024-04-13T22:04:07.980 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-49 zones={a2, a1} 2024-04-13T22:04:07.981 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:07.981 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:07.981 DEBUG:boto:Method: HEAD 2024-04-13T22:04:07.981 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:04:07.981 DEBUG:boto:Data: 2024-04-13T22:04:07.981 DEBUG:boto:Headers: {} 2024-04-13T22:04:07.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:07.981 DEBUG:boto:Port: 8001 2024-04-13T22:04:07.981 DEBUG:boto:Params: {} 2024-04-13T22:04:07.982 DEBUG:boto:Token: None 2024-04-13T22:04:07.982 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:07 GMT /vwnfns-49/ 2024-04-13T22:04:07.982 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WBMbO8qnTNYkWgPnQg/xsNYtwp0= 2024-04-13T22:04:07.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WBMbO8qnTNYkWgPnQg/xsNYtwp0=', 'Content-Length': '0'} 2024-04-13T22:04:07.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088790f7ae5ecc215-00661b0157-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:07.985 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:07.985 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:07.985 DEBUG:boto:Method: HEAD 2024-04-13T22:04:07.985 DEBUG:boto:Path: /vwnfns-49/ 2024-04-13T22:04:07.985 DEBUG:boto:Data: 2024-04-13T22:04:07.985 DEBUG:boto:Headers: {} 2024-04-13T22:04:07.985 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:07.985 DEBUG:boto:Port: 8000 2024-04-13T22:04:07.986 DEBUG:boto:Params: {} 2024-04-13T22:04:07.986 DEBUG:boto:Token: None 2024-04-13T22:04:07.986 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:07 GMT /vwnfns-49/ 2024-04-13T22:04:07.986 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WBMbO8qnTNYkWgPnQg/xsNYtwp0= 2024-04-13T22:04:07.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WBMbO8qnTNYkWgPnQg/xsNYtwp0=', 'Content-Length': '0'} 2024-04-13T22:04:07.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055993a05a6fcf47c-00661b0157-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:07.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:04:07.988 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:07.989 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:07.989 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:07.989 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:07.989 DEBUG:boto:Method: GET 2024-04-13T22:04:07.989 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:07.989 DEBUG:boto:Data: 2024-04-13T22:04:07.989 DEBUG:boto:Headers: {} 2024-04-13T22:04:07.989 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:07.989 DEBUG:boto:Port: 8001 2024-04-13T22:04:07.989 DEBUG:boto:Params: {} 2024-04-13T22:04:07.990 DEBUG:boto:Token: None 2024-04-13T22:04:07.990 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:07 GMT /vwnfns-49/?versions 2024-04-13T22:04:07.990 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lbiDRksTkBVz8bExXKATcJ9waiM= 2024-04-13T22:04:07.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lbiDRksTkBVz8bExXKATcJ9waiM=', 'Content-Length': '0'} 2024-04-13T22:04:07.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001add561407077afc-00661b0157-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:07.992 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:07.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:07.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:07.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:07.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:07.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:04:07.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:04:07.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:04:07.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:04:07.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:04:07.994 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:07.994 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:07.994 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:07.994 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:07.994 DEBUG:boto:Method: GET 2024-04-13T22:04:07.995 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:07.995 DEBUG:boto:Data: 2024-04-13T22:04:07.995 DEBUG:boto:Headers: {} 2024-04-13T22:04:07.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:07.995 DEBUG:boto:Port: 8000 2024-04-13T22:04:07.995 DEBUG:boto:Params: {} 2024-04-13T22:04:07.995 DEBUG:boto:Token: None 2024-04-13T22:04:07.995 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:07 GMT /vwnfns-49/?versions 2024-04-13T22:04:07.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lbiDRksTkBVz8bExXKATcJ9waiM= 2024-04-13T22:04:07.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lbiDRksTkBVz8bExXKATcJ9waiM=', 'Content-Length': '0'} 2024-04-13T22:04:07.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e534c937c43d4dc8-00661b0157-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:07.998 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:04:07.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:04:07.999 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:07.999 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.000 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:08.000 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:08.000 DEBUG:boto:Method: GET 2024-04-13T22:04:08.000 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:08.000 DEBUG:boto:Data: 2024-04-13T22:04:08.000 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.000 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.000 DEBUG:boto:Params: {} 2024-04-13T22:04:08.000 DEBUG:boto:Token: None 2024-04-13T22:04:08.001 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/?versions 2024-04-13T22:04:08.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WYI5sziRrBC74hH6WKzFGH/LtAo= 2024-04-13T22:04:08.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WYI5sziRrBC74hH6WKzFGH/LtAo=', 'Content-Length': '0'} 2024-04-13T22:04:08.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd721edca8cad265-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.003 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.004 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.004 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.004 DEBUG:boto:path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:08.004 DEBUG:boto:auth_path=/vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:08.004 DEBUG:boto:Method: GET 2024-04-13T22:04:08.004 DEBUG:boto:Path: /vwnfns-49/?versions&max-keys=999 2024-04-13T22:04:08.005 DEBUG:boto:Data: 2024-04-13T22:04:08.005 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.005 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.005 DEBUG:boto:Params: {} 2024-04-13T22:04:08.005 DEBUG:boto:Token: None 2024-04-13T22:04:08.005 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/?versions 2024-04-13T22:04:08.005 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WYI5sziRrBC74hH6WKzFGH/LtAo= 2024-04-13T22:04:08.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WYI5sziRrBC74hH6WKzFGH/LtAo=', 'Content-Length': '0'} 2024-04-13T22:04:08.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f284716960558fe4-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.008 DEBUG:boto:b'vwnfns-49999falseobj1nulltrue2024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:08.009 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.009 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.009 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.009 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.009 DEBUG:boto:Method: GET 2024-04-13T22:04:08.009 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.009 DEBUG:boto:Data: 2024-04-13T22:04:08.009 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.009 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.010 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.010 DEBUG:boto:Params: {} 2024-04-13T22:04:08.010 DEBUG:boto:Token: None 2024-04-13T22:04:08.010 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.010 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g= 2024-04-13T22:04:08.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g=', 'Content-Length': '0'} 2024-04-13T22:04:08.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006429c8a155c01085-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.054 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.055 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.055 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.055 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.055 DEBUG:boto:Method: GET 2024-04-13T22:04:08.055 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.055 DEBUG:boto:Data: 2024-04-13T22:04:08.055 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.055 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.055 DEBUG:boto:Params: {} 2024-04-13T22:04:08.056 DEBUG:boto:Token: None 2024-04-13T22:04:08.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g= 2024-04-13T22:04:08.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g=', 'Content-Length': '0'} 2024-04-13T22:04:08.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6700d0cb3148725-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.058 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.058 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.058 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.059 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.059 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.059 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.059 DEBUG:boto:Data: 2024-04-13T22:04:08.059 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.059 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.059 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.059 DEBUG:boto:Params: {} 2024-04-13T22:04:08.059 DEBUG:boto:Token: None 2024-04-13T22:04:08.059 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.060 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2VHhtwPpm5ZDIEXWRWzw0MwZTM= 2024-04-13T22:04:08.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2VHhtwPpm5ZDIEXWRWzw0MwZTM=', 'Content-Length': '0'} 2024-04-13T22:04:08.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d227cbbf1cc9177-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.062 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.062 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.062 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.062 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.062 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.062 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.062 DEBUG:boto:Data: 2024-04-13T22:04:08.063 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.063 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.063 DEBUG:boto:Params: {} 2024-04-13T22:04:08.063 DEBUG:boto:Token: None 2024-04-13T22:04:08.063 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2VHhtwPpm5ZDIEXWRWzw0MwZTM= 2024-04-13T22:04:08.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2VHhtwPpm5ZDIEXWRWzw0MwZTM=', 'Content-Length': '0'} 2024-04-13T22:04:08.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f815c574da6e05a-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:08.065 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.066 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.066 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.066 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.066 DEBUG:boto:Method: GET 2024-04-13T22:04:08.066 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.066 DEBUG:boto:Data: 2024-04-13T22:04:08.066 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.066 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.066 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.066 DEBUG:boto:Params: {} 2024-04-13T22:04:08.067 DEBUG:boto:Token: None 2024-04-13T22:04:08.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g= 2024-04-13T22:04:08.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g=', 'Content-Length': '0'} 2024-04-13T22:04:08.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f69e310a3ce6c52-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.069 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.069 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.069 DEBUG:boto:path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.069 DEBUG:boto:auth_path=/vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.069 DEBUG:boto:Method: GET 2024-04-13T22:04:08.069 DEBUG:boto:Path: /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.069 DEBUG:boto:Data: 2024-04-13T22:04:08.069 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.070 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.070 DEBUG:boto:Params: {} 2024-04-13T22:04:08.070 DEBUG:boto:Token: None 2024-04-13T22:04:08.070 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1?versionId=null 2024-04-13T22:04:08.070 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g= 2024-04-13T22:04:08.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CNkCERmvBXEsTKlMa+EXYGAbY0g=', 'Content-Length': '0'} 2024-04-13T22:04:08.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e45fde9912b8af0-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.072 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.072 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.072 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.072 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.072 DEBUG:boto:Method: GET 2024-04-13T22:04:08.073 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.073 DEBUG:boto:Data: 2024-04-13T22:04:08.073 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.073 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.073 DEBUG:boto:Params: {} 2024-04-13T22:04:08.073 DEBUG:boto:Token: None 2024-04-13T22:04:08.073 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.073 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8aacabc67206e4f-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.076 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.076 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.076 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.077 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.077 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.077 DEBUG:boto:Method: GET 2024-04-13T22:04:08.077 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.077 DEBUG:boto:Data: 2024-04-13T22:04:08.077 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.077 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.077 DEBUG:boto:Params: {} 2024-04-13T22:04:08.077 DEBUG:boto:Token: None 2024-04-13T22:04:08.078 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.078 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee2800314fc95e8-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.080 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.081 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.081 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.081 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.081 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:04:08.081 DEBUG:boto:Data: 2024-04-13T22:04:08.081 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.081 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.081 DEBUG:boto:Params: {} 2024-04-13T22:04:08.081 DEBUG:boto:Token: None 2024-04-13T22:04:08.082 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1 2024-04-13T22:04:08.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BNzDHQqk6yeoUYNFWPullyDt0OQ= 2024-04-13T22:04:08.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BNzDHQqk6yeoUYNFWPullyDt0OQ=', 'Content-Length': '0'} 2024-04-13T22:04:08.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c5137ef1f19c65e-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.084 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.084 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.084 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.084 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:04:08.084 DEBUG:boto:Data: 2024-04-13T22:04:08.084 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.084 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.084 DEBUG:boto:Params: {} 2024-04-13T22:04:08.085 DEBUG:boto:Token: None 2024-04-13T22:04:08.085 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1 2024-04-13T22:04:08.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BNzDHQqk6yeoUYNFWPullyDt0OQ= 2024-04-13T22:04:08.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BNzDHQqk6yeoUYNFWPullyDt0OQ=', 'Content-Length': '0'} 2024-04-13T22:04:08.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee7964530dad831-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:08.087 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.087 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.087 DEBUG:boto:Method: GET 2024-04-13T22:04:08.087 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:04:08.087 DEBUG:boto:Data: 2024-04-13T22:04:08.087 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.087 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.088 DEBUG:boto:Params: {} 2024-04-13T22:04:08.088 DEBUG:boto:Token: None 2024-04-13T22:04:08.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1 2024-04-13T22:04:08.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arXmmlODUTIfWK2UbWxhJDxOBMk= 2024-04-13T22:04:08.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arXmmlODUTIfWK2UbWxhJDxOBMk=', 'Content-Length': '0'} 2024-04-13T22:04:08.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000db8690f55119cb4-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.090 DEBUG:boto:path=/vwnfns-49/obj1 2024-04-13T22:04:08.090 DEBUG:boto:auth_path=/vwnfns-49/obj1 2024-04-13T22:04:08.090 DEBUG:boto:Method: GET 2024-04-13T22:04:08.090 DEBUG:boto:Path: /vwnfns-49/obj1 2024-04-13T22:04:08.090 DEBUG:boto:Data: 2024-04-13T22:04:08.090 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.091 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.091 DEBUG:boto:Params: {} 2024-04-13T22:04:08.091 DEBUG:boto:Token: None 2024-04-13T22:04:08.091 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj1 2024-04-13T22:04:08.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:arXmmlODUTIfWK2UbWxhJDxOBMk= 2024-04-13T22:04:08.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:arXmmlODUTIfWK2UbWxhJDxOBMk=', 'Content-Length': '0'} 2024-04-13T22:04:08.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000703d115531873cf0-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.093 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.093 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.093 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.094 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.094 DEBUG:boto:Method: GET 2024-04-13T22:04:08.094 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.094 DEBUG:boto:Data: 2024-04-13T22:04:08.094 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.094 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.094 DEBUG:boto:Params: {} 2024-04-13T22:04:08.094 DEBUG:boto:Token: None 2024-04-13T22:04:08.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8098198f014815f-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.097 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.097 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.097 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.098 DEBUG:boto:path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.098 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.098 DEBUG:boto:Method: GET 2024-04-13T22:04:08.098 DEBUG:boto:Path: /vwnfns-49/?prefix=obj1 2024-04-13T22:04:08.098 DEBUG:boto:Data: 2024-04-13T22:04:08.098 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.098 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.098 DEBUG:boto:Params: {} 2024-04-13T22:04:08.098 DEBUG:boto:Token: None 2024-04-13T22:04:08.099 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ad79b39d6c35303-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.101 DEBUG:boto:b'vwnfns-49obj11000falseobj12024-04-13T22:02:44.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:08.101 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.102 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.102 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.102 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.102 DEBUG:boto:Method: GET 2024-04-13T22:04:08.102 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.102 DEBUG:boto:Data: 2024-04-13T22:04:08.102 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.102 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.102 DEBUG:boto:Params: {} 2024-04-13T22:04:08.102 DEBUG:boto:Token: None 2024-04-13T22:04:08.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0= 2024-04-13T22:04:08.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0=', 'Content-Length': '0'} 2024-04-13T22:04:08.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccf07b30cc9111de-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.105 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.105 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.105 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.105 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.105 DEBUG:boto:Method: GET 2024-04-13T22:04:08.105 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.105 DEBUG:boto:Data: 2024-04-13T22:04:08.105 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.105 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.106 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.106 DEBUG:boto:Params: {} 2024-04-13T22:04:08.106 DEBUG:boto:Token: None 2024-04-13T22:04:08.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0= 2024-04-13T22:04:08.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0=', 'Content-Length': '0'} 2024-04-13T22:04:08.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000314acb01858128ca-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.108 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.109 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.109 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.109 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.109 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.109 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.109 DEBUG:boto:Data: 2024-04-13T22:04:08.109 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.109 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.109 DEBUG:boto:Params: {} 2024-04-13T22:04:08.109 DEBUG:boto:Token: None 2024-04-13T22:04:08.110 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YpL5xgStQnP/GVYxU1Eut6uZRNs= 2024-04-13T22:04:08.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YpL5xgStQnP/GVYxU1Eut6uZRNs=', 'Content-Length': '0'} 2024-04-13T22:04:08.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000229408ecbf5c40a0-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.112 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.112 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.112 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.112 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.112 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.112 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.112 DEBUG:boto:Data: 2024-04-13T22:04:08.112 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.112 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.113 DEBUG:boto:Params: {} 2024-04-13T22:04:08.113 DEBUG:boto:Token: None 2024-04-13T22:04:08.113 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YpL5xgStQnP/GVYxU1Eut6uZRNs= 2024-04-13T22:04:08.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YpL5xgStQnP/GVYxU1Eut6uZRNs=', 'Content-Length': '0'} 2024-04-13T22:04:08.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a129b946f3629c-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:08.115 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.115 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.115 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.115 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.116 DEBUG:boto:Method: GET 2024-04-13T22:04:08.116 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.116 DEBUG:boto:Data: 2024-04-13T22:04:08.116 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.116 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.116 DEBUG:boto:Params: {} 2024-04-13T22:04:08.116 DEBUG:boto:Token: None 2024-04-13T22:04:08.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0= 2024-04-13T22:04:08.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0=', 'Content-Length': '0'} 2024-04-13T22:04:08.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3730af7bde1df8a-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.119 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.119 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.119 DEBUG:boto:path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.119 DEBUG:boto:auth_path=/vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.119 DEBUG:boto:Method: GET 2024-04-13T22:04:08.119 DEBUG:boto:Path: /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.119 DEBUG:boto:Data: 2024-04-13T22:04:08.119 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.119 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.120 DEBUG:boto:Params: {} 2024-04-13T22:04:08.120 DEBUG:boto:Token: None 2024-04-13T22:04:08.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2?versionId=null 2024-04-13T22:04:08.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0= 2024-04-13T22:04:08.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dwgra6lJVr2QITB3UPsiPAwmwL0=', 'Content-Length': '0'} 2024-04-13T22:04:08.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34f607753517c19-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.122 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.122 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.122 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.122 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.122 DEBUG:boto:Method: GET 2024-04-13T22:04:08.123 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.123 DEBUG:boto:Data: 2024-04-13T22:04:08.123 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.123 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.123 DEBUG:boto:Params: {} 2024-04-13T22:04:08.123 DEBUG:boto:Token: None 2024-04-13T22:04:08.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da1537eee5c4e845-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.166 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.167 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.167 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.167 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.167 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.167 DEBUG:boto:Method: GET 2024-04-13T22:04:08.167 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.168 DEBUG:boto:Data: 2024-04-13T22:04:08.168 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.168 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.168 DEBUG:boto:Params: {} 2024-04-13T22:04:08.168 DEBUG:boto:Token: None 2024-04-13T22:04:08.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c37b6312a5763ff-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.171 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.171 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.171 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.172 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.172 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:04:08.172 DEBUG:boto:Data: 2024-04-13T22:04:08.172 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.172 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.172 DEBUG:boto:Params: {} 2024-04-13T22:04:08.172 DEBUG:boto:Token: None 2024-04-13T22:04:08.172 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2 2024-04-13T22:04:08.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rjNbCt0jEK6B/g2DvX/AMrfp7P8= 2024-04-13T22:04:08.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rjNbCt0jEK6B/g2DvX/AMrfp7P8=', 'Content-Length': '0'} 2024-04-13T22:04:08.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000459b7035295adc50-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.175 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.175 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.175 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.175 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:04:08.175 DEBUG:boto:Data: 2024-04-13T22:04:08.175 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.175 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.175 DEBUG:boto:Params: {} 2024-04-13T22:04:08.175 DEBUG:boto:Token: None 2024-04-13T22:04:08.176 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2 2024-04-13T22:04:08.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rjNbCt0jEK6B/g2DvX/AMrfp7P8= 2024-04-13T22:04:08.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rjNbCt0jEK6B/g2DvX/AMrfp7P8=', 'Content-Length': '0'} 2024-04-13T22:04:08.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001253d140304e90f2-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:08.178 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.178 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.178 DEBUG:boto:Method: GET 2024-04-13T22:04:08.178 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:04:08.178 DEBUG:boto:Data: 2024-04-13T22:04:08.178 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.179 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.179 DEBUG:boto:Params: {} 2024-04-13T22:04:08.179 DEBUG:boto:Token: None 2024-04-13T22:04:08.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2 2024-04-13T22:04:08.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jktUUJrws4TxQ7uQU7BgqXEbpn8= 2024-04-13T22:04:08.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jktUUJrws4TxQ7uQU7BgqXEbpn8=', 'Content-Length': '0'} 2024-04-13T22:04:08.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f19278aac81a2c5b-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.181 DEBUG:boto:path=/vwnfns-49/obj2 2024-04-13T22:04:08.181 DEBUG:boto:auth_path=/vwnfns-49/obj2 2024-04-13T22:04:08.181 DEBUG:boto:Method: GET 2024-04-13T22:04:08.181 DEBUG:boto:Path: /vwnfns-49/obj2 2024-04-13T22:04:08.182 DEBUG:boto:Data: 2024-04-13T22:04:08.182 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.182 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.182 DEBUG:boto:Params: {} 2024-04-13T22:04:08.182 DEBUG:boto:Token: None 2024-04-13T22:04:08.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj2 2024-04-13T22:04:08.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jktUUJrws4TxQ7uQU7BgqXEbpn8= 2024-04-13T22:04:08.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jktUUJrws4TxQ7uQU7BgqXEbpn8=', 'Content-Length': '0'} 2024-04-13T22:04:08.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce94e3e7e7acfd73-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.230 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.231 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.231 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.231 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.231 DEBUG:boto:Method: GET 2024-04-13T22:04:08.231 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.231 DEBUG:boto:Data: 2024-04-13T22:04:08.231 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.231 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.231 DEBUG:boto:Params: {} 2024-04-13T22:04:08.232 DEBUG:boto:Token: None 2024-04-13T22:04:08.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006611753470076f9d-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.235 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.235 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.235 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.235 DEBUG:boto:path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.235 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.235 DEBUG:boto:Method: GET 2024-04-13T22:04:08.236 DEBUG:boto:Path: /vwnfns-49/?prefix=obj2 2024-04-13T22:04:08.236 DEBUG:boto:Data: 2024-04-13T22:04:08.236 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.236 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.236 DEBUG:boto:Params: {} 2024-04-13T22:04:08.236 DEBUG:boto:Token: None 2024-04-13T22:04:08.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb57591bd9fcb67-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.239 DEBUG:boto:b'vwnfns-49obj21000falseobj22024-04-13T22:02:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:08.240 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.240 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.240 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.240 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.240 DEBUG:boto:Method: GET 2024-04-13T22:04:08.240 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.240 DEBUG:boto:Data: 2024-04-13T22:04:08.240 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.240 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.240 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.241 DEBUG:boto:Params: {} 2024-04-13T22:04:08.241 DEBUG:boto:Token: None 2024-04-13T22:04:08.241 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.241 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4= 2024-04-13T22:04:08.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4=', 'Content-Length': '0'} 2024-04-13T22:04:08.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b0c453818fa566a-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.286 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.287 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.287 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.287 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.287 DEBUG:boto:Method: GET 2024-04-13T22:04:08.287 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.287 DEBUG:boto:Data: 2024-04-13T22:04:08.287 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.287 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.287 DEBUG:boto:Params: {} 2024-04-13T22:04:08.287 DEBUG:boto:Token: None 2024-04-13T22:04:08.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4= 2024-04-13T22:04:08.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4=', 'Content-Length': '0'} 2024-04-13T22:04:08.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9d7a3c09124a98d-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.290 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.290 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.290 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.290 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.291 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.291 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.291 DEBUG:boto:Data: 2024-04-13T22:04:08.291 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.291 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.291 DEBUG:boto:Params: {} 2024-04-13T22:04:08.291 DEBUG:boto:Token: None 2024-04-13T22:04:08.291 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+CxtXaFiJznyu/u2r5EJse8slbk= 2024-04-13T22:04:08.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+CxtXaFiJznyu/u2r5EJse8slbk=', 'Content-Length': '0'} 2024-04-13T22:04:08.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef8c1162eb667777-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.294 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.294 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.294 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.294 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.294 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.294 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.294 DEBUG:boto:Data: 2024-04-13T22:04:08.294 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.294 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.294 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.294 DEBUG:boto:Params: {} 2024-04-13T22:04:08.295 DEBUG:boto:Token: None 2024-04-13T22:04:08.295 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+CxtXaFiJznyu/u2r5EJse8slbk= 2024-04-13T22:04:08.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+CxtXaFiJznyu/u2r5EJse8slbk=', 'Content-Length': '0'} 2024-04-13T22:04:08.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcef0c2df058e4e0-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:08.297 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.297 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.297 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.297 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.297 DEBUG:boto:Method: GET 2024-04-13T22:04:08.297 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.297 DEBUG:boto:Data: 2024-04-13T22:04:08.297 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.297 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.298 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.298 DEBUG:boto:Params: {} 2024-04-13T22:04:08.298 DEBUG:boto:Token: None 2024-04-13T22:04:08.298 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.298 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4= 2024-04-13T22:04:08.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4=', 'Content-Length': '0'} 2024-04-13T22:04:08.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f235d1110d8dec7e-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.300 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.300 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.300 DEBUG:boto:path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.300 DEBUG:boto:auth_path=/vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.300 DEBUG:boto:Method: GET 2024-04-13T22:04:08.300 DEBUG:boto:Path: /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.300 DEBUG:boto:Data: 2024-04-13T22:04:08.300 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.301 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.301 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.301 DEBUG:boto:Params: {} 2024-04-13T22:04:08.301 DEBUG:boto:Token: None 2024-04-13T22:04:08.301 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3?versionId=null 2024-04-13T22:04:08.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4= 2024-04-13T22:04:08.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IkwtEZXJZKjHuYlzrcu7k9KwdD4=', 'Content-Length': '0'} 2024-04-13T22:04:08.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e2bf032749f0fc2-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.304 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.304 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.304 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.304 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.304 DEBUG:boto:Method: GET 2024-04-13T22:04:08.304 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.305 DEBUG:boto:Data: 2024-04-13T22:04:08.305 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.305 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.305 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.305 DEBUG:boto:Params: {} 2024-04-13T22:04:08.305 DEBUG:boto:Token: None 2024-04-13T22:04:08.305 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cc8c5d78f4924cd-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.308 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.309 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.309 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.309 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.309 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.309 DEBUG:boto:Method: GET 2024-04-13T22:04:08.309 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.309 DEBUG:boto:Data: 2024-04-13T22:04:08.309 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.309 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.309 DEBUG:boto:Params: {} 2024-04-13T22:04:08.310 DEBUG:boto:Token: None 2024-04-13T22:04:08.310 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001543cb8b56c19049-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.313 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.313 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.313 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.313 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.313 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:04:08.313 DEBUG:boto:Data: 2024-04-13T22:04:08.313 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.314 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.314 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.314 DEBUG:boto:Params: {} 2024-04-13T22:04:08.314 DEBUG:boto:Token: None 2024-04-13T22:04:08.314 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3 2024-04-13T22:04:08.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gHpb9LoZhM9aWU8+5xrd6BoW4c0= 2024-04-13T22:04:08.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gHpb9LoZhM9aWU8+5xrd6BoW4c0=', 'Content-Length': '0'} 2024-04-13T22:04:08.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003463e2204196b7ee-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.316 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.316 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.316 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.316 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:04:08.316 DEBUG:boto:Data: 2024-04-13T22:04:08.316 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.316 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.317 DEBUG:boto:Params: {} 2024-04-13T22:04:08.317 DEBUG:boto:Token: None 2024-04-13T22:04:08.317 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3 2024-04-13T22:04:08.317 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gHpb9LoZhM9aWU8+5xrd6BoW4c0= 2024-04-13T22:04:08.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gHpb9LoZhM9aWU8+5xrd6BoW4c0=', 'Content-Length': '0'} 2024-04-13T22:04:08.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a24b2fc0e32e02cd-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:08.319 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.319 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.319 DEBUG:boto:Method: GET 2024-04-13T22:04:08.319 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:04:08.319 DEBUG:boto:Data: 2024-04-13T22:04:08.319 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.319 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.320 DEBUG:boto:Params: {} 2024-04-13T22:04:08.320 DEBUG:boto:Token: None 2024-04-13T22:04:08.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3 2024-04-13T22:04:08.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:STosgbyPIhAbw36PaWS8Qsdhmfo= 2024-04-13T22:04:08.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:STosgbyPIhAbw36PaWS8Qsdhmfo=', 'Content-Length': '0'} 2024-04-13T22:04:08.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007629a2d0f65bc4dc-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.322 DEBUG:boto:path=/vwnfns-49/obj3 2024-04-13T22:04:08.322 DEBUG:boto:auth_path=/vwnfns-49/obj3 2024-04-13T22:04:08.322 DEBUG:boto:Method: GET 2024-04-13T22:04:08.322 DEBUG:boto:Path: /vwnfns-49/obj3 2024-04-13T22:04:08.322 DEBUG:boto:Data: 2024-04-13T22:04:08.322 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.323 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.323 DEBUG:boto:Params: {} 2024-04-13T22:04:08.323 DEBUG:boto:Token: None 2024-04-13T22:04:08.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj3 2024-04-13T22:04:08.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:STosgbyPIhAbw36PaWS8Qsdhmfo= 2024-04-13T22:04:08.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:STosgbyPIhAbw36PaWS8Qsdhmfo=', 'Content-Length': '0'} 2024-04-13T22:04:08.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047048548d783efe5-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.366 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.367 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.367 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.367 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.367 DEBUG:boto:Method: GET 2024-04-13T22:04:08.367 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.367 DEBUG:boto:Data: 2024-04-13T22:04:08.367 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.367 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.367 DEBUG:boto:Params: {} 2024-04-13T22:04:08.367 DEBUG:boto:Token: None 2024-04-13T22:04:08.368 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca108d62e12b085-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.370 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.371 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.371 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.371 DEBUG:boto:path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.371 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.371 DEBUG:boto:Method: GET 2024-04-13T22:04:08.371 DEBUG:boto:Path: /vwnfns-49/?prefix=obj3 2024-04-13T22:04:08.371 DEBUG:boto:Data: 2024-04-13T22:04:08.371 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.371 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.372 DEBUG:boto:Params: {} 2024-04-13T22:04:08.372 DEBUG:boto:Token: None 2024-04-13T22:04:08.372 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037681477c6900ed1-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.374 DEBUG:boto:b'vwnfns-49obj31000falseobj32024-04-13T22:02:44.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:08.375 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.375 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.375 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.375 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.375 DEBUG:boto:Method: GET 2024-04-13T22:04:08.375 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.375 DEBUG:boto:Data: 2024-04-13T22:04:08.375 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.375 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.376 DEBUG:boto:Params: {} 2024-04-13T22:04:08.376 DEBUG:boto:Token: None 2024-04-13T22:04:08.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4= 2024-04-13T22:04:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4=', 'Content-Length': '0'} 2024-04-13T22:04:08.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0cac3f45ce584b5-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.378 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.378 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.378 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.378 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.378 DEBUG:boto:Method: GET 2024-04-13T22:04:08.378 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.378 DEBUG:boto:Data: 2024-04-13T22:04:08.378 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.379 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.379 DEBUG:boto:Params: {} 2024-04-13T22:04:08.379 DEBUG:boto:Token: None 2024-04-13T22:04:08.379 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.379 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4= 2024-04-13T22:04:08.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4=', 'Content-Length': '0'} 2024-04-13T22:04:08.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59714c12d4d79d4-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.381 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.381 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.382 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.382 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.382 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.382 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.382 DEBUG:boto:Data: 2024-04-13T22:04:08.382 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.382 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.382 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.382 DEBUG:boto:Params: {} 2024-04-13T22:04:08.382 DEBUG:boto:Token: None 2024-04-13T22:04:08.383 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HSmpusPeAsSE2X07MS7dmkCG42Y= 2024-04-13T22:04:08.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HSmpusPeAsSE2X07MS7dmkCG42Y=', 'Content-Length': '0'} 2024-04-13T22:04:08.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002264b9c77c2f70be-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.385 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.385 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.385 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.385 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.385 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.385 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.385 DEBUG:boto:Data: 2024-04-13T22:04:08.385 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.385 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.385 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.385 DEBUG:boto:Params: {} 2024-04-13T22:04:08.385 DEBUG:boto:Token: None 2024-04-13T22:04:08.386 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.386 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HSmpusPeAsSE2X07MS7dmkCG42Y= 2024-04-13T22:04:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HSmpusPeAsSE2X07MS7dmkCG42Y=', 'Content-Length': '0'} 2024-04-13T22:04:08.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fcd36d3b460b78d-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:08.388 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.388 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.388 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.388 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.388 DEBUG:boto:Method: GET 2024-04-13T22:04:08.388 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.388 DEBUG:boto:Data: 2024-04-13T22:04:08.388 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.389 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.389 DEBUG:boto:Params: {} 2024-04-13T22:04:08.389 DEBUG:boto:Token: None 2024-04-13T22:04:08.389 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.389 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4= 2024-04-13T22:04:08.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4=', 'Content-Length': '0'} 2024-04-13T22:04:08.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1d1aa4f10c45b02-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.391 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.391 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.391 DEBUG:boto:path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.391 DEBUG:boto:auth_path=/vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.391 DEBUG:boto:Method: GET 2024-04-13T22:04:08.391 DEBUG:boto:Path: /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.391 DEBUG:boto:Data: 2024-04-13T22:04:08.391 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.392 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.392 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.392 DEBUG:boto:Params: {} 2024-04-13T22:04:08.392 DEBUG:boto:Token: None 2024-04-13T22:04:08.392 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4?versionId=null 2024-04-13T22:04:08.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4= 2024-04-13T22:04:08.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d0zyMnAgubSiXsemQ/1rvxnL6G4=', 'Content-Length': '0'} 2024-04-13T22:04:08.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054b46f91d14153af-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.394 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.394 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.394 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.395 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.395 DEBUG:boto:Method: GET 2024-04-13T22:04:08.395 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.395 DEBUG:boto:Data: 2024-04-13T22:04:08.395 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.395 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.395 DEBUG:boto:Params: {} 2024-04-13T22:04:08.395 DEBUG:boto:Token: None 2024-04-13T22:04:08.395 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f372aec20e97d79-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.442 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.443 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.443 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.443 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.443 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.443 DEBUG:boto:Method: GET 2024-04-13T22:04:08.443 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.443 DEBUG:boto:Data: 2024-04-13T22:04:08.443 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.444 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.444 DEBUG:boto:Params: {} 2024-04-13T22:04:08.444 DEBUG:boto:Token: None 2024-04-13T22:04:08.444 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d704046b357abd7-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.446 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.447 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.447 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.447 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.447 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:04:08.447 DEBUG:boto:Data: 2024-04-13T22:04:08.447 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.447 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.447 DEBUG:boto:Params: {} 2024-04-13T22:04:08.448 DEBUG:boto:Token: None 2024-04-13T22:04:08.448 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4 2024-04-13T22:04:08.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PwoTqWl2k2rZ9ABjElN92ihMEQA= 2024-04-13T22:04:08.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PwoTqWl2k2rZ9ABjElN92ihMEQA=', 'Content-Length': '0'} 2024-04-13T22:04:08.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002794a2297ff2c864-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.450 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.450 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.450 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.450 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:04:08.450 DEBUG:boto:Data: 2024-04-13T22:04:08.450 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.450 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.450 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.450 DEBUG:boto:Params: {} 2024-04-13T22:04:08.451 DEBUG:boto:Token: None 2024-04-13T22:04:08.451 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4 2024-04-13T22:04:08.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PwoTqWl2k2rZ9ABjElN92ihMEQA= 2024-04-13T22:04:08.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PwoTqWl2k2rZ9ABjElN92ihMEQA=', 'Content-Length': '0'} 2024-04-13T22:04:08.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cdbf14fc40fbe51-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:08.453 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.453 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.453 DEBUG:boto:Method: GET 2024-04-13T22:04:08.453 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:04:08.453 DEBUG:boto:Data: 2024-04-13T22:04:08.453 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.453 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.453 DEBUG:boto:Params: {} 2024-04-13T22:04:08.453 DEBUG:boto:Token: None 2024-04-13T22:04:08.453 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4 2024-04-13T22:04:08.454 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+YT3Nizx8iMMqaaHATm/AMnVv4E= 2024-04-13T22:04:08.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+YT3Nizx8iMMqaaHATm/AMnVv4E=', 'Content-Length': '0'} 2024-04-13T22:04:08.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd89edb17e83ed20-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.455 DEBUG:boto:path=/vwnfns-49/obj4 2024-04-13T22:04:08.455 DEBUG:boto:auth_path=/vwnfns-49/obj4 2024-04-13T22:04:08.456 DEBUG:boto:Method: GET 2024-04-13T22:04:08.456 DEBUG:boto:Path: /vwnfns-49/obj4 2024-04-13T22:04:08.456 DEBUG:boto:Data: 2024-04-13T22:04:08.456 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.456 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.456 DEBUG:boto:Params: {} 2024-04-13T22:04:08.456 DEBUG:boto:Token: None 2024-04-13T22:04:08.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj4 2024-04-13T22:04:08.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+YT3Nizx8iMMqaaHATm/AMnVv4E= 2024-04-13T22:04:08.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+YT3Nizx8iMMqaaHATm/AMnVv4E=', 'Content-Length': '0'} 2024-04-13T22:04:08.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:02:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7df37fc603034e0-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.502 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.502 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.503 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.503 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.503 DEBUG:boto:Method: GET 2024-04-13T22:04:08.503 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.503 DEBUG:boto:Data: 2024-04-13T22:04:08.503 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.503 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.503 DEBUG:boto:Params: {} 2024-04-13T22:04:08.503 DEBUG:boto:Token: None 2024-04-13T22:04:08.504 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6aec5137d663e2-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.506 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.506 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.507 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.507 DEBUG:boto:path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.507 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.507 DEBUG:boto:Method: GET 2024-04-13T22:04:08.507 DEBUG:boto:Path: /vwnfns-49/?prefix=obj4 2024-04-13T22:04:08.507 DEBUG:boto:Data: 2024-04-13T22:04:08.507 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.507 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.507 DEBUG:boto:Params: {} 2024-04-13T22:04:08.507 DEBUG:boto:Token: None 2024-04-13T22:04:08.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c75749b54ad7f4-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.510 DEBUG:boto:b'vwnfns-49obj41000falseobj42024-04-13T22:02:44.531Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:04:08.510 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.510 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.510 DEBUG:boto:path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.511 DEBUG:boto:auth_path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.511 DEBUG:boto:Method: GET 2024-04-13T22:04:08.511 DEBUG:boto:Path: /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.511 DEBUG:boto:Data: 2024-04-13T22:04:08.511 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.511 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.511 DEBUG:boto:Params: {} 2024-04-13T22:04:08.511 DEBUG:boto:Token: None 2024-04-13T22:04:08.512 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg= 2024-04-13T22:04:08.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg=', 'Content-Length': '0'} 2024-04-13T22:04:08.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000546d6a10e4e8b91c-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.514 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.514 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.514 DEBUG:boto:path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.514 DEBUG:boto:auth_path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.514 DEBUG:boto:Method: GET 2024-04-13T22:04:08.514 DEBUG:boto:Path: /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.514 DEBUG:boto:Data: 2024-04-13T22:04:08.514 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.515 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.515 DEBUG:boto:Params: {} 2024-04-13T22:04:08.515 DEBUG:boto:Token: None 2024-04-13T22:04:08.515 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg= 2024-04-13T22:04:08.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg=', 'Content-Length': '0'} 2024-04-13T22:04:08.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097d08e9a9b44af6c-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.517 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.517 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.517 DEBUG:boto:path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.517 DEBUG:boto:auth_path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.517 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.518 DEBUG:boto:Path: /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.518 DEBUG:boto:Data: 2024-04-13T22:04:08.518 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.518 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.518 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.518 DEBUG:boto:Params: {} 2024-04-13T22:04:08.518 DEBUG:boto:Token: None 2024-04-13T22:04:08.518 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ndxvXBq6HTrwPmPnlMnaDwfpck4= 2024-04-13T22:04:08.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ndxvXBq6HTrwPmPnlMnaDwfpck4=', 'Content-Length': '0'} 2024-04-13T22:04:08.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b881135f625f646-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.520 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.520 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.521 DEBUG:boto:path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.521 DEBUG:boto:auth_path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.521 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.521 DEBUG:boto:Path: /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.521 DEBUG:boto:Data: 2024-04-13T22:04:08.521 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.521 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.521 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.521 DEBUG:boto:Params: {} 2024-04-13T22:04:08.521 DEBUG:boto:Token: None 2024-04-13T22:04:08.522 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ndxvXBq6HTrwPmPnlMnaDwfpck4= 2024-04-13T22:04:08.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ndxvXBq6HTrwPmPnlMnaDwfpck4=', 'Content-Length': '0'} 2024-04-13T22:04:08.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094f6e4286621118c-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:04:08.523 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.524 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.524 DEBUG:boto:path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.524 DEBUG:boto:auth_path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.524 DEBUG:boto:Method: GET 2024-04-13T22:04:08.524 DEBUG:boto:Path: /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.524 DEBUG:boto:Data: 2024-04-13T22:04:08.524 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.524 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.524 DEBUG:boto:Params: {} 2024-04-13T22:04:08.525 DEBUG:boto:Token: None 2024-04-13T22:04:08.525 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.525 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg= 2024-04-13T22:04:08.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg=', 'Content-Length': '0'} 2024-04-13T22:04:08.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000213bf4a4c9858a6f-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.527 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.527 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.527 DEBUG:boto:path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.527 DEBUG:boto:auth_path=/vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.527 DEBUG:boto:Method: GET 2024-04-13T22:04:08.527 DEBUG:boto:Path: /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.527 DEBUG:boto:Data: 2024-04-13T22:04:08.527 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.527 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.528 DEBUG:boto:Params: {} 2024-04-13T22:04:08.528 DEBUG:boto:Token: None 2024-04-13T22:04:08.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5?versionId=null 2024-04-13T22:04:08.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg= 2024-04-13T22:04:08.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QiH27GaCMOqce7zGBEtoQzEZCpg=', 'Content-Length': '0'} 2024-04-13T22:04:08.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd90ed81f597f7c-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.530 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.530 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.530 DEBUG:boto:path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.530 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.530 DEBUG:boto:Method: GET 2024-04-13T22:04:08.530 DEBUG:boto:Path: /vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.531 DEBUG:boto:Data: 2024-04-13T22:04:08.531 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.531 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.531 DEBUG:boto:Params: {} 2024-04-13T22:04:08.531 DEBUG:boto:Token: None 2024-04-13T22:04:08.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000341a38d87aabf0d2-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.534 DEBUG:boto:b'vwnfns-49obj51000falseobj52024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.534 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.534 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.534 DEBUG:boto:path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.534 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.535 DEBUG:boto:Method: GET 2024-04-13T22:04:08.535 DEBUG:boto:Path: /vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.535 DEBUG:boto:Data: 2024-04-13T22:04:08.535 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.535 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.535 DEBUG:boto:Params: {} 2024-04-13T22:04:08.535 DEBUG:boto:Token: None 2024-04-13T22:04:08.535 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd8721b36f5167fb-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.538 DEBUG:boto:b'vwnfns-49obj51000falseobj52024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.538 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.538 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.538 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.539 DEBUG:boto:Path: /vwnfns-49/obj5 2024-04-13T22:04:08.539 DEBUG:boto:Data: 2024-04-13T22:04:08.539 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.539 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.539 DEBUG:boto:Params: {} 2024-04-13T22:04:08.539 DEBUG:boto:Token: None 2024-04-13T22:04:08.539 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5 2024-04-13T22:04:08.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0duQrp7K1MWoZ5dDikRt3lFIQ3A= 2024-04-13T22:04:08.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0duQrp7K1MWoZ5dDikRt3lFIQ3A=', 'Content-Length': '0'} 2024-04-13T22:04:08.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067e00e252e02b87c-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.541 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.541 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.541 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.542 DEBUG:boto:Path: /vwnfns-49/obj5 2024-04-13T22:04:08.542 DEBUG:boto:Data: 2024-04-13T22:04:08.542 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.542 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.542 DEBUG:boto:Params: {} 2024-04-13T22:04:08.542 DEBUG:boto:Token: None 2024-04-13T22:04:08.542 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5 2024-04-13T22:04:08.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0duQrp7K1MWoZ5dDikRt3lFIQ3A= 2024-04-13T22:04:08.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0duQrp7K1MWoZ5dDikRt3lFIQ3A=', 'Content-Length': '0'} 2024-04-13T22:04:08.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000586bb83aae6ad359-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:04:08.544 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.544 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.544 DEBUG:boto:Method: GET 2024-04-13T22:04:08.544 DEBUG:boto:Path: /vwnfns-49/obj5 2024-04-13T22:04:08.544 DEBUG:boto:Data: 2024-04-13T22:04:08.544 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.545 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.545 DEBUG:boto:Params: {} 2024-04-13T22:04:08.545 DEBUG:boto:Token: None 2024-04-13T22:04:08.545 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5 2024-04-13T22:04:08.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2av72n9ydAxkgpgJ4htrNkQBjE= 2024-04-13T22:04:08.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2av72n9ydAxkgpgJ4htrNkQBjE=', 'Content-Length': '0'} 2024-04-13T22:04:08.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000433a0950a18c2e52-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.547 DEBUG:boto:path=/vwnfns-49/obj5 2024-04-13T22:04:08.547 DEBUG:boto:auth_path=/vwnfns-49/obj5 2024-04-13T22:04:08.547 DEBUG:boto:Method: GET 2024-04-13T22:04:08.547 DEBUG:boto:Path: /vwnfns-49/obj5 2024-04-13T22:04:08.547 DEBUG:boto:Data: 2024-04-13T22:04:08.547 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.548 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.548 DEBUG:boto:Params: {} 2024-04-13T22:04:08.548 DEBUG:boto:Token: None 2024-04-13T22:04:08.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj5 2024-04-13T22:04:08.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2av72n9ydAxkgpgJ4htrNkQBjE= 2024-04-13T22:04:08.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2av72n9ydAxkgpgJ4htrNkQBjE=', 'Content-Length': '0'} 2024-04-13T22:04:08.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d78cff2b7558310-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.550 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.550 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.550 DEBUG:boto:path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.550 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.550 DEBUG:boto:Method: GET 2024-04-13T22:04:08.550 DEBUG:boto:Path: /vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.550 DEBUG:boto:Data: 2024-04-13T22:04:08.551 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.551 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.551 DEBUG:boto:Params: {} 2024-04-13T22:04:08.551 DEBUG:boto:Token: None 2024-04-13T22:04:08.551 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cb47eb76905491b-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.598 DEBUG:boto:b'vwnfns-49obj51000falseobj52024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.599 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.599 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.599 DEBUG:boto:path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.599 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.599 DEBUG:boto:Method: GET 2024-04-13T22:04:08.599 DEBUG:boto:Path: /vwnfns-49/?prefix=obj5 2024-04-13T22:04:08.599 DEBUG:boto:Data: 2024-04-13T22:04:08.599 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.600 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.600 DEBUG:boto:Params: {} 2024-04-13T22:04:08.600 DEBUG:boto:Token: None 2024-04-13T22:04:08.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab124ce850a7afb9-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.603 DEBUG:boto:b'vwnfns-49obj51000falseobj52024-04-13T22:03:11.679Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:04:08.603 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.603 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.603 DEBUG:boto:path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.603 DEBUG:boto:auth_path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.603 DEBUG:boto:Method: GET 2024-04-13T22:04:08.603 DEBUG:boto:Path: /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.604 DEBUG:boto:Data: 2024-04-13T22:04:08.604 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.604 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.604 DEBUG:boto:Params: {} 2024-04-13T22:04:08.604 DEBUG:boto:Token: None 2024-04-13T22:04:08.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI= 2024-04-13T22:04:08.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI=', 'Content-Length': '0'} 2024-04-13T22:04:08.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61f12818b776c86-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.606 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.606 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.606 DEBUG:boto:path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.607 DEBUG:boto:auth_path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.607 DEBUG:boto:Method: GET 2024-04-13T22:04:08.607 DEBUG:boto:Path: /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.607 DEBUG:boto:Data: 2024-04-13T22:04:08.607 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.607 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.607 DEBUG:boto:Params: {} 2024-04-13T22:04:08.607 DEBUG:boto:Token: None 2024-04-13T22:04:08.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI= 2024-04-13T22:04:08.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI=', 'Content-Length': '0'} 2024-04-13T22:04:08.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006197ed8b747a7786-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.609 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.610 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.610 DEBUG:boto:path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.610 DEBUG:boto:auth_path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.610 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.610 DEBUG:boto:Path: /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.610 DEBUG:boto:Data: 2024-04-13T22:04:08.610 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.610 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.610 DEBUG:boto:Params: {} 2024-04-13T22:04:08.611 DEBUG:boto:Token: None 2024-04-13T22:04:08.611 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RnnnMHNlp1wGG0ys9A+8cMALLms= 2024-04-13T22:04:08.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RnnnMHNlp1wGG0ys9A+8cMALLms=', 'Content-Length': '0'} 2024-04-13T22:04:08.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bb622aeb666f841-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.613 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.613 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.613 DEBUG:boto:path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.613 DEBUG:boto:auth_path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.613 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.613 DEBUG:boto:Path: /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.613 DEBUG:boto:Data: 2024-04-13T22:04:08.613 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.613 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.613 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.613 DEBUG:boto:Params: {} 2024-04-13T22:04:08.614 DEBUG:boto:Token: None 2024-04-13T22:04:08.614 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.614 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RnnnMHNlp1wGG0ys9A+8cMALLms= 2024-04-13T22:04:08.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RnnnMHNlp1wGG0ys9A+8cMALLms=', 'Content-Length': '0'} 2024-04-13T22:04:08.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000493fc6b24f1dd72e-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:04:08.616 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.616 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.616 DEBUG:boto:path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.616 DEBUG:boto:auth_path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.616 DEBUG:boto:Method: GET 2024-04-13T22:04:08.616 DEBUG:boto:Path: /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.616 DEBUG:boto:Data: 2024-04-13T22:04:08.616 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.617 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.617 DEBUG:boto:Params: {} 2024-04-13T22:04:08.617 DEBUG:boto:Token: None 2024-04-13T22:04:08.617 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI= 2024-04-13T22:04:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI=', 'Content-Length': '0'} 2024-04-13T22:04:08.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065af56971dfffd31-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.619 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.619 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.619 DEBUG:boto:path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.619 DEBUG:boto:auth_path=/vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.619 DEBUG:boto:Method: GET 2024-04-13T22:04:08.619 DEBUG:boto:Path: /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.619 DEBUG:boto:Data: 2024-04-13T22:04:08.619 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.620 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.620 DEBUG:boto:Params: {} 2024-04-13T22:04:08.620 DEBUG:boto:Token: None 2024-04-13T22:04:08.620 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6?versionId=null 2024-04-13T22:04:08.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI= 2024-04-13T22:04:08.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mm9uTPmXnHet7duMyswZ8wWqfkI=', 'Content-Length': '0'} 2024-04-13T22:04:08.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f518da5a91100e3-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.622 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.622 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.622 DEBUG:boto:path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.622 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.622 DEBUG:boto:Method: GET 2024-04-13T22:04:08.623 DEBUG:boto:Path: /vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.623 DEBUG:boto:Data: 2024-04-13T22:04:08.623 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.623 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.623 DEBUG:boto:Params: {} 2024-04-13T22:04:08.623 DEBUG:boto:Token: None 2024-04-13T22:04:08.623 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f67de84fb22f30-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.625 DEBUG:boto:b'vwnfns-49obj61000falseobj62024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.626 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.626 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.626 DEBUG:boto:path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.626 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.626 DEBUG:boto:Method: GET 2024-04-13T22:04:08.626 DEBUG:boto:Path: /vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.626 DEBUG:boto:Data: 2024-04-13T22:04:08.626 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.627 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.627 DEBUG:boto:Params: {} 2024-04-13T22:04:08.627 DEBUG:boto:Token: None 2024-04-13T22:04:08.627 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.627 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b7caad2bd770d4-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.630 DEBUG:boto:b'vwnfns-49obj61000falseobj62024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.630 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.630 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.630 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.630 DEBUG:boto:Path: /vwnfns-49/obj6 2024-04-13T22:04:08.630 DEBUG:boto:Data: 2024-04-13T22:04:08.630 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.630 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.631 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.631 DEBUG:boto:Params: {} 2024-04-13T22:04:08.631 DEBUG:boto:Token: None 2024-04-13T22:04:08.631 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6 2024-04-13T22:04:08.631 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BZLnBrybi/iLcOJ/cc9RtQYJAUQ= 2024-04-13T22:04:08.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BZLnBrybi/iLcOJ/cc9RtQYJAUQ=', 'Content-Length': '0'} 2024-04-13T22:04:08.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0582ebf996c43ed-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.633 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.633 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.633 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.633 DEBUG:boto:Path: /vwnfns-49/obj6 2024-04-13T22:04:08.633 DEBUG:boto:Data: 2024-04-13T22:04:08.633 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.633 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.633 DEBUG:boto:Params: {} 2024-04-13T22:04:08.634 DEBUG:boto:Token: None 2024-04-13T22:04:08.634 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6 2024-04-13T22:04:08.634 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BZLnBrybi/iLcOJ/cc9RtQYJAUQ= 2024-04-13T22:04:08.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BZLnBrybi/iLcOJ/cc9RtQYJAUQ=', 'Content-Length': '0'} 2024-04-13T22:04:08.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3cbd4b326c6d089-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:04:08.636 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.636 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.636 DEBUG:boto:Method: GET 2024-04-13T22:04:08.636 DEBUG:boto:Path: /vwnfns-49/obj6 2024-04-13T22:04:08.636 DEBUG:boto:Data: 2024-04-13T22:04:08.636 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.636 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.636 DEBUG:boto:Params: {} 2024-04-13T22:04:08.636 DEBUG:boto:Token: None 2024-04-13T22:04:08.637 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6 2024-04-13T22:04:08.637 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gfxh1l9/H8XgmTsR53h3KCMMQfE= 2024-04-13T22:04:08.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gfxh1l9/H8XgmTsR53h3KCMMQfE=', 'Content-Length': '0'} 2024-04-13T22:04:08.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b399b362113bc5bd-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.638 DEBUG:boto:path=/vwnfns-49/obj6 2024-04-13T22:04:08.638 DEBUG:boto:auth_path=/vwnfns-49/obj6 2024-04-13T22:04:08.639 DEBUG:boto:Method: GET 2024-04-13T22:04:08.639 DEBUG:boto:Path: /vwnfns-49/obj6 2024-04-13T22:04:08.639 DEBUG:boto:Data: 2024-04-13T22:04:08.639 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.639 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.639 DEBUG:boto:Params: {} 2024-04-13T22:04:08.639 DEBUG:boto:Token: None 2024-04-13T22:04:08.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj6 2024-04-13T22:04:08.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gfxh1l9/H8XgmTsR53h3KCMMQfE= 2024-04-13T22:04:08.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gfxh1l9/H8XgmTsR53h3KCMMQfE=', 'Content-Length': '0'} 2024-04-13T22:04:08.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000090076f4d701cb0e-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.641 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.642 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.642 DEBUG:boto:path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.642 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.642 DEBUG:boto:Method: GET 2024-04-13T22:04:08.642 DEBUG:boto:Path: /vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.642 DEBUG:boto:Data: 2024-04-13T22:04:08.642 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.642 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.642 DEBUG:boto:Params: {} 2024-04-13T22:04:08.643 DEBUG:boto:Token: None 2024-04-13T22:04:08.643 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe845ced4dddfaf-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.686 DEBUG:boto:b'vwnfns-49obj61000falseobj62024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.687 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.687 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.687 DEBUG:boto:path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.687 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.687 DEBUG:boto:Method: GET 2024-04-13T22:04:08.687 DEBUG:boto:Path: /vwnfns-49/?prefix=obj6 2024-04-13T22:04:08.687 DEBUG:boto:Data: 2024-04-13T22:04:08.688 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.688 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.688 DEBUG:boto:Params: {} 2024-04-13T22:04:08.688 DEBUG:boto:Token: None 2024-04-13T22:04:08.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000982c284f9566bd1c-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.691 DEBUG:boto:b'vwnfns-49obj61000falseobj62024-04-13T22:03:11.726Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:04:08.691 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.691 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.691 DEBUG:boto:path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.692 DEBUG:boto:auth_path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.692 DEBUG:boto:Method: GET 2024-04-13T22:04:08.692 DEBUG:boto:Path: /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.692 DEBUG:boto:Data: 2024-04-13T22:04:08.692 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.692 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.692 DEBUG:boto:Params: {} 2024-04-13T22:04:08.692 DEBUG:boto:Token: None 2024-04-13T22:04:08.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.693 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k= 2024-04-13T22:04:08.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k=', 'Content-Length': '0'} 2024-04-13T22:04:08.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd214329f0ff5b73-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.694 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.695 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.695 DEBUG:boto:path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.695 DEBUG:boto:auth_path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.695 DEBUG:boto:Method: GET 2024-04-13T22:04:08.695 DEBUG:boto:Path: /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.695 DEBUG:boto:Data: 2024-04-13T22:04:08.695 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.695 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.695 DEBUG:boto:Params: {} 2024-04-13T22:04:08.695 DEBUG:boto:Token: None 2024-04-13T22:04:08.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k= 2024-04-13T22:04:08.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k=', 'Content-Length': '0'} 2024-04-13T22:04:08.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025e4078823c56f79-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.742 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.743 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.743 DEBUG:boto:path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.743 DEBUG:boto:auth_path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.743 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.743 DEBUG:boto:Path: /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.743 DEBUG:boto:Data: 2024-04-13T22:04:08.743 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.743 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.743 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.743 DEBUG:boto:Params: {} 2024-04-13T22:04:08.744 DEBUG:boto:Token: None 2024-04-13T22:04:08.744 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HxIEYTBojZ4x7WUKdHHwHaXvxR4= 2024-04-13T22:04:08.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HxIEYTBojZ4x7WUKdHHwHaXvxR4=', 'Content-Length': '0'} 2024-04-13T22:04:08.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b7ebe566e0aff02-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.746 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.746 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.746 DEBUG:boto:path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.746 DEBUG:boto:auth_path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.746 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.746 DEBUG:boto:Path: /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.746 DEBUG:boto:Data: 2024-04-13T22:04:08.747 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.747 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.747 DEBUG:boto:Params: {} 2024-04-13T22:04:08.747 DEBUG:boto:Token: None 2024-04-13T22:04:08.747 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.747 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HxIEYTBojZ4x7WUKdHHwHaXvxR4= 2024-04-13T22:04:08.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HxIEYTBojZ4x7WUKdHHwHaXvxR4=', 'Content-Length': '0'} 2024-04-13T22:04:08.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce0320a826a0cc5d-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:04:08.749 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.749 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.750 DEBUG:boto:path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.750 DEBUG:boto:auth_path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.750 DEBUG:boto:Method: GET 2024-04-13T22:04:08.750 DEBUG:boto:Path: /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.750 DEBUG:boto:Data: 2024-04-13T22:04:08.750 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.750 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.750 DEBUG:boto:Params: {} 2024-04-13T22:04:08.750 DEBUG:boto:Token: None 2024-04-13T22:04:08.751 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k= 2024-04-13T22:04:08.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k=', 'Content-Length': '0'} 2024-04-13T22:04:08.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c477c8d716696d29-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.752 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.752 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.753 DEBUG:boto:path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.753 DEBUG:boto:auth_path=/vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.753 DEBUG:boto:Method: GET 2024-04-13T22:04:08.753 DEBUG:boto:Path: /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.753 DEBUG:boto:Data: 2024-04-13T22:04:08.753 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.753 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.753 DEBUG:boto:Params: {} 2024-04-13T22:04:08.753 DEBUG:boto:Token: None 2024-04-13T22:04:08.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7?versionId=null 2024-04-13T22:04:08.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k= 2024-04-13T22:04:08.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qgDrqajkkjuGsOIMJiUxBsgx15k=', 'Content-Length': '0'} 2024-04-13T22:04:08.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d378177114b90310-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.756 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.756 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.756 DEBUG:boto:path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.756 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.756 DEBUG:boto:Method: GET 2024-04-13T22:04:08.756 DEBUG:boto:Path: /vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.756 DEBUG:boto:Data: 2024-04-13T22:04:08.756 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.756 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.756 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.757 DEBUG:boto:Params: {} 2024-04-13T22:04:08.757 DEBUG:boto:Token: None 2024-04-13T22:04:08.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.757 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095fecd9cbee0ea2-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.759 DEBUG:boto:b'vwnfns-49obj71000falseobj72024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.760 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.760 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.760 DEBUG:boto:path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.760 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.760 DEBUG:boto:Method: GET 2024-04-13T22:04:08.760 DEBUG:boto:Path: /vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.760 DEBUG:boto:Data: 2024-04-13T22:04:08.760 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.760 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.761 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.761 DEBUG:boto:Params: {} 2024-04-13T22:04:08.761 DEBUG:boto:Token: None 2024-04-13T22:04:08.761 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d1dad2a1a3233e5-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.763 DEBUG:boto:b'vwnfns-49obj71000falseobj72024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.764 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.764 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.764 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.764 DEBUG:boto:Path: /vwnfns-49/obj7 2024-04-13T22:04:08.764 DEBUG:boto:Data: 2024-04-13T22:04:08.764 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.764 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.764 DEBUG:boto:Params: {} 2024-04-13T22:04:08.765 DEBUG:boto:Token: None 2024-04-13T22:04:08.765 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7 2024-04-13T22:04:08.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0T85RjCK9PSFCNWCcpNLomyOW0I= 2024-04-13T22:04:08.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0T85RjCK9PSFCNWCcpNLomyOW0I=', 'Content-Length': '0'} 2024-04-13T22:04:08.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd1ef32ba164eff6-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.766 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.767 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.767 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.767 DEBUG:boto:Path: /vwnfns-49/obj7 2024-04-13T22:04:08.767 DEBUG:boto:Data: 2024-04-13T22:04:08.767 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.767 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.767 DEBUG:boto:Params: {} 2024-04-13T22:04:08.767 DEBUG:boto:Token: None 2024-04-13T22:04:08.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7 2024-04-13T22:04:08.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0T85RjCK9PSFCNWCcpNLomyOW0I= 2024-04-13T22:04:08.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0T85RjCK9PSFCNWCcpNLomyOW0I=', 'Content-Length': '0'} 2024-04-13T22:04:08.769 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eda6e07a171f172b-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:04:08.770 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.770 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.770 DEBUG:boto:Method: GET 2024-04-13T22:04:08.770 DEBUG:boto:Path: /vwnfns-49/obj7 2024-04-13T22:04:08.770 DEBUG:boto:Data: 2024-04-13T22:04:08.770 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.770 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.770 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.770 DEBUG:boto:Params: {} 2024-04-13T22:04:08.771 DEBUG:boto:Token: None 2024-04-13T22:04:08.771 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7 2024-04-13T22:04:08.771 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MPVtLJrJDO6uwlGZQMj7/spBTic= 2024-04-13T22:04:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MPVtLJrJDO6uwlGZQMj7/spBTic=', 'Content-Length': '0'} 2024-04-13T22:04:08.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7120f48e90e3332-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.814 DEBUG:boto:path=/vwnfns-49/obj7 2024-04-13T22:04:08.815 DEBUG:boto:auth_path=/vwnfns-49/obj7 2024-04-13T22:04:08.815 DEBUG:boto:Method: GET 2024-04-13T22:04:08.815 DEBUG:boto:Path: /vwnfns-49/obj7 2024-04-13T22:04:08.815 DEBUG:boto:Data: 2024-04-13T22:04:08.815 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.815 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.815 DEBUG:boto:Params: {} 2024-04-13T22:04:08.815 DEBUG:boto:Token: None 2024-04-13T22:04:08.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj7 2024-04-13T22:04:08.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MPVtLJrJDO6uwlGZQMj7/spBTic= 2024-04-13T22:04:08.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MPVtLJrJDO6uwlGZQMj7/spBTic=', 'Content-Length': '0'} 2024-04-13T22:04:08.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a69cba00dc7ebdbb-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.818 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.818 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.818 DEBUG:boto:path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.818 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.818 DEBUG:boto:Method: GET 2024-04-13T22:04:08.818 DEBUG:boto:Path: /vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.818 DEBUG:boto:Data: 2024-04-13T22:04:08.818 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.819 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.819 DEBUG:boto:Params: {} 2024-04-13T22:04:08.819 DEBUG:boto:Token: None 2024-04-13T22:04:08.819 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4e92e521e403bb-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.821 DEBUG:boto:b'vwnfns-49obj71000falseobj72024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.822 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.822 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.822 DEBUG:boto:path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.822 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.822 DEBUG:boto:Method: GET 2024-04-13T22:04:08.822 DEBUG:boto:Path: /vwnfns-49/?prefix=obj7 2024-04-13T22:04:08.822 DEBUG:boto:Data: 2024-04-13T22:04:08.822 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.823 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.823 DEBUG:boto:Params: {} 2024-04-13T22:04:08.823 DEBUG:boto:Token: None 2024-04-13T22:04:08.823 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034732f0c9a5fb835-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.825 DEBUG:boto:b'vwnfns-49obj71000falseobj72024-04-13T22:03:11.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:04:08.826 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.826 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.826 DEBUG:boto:path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.826 DEBUG:boto:auth_path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.826 DEBUG:boto:Method: GET 2024-04-13T22:04:08.826 DEBUG:boto:Path: /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.826 DEBUG:boto:Data: 2024-04-13T22:04:08.827 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.827 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.827 DEBUG:boto:Params: {} 2024-04-13T22:04:08.827 DEBUG:boto:Token: None 2024-04-13T22:04:08.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM= 2024-04-13T22:04:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM=', 'Content-Length': '0'} 2024-04-13T22:04:08.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004913a78bae6c2260-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.829 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.829 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.829 DEBUG:boto:path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.830 DEBUG:boto:auth_path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.830 DEBUG:boto:Method: GET 2024-04-13T22:04:08.830 DEBUG:boto:Path: /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.830 DEBUG:boto:Data: 2024-04-13T22:04:08.830 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.830 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.830 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.830 DEBUG:boto:Params: {} 2024-04-13T22:04:08.830 DEBUG:boto:Token: None 2024-04-13T22:04:08.831 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.831 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM= 2024-04-13T22:04:08.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM=', 'Content-Length': '0'} 2024-04-13T22:04:08.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0c9e3c469ec26d4-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.833 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.833 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.833 DEBUG:boto:path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.833 DEBUG:boto:auth_path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.833 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.833 DEBUG:boto:Path: /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.834 DEBUG:boto:Data: 2024-04-13T22:04:08.834 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.834 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.834 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.834 DEBUG:boto:Params: {} 2024-04-13T22:04:08.834 DEBUG:boto:Token: None 2024-04-13T22:04:08.834 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.834 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ve0ugb1tjZbOl3yanjj0NwVW2UA= 2024-04-13T22:04:08.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ve0ugb1tjZbOl3yanjj0NwVW2UA=', 'Content-Length': '0'} 2024-04-13T22:04:08.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067d3ed7c75cb0e3b-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.836 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.836 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.836 DEBUG:boto:path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.836 DEBUG:boto:auth_path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.836 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.837 DEBUG:boto:Path: /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.837 DEBUG:boto:Data: 2024-04-13T22:04:08.837 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.837 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.837 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.837 DEBUG:boto:Params: {} 2024-04-13T22:04:08.837 DEBUG:boto:Token: None 2024-04-13T22:04:08.837 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.837 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ve0ugb1tjZbOl3yanjj0NwVW2UA= 2024-04-13T22:04:08.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ve0ugb1tjZbOl3yanjj0NwVW2UA=', 'Content-Length': '0'} 2024-04-13T22:04:08.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057183b5793604db9-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:04:08.840 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.840 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.840 DEBUG:boto:path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.840 DEBUG:boto:auth_path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.840 DEBUG:boto:Method: GET 2024-04-13T22:04:08.840 DEBUG:boto:Path: /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.840 DEBUG:boto:Data: 2024-04-13T22:04:08.840 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.840 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.841 DEBUG:boto:Params: {} 2024-04-13T22:04:08.841 DEBUG:boto:Token: None 2024-04-13T22:04:08.841 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.841 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM= 2024-04-13T22:04:08.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM=', 'Content-Length': '0'} 2024-04-13T22:04:08.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a7be09dabc034e6-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.843 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.843 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.843 DEBUG:boto:path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.843 DEBUG:boto:auth_path=/vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.843 DEBUG:boto:Method: GET 2024-04-13T22:04:08.843 DEBUG:boto:Path: /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.843 DEBUG:boto:Data: 2024-04-13T22:04:08.843 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.843 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.844 DEBUG:boto:Params: {} 2024-04-13T22:04:08.844 DEBUG:boto:Token: None 2024-04-13T22:04:08.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8?versionId=null 2024-04-13T22:04:08.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM= 2024-04-13T22:04:08.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pjcQTqiWGjjewMwEhlsAErNNrAM=', 'Content-Length': '0'} 2024-04-13T22:04:08.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec8a366519e796b-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.886 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.887 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.887 DEBUG:boto:path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.887 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.887 DEBUG:boto:Method: GET 2024-04-13T22:04:08.887 DEBUG:boto:Path: /vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.887 DEBUG:boto:Data: 2024-04-13T22:04:08.887 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.887 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.887 DEBUG:boto:Params: {} 2024-04-13T22:04:08.888 DEBUG:boto:Token: None 2024-04-13T22:04:08.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a7a6696a9f7887-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.890 DEBUG:boto:b'vwnfns-49obj81000falseobj82024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.891 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.891 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.891 DEBUG:boto:path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.891 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.891 DEBUG:boto:Method: GET 2024-04-13T22:04:08.891 DEBUG:boto:Path: /vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.891 DEBUG:boto:Data: 2024-04-13T22:04:08.891 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.892 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.892 DEBUG:boto:Params: {} 2024-04-13T22:04:08.892 DEBUG:boto:Token: None 2024-04-13T22:04:08.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda5965dcdd30503-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.894 DEBUG:boto:b'vwnfns-49obj81000falseobj82024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.894 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.895 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.895 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.895 DEBUG:boto:Path: /vwnfns-49/obj8 2024-04-13T22:04:08.895 DEBUG:boto:Data: 2024-04-13T22:04:08.895 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.895 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.895 DEBUG:boto:Params: {} 2024-04-13T22:04:08.895 DEBUG:boto:Token: None 2024-04-13T22:04:08.896 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8 2024-04-13T22:04:08.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4jzh1RKMnOX+XwO43bPAGgxDyik= 2024-04-13T22:04:08.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4jzh1RKMnOX+XwO43bPAGgxDyik=', 'Content-Length': '0'} 2024-04-13T22:04:08.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee460d3fb90b1965-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.897 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.897 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.898 DEBUG:boto:Method: HEAD 2024-04-13T22:04:08.898 DEBUG:boto:Path: /vwnfns-49/obj8 2024-04-13T22:04:08.898 DEBUG:boto:Data: 2024-04-13T22:04:08.898 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.898 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.898 DEBUG:boto:Params: {} 2024-04-13T22:04:08.898 DEBUG:boto:Token: None 2024-04-13T22:04:08.898 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8 2024-04-13T22:04:08.898 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4jzh1RKMnOX+XwO43bPAGgxDyik= 2024-04-13T22:04:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4jzh1RKMnOX+XwO43bPAGgxDyik=', 'Content-Length': '0'} 2024-04-13T22:04:08.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004af53000a2717520-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:04:08.901 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.901 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.901 DEBUG:boto:Method: GET 2024-04-13T22:04:08.901 DEBUG:boto:Path: /vwnfns-49/obj8 2024-04-13T22:04:08.901 DEBUG:boto:Data: 2024-04-13T22:04:08.901 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.901 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.902 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.902 DEBUG:boto:Params: {} 2024-04-13T22:04:08.902 DEBUG:boto:Token: None 2024-04-13T22:04:08.902 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8 2024-04-13T22:04:08.902 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XxQvrfu11F0RNksAnCno2XDu9BY= 2024-04-13T22:04:08.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XxQvrfu11F0RNksAnCno2XDu9BY=', 'Content-Length': '0'} 2024-04-13T22:04:08.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb10c253c64c476a-00661b0158-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.904 DEBUG:boto:path=/vwnfns-49/obj8 2024-04-13T22:04:08.904 DEBUG:boto:auth_path=/vwnfns-49/obj8 2024-04-13T22:04:08.904 DEBUG:boto:Method: GET 2024-04-13T22:04:08.904 DEBUG:boto:Path: /vwnfns-49/obj8 2024-04-13T22:04:08.904 DEBUG:boto:Data: 2024-04-13T22:04:08.904 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.905 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.905 DEBUG:boto:Params: {} 2024-04-13T22:04:08.905 DEBUG:boto:Token: None 2024-04-13T22:04:08.905 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/obj8 2024-04-13T22:04:08.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XxQvrfu11F0RNksAnCno2XDu9BY= 2024-04-13T22:04:08.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XxQvrfu11F0RNksAnCno2XDu9BY=', 'Content-Length': '0'} 2024-04-13T22:04:08.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:03:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-49:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031aa32e6f33fa203-00661b0158-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.907 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.907 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.907 DEBUG:boto:path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.907 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.907 DEBUG:boto:Method: GET 2024-04-13T22:04:08.908 DEBUG:boto:Path: /vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.908 DEBUG:boto:Data: 2024-04-13T22:04:08.908 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:08.908 DEBUG:boto:Port: 8001 2024-04-13T22:04:08.908 DEBUG:boto:Params: {} 2024-04-13T22:04:08.908 DEBUG:boto:Token: None 2024-04-13T22:04:08.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa78c9b2362fc0a-00661b0158-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.911 DEBUG:boto:b'vwnfns-49obj81000falseobj82024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.911 DEBUG:boto:path=/vwnfns-49/ 2024-04-13T22:04:08.911 DEBUG:boto:auth_path=/vwnfns-49/ 2024-04-13T22:04:08.911 DEBUG:boto:path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.911 DEBUG:boto:auth_path=/vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.911 DEBUG:boto:Method: GET 2024-04-13T22:04:08.912 DEBUG:boto:Path: /vwnfns-49/?prefix=obj8 2024-04-13T22:04:08.912 DEBUG:boto:Data: 2024-04-13T22:04:08.912 DEBUG:boto:Headers: {} 2024-04-13T22:04:08.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:08.912 DEBUG:boto:Port: 8000 2024-04-13T22:04:08.912 DEBUG:boto:Params: {} 2024-04-13T22:04:08.912 DEBUG:boto:Token: None 2024-04-13T22:04:08.912 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:08 GMT /vwnfns-49/ 2024-04-13T22:04:08.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk= 2024-04-13T22:04:08.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:08 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w6+VAb0WfDU/Jpe/BEIppeQtQRk=', 'Content-Length': '0'} 2024-04-13T22:04:08.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eddcbe0cb737020-00661b0158-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:08.915 DEBUG:boto:b'vwnfns-49obj81000falseobj82024-04-13T22:03:11.782Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:08.915 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-49 zones={a2, a1} 2024-04-13T22:04:08.915 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-13T22:04:08.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:08.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:08.916 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:08.963 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:08.963 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:09.009 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:04:09.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.009 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:04:09.009 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:03:11.543799Z" 2024-04-13T22:04:09.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005728", 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:03:20.335505Z" 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004031", 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:04:06.364661Z" 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-13T22:04:09.010 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:03:20.503993Z" 2024-04-13T22:04:09.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:09.011 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:04:09.012 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000004031', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:09.012 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:04:09.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:09.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:09.012 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:09.062 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:09.062 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.106 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:11.543799Z" 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:09.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005728", 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.335505Z" 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:09.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003890", 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:19.224333Z" 2024-04-13T22:04:09.109 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:09.110 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.503993Z" 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:09Z" 2024-04-13T22:04:09.111 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:09.112 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:11.543799Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005728", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.335505Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003890", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:19.224333Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009525", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.503993Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:04:09Z" } 2024-04-13T22:04:09.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:11.543799Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005728', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.335505Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003890', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:19.224333Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009525', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.503993Z'}}] 2024-04-13T22:04:09.113 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000004031', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:09.113 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000003890', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:09.113 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000004031 target=00000000000000000000:00000000000000003890 2024-04-13T22:04:14.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:14.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:14.115 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:14.172 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:14.173 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:14.221 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:11.543799Z" 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005728", 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.335505Z" 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003890", 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:19.224333Z" 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.503993Z" 2024-04-13T22:04:14.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:14Z" 2024-04-13T22:04:14.226 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:14.227 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:11.543799Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005728", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.335505Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003890", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:19.224333Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009525", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.503993Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:04:14Z" } 2024-04-13T22:04:14.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:11.543799Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005728', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.335505Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003890', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:19.224333Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009525', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.503993Z'}}] 2024-04-13T22:04:14.227 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000004031', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:14.228 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000003890', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:14.228 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000004031 target=00000000000000000000:00000000000000003890 2024-04-13T22:04:19.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:19.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:19.230 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:19.287 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:19.288 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:19.335 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:11.543799Z" 2024-04-13T22:04:19.336 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005728", 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:19.337 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.335505Z" 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003890", 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:19.338 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:19.224333Z" 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:19.339 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.503993Z" 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:04:19.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:19.341 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:19Z" 2024-04-13T22:04:19.341 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:19.342 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:11.543799Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005728", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.335505Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003890", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:19.224333Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009525", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.503993Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:04:19Z" } 2024-04-13T22:04:19.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:11.543799Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005728', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.335505Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003890', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:19.224333Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009525', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.503993Z'}}] 2024-04-13T22:04:19.342 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000004031', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:19.342 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000003890', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:19.342 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000004031 target=00000000000000000000:00000000000000003890 2024-04-13T22:04:24.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.343 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:24.403 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:24.403 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:24.453 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:11.543799Z" 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005728", 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.335505Z" 2024-04-13T22:04:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004031", 2024-04-13T22:04:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:06.364661Z" 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000009525", 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:20.503993Z" 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:24Z" 2024-04-13T22:04:24.460 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:24.461 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:11.543799Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005728", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.335505Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004031", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:04:06.364661Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009525", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:20.503993Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:04:24Z" } 2024-04-13T22:04:24.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:11.543799Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005728', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.335505Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004031', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:04:06.364661Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009525', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:20.503993Z'}}] 2024-04-13T22:04:24.461 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000004031', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:24.462 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000005728', 2: '00000000000000000000:00000000000000004031', 3: '00000000000000000000:00000000000000009525'} 2024-04-13T22:04:24.462 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:04:24.462 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-13T22:04:24.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.462 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:24.510 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:24.510 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005021", 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:02:23.481807Z" 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:03:41.179779Z" 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003044", 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:03:41.181350Z" 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000008538", 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:03:41.182735Z" 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.559 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:04:24.560 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000004174', 2: '00000000000000000000:00000000000000003044', 3: '00000000000000000000:00000000000000008538'} 2024-04-13T22:04:24.561 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:04:24.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.561 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:24.611 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:24.611 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:24.652 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005021", 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:02:23.481807Z" 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:41.179779Z" 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000003044", 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:41.181350Z" 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000008538", 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:03:41.182735Z" 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:04:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:04:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:24Z" 2024-04-13T22:04:24.658 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:24.659 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005021", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:02:23.481807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:41.179779Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003044", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:41.181350Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008538", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:03:41.182735Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:04:24Z" } 2024-04-13T22:04:24.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005021', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:02:23.481807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:41.179779Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003044', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:41.181350Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008538', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:03:41.182735Z'}}] 2024-04-13T22:04:24.659 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000004174', 2: '00000000000000000000:00000000000000003044', 3: '00000000000000000000:00000000000000008538'} 2024-04-13T22:04:24.659 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005021', 1: '00000000000000000000:00000000000000004174', 2: '00000000000000000000:00000000000000003044', 3: '00000000000000000000:00000000000000008538'} 2024-04-13T22:04:24.659 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:04:24.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2024-04-13T22:04:24.662 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-50 2024-04-13T22:04:24.662 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:24.662 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:24.662 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.663 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:24.663 DEBUG:boto:Data: 2024-04-13T22:04:24.663 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.663 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.664 DEBUG:boto:Params: {} 2024-04-13T22:04:24.664 DEBUG:boto:Token: None 2024-04-13T22:04:24.664 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/ 2024-04-13T22:04:24.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:60AvT8QXrkQruBZj8Tu914yPK+8= 2024-04-13T22:04:24.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:60AvT8QXrkQruBZj8Tu914yPK+8=', 'Content-Length': '0'} 2024-04-13T22:04:24.679 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1226cd491a799ee-00661b0168-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.679 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-51 2024-04-13T22:04:24.679 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:24.680 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:24.680 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.680 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:24.680 DEBUG:boto:Data: 2024-04-13T22:04:24.680 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.680 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.680 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.680 DEBUG:boto:Params: {} 2024-04-13T22:04:24.680 DEBUG:boto:Token: None 2024-04-13T22:04:24.680 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/ 2024-04-13T22:04:24.681 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yaI+cUgxNldrupbZ6FmQLungBUU= 2024-04-13T22:04:24.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yaI+cUgxNldrupbZ6FmQLungBUU=', 'Content-Length': '0'} 2024-04-13T22:04:24.703 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a40af59706f4203-00661b0168-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.703 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:24.703 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:24.703 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.703 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:24.703 DEBUG:boto:Data: 2024-04-13T22:04:24.703 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.704 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.704 DEBUG:boto:Params: {} 2024-04-13T22:04:24.704 DEBUG:boto:Token: None 2024-04-13T22:04:24.704 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/ 2024-04-13T22:04:24.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA= 2024-04-13T22:04:24.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA=', 'Content-Length': '0'} 2024-04-13T22:04:24.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1e120f47ef45e01-00661b0168-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.706 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:24.706 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:24.707 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.707 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:04:24.707 DEBUG:boto:Data: 2024-04-13T22:04:24.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.707 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.707 DEBUG:boto:Params: {} 2024-04-13T22:04:24.707 DEBUG:boto:Token: None 2024-04-13T22:04:24.708 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/obj1 2024-04-13T22:04:24.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DQOU3umsk6rO/Xn1+/kqQ6O/b1Y= 2024-04-13T22:04:24.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DQOU3umsk6rO/Xn1+/kqQ6O/b1Y='} 2024-04-13T22:04:24.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0b3d3ee997110e1-00661b0168-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.751 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:24.751 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:24.751 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.751 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:24.751 DEBUG:boto:Data: 2024-04-13T22:04:24.751 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.752 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.752 DEBUG:boto:Params: {} 2024-04-13T22:04:24.752 DEBUG:boto:Token: None 2024-04-13T22:04:24.752 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/ 2024-04-13T22:04:24.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA= 2024-04-13T22:04:24.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA=', 'Content-Length': '0'} 2024-04-13T22:04:24.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090d7e260d7417327-00661b0168-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.754 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:24.755 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:24.755 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.755 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:04:24.755 DEBUG:boto:Data: 2024-04-13T22:04:24.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.755 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.755 DEBUG:boto:Params: {} 2024-04-13T22:04:24.755 DEBUG:boto:Token: None 2024-04-13T22:04:24.755 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/obj2 2024-04-13T22:04:24.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DCMlyHuk1Ka3iQY4acjUMM46MPk= 2024-04-13T22:04:24.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DCMlyHuk1Ka3iQY4acjUMM46MPk='} 2024-04-13T22:04:24.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a680cdaa0a7c291e-00661b0168-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.760 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:24.760 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:24.761 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.761 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:24.761 DEBUG:boto:Data: 2024-04-13T22:04:24.761 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.761 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.761 DEBUG:boto:Params: {} 2024-04-13T22:04:24.761 DEBUG:boto:Token: None 2024-04-13T22:04:24.761 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/ 2024-04-13T22:04:24.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA= 2024-04-13T22:04:24.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA=', 'Content-Length': '0'} 2024-04-13T22:04:24.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e894e2e76b4c2b0-00661b0168-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.764 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:24.764 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:24.764 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.764 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:04:24.764 DEBUG:boto:Data: 2024-04-13T22:04:24.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.765 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.765 DEBUG:boto:Params: {} 2024-04-13T22:04:24.765 DEBUG:boto:Token: None 2024-04-13T22:04:24.765 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/obj3 2024-04-13T22:04:24.766 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2RN7Z8U6SPndTYbV8AYM1eQv2tg= 2024-04-13T22:04:24.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2RN7Z8U6SPndTYbV8AYM1eQv2tg='} 2024-04-13T22:04:24.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff1268f8270d591b-00661b0168-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.811 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:24.811 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:24.811 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.811 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:24.811 DEBUG:boto:Data: 2024-04-13T22:04:24.811 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.811 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.811 DEBUG:boto:Params: {} 2024-04-13T22:04:24.812 DEBUG:boto:Token: None 2024-04-13T22:04:24.812 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/ 2024-04-13T22:04:24.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA= 2024-04-13T22:04:24.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l6y5G5HKDDobt1mAvvmbtAKUerA=', 'Content-Length': '0'} 2024-04-13T22:04:24.814 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022d5214f21a7b4dc-00661b0168-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.814 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:24.815 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:24.815 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.815 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:04:24.815 DEBUG:boto:Data: 2024-04-13T22:04:24.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:24.816 DEBUG:boto:Port: 8000 2024-04-13T22:04:24.816 DEBUG:boto:Params: {} 2024-04-13T22:04:24.816 DEBUG:boto:Token: None 2024-04-13T22:04:24.817 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-50/obj4 2024-04-13T22:04:24.817 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LxtHzsNX5H8IHjZwlYRt7vaIWpQ= 2024-04-13T22:04:24.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LxtHzsNX5H8IHjZwlYRt7vaIWpQ='} 2024-04-13T22:04:24.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053078070d6e64f6d-00661b0168-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.822 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:24.823 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:24.823 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.823 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:24.823 DEBUG:boto:Data: 2024-04-13T22:04:24.824 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.824 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.824 DEBUG:boto:Params: {} 2024-04-13T22:04:24.824 DEBUG:boto:Token: None 2024-04-13T22:04:24.825 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/ 2024-04-13T22:04:24.825 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI= 2024-04-13T22:04:24.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI=', 'Content-Length': '0'} 2024-04-13T22:04:24.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000795e34fb4a59c7ef-00661b0168-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.828 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:24.828 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:24.829 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.829 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:04:24.829 DEBUG:boto:Data: 2024-04-13T22:04:24.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.829 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.830 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.830 DEBUG:boto:Params: {} 2024-04-13T22:04:24.830 DEBUG:boto:Token: None 2024-04-13T22:04:24.830 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/obj1 2024-04-13T22:04:24.831 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d1AZAErgOwBC5JBnUMZ4be4lsiI= 2024-04-13T22:04:24.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d1AZAErgOwBC5JBnUMZ4be4lsiI='} 2024-04-13T22:04:24.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000816b3a6c47e67782-00661b0168-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.875 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:24.875 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:24.875 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.875 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:24.875 DEBUG:boto:Data: 2024-04-13T22:04:24.875 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.875 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.875 DEBUG:boto:Params: {} 2024-04-13T22:04:24.875 DEBUG:boto:Token: None 2024-04-13T22:04:24.876 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/ 2024-04-13T22:04:24.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI= 2024-04-13T22:04:24.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI=', 'Content-Length': '0'} 2024-04-13T22:04:24.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058ae7fa16dc1eabe-00661b0168-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.879 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:24.879 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:24.879 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.879 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:04:24.879 DEBUG:boto:Data: 2024-04-13T22:04:24.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.879 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.879 DEBUG:boto:Params: {} 2024-04-13T22:04:24.879 DEBUG:boto:Token: None 2024-04-13T22:04:24.880 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/obj2 2024-04-13T22:04:24.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sg6J1jkA1+aXyYnapisrnOxAhJY= 2024-04-13T22:04:24.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sg6J1jkA1+aXyYnapisrnOxAhJY='} 2024-04-13T22:04:24.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f176e653a45a472-00661b0168-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.886 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:24.886 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:24.886 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.887 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:24.887 DEBUG:boto:Data: 2024-04-13T22:04:24.887 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.887 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.887 DEBUG:boto:Params: {} 2024-04-13T22:04:24.887 DEBUG:boto:Token: None 2024-04-13T22:04:24.887 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/ 2024-04-13T22:04:24.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI= 2024-04-13T22:04:24.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI=', 'Content-Length': '0'} 2024-04-13T22:04:24.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5b9a3c02677f0cd-00661b0168-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.889 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:24.889 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:24.890 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.890 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:04:24.890 DEBUG:boto:Data: 2024-04-13T22:04:24.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.890 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.890 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.890 DEBUG:boto:Params: {} 2024-04-13T22:04:24.890 DEBUG:boto:Token: None 2024-04-13T22:04:24.890 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/obj3 2024-04-13T22:04:24.891 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fP/qQHuyaNuE7dpjedAUFgY/enE= 2024-04-13T22:04:24.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fP/qQHuyaNuE7dpjedAUFgY/enE='} 2024-04-13T22:04:24.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d02eec56b083fa88-00661b0168-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.896 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:24.896 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:24.896 DEBUG:boto:Method: HEAD 2024-04-13T22:04:24.896 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:24.896 DEBUG:boto:Data: 2024-04-13T22:04:24.896 DEBUG:boto:Headers: {} 2024-04-13T22:04:24.897 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.897 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.897 DEBUG:boto:Params: {} 2024-04-13T22:04:24.897 DEBUG:boto:Token: None 2024-04-13T22:04:24.897 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/ 2024-04-13T22:04:24.897 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI= 2024-04-13T22:04:24.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkyq3wnNKlgRl2CGxisP01IpjEI=', 'Content-Length': '0'} 2024-04-13T22:04:24.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089530fdfb3244d81-00661b0168-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.899 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:24.899 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:24.899 DEBUG:boto:Method: PUT 2024-04-13T22:04:24.899 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:04:24.899 DEBUG:boto:Data: 2024-04-13T22:04:24.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:04:24.900 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:24.900 DEBUG:boto:Port: 8001 2024-04-13T22:04:24.900 DEBUG:boto:Params: {} 2024-04-13T22:04:24.900 DEBUG:boto:Token: None 2024-04-13T22:04:24.900 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:04:24 GMT /vwnfns-51/obj4 2024-04-13T22:04:24.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z9ADKt1vDEUSHA7QYbV/sXXdxEg= 2024-04-13T22:04:24.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:04:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z9ADKt1vDEUSHA7QYbV/sXXdxEg='} 2024-04-13T22:04:24.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063e0af4958a6b4ca-00661b0168-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:24.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:24.943 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:25.034 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:04:25.046 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:04:25.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.046 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045864.694964_643.1", 2024-04-13T22:04:25.046 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:04:24.694964Z" 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:25.047 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:04:25.048 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:25.048 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:04:25.049 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:25.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:04:25.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:25.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:25.049 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:25.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:25.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:25.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:25.126+0000 7fa4d7f95780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:25.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:25.127+0000 7fa4d7f95780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:25.140 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:25.140 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:25.140 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:25.140 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:25.141 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045864.694964_643.1", 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:24.694964Z", 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.142 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.143 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:25.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:25Z" 2024-04-13T22:04:25.146 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:25.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045864.694964_643.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:04:24.694964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:04:25Z" } 2024-04-13T22:04:25.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045864.694964_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:04:24.694964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:04:25.147 DEBUG:rgw_multi.tests:log_status={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:25.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:25.148 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:04:25.148 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-50 2024-04-13T22:04:25.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:25.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:25.148 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:04:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:25.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:25.237+0000 7f3c1fe99780 1 bucket sync caught up with source: 2024-04-13T22:04:25.238 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.45.6, , 00000000001.38.6, , 00000000001.54.6, , , 00000000001.21.6, , , ] 2024-04-13T22:04:25.238 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.45.6, , 00000000001.38.6, , 00000000001.54.6, , , 00000000001.21.6, , , ] 2024-04-13T22:04:25.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:25.237+0000 7f3c1fe99780 0 bucket checkpoint complete 2024-04-13T22:04:25.252 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-50 2024-04-13T22:04:25.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:25.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:25.253 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:04:25.308 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:25.308 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:25.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:25.344+0000 7f6023d7f780 1 waiting to reach incremental sync.. 2024-04-13T22:04:30.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:30.345+0000 7f6023d7f780 1 bucket sync caught up with source: 2024-04-13T22:04:30.346 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.987.6, , 00000000001.1740.6, , 00000000001.2800.6, , , 00000000001.434.6, , , ] 2024-04-13T22:04:30.346 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.987.6, , 00000000001.1740.6, , 00000000001.2800.6, , , 00000000001.434.6, , , ] 2024-04-13T22:04:30.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:30.345+0000 7f6023d7f780 0 bucket checkpoint complete 2024-04-13T22:04:30.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-50 zones={a2, a1} 2024-04-13T22:04:30.363 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.363 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.363 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.363 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:30.363 DEBUG:boto:Data: 2024-04-13T22:04:30.363 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.363 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.364 DEBUG:boto:Params: {} 2024-04-13T22:04:30.364 DEBUG:boto:Token: None 2024-04-13T22:04:30.364 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cn65IyLRrw6jnFyTYCN05jfs4F0= 2024-04-13T22:04:30.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cn65IyLRrw6jnFyTYCN05jfs4F0=', 'Content-Length': '0'} 2024-04-13T22:04:30.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af95fafd80cd0712-00661b016e-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.366 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.367 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.367 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.367 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:04:30.367 DEBUG:boto:Data: 2024-04-13T22:04:30.367 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.367 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.367 DEBUG:boto:Params: {} 2024-04-13T22:04:30.367 DEBUG:boto:Token: None 2024-04-13T22:04:30.367 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cn65IyLRrw6jnFyTYCN05jfs4F0= 2024-04-13T22:04:30.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cn65IyLRrw6jnFyTYCN05jfs4F0=', 'Content-Length': '0'} 2024-04-13T22:04:30.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002ca9251f76e8637-00661b016e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.370 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:04:30.370 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.370 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.370 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.371 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.371 DEBUG:boto:Method: GET 2024-04-13T22:04:30.371 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.371 DEBUG:boto:Data: 2024-04-13T22:04:30.371 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.371 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.371 DEBUG:boto:Params: {} 2024-04-13T22:04:30.371 DEBUG:boto:Token: None 2024-04-13T22:04:30.371 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/?versions 2024-04-13T22:04:30.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM= 2024-04-13T22:04:30.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM=', 'Content-Length': '0'} 2024-04-13T22:04:30.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd477234c61eca6-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.374 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:30.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:30.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:30.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:30.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:04:30.375 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.375 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.375 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.376 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.376 DEBUG:boto:Method: GET 2024-04-13T22:04:30.376 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.376 DEBUG:boto:Data: 2024-04-13T22:04:30.376 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.376 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.376 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.376 DEBUG:boto:Params: {} 2024-04-13T22:04:30.376 DEBUG:boto:Token: None 2024-04-13T22:04:30.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/?versions 2024-04-13T22:04:30.377 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM= 2024-04-13T22:04:30.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM=', 'Content-Length': '0'} 2024-04-13T22:04:30.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c33984bedda3eba-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.379 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:30.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:30.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:30.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:30.380 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.380 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.380 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.380 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.380 DEBUG:boto:Method: GET 2024-04-13T22:04:30.380 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.380 DEBUG:boto:Data: 2024-04-13T22:04:30.380 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.380 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.380 DEBUG:boto:Params: {} 2024-04-13T22:04:30.381 DEBUG:boto:Token: None 2024-04-13T22:04:30.381 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/?versions 2024-04-13T22:04:30.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM= 2024-04-13T22:04:30.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM=', 'Content-Length': '0'} 2024-04-13T22:04:30.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2aacfa04dcbd7d-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.383 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.384 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.384 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.384 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.384 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.384 DEBUG:boto:Method: GET 2024-04-13T22:04:30.384 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:04:30.384 DEBUG:boto:Data: 2024-04-13T22:04:30.384 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.384 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.384 DEBUG:boto:Params: {} 2024-04-13T22:04:30.384 DEBUG:boto:Token: None 2024-04-13T22:04:30.385 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/?versions 2024-04-13T22:04:30.385 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM= 2024-04-13T22:04:30.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4PeMhoUPx36LJwbfJKbuLurB+OM=', 'Content-Length': '0'} 2024-04-13T22:04:30.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052f65c8d8c948518-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.387 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:30.388 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.388 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.388 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.388 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.388 DEBUG:boto:Method: GET 2024-04-13T22:04:30.388 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.388 DEBUG:boto:Data: 2024-04-13T22:04:30.388 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.388 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.388 DEBUG:boto:Params: {} 2024-04-13T22:04:30.388 DEBUG:boto:Token: None 2024-04-13T22:04:30.389 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.389 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE= 2024-04-13T22:04:30.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE=', 'Content-Length': '0'} 2024-04-13T22:04:30.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001977b6009ebd7b10-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.391 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.391 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.391 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.391 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.391 DEBUG:boto:Method: GET 2024-04-13T22:04:30.391 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.391 DEBUG:boto:Data: 2024-04-13T22:04:30.391 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.392 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.392 DEBUG:boto:Params: {} 2024-04-13T22:04:30.392 DEBUG:boto:Token: None 2024-04-13T22:04:30.392 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE= 2024-04-13T22:04:30.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE=', 'Content-Length': '0'} 2024-04-13T22:04:30.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f036b76f6fbb954-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.434 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.435 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.435 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.435 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.435 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.435 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.435 DEBUG:boto:Data: 2024-04-13T22:04:30.435 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.435 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.435 DEBUG:boto:Params: {} 2024-04-13T22:04:30.436 DEBUG:boto:Token: None 2024-04-13T22:04:30.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wRl7lnNpGskkWj5PbDt+BB+FKSI= 2024-04-13T22:04:30.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wRl7lnNpGskkWj5PbDt+BB+FKSI=', 'Content-Length': '0'} 2024-04-13T22:04:30.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f44e8668cd7e6215-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.438 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.438 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.438 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.438 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.439 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.439 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.439 DEBUG:boto:Data: 2024-04-13T22:04:30.439 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.439 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.439 DEBUG:boto:Params: {} 2024-04-13T22:04:30.439 DEBUG:boto:Token: None 2024-04-13T22:04:30.441 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wRl7lnNpGskkWj5PbDt+BB+FKSI= 2024-04-13T22:04:30.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wRl7lnNpGskkWj5PbDt+BB+FKSI=', 'Content-Length': '0'} 2024-04-13T22:04:30.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000582993b8429d7ddc-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:30.443 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.443 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.443 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.443 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.444 DEBUG:boto:Method: GET 2024-04-13T22:04:30.444 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.444 DEBUG:boto:Data: 2024-04-13T22:04:30.444 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.444 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.444 DEBUG:boto:Params: {} 2024-04-13T22:04:30.444 DEBUG:boto:Token: None 2024-04-13T22:04:30.444 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE= 2024-04-13T22:04:30.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE=', 'Content-Length': '0'} 2024-04-13T22:04:30.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd6461ecbd23b707-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.447 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.447 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.447 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.447 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.447 DEBUG:boto:Method: GET 2024-04-13T22:04:30.447 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.447 DEBUG:boto:Data: 2024-04-13T22:04:30.447 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.448 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.448 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.448 DEBUG:boto:Params: {} 2024-04-13T22:04:30.448 DEBUG:boto:Token: None 2024-04-13T22:04:30.448 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:04:30.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE= 2024-04-13T22:04:30.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zYSn4zFwXJCGmh6ma7pXZV6CSRE=', 'Content-Length': '0'} 2024-04-13T22:04:30.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000252bcd9628ddd8c8-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.450 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.450 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.451 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.451 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.451 DEBUG:boto:Method: GET 2024-04-13T22:04:30.451 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.451 DEBUG:boto:Data: 2024-04-13T22:04:30.451 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.451 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.451 DEBUG:boto:Params: {} 2024-04-13T22:04:30.451 DEBUG:boto:Token: None 2024-04-13T22:04:30.452 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc0c903549954eb-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.494 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.495 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.495 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.495 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.495 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.495 DEBUG:boto:Method: GET 2024-04-13T22:04:30.495 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.495 DEBUG:boto:Data: 2024-04-13T22:04:30.496 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.496 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.496 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.496 DEBUG:boto:Params: {} 2024-04-13T22:04:30.496 DEBUG:boto:Token: None 2024-04-13T22:04:30.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077db37df52f7484a-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.498 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.499 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.499 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.499 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.499 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:04:30.499 DEBUG:boto:Data: 2024-04-13T22:04:30.499 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.499 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.499 DEBUG:boto:Params: {} 2024-04-13T22:04:30.500 DEBUG:boto:Token: None 2024-04-13T22:04:30.500 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1 2024-04-13T22:04:30.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WSRYVCxOVrYJNremxVIrAAV8nqc= 2024-04-13T22:04:30.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WSRYVCxOVrYJNremxVIrAAV8nqc=', 'Content-Length': '0'} 2024-04-13T22:04:30.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1486ef74fbc83ac-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.502 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.502 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.502 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.502 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:04:30.502 DEBUG:boto:Data: 2024-04-13T22:04:30.502 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.502 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.502 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.503 DEBUG:boto:Params: {} 2024-04-13T22:04:30.503 DEBUG:boto:Token: None 2024-04-13T22:04:30.503 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1 2024-04-13T22:04:30.503 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WSRYVCxOVrYJNremxVIrAAV8nqc= 2024-04-13T22:04:30.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WSRYVCxOVrYJNremxVIrAAV8nqc=', 'Content-Length': '0'} 2024-04-13T22:04:30.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000043b61449bf0714a-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:30.505 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.505 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.505 DEBUG:boto:Method: GET 2024-04-13T22:04:30.505 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:04:30.505 DEBUG:boto:Data: 2024-04-13T22:04:30.505 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.506 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.506 DEBUG:boto:Params: {} 2024-04-13T22:04:30.506 DEBUG:boto:Token: None 2024-04-13T22:04:30.506 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1 2024-04-13T22:04:30.506 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gPe+E6m5w5zimuJyjh8OaU8sins= 2024-04-13T22:04:30.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gPe+E6m5w5zimuJyjh8OaU8sins=', 'Content-Length': '0'} 2024-04-13T22:04:30.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cef40f7b367bcf52-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.508 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:04:30.508 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:04:30.508 DEBUG:boto:Method: GET 2024-04-13T22:04:30.509 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:04:30.509 DEBUG:boto:Data: 2024-04-13T22:04:30.509 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.509 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.509 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.509 DEBUG:boto:Params: {} 2024-04-13T22:04:30.509 DEBUG:boto:Token: None 2024-04-13T22:04:30.509 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj1 2024-04-13T22:04:30.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gPe+E6m5w5zimuJyjh8OaU8sins= 2024-04-13T22:04:30.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gPe+E6m5w5zimuJyjh8OaU8sins=', 'Content-Length': '0'} 2024-04-13T22:04:30.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037ae6618d6a90aa3-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.558 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.558 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.559 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.559 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.559 DEBUG:boto:Method: GET 2024-04-13T22:04:30.559 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.559 DEBUG:boto:Data: 2024-04-13T22:04:30.559 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.559 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.559 DEBUG:boto:Params: {} 2024-04-13T22:04:30.559 DEBUG:boto:Token: None 2024-04-13T22:04:30.560 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c4979f466a7157e-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.562 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.563 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.563 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.563 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.563 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.563 DEBUG:boto:Method: GET 2024-04-13T22:04:30.563 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:04:30.563 DEBUG:boto:Data: 2024-04-13T22:04:30.563 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.563 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.563 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.564 DEBUG:boto:Params: {} 2024-04-13T22:04:30.564 DEBUG:boto:Token: None 2024-04-13T22:04:30.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008676ff51c30ed786-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.566 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:30.567 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.567 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.567 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.567 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.567 DEBUG:boto:Method: GET 2024-04-13T22:04:30.567 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.568 DEBUG:boto:Data: 2024-04-13T22:04:30.568 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.568 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.568 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.568 DEBUG:boto:Params: {} 2024-04-13T22:04:30.568 DEBUG:boto:Token: None 2024-04-13T22:04:30.568 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE= 2024-04-13T22:04:30.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE=', 'Content-Length': '0'} 2024-04-13T22:04:30.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053433eba7a8aed69-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.570 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.570 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.570 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.570 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.570 DEBUG:boto:Method: GET 2024-04-13T22:04:30.571 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.571 DEBUG:boto:Data: 2024-04-13T22:04:30.571 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.571 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.571 DEBUG:boto:Params: {} 2024-04-13T22:04:30.571 DEBUG:boto:Token: None 2024-04-13T22:04:30.571 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE= 2024-04-13T22:04:30.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE=', 'Content-Length': '0'} 2024-04-13T22:04:30.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000931647a0eb1a79f4-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.574 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.574 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.574 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.574 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.574 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.574 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.574 DEBUG:boto:Data: 2024-04-13T22:04:30.574 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.574 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.574 DEBUG:boto:Params: {} 2024-04-13T22:04:30.575 DEBUG:boto:Token: None 2024-04-13T22:04:30.575 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ThcmxQURfA1+YGKus1Udjg83TNU= 2024-04-13T22:04:30.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ThcmxQURfA1+YGKus1Udjg83TNU=', 'Content-Length': '0'} 2024-04-13T22:04:30.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adab46c4d8bc0346-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.577 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.577 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.577 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.577 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.577 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.577 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.577 DEBUG:boto:Data: 2024-04-13T22:04:30.578 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.578 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.578 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.578 DEBUG:boto:Params: {} 2024-04-13T22:04:30.578 DEBUG:boto:Token: None 2024-04-13T22:04:30.578 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.578 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ThcmxQURfA1+YGKus1Udjg83TNU= 2024-04-13T22:04:30.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ThcmxQURfA1+YGKus1Udjg83TNU=', 'Content-Length': '0'} 2024-04-13T22:04:30.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d03753407f9cc07-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:30.580 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.580 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.580 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.580 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.581 DEBUG:boto:Method: GET 2024-04-13T22:04:30.581 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.581 DEBUG:boto:Data: 2024-04-13T22:04:30.581 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.581 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.581 DEBUG:boto:Params: {} 2024-04-13T22:04:30.581 DEBUG:boto:Token: None 2024-04-13T22:04:30.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.581 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE= 2024-04-13T22:04:30.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE=', 'Content-Length': '0'} 2024-04-13T22:04:30.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcb64d1e162c5933-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.584 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.584 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.584 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.584 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.584 DEBUG:boto:Method: GET 2024-04-13T22:04:30.584 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.584 DEBUG:boto:Data: 2024-04-13T22:04:30.584 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.584 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.584 DEBUG:boto:Params: {} 2024-04-13T22:04:30.585 DEBUG:boto:Token: None 2024-04-13T22:04:30.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:04:30.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE= 2024-04-13T22:04:30.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5cJLOiSfmobREOEA5eG25nNcsYE=', 'Content-Length': '0'} 2024-04-13T22:04:30.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c1955aaf6531d8-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.587 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.587 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.587 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.587 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.587 DEBUG:boto:Method: GET 2024-04-13T22:04:30.587 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.587 DEBUG:boto:Data: 2024-04-13T22:04:30.587 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.588 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.588 DEBUG:boto:Params: {} 2024-04-13T22:04:30.588 DEBUG:boto:Token: None 2024-04-13T22:04:30.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ebd2775adf57ab-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.591 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.591 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.591 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.591 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.591 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.591 DEBUG:boto:Method: GET 2024-04-13T22:04:30.591 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.592 DEBUG:boto:Data: 2024-04-13T22:04:30.592 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.592 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.592 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.592 DEBUG:boto:Params: {} 2024-04-13T22:04:30.592 DEBUG:boto:Token: None 2024-04-13T22:04:30.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079d6bff51c4fe2af-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.638 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.639 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.639 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.639 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.639 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:04:30.639 DEBUG:boto:Data: 2024-04-13T22:04:30.639 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.640 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.640 DEBUG:boto:Params: {} 2024-04-13T22:04:30.640 DEBUG:boto:Token: None 2024-04-13T22:04:30.640 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2 2024-04-13T22:04:30.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2Y/JPb3vqnRPVYYntsvdq3Usrek= 2024-04-13T22:04:30.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2Y/JPb3vqnRPVYYntsvdq3Usrek=', 'Content-Length': '0'} 2024-04-13T22:04:30.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d49c7be988a87d8-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.642 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.642 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.643 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.643 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:04:30.643 DEBUG:boto:Data: 2024-04-13T22:04:30.643 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.643 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.643 DEBUG:boto:Params: {} 2024-04-13T22:04:30.643 DEBUG:boto:Token: None 2024-04-13T22:04:30.643 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2 2024-04-13T22:04:30.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2Y/JPb3vqnRPVYYntsvdq3Usrek= 2024-04-13T22:04:30.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2Y/JPb3vqnRPVYYntsvdq3Usrek=', 'Content-Length': '0'} 2024-04-13T22:04:30.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4454b15b1770e86-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:30.645 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.645 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.646 DEBUG:boto:Method: GET 2024-04-13T22:04:30.646 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:04:30.646 DEBUG:boto:Data: 2024-04-13T22:04:30.646 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.646 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.646 DEBUG:boto:Params: {} 2024-04-13T22:04:30.646 DEBUG:boto:Token: None 2024-04-13T22:04:30.646 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2 2024-04-13T22:04:30.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0+E6e/aeSZUXcikMD9E35GMoQio= 2024-04-13T22:04:30.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0+E6e/aeSZUXcikMD9E35GMoQio=', 'Content-Length': '0'} 2024-04-13T22:04:30.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032d76a65edb7aaee-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.648 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:04:30.649 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:04:30.649 DEBUG:boto:Method: GET 2024-04-13T22:04:30.649 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:04:30.649 DEBUG:boto:Data: 2024-04-13T22:04:30.649 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.649 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.649 DEBUG:boto:Params: {} 2024-04-13T22:04:30.649 DEBUG:boto:Token: None 2024-04-13T22:04:30.649 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj2 2024-04-13T22:04:30.650 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0+E6e/aeSZUXcikMD9E35GMoQio= 2024-04-13T22:04:30.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0+E6e/aeSZUXcikMD9E35GMoQio=', 'Content-Length': '0'} 2024-04-13T22:04:30.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ec2e11a9d885132-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.652 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.652 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.652 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.652 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.652 DEBUG:boto:Method: GET 2024-04-13T22:04:30.652 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.652 DEBUG:boto:Data: 2024-04-13T22:04:30.652 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.652 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.652 DEBUG:boto:Params: {} 2024-04-13T22:04:30.653 DEBUG:boto:Token: None 2024-04-13T22:04:30.653 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007035d37f7acb1cb2-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.655 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.656 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.656 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.656 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.656 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.656 DEBUG:boto:Method: GET 2024-04-13T22:04:30.656 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:04:30.656 DEBUG:boto:Data: 2024-04-13T22:04:30.657 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.657 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.657 DEBUG:boto:Params: {} 2024-04-13T22:04:30.657 DEBUG:boto:Token: None 2024-04-13T22:04:30.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050e8490ec08924ec-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.660 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:30.660 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.660 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.660 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.660 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.660 DEBUG:boto:Method: GET 2024-04-13T22:04:30.660 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.660 DEBUG:boto:Data: 2024-04-13T22:04:30.661 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.661 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.661 DEBUG:boto:Params: {} 2024-04-13T22:04:30.661 DEBUG:boto:Token: None 2024-04-13T22:04:30.661 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.661 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc= 2024-04-13T22:04:30.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc=', 'Content-Length': '0'} 2024-04-13T22:04:30.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad0d7d7fed2392e-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.710 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.710 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.711 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.711 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.711 DEBUG:boto:Method: GET 2024-04-13T22:04:30.711 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.711 DEBUG:boto:Data: 2024-04-13T22:04:30.711 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.711 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.711 DEBUG:boto:Params: {} 2024-04-13T22:04:30.711 DEBUG:boto:Token: None 2024-04-13T22:04:30.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc= 2024-04-13T22:04:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc=', 'Content-Length': '0'} 2024-04-13T22:04:30.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000831aacf88c3b7b5a-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.714 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.714 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.714 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.714 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.714 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.714 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.715 DEBUG:boto:Data: 2024-04-13T22:04:30.715 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.715 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.715 DEBUG:boto:Params: {} 2024-04-13T22:04:30.715 DEBUG:boto:Token: None 2024-04-13T22:04:30.715 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3VFYNdU3cHQ7lrG0ubHoBJGJTw= 2024-04-13T22:04:30.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3VFYNdU3cHQ7lrG0ubHoBJGJTw=', 'Content-Length': '0'} 2024-04-13T22:04:30.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d7d885ec314758c-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.717 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.717 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.717 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.717 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.717 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.718 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.718 DEBUG:boto:Data: 2024-04-13T22:04:30.718 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.718 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.718 DEBUG:boto:Params: {} 2024-04-13T22:04:30.718 DEBUG:boto:Token: None 2024-04-13T22:04:30.718 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z3VFYNdU3cHQ7lrG0ubHoBJGJTw= 2024-04-13T22:04:30.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z3VFYNdU3cHQ7lrG0ubHoBJGJTw=', 'Content-Length': '0'} 2024-04-13T22:04:30.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028838e9a0800b35e-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:30.720 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.720 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.720 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.720 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.721 DEBUG:boto:Method: GET 2024-04-13T22:04:30.721 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.721 DEBUG:boto:Data: 2024-04-13T22:04:30.721 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.721 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.721 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.721 DEBUG:boto:Params: {} 2024-04-13T22:04:30.721 DEBUG:boto:Token: None 2024-04-13T22:04:30.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc= 2024-04-13T22:04:30.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc=', 'Content-Length': '0'} 2024-04-13T22:04:30.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea68d4c0672fc3b9-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.723 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.723 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.723 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.724 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.724 DEBUG:boto:Method: GET 2024-04-13T22:04:30.724 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.724 DEBUG:boto:Data: 2024-04-13T22:04:30.724 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.724 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.724 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.724 DEBUG:boto:Params: {} 2024-04-13T22:04:30.724 DEBUG:boto:Token: None 2024-04-13T22:04:30.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:04:30.725 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc= 2024-04-13T22:04:30.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PTvYwzxdlsVgY46FayfFUi+ZcBc=', 'Content-Length': '0'} 2024-04-13T22:04:30.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001406bdf5bd28214d-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.727 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.727 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.727 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.727 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.727 DEBUG:boto:Method: GET 2024-04-13T22:04:30.727 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.727 DEBUG:boto:Data: 2024-04-13T22:04:30.727 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.727 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.728 DEBUG:boto:Params: {} 2024-04-13T22:04:30.728 DEBUG:boto:Token: None 2024-04-13T22:04:30.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7fd6cfcde0ac8a4-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.730 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.731 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.731 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.731 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.731 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.731 DEBUG:boto:Method: GET 2024-04-13T22:04:30.731 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.731 DEBUG:boto:Data: 2024-04-13T22:04:30.731 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.731 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.731 DEBUG:boto:Params: {} 2024-04-13T22:04:30.732 DEBUG:boto:Token: None 2024-04-13T22:04:30.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6bb888129123080-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.734 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.734 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.734 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.735 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.735 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:04:30.735 DEBUG:boto:Data: 2024-04-13T22:04:30.735 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.735 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.735 DEBUG:boto:Params: {} 2024-04-13T22:04:30.735 DEBUG:boto:Token: None 2024-04-13T22:04:30.735 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3 2024-04-13T22:04:30.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tSOqVjQF7RJ1xd3VDq3Z5XB7nhs= 2024-04-13T22:04:30.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tSOqVjQF7RJ1xd3VDq3Z5XB7nhs=', 'Content-Length': '0'} 2024-04-13T22:04:30.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002308534263037e4e-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.737 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.737 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.737 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.738 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:04:30.738 DEBUG:boto:Data: 2024-04-13T22:04:30.738 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.738 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.738 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.738 DEBUG:boto:Params: {} 2024-04-13T22:04:30.738 DEBUG:boto:Token: None 2024-04-13T22:04:30.738 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3 2024-04-13T22:04:30.738 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tSOqVjQF7RJ1xd3VDq3Z5XB7nhs= 2024-04-13T22:04:30.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tSOqVjQF7RJ1xd3VDq3Z5XB7nhs=', 'Content-Length': '0'} 2024-04-13T22:04:30.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003750158ef82f245e-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:30.740 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.740 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.740 DEBUG:boto:Method: GET 2024-04-13T22:04:30.741 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:04:30.741 DEBUG:boto:Data: 2024-04-13T22:04:30.741 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.741 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.741 DEBUG:boto:Params: {} 2024-04-13T22:04:30.741 DEBUG:boto:Token: None 2024-04-13T22:04:30.741 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3 2024-04-13T22:04:30.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rmJ8ZcKjOVcvin+8U2dO3opTD2I= 2024-04-13T22:04:30.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rmJ8ZcKjOVcvin+8U2dO3opTD2I=', 'Content-Length': '0'} 2024-04-13T22:04:30.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de12322700efe88c-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.743 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:04:30.743 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:04:30.744 DEBUG:boto:Method: GET 2024-04-13T22:04:30.744 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:04:30.744 DEBUG:boto:Data: 2024-04-13T22:04:30.744 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.744 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.744 DEBUG:boto:Params: {} 2024-04-13T22:04:30.744 DEBUG:boto:Token: None 2024-04-13T22:04:30.744 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj3 2024-04-13T22:04:30.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rmJ8ZcKjOVcvin+8U2dO3opTD2I= 2024-04-13T22:04:30.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rmJ8ZcKjOVcvin+8U2dO3opTD2I=', 'Content-Length': '0'} 2024-04-13T22:04:30.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc974bbd2f815009-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.746 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.746 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.746 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.746 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.747 DEBUG:boto:Method: GET 2024-04-13T22:04:30.747 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.747 DEBUG:boto:Data: 2024-04-13T22:04:30.747 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.747 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.747 DEBUG:boto:Params: {} 2024-04-13T22:04:30.747 DEBUG:boto:Token: None 2024-04-13T22:04:30.747 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.747 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074234890a056aaa3-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.790 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.791 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.791 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.791 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.791 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.791 DEBUG:boto:Method: GET 2024-04-13T22:04:30.791 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:04:30.791 DEBUG:boto:Data: 2024-04-13T22:04:30.792 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.792 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.792 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.792 DEBUG:boto:Params: {} 2024-04-13T22:04:30.792 DEBUG:boto:Token: None 2024-04-13T22:04:30.792 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7426cb136d5ad19-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.795 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:30.795 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.795 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.795 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.795 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.795 DEBUG:boto:Method: GET 2024-04-13T22:04:30.795 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.796 DEBUG:boto:Data: 2024-04-13T22:04:30.796 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.796 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.796 DEBUG:boto:Params: {} 2024-04-13T22:04:30.796 DEBUG:boto:Token: None 2024-04-13T22:04:30.796 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE= 2024-04-13T22:04:30.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE=', 'Content-Length': '0'} 2024-04-13T22:04:30.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f15c9365a433558c-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.798 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.798 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.800 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.801 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.801 DEBUG:boto:Method: GET 2024-04-13T22:04:30.801 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.801 DEBUG:boto:Data: 2024-04-13T22:04:30.801 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.801 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.801 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.801 DEBUG:boto:Params: {} 2024-04-13T22:04:30.802 DEBUG:boto:Token: None 2024-04-13T22:04:30.802 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.802 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE= 2024-04-13T22:04:30.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE=', 'Content-Length': '0'} 2024-04-13T22:04:30.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaa3b8969acc21d1-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.846 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.846 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.847 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.847 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.847 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.847 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.847 DEBUG:boto:Data: 2024-04-13T22:04:30.847 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.847 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.847 DEBUG:boto:Params: {} 2024-04-13T22:04:30.847 DEBUG:boto:Token: None 2024-04-13T22:04:30.848 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1cqsw4vga6qdK/UzaJ504Z3sUKI= 2024-04-13T22:04:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1cqsw4vga6qdK/UzaJ504Z3sUKI=', 'Content-Length': '0'} 2024-04-13T22:04:30.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000812a864d1906894a-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.850 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.850 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.850 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.850 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.850 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.850 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.850 DEBUG:boto:Data: 2024-04-13T22:04:30.850 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.851 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.851 DEBUG:boto:Params: {} 2024-04-13T22:04:30.851 DEBUG:boto:Token: None 2024-04-13T22:04:30.851 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1cqsw4vga6qdK/UzaJ504Z3sUKI= 2024-04-13T22:04:30.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1cqsw4vga6qdK/UzaJ504Z3sUKI=', 'Content-Length': '0'} 2024-04-13T22:04:30.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005921e49a9bc8c7d4-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:30.853 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.853 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.853 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.853 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.854 DEBUG:boto:Method: GET 2024-04-13T22:04:30.854 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.854 DEBUG:boto:Data: 2024-04-13T22:04:30.854 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.854 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.854 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.854 DEBUG:boto:Params: {} 2024-04-13T22:04:30.854 DEBUG:boto:Token: None 2024-04-13T22:04:30.855 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.855 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE= 2024-04-13T22:04:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE=', 'Content-Length': '0'} 2024-04-13T22:04:30.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079d9e0aab6f437a9-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.857 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.857 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.857 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.857 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.857 DEBUG:boto:Method: GET 2024-04-13T22:04:30.857 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.857 DEBUG:boto:Data: 2024-04-13T22:04:30.858 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.858 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.858 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.858 DEBUG:boto:Params: {} 2024-04-13T22:04:30.858 DEBUG:boto:Token: None 2024-04-13T22:04:30.858 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:04:30.858 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE= 2024-04-13T22:04:30.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aLZZT9kWOhRB17pJHU9XZkMXfVE=', 'Content-Length': '0'} 2024-04-13T22:04:30.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4978c0f607d8379-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.860 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.860 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.860 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.861 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.861 DEBUG:boto:Method: GET 2024-04-13T22:04:30.861 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.861 DEBUG:boto:Data: 2024-04-13T22:04:30.861 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.861 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.861 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.861 DEBUG:boto:Params: {} 2024-04-13T22:04:30.861 DEBUG:boto:Token: None 2024-04-13T22:04:30.861 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.862 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e43e4a1fda16ab5-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.864 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.864 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.864 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.864 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.864 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.864 DEBUG:boto:Method: GET 2024-04-13T22:04:30.865 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.865 DEBUG:boto:Data: 2024-04-13T22:04:30.865 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.865 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.865 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.865 DEBUG:boto:Params: {} 2024-04-13T22:04:30.865 DEBUG:boto:Token: None 2024-04-13T22:04:30.865 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.865 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cfbca69b95d2005-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.868 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.868 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.868 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.868 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.868 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:04:30.869 DEBUG:boto:Data: 2024-04-13T22:04:30.869 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.869 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.869 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.869 DEBUG:boto:Params: {} 2024-04-13T22:04:30.869 DEBUG:boto:Token: None 2024-04-13T22:04:30.869 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4 2024-04-13T22:04:30.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SA61asK2uUVOOLl5Qs9iWc6sI/U= 2024-04-13T22:04:30.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SA61asK2uUVOOLl5Qs9iWc6sI/U=', 'Content-Length': '0'} 2024-04-13T22:04:30.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000692a46dd087d46ff-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.871 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.871 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.871 DEBUG:boto:Method: HEAD 2024-04-13T22:04:30.871 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:04:30.871 DEBUG:boto:Data: 2024-04-13T22:04:30.872 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.872 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.872 DEBUG:boto:Params: {} 2024-04-13T22:04:30.872 DEBUG:boto:Token: None 2024-04-13T22:04:30.872 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4 2024-04-13T22:04:30.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SA61asK2uUVOOLl5Qs9iWc6sI/U= 2024-04-13T22:04:30.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SA61asK2uUVOOLl5Qs9iWc6sI/U=', 'Content-Length': '0'} 2024-04-13T22:04:30.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000094a079b914b10a0-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:30.874 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.874 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.874 DEBUG:boto:Method: GET 2024-04-13T22:04:30.874 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:04:30.875 DEBUG:boto:Data: 2024-04-13T22:04:30.875 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.875 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.875 DEBUG:boto:Params: {} 2024-04-13T22:04:30.875 DEBUG:boto:Token: None 2024-04-13T22:04:30.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4 2024-04-13T22:04:30.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6f++zMcmmnNb3M7hL+5Oze2MSNE= 2024-04-13T22:04:30.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6f++zMcmmnNb3M7hL+5Oze2MSNE=', 'Content-Length': '0'} 2024-04-13T22:04:30.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff9a98defeee25c7-00661b016e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.918 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:04:30.919 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:04:30.919 DEBUG:boto:Method: GET 2024-04-13T22:04:30.919 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:04:30.919 DEBUG:boto:Data: 2024-04-13T22:04:30.919 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.919 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.919 DEBUG:boto:Params: {} 2024-04-13T22:04:30.920 DEBUG:boto:Token: None 2024-04-13T22:04:30.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/obj4 2024-04-13T22:04:30.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6f++zMcmmnNb3M7hL+5Oze2MSNE= 2024-04-13T22:04:30.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6f++zMcmmnNb3M7hL+5Oze2MSNE=', 'Content-Length': '0'} 2024-04-13T22:04:30.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000385e745420d59b0b-00661b016e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.922 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.923 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.923 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.923 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.923 DEBUG:boto:Method: GET 2024-04-13T22:04:30.923 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.923 DEBUG:boto:Data: 2024-04-13T22:04:30.923 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:30.923 DEBUG:boto:Port: 8001 2024-04-13T22:04:30.923 DEBUG:boto:Params: {} 2024-04-13T22:04:30.924 DEBUG:boto:Token: None 2024-04-13T22:04:30.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fa10cb18e37738-00661b016e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.926 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.927 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:04:30.927 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:04:30.927 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.927 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.927 DEBUG:boto:Method: GET 2024-04-13T22:04:30.927 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:04:30.927 DEBUG:boto:Data: 2024-04-13T22:04:30.927 DEBUG:boto:Headers: {} 2024-04-13T22:04:30.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:30.927 DEBUG:boto:Port: 8000 2024-04-13T22:04:30.927 DEBUG:boto:Params: {} 2024-04-13T22:04:30.928 DEBUG:boto:Token: None 2024-04-13T22:04:30.928 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:30 GMT /vwnfns-50/ 2024-04-13T22:04:30.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM= 2024-04-13T22:04:30.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oDTxM+mKMflS8Z5z+2yB5Js7iHM=', 'Content-Length': '0'} 2024-04-13T22:04:30.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088794bc504d80d79-00661b016e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:30.930 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:30.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-50 zones={a2, a1} 2024-04-13T22:04:30.930 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-51 2024-04-13T22:04:30.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:30.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:30.931 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-51 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:04:30.986 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:30.987 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:31.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:31.021+0000 7f3c501c9780 1 waiting to reach incremental sync.. 2024-04-13T22:04:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:36.022+0000 7f3c501c9780 1 waiting to reach incremental sync.. 2024-04-13T22:04:41.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:41.023+0000 7f3c501c9780 1 waiting to reach incremental sync.. 2024-04-13T22:04:46.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:46.025+0000 7f3c501c9780 1 bucket sync caught up with source: 2024-04-13T22:04:46.026 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.32.6, , 00000000001.18.6, , 00000000001.18.6, , , 00000000001.40.6, , , ] 2024-04-13T22:04:46.026 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.32.6, , 00000000001.18.6, , 00000000001.18.6, , , 00000000001.40.6, , , ] 2024-04-13T22:04:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:46.025+0000 7f3c501c9780 0 bucket checkpoint complete 2024-04-13T22:04:46.043 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-51 2024-04-13T22:04:46.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:46.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:04:46.044 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-51 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:04:46.105 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:46.105 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:46.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:46.142+0000 7f421520c780 1 bucket sync caught up with source: 2024-04-13T22:04:46.142 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.538.6, , 00000000001.2004.6, , 00000000001.806.6, , , 00000000001.1738.6, , , ] 2024-04-13T22:04:46.142 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.538.6, , 00000000001.2004.6, , 00000000001.806.6, , , 00000000001.1738.6, , , ] 2024-04-13T22:04:46.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:46.142+0000 7f421520c780 0 bucket checkpoint complete 2024-04-13T22:04:46.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-51 zones={a2, a1} 2024-04-13T22:04:46.155 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.155 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.155 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.155 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:46.155 DEBUG:boto:Data: 2024-04-13T22:04:46.155 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.156 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.156 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.156 DEBUG:boto:Params: {} 2024-04-13T22:04:46.156 DEBUG:boto:Token: None 2024-04-13T22:04:46.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/TY0GZFgsRisgeVH3PEMk0XDf+o= 2024-04-13T22:04:46.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/TY0GZFgsRisgeVH3PEMk0XDf+o=', 'Content-Length': '0'} 2024-04-13T22:04:46.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d06bac5cd856d028-00661b017e-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.159 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.159 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.159 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.159 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:04:46.159 DEBUG:boto:Data: 2024-04-13T22:04:46.159 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.160 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.160 DEBUG:boto:Params: {} 2024-04-13T22:04:46.160 DEBUG:boto:Token: None 2024-04-13T22:04:46.160 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/TY0GZFgsRisgeVH3PEMk0XDf+o= 2024-04-13T22:04:46.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/TY0GZFgsRisgeVH3PEMk0XDf+o=', 'Content-Length': '0'} 2024-04-13T22:04:46.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef7c9db87ebec89e-00661b017e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.162 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:04:46.162 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.163 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.163 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.163 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.163 DEBUG:boto:Method: GET 2024-04-13T22:04:46.163 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.163 DEBUG:boto:Data: 2024-04-13T22:04:46.163 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.163 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.163 DEBUG:boto:Params: {} 2024-04-13T22:04:46.164 DEBUG:boto:Token: None 2024-04-13T22:04:46.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/?versions 2024-04-13T22:04:46.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA= 2024-04-13T22:04:46.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA=', 'Content-Length': '0'} 2024-04-13T22:04:46.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73ad3a7b15341c3-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.166 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:46.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:46.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:46.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:46.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:04:46.167 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.167 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.167 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.168 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.168 DEBUG:boto:Method: GET 2024-04-13T22:04:46.168 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.168 DEBUG:boto:Data: 2024-04-13T22:04:46.168 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.168 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.168 DEBUG:boto:Params: {} 2024-04-13T22:04:46.168 DEBUG:boto:Token: None 2024-04-13T22:04:46.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/?versions 2024-04-13T22:04:46.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA= 2024-04-13T22:04:46.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA=', 'Content-Length': '0'} 2024-04-13T22:04:46.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015beeeb5fd587a90-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.170 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:04:46.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:04:46.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:04:46.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:04:46.171 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.171 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.172 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.172 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.172 DEBUG:boto:Method: GET 2024-04-13T22:04:46.172 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.172 DEBUG:boto:Data: 2024-04-13T22:04:46.172 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.172 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.172 DEBUG:boto:Params: {} 2024-04-13T22:04:46.172 DEBUG:boto:Token: None 2024-04-13T22:04:46.173 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/?versions 2024-04-13T22:04:46.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA= 2024-04-13T22:04:46.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA=', 'Content-Length': '0'} 2024-04-13T22:04:46.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c95af3c3c13487-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.175 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.175 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.175 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.176 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.176 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.176 DEBUG:boto:Method: GET 2024-04-13T22:04:46.176 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:04:46.176 DEBUG:boto:Data: 2024-04-13T22:04:46.176 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.176 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.176 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.176 DEBUG:boto:Params: {} 2024-04-13T22:04:46.176 DEBUG:boto:Token: None 2024-04-13T22:04:46.177 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/?versions 2024-04-13T22:04:46.177 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA= 2024-04-13T22:04:46.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ePXhY341jKiFRkN9k0YetVIf7uA=', 'Content-Length': '0'} 2024-04-13T22:04:46.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da5bd1691ab6f577-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.179 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:46.179 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.180 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.180 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.180 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.180 DEBUG:boto:Method: GET 2024-04-13T22:04:46.180 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.180 DEBUG:boto:Data: 2024-04-13T22:04:46.180 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.180 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.180 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.180 DEBUG:boto:Params: {} 2024-04-13T22:04:46.180 DEBUG:boto:Token: None 2024-04-13T22:04:46.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc= 2024-04-13T22:04:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc=', 'Content-Length': '0'} 2024-04-13T22:04:46.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000213a7e7709a018c7-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.230 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.231 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.231 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.231 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.231 DEBUG:boto:Method: GET 2024-04-13T22:04:46.231 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.231 DEBUG:boto:Data: 2024-04-13T22:04:46.231 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.231 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.231 DEBUG:boto:Params: {} 2024-04-13T22:04:46.231 DEBUG:boto:Token: None 2024-04-13T22:04:46.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc= 2024-04-13T22:04:46.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc=', 'Content-Length': '0'} 2024-04-13T22:04:46.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1b731d9dd7a5e9e-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.234 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.234 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.234 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.234 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.234 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.234 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.234 DEBUG:boto:Data: 2024-04-13T22:04:46.235 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.235 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.235 DEBUG:boto:Params: {} 2024-04-13T22:04:46.235 DEBUG:boto:Token: None 2024-04-13T22:04:46.235 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jOH+sIafVr0zGHVfSf1bY2ETPnk= 2024-04-13T22:04:46.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jOH+sIafVr0zGHVfSf1bY2ETPnk=', 'Content-Length': '0'} 2024-04-13T22:04:46.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5d06033af7932b7-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.237 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.237 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.237 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.237 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.237 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.237 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.237 DEBUG:boto:Data: 2024-04-13T22:04:46.237 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.237 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.238 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.238 DEBUG:boto:Params: {} 2024-04-13T22:04:46.238 DEBUG:boto:Token: None 2024-04-13T22:04:46.238 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.238 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jOH+sIafVr0zGHVfSf1bY2ETPnk= 2024-04-13T22:04:46.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jOH+sIafVr0zGHVfSf1bY2ETPnk=', 'Content-Length': '0'} 2024-04-13T22:04:46.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa05dfd3473fc03b-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:46.240 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.240 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.240 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.240 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.240 DEBUG:boto:Method: GET 2024-04-13T22:04:46.240 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.240 DEBUG:boto:Data: 2024-04-13T22:04:46.240 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.241 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.241 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.241 DEBUG:boto:Params: {} 2024-04-13T22:04:46.241 DEBUG:boto:Token: None 2024-04-13T22:04:46.241 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.241 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc= 2024-04-13T22:04:46.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc=', 'Content-Length': '0'} 2024-04-13T22:04:46.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000db467e74751cf2-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.243 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.243 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.243 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.243 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.243 DEBUG:boto:Method: GET 2024-04-13T22:04:46.243 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.243 DEBUG:boto:Data: 2024-04-13T22:04:46.243 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.243 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.244 DEBUG:boto:Params: {} 2024-04-13T22:04:46.244 DEBUG:boto:Token: None 2024-04-13T22:04:46.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:04:46.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc= 2024-04-13T22:04:46.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ut2IYeFfy3WG5dq8zaZNmJCrfsc=', 'Content-Length': '0'} 2024-04-13T22:04:46.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c432414e6e081e4-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.246 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.246 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.246 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.246 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.246 DEBUG:boto:Method: GET 2024-04-13T22:04:46.246 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.246 DEBUG:boto:Data: 2024-04-13T22:04:46.246 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.247 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.247 DEBUG:boto:Params: {} 2024-04-13T22:04:46.247 DEBUG:boto:Token: None 2024-04-13T22:04:46.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001942e31c878d974c-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.249 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.250 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.250 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.250 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.250 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.250 DEBUG:boto:Method: GET 2024-04-13T22:04:46.250 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.250 DEBUG:boto:Data: 2024-04-13T22:04:46.250 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.251 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.251 DEBUG:boto:Params: {} 2024-04-13T22:04:46.251 DEBUG:boto:Token: None 2024-04-13T22:04:46.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a531e66c495895da-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.253 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.254 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.254 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.254 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.254 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:04:46.254 DEBUG:boto:Data: 2024-04-13T22:04:46.254 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.254 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.255 DEBUG:boto:Params: {} 2024-04-13T22:04:46.255 DEBUG:boto:Token: None 2024-04-13T22:04:46.255 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1 2024-04-13T22:04:46.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BJyBqvVYMqMDLc6pNWvNUmWIxvk= 2024-04-13T22:04:46.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BJyBqvVYMqMDLc6pNWvNUmWIxvk=', 'Content-Length': '0'} 2024-04-13T22:04:46.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c063169a5138ac57-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.257 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.257 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.257 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.257 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:04:46.257 DEBUG:boto:Data: 2024-04-13T22:04:46.257 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.257 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.257 DEBUG:boto:Params: {} 2024-04-13T22:04:46.257 DEBUG:boto:Token: None 2024-04-13T22:04:46.258 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1 2024-04-13T22:04:46.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BJyBqvVYMqMDLc6pNWvNUmWIxvk= 2024-04-13T22:04:46.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BJyBqvVYMqMDLc6pNWvNUmWIxvk=', 'Content-Length': '0'} 2024-04-13T22:04:46.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acaa1203f063ff74-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:04:46.260 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.260 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.260 DEBUG:boto:Method: GET 2024-04-13T22:04:46.260 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:04:46.260 DEBUG:boto:Data: 2024-04-13T22:04:46.260 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.260 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.260 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.260 DEBUG:boto:Params: {} 2024-04-13T22:04:46.260 DEBUG:boto:Token: None 2024-04-13T22:04:46.261 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1 2024-04-13T22:04:46.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U5nnENiXhFz+mjH5xFT0YHHajOQ= 2024-04-13T22:04:46.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U5nnENiXhFz+mjH5xFT0YHHajOQ=', 'Content-Length': '0'} 2024-04-13T22:04:46.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ff03b2b89b9b5aa-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.262 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:04:46.262 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:04:46.263 DEBUG:boto:Method: GET 2024-04-13T22:04:46.263 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:04:46.263 DEBUG:boto:Data: 2024-04-13T22:04:46.263 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.263 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.263 DEBUG:boto:Params: {} 2024-04-13T22:04:46.263 DEBUG:boto:Token: None 2024-04-13T22:04:46.263 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj1 2024-04-13T22:04:46.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U5nnENiXhFz+mjH5xFT0YHHajOQ= 2024-04-13T22:04:46.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U5nnENiXhFz+mjH5xFT0YHHajOQ=', 'Content-Length': '0'} 2024-04-13T22:04:46.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d720e7b786194eb2-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.265 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.265 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.266 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.266 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.266 DEBUG:boto:Method: GET 2024-04-13T22:04:46.266 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.266 DEBUG:boto:Data: 2024-04-13T22:04:46.266 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.266 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.266 DEBUG:boto:Params: {} 2024-04-13T22:04:46.266 DEBUG:boto:Token: None 2024-04-13T22:04:46.267 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9341351af0b302-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.310 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.311 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.311 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.311 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.311 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.311 DEBUG:boto:Method: GET 2024-04-13T22:04:46.311 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:04:46.312 DEBUG:boto:Data: 2024-04-13T22:04:46.312 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.312 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.312 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.312 DEBUG:boto:Params: {} 2024-04-13T22:04:46.312 DEBUG:boto:Token: None 2024-04-13T22:04:46.312 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7437d3102bccbd-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.315 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:46.315 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.315 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.315 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.315 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.315 DEBUG:boto:Method: GET 2024-04-13T22:04:46.316 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.316 DEBUG:boto:Data: 2024-04-13T22:04:46.316 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.316 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.316 DEBUG:boto:Params: {} 2024-04-13T22:04:46.316 DEBUG:boto:Token: None 2024-04-13T22:04:46.316 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4= 2024-04-13T22:04:46.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4=', 'Content-Length': '0'} 2024-04-13T22:04:46.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59818b1b29caeda-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.318 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.318 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.319 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.319 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.319 DEBUG:boto:Method: GET 2024-04-13T22:04:46.319 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.319 DEBUG:boto:Data: 2024-04-13T22:04:46.319 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.319 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.319 DEBUG:boto:Params: {} 2024-04-13T22:04:46.319 DEBUG:boto:Token: None 2024-04-13T22:04:46.319 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4= 2024-04-13T22:04:46.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4=', 'Content-Length': '0'} 2024-04-13T22:04:46.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a6d00a52ad8a629-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.322 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.322 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.322 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.322 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.322 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.322 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.322 DEBUG:boto:Data: 2024-04-13T22:04:46.322 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.322 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.323 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.323 DEBUG:boto:Params: {} 2024-04-13T22:04:46.323 DEBUG:boto:Token: None 2024-04-13T22:04:46.323 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LoRTGY/39SFfQcv7Nxich2JVOkY= 2024-04-13T22:04:46.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LoRTGY/39SFfQcv7Nxich2JVOkY=', 'Content-Length': '0'} 2024-04-13T22:04:46.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc5da2335d151262-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.325 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.325 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.325 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.325 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.325 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.325 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.325 DEBUG:boto:Data: 2024-04-13T22:04:46.325 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.325 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.326 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.326 DEBUG:boto:Params: {} 2024-04-13T22:04:46.326 DEBUG:boto:Token: None 2024-04-13T22:04:46.326 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LoRTGY/39SFfQcv7Nxich2JVOkY= 2024-04-13T22:04:46.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LoRTGY/39SFfQcv7Nxich2JVOkY=', 'Content-Length': '0'} 2024-04-13T22:04:46.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e26dfb3f3b5f979a-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:46.328 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.328 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.328 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.328 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.328 DEBUG:boto:Method: GET 2024-04-13T22:04:46.328 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.328 DEBUG:boto:Data: 2024-04-13T22:04:46.329 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.329 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.329 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.329 DEBUG:boto:Params: {} 2024-04-13T22:04:46.329 DEBUG:boto:Token: None 2024-04-13T22:04:46.329 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4= 2024-04-13T22:04:46.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4=', 'Content-Length': '0'} 2024-04-13T22:04:46.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003af209aae4ea0c30-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.331 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.331 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.331 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.331 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.331 DEBUG:boto:Method: GET 2024-04-13T22:04:46.331 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.331 DEBUG:boto:Data: 2024-04-13T22:04:46.332 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.332 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.332 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.332 DEBUG:boto:Params: {} 2024-04-13T22:04:46.332 DEBUG:boto:Token: None 2024-04-13T22:04:46.332 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:04:46.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4= 2024-04-13T22:04:46.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i7UIycmMsT5N83zKRzsyK1jsOu4=', 'Content-Length': '0'} 2024-04-13T22:04:46.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b9a9c12188ff968-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.334 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.334 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.334 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.334 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.334 DEBUG:boto:Method: GET 2024-04-13T22:04:46.335 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.335 DEBUG:boto:Data: 2024-04-13T22:04:46.335 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.335 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.335 DEBUG:boto:Params: {} 2024-04-13T22:04:46.335 DEBUG:boto:Token: None 2024-04-13T22:04:46.335 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be31ef0887c383e2-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.338 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.338 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.338 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.338 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.338 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.338 DEBUG:boto:Method: GET 2024-04-13T22:04:46.338 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.339 DEBUG:boto:Data: 2024-04-13T22:04:46.339 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.339 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.339 DEBUG:boto:Params: {} 2024-04-13T22:04:46.339 DEBUG:boto:Token: None 2024-04-13T22:04:46.339 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e402525398d042a6-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.341 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.341 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.342 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.342 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.342 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:04:46.342 DEBUG:boto:Data: 2024-04-13T22:04:46.342 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.342 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.342 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.342 DEBUG:boto:Params: {} 2024-04-13T22:04:46.342 DEBUG:boto:Token: None 2024-04-13T22:04:46.343 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2 2024-04-13T22:04:46.343 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q6PXrPvkfXEUUSCdW6TTdC9/NyQ= 2024-04-13T22:04:46.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q6PXrPvkfXEUUSCdW6TTdC9/NyQ=', 'Content-Length': '0'} 2024-04-13T22:04:46.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a121ad504f6ebb97-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.344 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.344 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.345 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.345 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:04:46.345 DEBUG:boto:Data: 2024-04-13T22:04:46.345 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.345 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.345 DEBUG:boto:Params: {} 2024-04-13T22:04:46.345 DEBUG:boto:Token: None 2024-04-13T22:04:46.345 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2 2024-04-13T22:04:46.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q6PXrPvkfXEUUSCdW6TTdC9/NyQ= 2024-04-13T22:04:46.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q6PXrPvkfXEUUSCdW6TTdC9/NyQ=', 'Content-Length': '0'} 2024-04-13T22:04:46.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000081395fd5b6a1c10-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:04:46.347 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.348 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.348 DEBUG:boto:Method: GET 2024-04-13T22:04:46.348 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:04:46.348 DEBUG:boto:Data: 2024-04-13T22:04:46.348 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.348 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.348 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.348 DEBUG:boto:Params: {} 2024-04-13T22:04:46.348 DEBUG:boto:Token: None 2024-04-13T22:04:46.348 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2 2024-04-13T22:04:46.349 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EU+9SQQYKEJxw6I+FKy3Lc7IRUI= 2024-04-13T22:04:46.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EU+9SQQYKEJxw6I+FKy3Lc7IRUI=', 'Content-Length': '0'} 2024-04-13T22:04:46.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf1ac0bc0e5f4f44-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.350 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:04:46.350 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:04:46.351 DEBUG:boto:Method: GET 2024-04-13T22:04:46.351 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:04:46.351 DEBUG:boto:Data: 2024-04-13T22:04:46.351 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.351 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.351 DEBUG:boto:Params: {} 2024-04-13T22:04:46.351 DEBUG:boto:Token: None 2024-04-13T22:04:46.351 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj2 2024-04-13T22:04:46.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EU+9SQQYKEJxw6I+FKy3Lc7IRUI= 2024-04-13T22:04:46.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EU+9SQQYKEJxw6I+FKy3Lc7IRUI=', 'Content-Length': '0'} 2024-04-13T22:04:46.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002093b434ffeae1a-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.353 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.353 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.353 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.353 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.353 DEBUG:boto:Method: GET 2024-04-13T22:04:46.354 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.354 DEBUG:boto:Data: 2024-04-13T22:04:46.354 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.354 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.354 DEBUG:boto:Params: {} 2024-04-13T22:04:46.354 DEBUG:boto:Token: None 2024-04-13T22:04:46.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000497f114544929eed-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.356 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.357 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.357 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.357 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.357 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.357 DEBUG:boto:Method: GET 2024-04-13T22:04:46.357 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:04:46.357 DEBUG:boto:Data: 2024-04-13T22:04:46.357 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.358 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.358 DEBUG:boto:Params: {} 2024-04-13T22:04:46.358 DEBUG:boto:Token: None 2024-04-13T22:04:46.358 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099010d46441b525d-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.360 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:46.361 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.361 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.361 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.361 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.361 DEBUG:boto:Method: GET 2024-04-13T22:04:46.361 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.361 DEBUG:boto:Data: 2024-04-13T22:04:46.361 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.362 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.362 DEBUG:boto:Params: {} 2024-04-13T22:04:46.362 DEBUG:boto:Token: None 2024-04-13T22:04:46.362 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.362 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34= 2024-04-13T22:04:46.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34=', 'Content-Length': '0'} 2024-04-13T22:04:46.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4faceb48848d417-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.364 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.364 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.364 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.364 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.364 DEBUG:boto:Method: GET 2024-04-13T22:04:46.364 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.364 DEBUG:boto:Data: 2024-04-13T22:04:46.364 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.364 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.364 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.364 DEBUG:boto:Params: {} 2024-04-13T22:04:46.365 DEBUG:boto:Token: None 2024-04-13T22:04:46.365 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34= 2024-04-13T22:04:46.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34=', 'Content-Length': '0'} 2024-04-13T22:04:46.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002661e55fb0acd6c5-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.367 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.367 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.367 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.367 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.367 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.367 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.367 DEBUG:boto:Data: 2024-04-13T22:04:46.367 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.367 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.367 DEBUG:boto:Params: {} 2024-04-13T22:04:46.368 DEBUG:boto:Token: None 2024-04-13T22:04:46.368 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FHI1VN/M8m0x35Cyzp+Rv/Acy4E= 2024-04-13T22:04:46.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FHI1VN/M8m0x35Cyzp+Rv/Acy4E=', 'Content-Length': '0'} 2024-04-13T22:04:46.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086a6b6d83bc3b818-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.369 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.370 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.370 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.370 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.370 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.370 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.370 DEBUG:boto:Data: 2024-04-13T22:04:46.370 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.370 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.370 DEBUG:boto:Params: {} 2024-04-13T22:04:46.370 DEBUG:boto:Token: None 2024-04-13T22:04:46.371 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FHI1VN/M8m0x35Cyzp+Rv/Acy4E= 2024-04-13T22:04:46.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FHI1VN/M8m0x35Cyzp+Rv/Acy4E=', 'Content-Length': '0'} 2024-04-13T22:04:46.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f9a473a750bd810-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:46.373 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.373 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.373 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.373 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.373 DEBUG:boto:Method: GET 2024-04-13T22:04:46.373 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.373 DEBUG:boto:Data: 2024-04-13T22:04:46.373 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.373 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.374 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.374 DEBUG:boto:Params: {} 2024-04-13T22:04:46.374 DEBUG:boto:Token: None 2024-04-13T22:04:46.374 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.374 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34= 2024-04-13T22:04:46.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34=', 'Content-Length': '0'} 2024-04-13T22:04:46.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8a81a1265f1cb72-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.376 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.376 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.376 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.376 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.376 DEBUG:boto:Method: GET 2024-04-13T22:04:46.376 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.376 DEBUG:boto:Data: 2024-04-13T22:04:46.376 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.377 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.377 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.377 DEBUG:boto:Params: {} 2024-04-13T22:04:46.377 DEBUG:boto:Token: None 2024-04-13T22:04:46.377 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:04:46.377 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34= 2024-04-13T22:04:46.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGvHcGH9H1XGqNyDRRVF+4ir+34=', 'Content-Length': '0'} 2024-04-13T22:04:46.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069616485cc99136c-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.422 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.423 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.423 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.423 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.423 DEBUG:boto:Method: GET 2024-04-13T22:04:46.423 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.423 DEBUG:boto:Data: 2024-04-13T22:04:46.423 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.424 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.424 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.424 DEBUG:boto:Params: {} 2024-04-13T22:04:46.424 DEBUG:boto:Token: None 2024-04-13T22:04:46.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038bacaee7bd771ed-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.427 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.427 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.427 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.427 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.427 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.427 DEBUG:boto:Method: GET 2024-04-13T22:04:46.427 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.427 DEBUG:boto:Data: 2024-04-13T22:04:46.428 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.428 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.428 DEBUG:boto:Params: {} 2024-04-13T22:04:46.428 DEBUG:boto:Token: None 2024-04-13T22:04:46.428 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.428 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027c17a5a7f8a59bc-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.430 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.431 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.431 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.431 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.431 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:04:46.431 DEBUG:boto:Data: 2024-04-13T22:04:46.431 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.431 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.431 DEBUG:boto:Params: {} 2024-04-13T22:04:46.432 DEBUG:boto:Token: None 2024-04-13T22:04:46.432 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3 2024-04-13T22:04:46.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qhTSEoNM6EMC5b1E05KItlYqWv0= 2024-04-13T22:04:46.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qhTSEoNM6EMC5b1E05KItlYqWv0=', 'Content-Length': '0'} 2024-04-13T22:04:46.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036778310e69e5114-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.434 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.434 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.434 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.434 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:04:46.434 DEBUG:boto:Data: 2024-04-13T22:04:46.434 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.434 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.434 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.434 DEBUG:boto:Params: {} 2024-04-13T22:04:46.434 DEBUG:boto:Token: None 2024-04-13T22:04:46.435 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3 2024-04-13T22:04:46.435 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qhTSEoNM6EMC5b1E05KItlYqWv0= 2024-04-13T22:04:46.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qhTSEoNM6EMC5b1E05KItlYqWv0=', 'Content-Length': '0'} 2024-04-13T22:04:46.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad74eff5fa9afe38-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:04:46.437 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.437 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.437 DEBUG:boto:Method: GET 2024-04-13T22:04:46.437 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:04:46.437 DEBUG:boto:Data: 2024-04-13T22:04:46.437 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.437 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.437 DEBUG:boto:Params: {} 2024-04-13T22:04:46.438 DEBUG:boto:Token: None 2024-04-13T22:04:46.438 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3 2024-04-13T22:04:46.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2/7xtnPwohoTcBuIBMPL+1vl8dY= 2024-04-13T22:04:46.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2/7xtnPwohoTcBuIBMPL+1vl8dY=', 'Content-Length': '0'} 2024-04-13T22:04:46.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ab238ce8a96c648-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.482 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:04:46.482 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:04:46.483 DEBUG:boto:Method: GET 2024-04-13T22:04:46.483 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:04:46.483 DEBUG:boto:Data: 2024-04-13T22:04:46.483 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.483 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.483 DEBUG:boto:Params: {} 2024-04-13T22:04:46.483 DEBUG:boto:Token: None 2024-04-13T22:04:46.483 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj3 2024-04-13T22:04:46.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2/7xtnPwohoTcBuIBMPL+1vl8dY= 2024-04-13T22:04:46.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2/7xtnPwohoTcBuIBMPL+1vl8dY=', 'Content-Length': '0'} 2024-04-13T22:04:46.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f497c92e5fd6bff7-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.486 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.486 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.486 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.486 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.486 DEBUG:boto:Method: GET 2024-04-13T22:04:46.486 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.486 DEBUG:boto:Data: 2024-04-13T22:04:46.486 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.487 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.487 DEBUG:boto:Params: {} 2024-04-13T22:04:46.487 DEBUG:boto:Token: None 2024-04-13T22:04:46.487 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.487 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8859ecf36e7e789-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.489 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.489 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.490 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.490 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.490 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.490 DEBUG:boto:Method: GET 2024-04-13T22:04:46.490 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:04:46.490 DEBUG:boto:Data: 2024-04-13T22:04:46.490 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.490 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.490 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.490 DEBUG:boto:Params: {} 2024-04-13T22:04:46.491 DEBUG:boto:Token: None 2024-04-13T22:04:46.491 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002976db43ce8f99d6-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.493 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:46.493 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.493 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.494 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.494 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.494 DEBUG:boto:Method: GET 2024-04-13T22:04:46.494 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.494 DEBUG:boto:Data: 2024-04-13T22:04:46.494 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.494 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.494 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.494 DEBUG:boto:Params: {} 2024-04-13T22:04:46.494 DEBUG:boto:Token: None 2024-04-13T22:04:46.495 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI= 2024-04-13T22:04:46.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI=', 'Content-Length': '0'} 2024-04-13T22:04:46.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a15428bcb32ead0-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.496 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.497 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.497 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.497 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.497 DEBUG:boto:Method: GET 2024-04-13T22:04:46.497 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.497 DEBUG:boto:Data: 2024-04-13T22:04:46.497 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.497 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.497 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.497 DEBUG:boto:Params: {} 2024-04-13T22:04:46.498 DEBUG:boto:Token: None 2024-04-13T22:04:46.498 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.498 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI= 2024-04-13T22:04:46.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI=', 'Content-Length': '0'} 2024-04-13T22:04:46.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003153a6121029ed18-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.500 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.500 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.500 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.500 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.500 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.500 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.500 DEBUG:boto:Data: 2024-04-13T22:04:46.500 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.500 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.501 DEBUG:boto:Params: {} 2024-04-13T22:04:46.501 DEBUG:boto:Token: None 2024-04-13T22:04:46.501 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K8qG6BT5qtIzcIv1YzbZ9XH+vnY= 2024-04-13T22:04:46.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K8qG6BT5qtIzcIv1YzbZ9XH+vnY=', 'Content-Length': '0'} 2024-04-13T22:04:46.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc9a4e8a9acef5e4-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.503 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.503 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.503 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.503 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.503 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.503 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.503 DEBUG:boto:Data: 2024-04-13T22:04:46.503 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.503 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.504 DEBUG:boto:Params: {} 2024-04-13T22:04:46.504 DEBUG:boto:Token: None 2024-04-13T22:04:46.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K8qG6BT5qtIzcIv1YzbZ9XH+vnY= 2024-04-13T22:04:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K8qG6BT5qtIzcIv1YzbZ9XH+vnY=', 'Content-Length': '0'} 2024-04-13T22:04:46.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f52060766cceca0-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:46.506 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.506 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.506 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.506 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.506 DEBUG:boto:Method: GET 2024-04-13T22:04:46.507 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.507 DEBUG:boto:Data: 2024-04-13T22:04:46.507 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.507 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.507 DEBUG:boto:Params: {} 2024-04-13T22:04:46.507 DEBUG:boto:Token: None 2024-04-13T22:04:46.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI= 2024-04-13T22:04:46.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI=', 'Content-Length': '0'} 2024-04-13T22:04:46.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbf301b6ee2295ff-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.509 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.509 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.510 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.510 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.510 DEBUG:boto:Method: GET 2024-04-13T22:04:46.510 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.510 DEBUG:boto:Data: 2024-04-13T22:04:46.510 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.510 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.510 DEBUG:boto:Params: {} 2024-04-13T22:04:46.510 DEBUG:boto:Token: None 2024-04-13T22:04:46.511 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:04:46.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI= 2024-04-13T22:04:46.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eTvCu9U5D5R+WXSpKYJy9RJytiI=', 'Content-Length': '0'} 2024-04-13T22:04:46.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3cc449b802a2d10-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.512 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.512 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.513 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.513 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.513 DEBUG:boto:Method: GET 2024-04-13T22:04:46.513 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.513 DEBUG:boto:Data: 2024-04-13T22:04:46.513 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.513 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.513 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.513 DEBUG:boto:Params: {} 2024-04-13T22:04:46.513 DEBUG:boto:Token: None 2024-04-13T22:04:46.513 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.514 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420ee4173706fe00-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.516 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.516 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.517 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.517 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.517 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.517 DEBUG:boto:Method: GET 2024-04-13T22:04:46.517 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.517 DEBUG:boto:Data: 2024-04-13T22:04:46.517 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.517 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.517 DEBUG:boto:Params: {} 2024-04-13T22:04:46.517 DEBUG:boto:Token: None 2024-04-13T22:04:46.518 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e6199a6b9d74016-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.520 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.520 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.520 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.521 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.521 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:04:46.521 DEBUG:boto:Data: 2024-04-13T22:04:46.521 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.521 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.521 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.521 DEBUG:boto:Params: {} 2024-04-13T22:04:46.521 DEBUG:boto:Token: None 2024-04-13T22:04:46.521 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4 2024-04-13T22:04:46.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yoQvdWwYm8IulqAcWTJoehRgFlE= 2024-04-13T22:04:46.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yoQvdWwYm8IulqAcWTJoehRgFlE=', 'Content-Length': '0'} 2024-04-13T22:04:46.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006955d950e106f8b4-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.523 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.523 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.523 DEBUG:boto:Method: HEAD 2024-04-13T22:04:46.524 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:04:46.524 DEBUG:boto:Data: 2024-04-13T22:04:46.524 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.524 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.524 DEBUG:boto:Params: {} 2024-04-13T22:04:46.524 DEBUG:boto:Token: None 2024-04-13T22:04:46.524 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4 2024-04-13T22:04:46.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yoQvdWwYm8IulqAcWTJoehRgFlE= 2024-04-13T22:04:46.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yoQvdWwYm8IulqAcWTJoehRgFlE=', 'Content-Length': '0'} 2024-04-13T22:04:46.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004659ad0c397a7839-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:04:46.526 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.527 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.527 DEBUG:boto:Method: GET 2024-04-13T22:04:46.527 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:04:46.527 DEBUG:boto:Data: 2024-04-13T22:04:46.527 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.527 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.527 DEBUG:boto:Params: {} 2024-04-13T22:04:46.527 DEBUG:boto:Token: None 2024-04-13T22:04:46.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4 2024-04-13T22:04:46.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NQdQniSZ/DNNkS9NBfOO70zCxAc= 2024-04-13T22:04:46.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NQdQniSZ/DNNkS9NBfOO70zCxAc=', 'Content-Length': '0'} 2024-04-13T22:04:46.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee3e2f2cb05cad4-00661b017e-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.574 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:04:46.574 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:04:46.575 DEBUG:boto:Method: GET 2024-04-13T22:04:46.575 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:04:46.575 DEBUG:boto:Data: 2024-04-13T22:04:46.575 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.575 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.575 DEBUG:boto:Params: {} 2024-04-13T22:04:46.575 DEBUG:boto:Token: None 2024-04-13T22:04:46.576 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/obj4 2024-04-13T22:04:46.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NQdQniSZ/DNNkS9NBfOO70zCxAc= 2024-04-13T22:04:46.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NQdQniSZ/DNNkS9NBfOO70zCxAc=', 'Content-Length': '0'} 2024-04-13T22:04:46.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000360785af77962df2-00661b017e-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.578 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.578 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.578 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.578 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.578 DEBUG:boto:Method: GET 2024-04-13T22:04:46.578 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.578 DEBUG:boto:Data: 2024-04-13T22:04:46.578 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.578 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:04:46.578 DEBUG:boto:Port: 8001 2024-04-13T22:04:46.579 DEBUG:boto:Params: {} 2024-04-13T22:04:46.579 DEBUG:boto:Token: None 2024-04-13T22:04:46.579 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.579 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a221c78626dff23a-00661b017e-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.581 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.581 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:04:46.581 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:04:46.582 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.582 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.582 DEBUG:boto:Method: GET 2024-04-13T22:04:46.582 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:04:46.582 DEBUG:boto:Data: 2024-04-13T22:04:46.582 DEBUG:boto:Headers: {} 2024-04-13T22:04:46.582 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:04:46.582 DEBUG:boto:Port: 8000 2024-04-13T22:04:46.582 DEBUG:boto:Params: {} 2024-04-13T22:04:46.582 DEBUG:boto:Token: None 2024-04-13T22:04:46.583 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:04:46 GMT /vwnfns-51/ 2024-04-13T22:04:46.583 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I= 2024-04-13T22:04:46.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:04:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1BHuyhYd/Math7mRR9keQQgrE2I=', 'Content-Length': '0'} 2024-04-13T22:04:46.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d326559ba6eb99d-00661b017e-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:04:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:04:46.585 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:04:46.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-51 zones={a2, a1} 2024-04-13T22:04:46.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vwnfns-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:04:46.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vwnfns-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:04:46.586 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vwnfns-50 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:04:46.635 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:46.635 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:46.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vwnfns-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:04:46.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vwnfns-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:04:46.700 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vwnfns-51 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:04:46.748 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:46.748 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:46.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:46.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:46.813 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:46.860 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:46.860 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:46.889 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045886.800232_649.1", 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:04:46.800232Z" 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.902 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:46.903 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:04:46.904 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:46.905 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:04:46.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:46.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:46.905 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:46.953 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:46.953 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:46.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:46.983+0000 7f779d3c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:46.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:46.984+0000 7f779d3c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:46.996 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045864.694964_643.1", 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:24.694964Z", 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:46.997 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:04:46.998 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:46.999 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:47.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:46Z" 2024-04-13T22:04:47.001 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:47.002 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045864.694964_643.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:04:24.694964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:04:46Z" } 2024-04-13T22:04:47.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045864.694964_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:04:24.694964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:04:47.003 DEBUG:rgw_multi.tests:log_status={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:47.003 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:47.003 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045886.800232_649.1 target=1_1713045864.694964_643.1 2024-04-13T22:04:52.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:52.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:52.005 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:52.060 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:52.061 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:52.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:52.093+0000 7fd33e838780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:52.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:52.094+0000 7fd33e838780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:52.107 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:52.107 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:52.108 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045864.694964_643.1", 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:24.694964Z", 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:52.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:52.110 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:52.111 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:52.112 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:52.113 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:04:52.114 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:04:52.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:52.114 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:52Z" 2024-04-13T22:04:52.114 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:52.115 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045864.694964_643.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:04:24.694964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:04:52Z" } 2024-04-13T22:04:52.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045864.694964_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:04:24.694964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:04:52.116 DEBUG:rgw_multi.tests:log_status={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:52.116 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:52.116 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045886.800232_649.1 target=1_1713045864.694964_643.1 2024-04-13T22:04:57.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:57.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:04:57.118 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:04:57.173 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:04:57.173 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:04:57.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:57.205+0000 7f8355491780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:57.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:04:57.206+0000 7f8355491780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:04:57.220 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:04:57.220 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:57.221 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045864.694964_643.1", 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:24.694964Z", 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:04:57.222 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:57.223 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:57.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:04:57.225 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:04:57Z" 2024-04-13T22:04:57.226 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:04:57.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045864.694964_643.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:04:24.694964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:04:57Z" } 2024-04-13T22:04:57.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045864.694964_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:04:24.694964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:04:57.228 DEBUG:rgw_multi.tests:log_status={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:57.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:04:57.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045886.800232_649.1 target=1_1713045864.694964_643.1 2024-04-13T22:05:02.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:02.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:02.229 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:02.286 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:02.286 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:02.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:02.319+0000 7fb7c2076780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:05:02.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:02.320+0000 7fb7c2076780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:05:02.335 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:05:02.335 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:05:02.335 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:05:02.335 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045864.694964_643.1", 2024-04-13T22:05:02.336 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:24.694964Z", 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:02.337 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:05:02.338 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:02.339 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:02.340 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:05:02Z" 2024-04-13T22:05:02.341 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:05:02.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045864.694964_643.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:04:24.694964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:05:02Z" } 2024-04-13T22:05:02.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045864.694964_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:04:24.694964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:05:02.343 DEBUG:rgw_multi.tests:log_status={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:02.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045864.694964_643.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:02.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045886.800232_649.1 target=1_1713045864.694964_643.1 2024-04-13T22:05:07.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:07.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:07.345 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:07.401 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:07.402 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:07.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:07.435+0000 7f0e9517d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:05:07.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:07.436+0000 7f0e9517d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:07.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045886.800232_649.1", 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:04:46.800232Z", 2024-04-13T22:05:07.452 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:07.453 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:07.454 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:07.455 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:07.456 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:05:07.457 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:05:07.457 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:05:07.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:07.457 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:05:07Z" 2024-04-13T22:05:07.457 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:05:07.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045886.800232_649.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:04:46.800232Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:05:07Z" } 2024-04-13T22:05:07.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045886.800232_649.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:04:46.800232Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:05:07.458 DEBUG:rgw_multi.tests:log_status={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:07.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045886.800232_649.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:07.458 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:05:07.459 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:07.459 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:07.459 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.459 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:05:07.459 DEBUG:boto:Data: 2024-04-13T22:05:07.459 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.459 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.460 DEBUG:boto:Params: {} 2024-04-13T22:05:07.460 DEBUG:boto:Token: None 2024-04-13T22:05:07.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/ 2024-04-13T22:05:07.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g= 2024-04-13T22:05:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g=', 'Content-Length': '0'} 2024-04-13T22:05:07.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ca0491f0bc1c809-00661b0193-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.463 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:07.463 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:07.463 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.463 DEBUG:boto:Path: /vwnfns-50/obj5 2024-04-13T22:05:07.463 DEBUG:boto:Data: 2024-04-13T22:05:07.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.464 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.464 DEBUG:boto:Params: {} 2024-04-13T22:05:07.464 DEBUG:boto:Token: None 2024-04-13T22:05:07.464 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/obj5 2024-04-13T22:05:07.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z+YhiEGSPMcOvlkVwAcZrgQK7jE= 2024-04-13T22:05:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z+YhiEGSPMcOvlkVwAcZrgQK7jE='} 2024-04-13T22:05:07.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f44c2a7017bf6118-00661b0193-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.507 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:07.507 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:07.507 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.507 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:05:07.507 DEBUG:boto:Data: 2024-04-13T22:05:07.507 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.507 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.507 DEBUG:boto:Params: {} 2024-04-13T22:05:07.507 DEBUG:boto:Token: None 2024-04-13T22:05:07.507 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/ 2024-04-13T22:05:07.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g= 2024-04-13T22:05:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g=', 'Content-Length': '0'} 2024-04-13T22:05:07.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072acc1908cfea323-00661b0193-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.510 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:07.510 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:07.510 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.510 DEBUG:boto:Path: /vwnfns-50/obj6 2024-04-13T22:05:07.510 DEBUG:boto:Data: 2024-04-13T22:05:07.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.511 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.511 DEBUG:boto:Params: {} 2024-04-13T22:05:07.511 DEBUG:boto:Token: None 2024-04-13T22:05:07.511 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/obj6 2024-04-13T22:05:07.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LY0kTeRotuoQP4jnaVf6Zg4KD0I= 2024-04-13T22:05:07.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LY0kTeRotuoQP4jnaVf6Zg4KD0I='} 2024-04-13T22:05:07.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038f7898422ed4b88-00661b0193-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.519 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:07.519 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:07.519 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.519 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:05:07.519 DEBUG:boto:Data: 2024-04-13T22:05:07.519 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.519 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.520 DEBUG:boto:Params: {} 2024-04-13T22:05:07.520 DEBUG:boto:Token: None 2024-04-13T22:05:07.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/ 2024-04-13T22:05:07.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g= 2024-04-13T22:05:07.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g=', 'Content-Length': '0'} 2024-04-13T22:05:07.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc28dedacdde5117-00661b0193-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.523 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:07.523 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:07.523 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.523 DEBUG:boto:Path: /vwnfns-50/obj7 2024-04-13T22:05:07.523 DEBUG:boto:Data: 2024-04-13T22:05:07.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.524 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.524 DEBUG:boto:Params: {} 2024-04-13T22:05:07.524 DEBUG:boto:Token: None 2024-04-13T22:05:07.524 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/obj7 2024-04-13T22:05:07.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jXbFi3mSBFLw4yA84iyjRTpbBnQ= 2024-04-13T22:05:07.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jXbFi3mSBFLw4yA84iyjRTpbBnQ='} 2024-04-13T22:05:07.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009123ab1e34e23116-00661b0193-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.571 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:07.571 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:07.571 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.571 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:05:07.571 DEBUG:boto:Data: 2024-04-13T22:05:07.571 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.571 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.571 DEBUG:boto:Params: {} 2024-04-13T22:05:07.571 DEBUG:boto:Token: None 2024-04-13T22:05:07.572 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/ 2024-04-13T22:05:07.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g= 2024-04-13T22:05:07.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pJpwv9bBDsfgqiuuUtW2/LAi28g=', 'Content-Length': '0'} 2024-04-13T22:05:07.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0f3826fe2261922-00661b0193-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.574 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:07.574 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:07.574 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.574 DEBUG:boto:Path: /vwnfns-50/obj8 2024-04-13T22:05:07.574 DEBUG:boto:Data: 2024-04-13T22:05:07.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:07.575 DEBUG:boto:Port: 8000 2024-04-13T22:05:07.575 DEBUG:boto:Params: {} 2024-04-13T22:05:07.575 DEBUG:boto:Token: None 2024-04-13T22:05:07.575 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-50/obj8 2024-04-13T22:05:07.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a9/znAbey0H2bCwoeF6EEdWTA24= 2024-04-13T22:05:07.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a9/znAbey0H2bCwoeF6EEdWTA24='} 2024-04-13T22:05:07.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f88a245c888da94f-00661b0193-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.581 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:07.581 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:07.581 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.581 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:05:07.581 DEBUG:boto:Data: 2024-04-13T22:05:07.581 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.581 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.581 DEBUG:boto:Params: {} 2024-04-13T22:05:07.582 DEBUG:boto:Token: None 2024-04-13T22:05:07.582 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/ 2024-04-13T22:05:07.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA= 2024-04-13T22:05:07.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA=', 'Content-Length': '0'} 2024-04-13T22:05:07.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1d2460880a15b8b-00661b0193-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.584 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:07.585 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:07.585 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.585 DEBUG:boto:Path: /vwnfns-51/obj5 2024-04-13T22:05:07.585 DEBUG:boto:Data: 2024-04-13T22:05:07.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.585 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.585 DEBUG:boto:Params: {} 2024-04-13T22:05:07.585 DEBUG:boto:Token: None 2024-04-13T22:05:07.586 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/obj5 2024-04-13T22:05:07.586 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:giW5aqCNzsB1pFAkA4jyiIdZho0= 2024-04-13T22:05:07.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:giW5aqCNzsB1pFAkA4jyiIdZho0='} 2024-04-13T22:05:07.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cfc272568ece750-00661b0193-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.590 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:07.590 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:07.590 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.590 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:05:07.590 DEBUG:boto:Data: 2024-04-13T22:05:07.590 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.590 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.590 DEBUG:boto:Params: {} 2024-04-13T22:05:07.591 DEBUG:boto:Token: None 2024-04-13T22:05:07.591 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/ 2024-04-13T22:05:07.591 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA= 2024-04-13T22:05:07.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA=', 'Content-Length': '0'} 2024-04-13T22:05:07.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e12e607052c4e97c-00661b0193-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.593 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:07.593 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:07.593 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.593 DEBUG:boto:Path: /vwnfns-51/obj6 2024-04-13T22:05:07.593 DEBUG:boto:Data: 2024-04-13T22:05:07.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.594 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.594 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.594 DEBUG:boto:Params: {} 2024-04-13T22:05:07.594 DEBUG:boto:Token: None 2024-04-13T22:05:07.594 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/obj6 2024-04-13T22:05:07.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B0I+O8TI0O7WNwq3S2UcD2xs5+Q= 2024-04-13T22:05:07.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B0I+O8TI0O7WNwq3S2UcD2xs5+Q='} 2024-04-13T22:05:07.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b41d1526f2613bb-00661b0193-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.598 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:07.598 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:07.598 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.599 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:05:07.599 DEBUG:boto:Data: 2024-04-13T22:05:07.599 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.599 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.599 DEBUG:boto:Params: {} 2024-04-13T22:05:07.599 DEBUG:boto:Token: None 2024-04-13T22:05:07.599 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/ 2024-04-13T22:05:07.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA= 2024-04-13T22:05:07.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA=', 'Content-Length': '0'} 2024-04-13T22:05:07.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e296ffd2c49264e7-00661b0193-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.602 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:07.602 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:07.602 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.602 DEBUG:boto:Path: /vwnfns-51/obj7 2024-04-13T22:05:07.602 DEBUG:boto:Data: 2024-04-13T22:05:07.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.602 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.602 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.602 DEBUG:boto:Params: {} 2024-04-13T22:05:07.603 DEBUG:boto:Token: None 2024-04-13T22:05:07.603 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/obj7 2024-04-13T22:05:07.603 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UH9q6tJuTMvStIK7BXxyGmV9Lq8= 2024-04-13T22:05:07.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UH9q6tJuTMvStIK7BXxyGmV9Lq8='} 2024-04-13T22:05:07.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4b81efe95188b56-00661b0193-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.607 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:07.607 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:07.607 DEBUG:boto:Method: HEAD 2024-04-13T22:05:07.607 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:05:07.607 DEBUG:boto:Data: 2024-04-13T22:05:07.607 DEBUG:boto:Headers: {} 2024-04-13T22:05:07.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.608 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.608 DEBUG:boto:Params: {} 2024-04-13T22:05:07.608 DEBUG:boto:Token: None 2024-04-13T22:05:07.608 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/ 2024-04-13T22:05:07.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA= 2024-04-13T22:05:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8jM5mYnGpxCL+CWN+BP+InTWQqA=', 'Content-Length': '0'} 2024-04-13T22:05:07.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a3d0b5eb277f78f-00661b0193-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.610 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:07.610 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:07.610 DEBUG:boto:Method: PUT 2024-04-13T22:05:07.611 DEBUG:boto:Path: /vwnfns-51/obj8 2024-04-13T22:05:07.611 DEBUG:boto:Data: 2024-04-13T22:05:07.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:05:07.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:07.611 DEBUG:boto:Port: 8001 2024-04-13T22:05:07.611 DEBUG:boto:Params: {} 2024-04-13T22:05:07.611 DEBUG:boto:Token: None 2024-04-13T22:05:07.611 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:05:07 GMT /vwnfns-51/obj8 2024-04-13T22:05:07.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6rD7+jwUaV9bHhI3chbPNMWb/tQ= 2024-04-13T22:05:07.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6rD7+jwUaV9bHhI3chbPNMWb/tQ='} 2024-04-13T22:05:07.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042eedef115230fea-00661b0193-1a1e-a2'), ('Date', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:07.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vwnfns-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:05:07.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vwnfns-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:05:07.616 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vwnfns-50 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:05:07.664 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:07.664 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:07.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vwnfns-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:05:07.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vwnfns-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:05:07.733 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vwnfns-51 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:05:07.782 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:07.782 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:07.848 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-50 2024-04-13T22:05:07.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:07.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:07.848 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:05:07.902 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:07.902 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:07.936+0000 7f1fd4d79780 1 waiting to reach incremental sync.. 2024-04-13T22:05:12.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:12.937+0000 7f1fd4d79780 1 bucket sync caught up with source: 2024-04-13T22:05:12.939 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.49.1, 00000000002.115.1, 00000000004.45.1, 00000000002.35.1, 00000000004.59.1, 00000000002.44.1, 00000000002.44.1, 00000000004.25.1, 00000000002.31.1, 00000000002.22.1, 00000000002.42.1] 2024-04-13T22:05:12.939 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.49.1, 00000000002.115.1, 00000000004.45.1, 00000000002.35.1, 00000000004.59.1, 00000000002.44.1, 00000000002.44.1, 00000000004.25.1, 00000000002.31.1, 00000000002.22.1, 00000000002.42.1] 2024-04-13T22:05:12.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:12.937+0000 7f1fd4d79780 0 bucket checkpoint complete 2024-04-13T22:05:12.957 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-50 2024-04-13T22:05:12.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:12.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:12.959 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:05:13.020 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:13.021 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:13.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:13.056+0000 7f407cf26780 1 waiting to reach incremental sync.. 2024-04-13T22:05:18.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:18.058+0000 7f407cf26780 1 waiting to reach incremental sync.. 2024-04-13T22:05:23.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:23.058+0000 7f407cf26780 1 waiting to reach incremental sync.. 2024-04-13T22:05:28.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:28.060+0000 7f407cf26780 1 bucket sync caught up with source: 2024-04-13T22:05:28.060 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.993.1, 00000000002.736.1, 00000000004.1750.1, 00000000002.1741.1, 00000000004.2808.1, 00000000002.972.1, 00000000002.1394.1, 00000000004.440.1, 00000000002.387.1, 00000000002.2014.1, 00000000002.1105.1] 2024-04-13T22:05:28.060 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.988.1, 00000000001.735.1, 00000000002.1741.1, 00000000001.1740.1, 00000000002.2801.1, 00000000001.969.1, 00000000001.1393.1, 00000000002.435.1, 00000000001.386.1, 00000000001.2005.1, 00000000001.1104.1] 2024-04-13T22:05:28.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:28.060+0000 7f407cf26780 0 bucket checkpoint complete 2024-04-13T22:05:28.074 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-50 zones={a2, a1} 2024-04-13T22:05:28.074 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.074 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.074 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.074 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:05:28.075 DEBUG:boto:Data: 2024-04-13T22:05:28.075 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.075 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.075 DEBUG:boto:Params: {} 2024-04-13T22:05:28.075 DEBUG:boto:Token: None 2024-04-13T22:05:28.075 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zHcr79cGekqq8WM0cmEtLYjxXJs= 2024-04-13T22:05:28.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zHcr79cGekqq8WM0cmEtLYjxXJs=', 'Content-Length': '0'} 2024-04-13T22:05:28.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd5bc623b2ee81d4-00661b01a8-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.078 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.078 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.078 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.079 DEBUG:boto:Path: /vwnfns-50/ 2024-04-13T22:05:28.079 DEBUG:boto:Data: 2024-04-13T22:05:28.079 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.079 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.079 DEBUG:boto:Params: {} 2024-04-13T22:05:28.079 DEBUG:boto:Token: None 2024-04-13T22:05:28.079 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zHcr79cGekqq8WM0cmEtLYjxXJs= 2024-04-13T22:05:28.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zHcr79cGekqq8WM0cmEtLYjxXJs=', 'Content-Length': '0'} 2024-04-13T22:05:28.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002d746c7fb731260-00661b01a8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.082 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:05:28.082 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.082 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.082 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.082 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.082 DEBUG:boto:Method: GET 2024-04-13T22:05:28.082 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.082 DEBUG:boto:Data: 2024-04-13T22:05:28.083 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.083 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.083 DEBUG:boto:Params: {} 2024-04-13T22:05:28.083 DEBUG:boto:Token: None 2024-04-13T22:05:28.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/?versions 2024-04-13T22:05:28.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ= 2024-04-13T22:05:28.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ=', 'Content-Length': '0'} 2024-04-13T22:05:28.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084056c34eff2b948-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.086 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:05:28.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:05:28.088 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.088 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.088 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.088 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.088 DEBUG:boto:Method: GET 2024-04-13T22:05:28.088 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.088 DEBUG:boto:Data: 2024-04-13T22:05:28.088 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.088 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.088 DEBUG:boto:Params: {} 2024-04-13T22:05:28.089 DEBUG:boto:Token: None 2024-04-13T22:05:28.089 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/?versions 2024-04-13T22:05:28.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ= 2024-04-13T22:05:28.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ=', 'Content-Length': '0'} 2024-04-13T22:05:28.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89662d0100c6ec6-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.091 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:05:28.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:05:28.093 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.093 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.093 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.093 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.093 DEBUG:boto:Method: GET 2024-04-13T22:05:28.093 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.093 DEBUG:boto:Data: 2024-04-13T22:05:28.093 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.093 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.093 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.093 DEBUG:boto:Params: {} 2024-04-13T22:05:28.094 DEBUG:boto:Token: None 2024-04-13T22:05:28.094 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/?versions 2024-04-13T22:05:28.094 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ= 2024-04-13T22:05:28.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ=', 'Content-Length': '0'} 2024-04-13T22:05:28.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f58922bc29aa17-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.097 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.097 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.097 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.097 DEBUG:boto:path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.097 DEBUG:boto:auth_path=/vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.098 DEBUG:boto:Method: GET 2024-04-13T22:05:28.098 DEBUG:boto:Path: /vwnfns-50/?versions&max-keys=999 2024-04-13T22:05:28.098 DEBUG:boto:Data: 2024-04-13T22:05:28.098 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.098 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.098 DEBUG:boto:Params: {} 2024-04-13T22:05:28.098 DEBUG:boto:Token: None 2024-04-13T22:05:28.098 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/?versions 2024-04-13T22:05:28.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ= 2024-04-13T22:05:28.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/VxD0ahLKnOj6O0tQambr9F1QSQ=', 'Content-Length': '0'} 2024-04-13T22:05:28.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5dc545450db6c40-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.101 DEBUG:boto:b'vwnfns-50999falseobj1nulltrue2024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:05:28.102 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.102 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.102 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.102 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.102 DEBUG:boto:Method: GET 2024-04-13T22:05:28.102 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.102 DEBUG:boto:Data: 2024-04-13T22:05:28.102 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.103 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.103 DEBUG:boto:Params: {} 2024-04-13T22:05:28.103 DEBUG:boto:Token: None 2024-04-13T22:05:28.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY= 2024-04-13T22:05:28.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY=', 'Content-Length': '0'} 2024-04-13T22:05:28.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd7b1b40048b142b-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.106 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.106 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.106 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.106 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.106 DEBUG:boto:Method: GET 2024-04-13T22:05:28.106 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.106 DEBUG:boto:Data: 2024-04-13T22:05:28.106 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.106 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.107 DEBUG:boto:Params: {} 2024-04-13T22:05:28.107 DEBUG:boto:Token: None 2024-04-13T22:05:28.107 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY= 2024-04-13T22:05:28.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY=', 'Content-Length': '0'} 2024-04-13T22:05:28.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9be34bf65fbf928-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.109 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.109 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.109 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.109 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.109 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.109 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.110 DEBUG:boto:Data: 2024-04-13T22:05:28.110 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.110 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.110 DEBUG:boto:Params: {} 2024-04-13T22:05:28.110 DEBUG:boto:Token: None 2024-04-13T22:05:28.110 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q/hI4y9WZw72gFaKAh7Z/tDiPhg= 2024-04-13T22:05:28.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q/hI4y9WZw72gFaKAh7Z/tDiPhg=', 'Content-Length': '0'} 2024-04-13T22:05:28.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000017f40370ca4f815-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.112 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.112 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.112 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.112 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.112 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.113 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.113 DEBUG:boto:Data: 2024-04-13T22:05:28.113 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.113 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.113 DEBUG:boto:Params: {} 2024-04-13T22:05:28.113 DEBUG:boto:Token: None 2024-04-13T22:05:28.113 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q/hI4y9WZw72gFaKAh7Z/tDiPhg= 2024-04-13T22:05:28.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q/hI4y9WZw72gFaKAh7Z/tDiPhg=', 'Content-Length': '0'} 2024-04-13T22:05:28.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cd113523153e117-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:05:28.115 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.115 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.115 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.115 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.116 DEBUG:boto:Method: GET 2024-04-13T22:05:28.116 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.116 DEBUG:boto:Data: 2024-04-13T22:05:28.116 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.116 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.116 DEBUG:boto:Params: {} 2024-04-13T22:05:28.116 DEBUG:boto:Token: None 2024-04-13T22:05:28.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY= 2024-04-13T22:05:28.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY=', 'Content-Length': '0'} 2024-04-13T22:05:28.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082b8a7f44754ba28-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.118 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.118 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.119 DEBUG:boto:path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.119 DEBUG:boto:auth_path=/vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.119 DEBUG:boto:Method: GET 2024-04-13T22:05:28.119 DEBUG:boto:Path: /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.119 DEBUG:boto:Data: 2024-04-13T22:05:28.119 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.119 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.119 DEBUG:boto:Params: {} 2024-04-13T22:05:28.119 DEBUG:boto:Token: None 2024-04-13T22:05:28.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1?versionId=null 2024-04-13T22:05:28.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY= 2024-04-13T22:05:28.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jdc+nUKd9qUxby1fLZ2rxXctvdY=', 'Content-Length': '0'} 2024-04-13T22:05:28.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062886c6b4aebc53d-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.166 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.167 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.167 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.167 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.167 DEBUG:boto:Method: GET 2024-04-13T22:05:28.167 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.167 DEBUG:boto:Data: 2024-04-13T22:05:28.167 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.167 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.168 DEBUG:boto:Params: {} 2024-04-13T22:05:28.168 DEBUG:boto:Token: None 2024-04-13T22:05:28.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a50ff69ae1d10ea8-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.170 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.171 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.171 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.171 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.171 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.171 DEBUG:boto:Method: GET 2024-04-13T22:05:28.171 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.171 DEBUG:boto:Data: 2024-04-13T22:05:28.172 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.172 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.172 DEBUG:boto:Params: {} 2024-04-13T22:05:28.172 DEBUG:boto:Token: None 2024-04-13T22:05:28.172 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bc098ba13d600b7-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.174 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.175 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.175 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.175 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.175 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:05:28.175 DEBUG:boto:Data: 2024-04-13T22:05:28.175 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.175 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.176 DEBUG:boto:Params: {} 2024-04-13T22:05:28.176 DEBUG:boto:Token: None 2024-04-13T22:05:28.176 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1 2024-04-13T22:05:28.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S1rflH7Bh+v5zgLpQq4kd7hnArs= 2024-04-13T22:05:28.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S1rflH7Bh+v5zgLpQq4kd7hnArs=', 'Content-Length': '0'} 2024-04-13T22:05:28.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c4230a00a05537a-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.178 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.178 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.178 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.178 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:05:28.178 DEBUG:boto:Data: 2024-04-13T22:05:28.179 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.179 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.179 DEBUG:boto:Params: {} 2024-04-13T22:05:28.179 DEBUG:boto:Token: None 2024-04-13T22:05:28.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1 2024-04-13T22:05:28.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S1rflH7Bh+v5zgLpQq4kd7hnArs= 2024-04-13T22:05:28.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S1rflH7Bh+v5zgLpQq4kd7hnArs=', 'Content-Length': '0'} 2024-04-13T22:05:28.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd1b37ddb777e166-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:05:28.181 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.181 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.181 DEBUG:boto:Method: GET 2024-04-13T22:05:28.181 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:05:28.182 DEBUG:boto:Data: 2024-04-13T22:05:28.182 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.182 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.182 DEBUG:boto:Params: {} 2024-04-13T22:05:28.182 DEBUG:boto:Token: None 2024-04-13T22:05:28.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1 2024-04-13T22:05:28.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tHAuIwmgjONgkBz9XNUW+QZQp4k= 2024-04-13T22:05:28.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tHAuIwmgjONgkBz9XNUW+QZQp4k=', 'Content-Length': '0'} 2024-04-13T22:05:28.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000775e02291c495d88-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.230 DEBUG:boto:path=/vwnfns-50/obj1 2024-04-13T22:05:28.231 DEBUG:boto:auth_path=/vwnfns-50/obj1 2024-04-13T22:05:28.231 DEBUG:boto:Method: GET 2024-04-13T22:05:28.231 DEBUG:boto:Path: /vwnfns-50/obj1 2024-04-13T22:05:28.231 DEBUG:boto:Data: 2024-04-13T22:05:28.231 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.231 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.232 DEBUG:boto:Params: {} 2024-04-13T22:05:28.232 DEBUG:boto:Token: None 2024-04-13T22:05:28.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj1 2024-04-13T22:05:28.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tHAuIwmgjONgkBz9XNUW+QZQp4k= 2024-04-13T22:05:28.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tHAuIwmgjONgkBz9XNUW+QZQp4k=', 'Content-Length': '0'} 2024-04-13T22:05:28.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b3938cda69d39b7-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.234 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.234 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.235 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.235 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.235 DEBUG:boto:Method: GET 2024-04-13T22:05:28.235 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.235 DEBUG:boto:Data: 2024-04-13T22:05:28.235 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.235 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.235 DEBUG:boto:Params: {} 2024-04-13T22:05:28.235 DEBUG:boto:Token: None 2024-04-13T22:05:28.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da46470f0eb9e30-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.238 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.239 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.239 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.239 DEBUG:boto:path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.239 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.239 DEBUG:boto:Method: GET 2024-04-13T22:05:28.239 DEBUG:boto:Path: /vwnfns-50/?prefix=obj1 2024-04-13T22:05:28.239 DEBUG:boto:Data: 2024-04-13T22:05:28.239 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.240 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.240 DEBUG:boto:Params: {} 2024-04-13T22:05:28.240 DEBUG:boto:Token: None 2024-04-13T22:05:28.240 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e24f5904063237-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.242 DEBUG:boto:b'vwnfns-50obj11000falseobj12024-04-13T22:04:24.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:05:28.243 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.243 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.243 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.243 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.243 DEBUG:boto:Method: GET 2024-04-13T22:05:28.243 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.244 DEBUG:boto:Data: 2024-04-13T22:05:28.244 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.244 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.244 DEBUG:boto:Params: {} 2024-04-13T22:05:28.244 DEBUG:boto:Token: None 2024-04-13T22:05:28.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs= 2024-04-13T22:05:28.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs=', 'Content-Length': '0'} 2024-04-13T22:05:28.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9635967cd7a895a-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.246 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.246 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.246 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.246 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.247 DEBUG:boto:Method: GET 2024-04-13T22:05:28.247 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.247 DEBUG:boto:Data: 2024-04-13T22:05:28.247 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.247 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.247 DEBUG:boto:Params: {} 2024-04-13T22:05:28.247 DEBUG:boto:Token: None 2024-04-13T22:05:28.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs= 2024-04-13T22:05:28.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs=', 'Content-Length': '0'} 2024-04-13T22:05:28.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000764ba131655aae64-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.290 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.291 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.291 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.291 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.291 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.291 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.291 DEBUG:boto:Data: 2024-04-13T22:05:28.291 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.291 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.292 DEBUG:boto:Params: {} 2024-04-13T22:05:28.292 DEBUG:boto:Token: None 2024-04-13T22:05:28.292 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xQQXutaDTWuAsIHeSvBeyBJQi9A= 2024-04-13T22:05:28.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xQQXutaDTWuAsIHeSvBeyBJQi9A=', 'Content-Length': '0'} 2024-04-13T22:05:28.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1ddb0ee3ba2a89c-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.294 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.294 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.294 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.294 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.294 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.295 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.295 DEBUG:boto:Data: 2024-04-13T22:05:28.295 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.295 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.295 DEBUG:boto:Params: {} 2024-04-13T22:05:28.295 DEBUG:boto:Token: None 2024-04-13T22:05:28.295 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xQQXutaDTWuAsIHeSvBeyBJQi9A= 2024-04-13T22:05:28.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xQQXutaDTWuAsIHeSvBeyBJQi9A=', 'Content-Length': '0'} 2024-04-13T22:05:28.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084b7d5ec6f9e2f8a-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:05:28.297 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.297 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.298 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.298 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.298 DEBUG:boto:Method: GET 2024-04-13T22:05:28.298 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.298 DEBUG:boto:Data: 2024-04-13T22:05:28.298 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.298 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.298 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.298 DEBUG:boto:Params: {} 2024-04-13T22:05:28.299 DEBUG:boto:Token: None 2024-04-13T22:05:28.299 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.299 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs= 2024-04-13T22:05:28.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs=', 'Content-Length': '0'} 2024-04-13T22:05:28.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b60c2cbeda8174b-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.301 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.301 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.301 DEBUG:boto:path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.301 DEBUG:boto:auth_path=/vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.301 DEBUG:boto:Method: GET 2024-04-13T22:05:28.301 DEBUG:boto:Path: /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.302 DEBUG:boto:Data: 2024-04-13T22:05:28.302 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.302 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.302 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.302 DEBUG:boto:Params: {} 2024-04-13T22:05:28.302 DEBUG:boto:Token: None 2024-04-13T22:05:28.302 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2?versionId=null 2024-04-13T22:05:28.302 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs= 2024-04-13T22:05:28.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ccodaCzgr9pW1JGB0KJj98aUhLs=', 'Content-Length': '0'} 2024-04-13T22:05:28.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000272e7b801ca6759-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.304 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.304 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.304 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.304 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.305 DEBUG:boto:Method: GET 2024-04-13T22:05:28.305 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.305 DEBUG:boto:Data: 2024-04-13T22:05:28.305 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.305 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.305 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.305 DEBUG:boto:Params: {} 2024-04-13T22:05:28.305 DEBUG:boto:Token: None 2024-04-13T22:05:28.305 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.306 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e855b9810868ff-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.308 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.308 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.308 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.308 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.308 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.309 DEBUG:boto:Method: GET 2024-04-13T22:05:28.309 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.309 DEBUG:boto:Data: 2024-04-13T22:05:28.309 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.309 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.309 DEBUG:boto:Params: {} 2024-04-13T22:05:28.309 DEBUG:boto:Token: None 2024-04-13T22:05:28.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.309 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000057b6943bc8a3f48-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.312 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.312 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.312 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.312 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.312 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:05:28.312 DEBUG:boto:Data: 2024-04-13T22:05:28.313 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.313 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.313 DEBUG:boto:Params: {} 2024-04-13T22:05:28.313 DEBUG:boto:Token: None 2024-04-13T22:05:28.313 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2 2024-04-13T22:05:28.313 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zD6uyGn3pxnx7A5xS1hlNv3cu8Q= 2024-04-13T22:05:28.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zD6uyGn3pxnx7A5xS1hlNv3cu8Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce7f82d070cfe612-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.315 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.315 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.315 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.315 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:05:28.315 DEBUG:boto:Data: 2024-04-13T22:05:28.315 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.316 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.316 DEBUG:boto:Params: {} 2024-04-13T22:05:28.316 DEBUG:boto:Token: None 2024-04-13T22:05:28.316 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2 2024-04-13T22:05:28.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zD6uyGn3pxnx7A5xS1hlNv3cu8Q= 2024-04-13T22:05:28.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zD6uyGn3pxnx7A5xS1hlNv3cu8Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000594c5df46428a85e-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:05:28.318 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.318 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.318 DEBUG:boto:Method: GET 2024-04-13T22:05:28.318 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:05:28.318 DEBUG:boto:Data: 2024-04-13T22:05:28.319 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.319 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.319 DEBUG:boto:Params: {} 2024-04-13T22:05:28.319 DEBUG:boto:Token: None 2024-04-13T22:05:28.319 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2 2024-04-13T22:05:28.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2xGtZGTmTaC18FmsVeJUAXg4D7A= 2024-04-13T22:05:28.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2xGtZGTmTaC18FmsVeJUAXg4D7A=', 'Content-Length': '0'} 2024-04-13T22:05:28.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d32822ac935bb471-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.321 DEBUG:boto:path=/vwnfns-50/obj2 2024-04-13T22:05:28.321 DEBUG:boto:auth_path=/vwnfns-50/obj2 2024-04-13T22:05:28.321 DEBUG:boto:Method: GET 2024-04-13T22:05:28.322 DEBUG:boto:Path: /vwnfns-50/obj2 2024-04-13T22:05:28.322 DEBUG:boto:Data: 2024-04-13T22:05:28.322 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.322 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.322 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.322 DEBUG:boto:Params: {} 2024-04-13T22:05:28.322 DEBUG:boto:Token: None 2024-04-13T22:05:28.322 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj2 2024-04-13T22:05:28.322 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2xGtZGTmTaC18FmsVeJUAXg4D7A= 2024-04-13T22:05:28.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2xGtZGTmTaC18FmsVeJUAXg4D7A=', 'Content-Length': '0'} 2024-04-13T22:05:28.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6f63c640417fd7a-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.324 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.324 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.325 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.325 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.325 DEBUG:boto:Method: GET 2024-04-13T22:05:28.325 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.325 DEBUG:boto:Data: 2024-04-13T22:05:28.325 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.325 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.325 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.325 DEBUG:boto:Params: {} 2024-04-13T22:05:28.325 DEBUG:boto:Token: None 2024-04-13T22:05:28.326 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35169351e61e127-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.328 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.329 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.329 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.329 DEBUG:boto:path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.329 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.329 DEBUG:boto:Method: GET 2024-04-13T22:05:28.329 DEBUG:boto:Path: /vwnfns-50/?prefix=obj2 2024-04-13T22:05:28.330 DEBUG:boto:Data: 2024-04-13T22:05:28.330 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.330 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.330 DEBUG:boto:Params: {} 2024-04-13T22:05:28.330 DEBUG:boto:Token: None 2024-04-13T22:05:28.330 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.330 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed2b2185a0cc518-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.332 DEBUG:boto:b'vwnfns-50obj21000falseobj22024-04-13T22:04:24.758Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:05:28.333 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.333 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.333 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.333 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.333 DEBUG:boto:Method: GET 2024-04-13T22:05:28.333 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.333 DEBUG:boto:Data: 2024-04-13T22:05:28.333 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.334 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.334 DEBUG:boto:Params: {} 2024-04-13T22:05:28.334 DEBUG:boto:Token: None 2024-04-13T22:05:28.334 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA= 2024-04-13T22:05:28.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA=', 'Content-Length': '0'} 2024-04-13T22:05:28.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000721a9d299a1f67c7-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.336 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.336 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.336 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.336 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.336 DEBUG:boto:Method: GET 2024-04-13T22:05:28.337 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.337 DEBUG:boto:Data: 2024-04-13T22:05:28.337 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.337 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.337 DEBUG:boto:Params: {} 2024-04-13T22:05:28.337 DEBUG:boto:Token: None 2024-04-13T22:05:28.337 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA= 2024-04-13T22:05:28.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA=', 'Content-Length': '0'} 2024-04-13T22:05:28.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d47e760e2313d60-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.339 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.339 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.339 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.340 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.340 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.340 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.340 DEBUG:boto:Data: 2024-04-13T22:05:28.340 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.340 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.340 DEBUG:boto:Params: {} 2024-04-13T22:05:28.340 DEBUG:boto:Token: None 2024-04-13T22:05:28.340 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.341 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SsVnloRwLfZTR+sA1OqslB/qsdI= 2024-04-13T22:05:28.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SsVnloRwLfZTR+sA1OqslB/qsdI=', 'Content-Length': '0'} 2024-04-13T22:05:28.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000527160dcd958720b-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.342 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.343 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.343 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.343 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.343 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.343 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.343 DEBUG:boto:Data: 2024-04-13T22:05:28.343 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.343 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.343 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.343 DEBUG:boto:Params: {} 2024-04-13T22:05:28.344 DEBUG:boto:Token: None 2024-04-13T22:05:28.344 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SsVnloRwLfZTR+sA1OqslB/qsdI= 2024-04-13T22:05:28.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SsVnloRwLfZTR+sA1OqslB/qsdI=', 'Content-Length': '0'} 2024-04-13T22:05:28.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000791b37cbef6e0e6-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:05:28.346 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.346 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.346 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.346 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.346 DEBUG:boto:Method: GET 2024-04-13T22:05:28.346 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.346 DEBUG:boto:Data: 2024-04-13T22:05:28.347 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.347 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.347 DEBUG:boto:Params: {} 2024-04-13T22:05:28.347 DEBUG:boto:Token: None 2024-04-13T22:05:28.347 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.347 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA= 2024-04-13T22:05:28.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA=', 'Content-Length': '0'} 2024-04-13T22:05:28.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0e2e0fa3ab7b8e4-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.390 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.393 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.394 DEBUG:boto:path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.394 DEBUG:boto:auth_path=/vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.394 DEBUG:boto:Method: GET 2024-04-13T22:05:28.394 DEBUG:boto:Path: /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.394 DEBUG:boto:Data: 2024-04-13T22:05:28.394 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.394 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.394 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.394 DEBUG:boto:Params: {} 2024-04-13T22:05:28.394 DEBUG:boto:Token: None 2024-04-13T22:05:28.395 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3?versionId=null 2024-04-13T22:05:28.395 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA= 2024-04-13T22:05:28.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pn6xoy99HlX/28YK03hogFESESA=', 'Content-Length': '0'} 2024-04-13T22:05:28.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f555af65c39229a-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.397 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.397 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.397 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.397 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.397 DEBUG:boto:Method: GET 2024-04-13T22:05:28.397 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.398 DEBUG:boto:Data: 2024-04-13T22:05:28.398 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.398 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.398 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.398 DEBUG:boto:Params: {} 2024-04-13T22:05:28.398 DEBUG:boto:Token: None 2024-04-13T22:05:28.398 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.398 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e91502a9a07342b6-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.400 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.401 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.401 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.401 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.401 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.401 DEBUG:boto:Method: GET 2024-04-13T22:05:28.401 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.401 DEBUG:boto:Data: 2024-04-13T22:05:28.402 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.402 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.402 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.402 DEBUG:boto:Params: {} 2024-04-13T22:05:28.402 DEBUG:boto:Token: None 2024-04-13T22:05:28.402 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.402 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006102a748665fde8f-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.404 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.405 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.405 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.405 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.405 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:05:28.405 DEBUG:boto:Data: 2024-04-13T22:05:28.405 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.405 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.405 DEBUG:boto:Params: {} 2024-04-13T22:05:28.406 DEBUG:boto:Token: None 2024-04-13T22:05:28.406 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3 2024-04-13T22:05:28.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OgJBLp1geTQRYyLJXJuDNqzZQ1Q= 2024-04-13T22:05:28.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OgJBLp1geTQRYyLJXJuDNqzZQ1Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcb726540591832a-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.407 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.408 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.408 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.408 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:05:28.408 DEBUG:boto:Data: 2024-04-13T22:05:28.408 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.408 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.408 DEBUG:boto:Params: {} 2024-04-13T22:05:28.408 DEBUG:boto:Token: None 2024-04-13T22:05:28.408 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3 2024-04-13T22:05:28.409 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OgJBLp1geTQRYyLJXJuDNqzZQ1Q= 2024-04-13T22:05:28.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OgJBLp1geTQRYyLJXJuDNqzZQ1Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000067af2124969777f-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:05:28.410 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.410 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.411 DEBUG:boto:Method: GET 2024-04-13T22:05:28.411 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:05:28.411 DEBUG:boto:Data: 2024-04-13T22:05:28.411 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.411 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.411 DEBUG:boto:Params: {} 2024-04-13T22:05:28.411 DEBUG:boto:Token: None 2024-04-13T22:05:28.411 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3 2024-04-13T22:05:28.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GVKX4/zJi7I6NqieXeiXpK4qJhY= 2024-04-13T22:05:28.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GVKX4/zJi7I6NqieXeiXpK4qJhY=', 'Content-Length': '0'} 2024-04-13T22:05:28.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000917064648dded346-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.414 DEBUG:boto:path=/vwnfns-50/obj3 2024-04-13T22:05:28.414 DEBUG:boto:auth_path=/vwnfns-50/obj3 2024-04-13T22:05:28.414 DEBUG:boto:Method: GET 2024-04-13T22:05:28.414 DEBUG:boto:Path: /vwnfns-50/obj3 2024-04-13T22:05:28.414 DEBUG:boto:Data: 2024-04-13T22:05:28.414 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.414 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.414 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.414 DEBUG:boto:Params: {} 2024-04-13T22:05:28.414 DEBUG:boto:Token: None 2024-04-13T22:05:28.415 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj3 2024-04-13T22:05:28.415 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GVKX4/zJi7I6NqieXeiXpK4qJhY= 2024-04-13T22:05:28.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GVKX4/zJi7I6NqieXeiXpK4qJhY=', 'Content-Length': '0'} 2024-04-13T22:05:28.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c123e268313e00a9-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.416 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.417 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.417 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.417 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.417 DEBUG:boto:Method: GET 2024-04-13T22:05:28.417 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.417 DEBUG:boto:Data: 2024-04-13T22:05:28.417 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.417 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.417 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.417 DEBUG:boto:Params: {} 2024-04-13T22:05:28.417 DEBUG:boto:Token: None 2024-04-13T22:05:28.418 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.418 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037eff4187545d541-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.420 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.420 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.421 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.421 DEBUG:boto:path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.421 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.421 DEBUG:boto:Method: GET 2024-04-13T22:05:28.421 DEBUG:boto:Path: /vwnfns-50/?prefix=obj3 2024-04-13T22:05:28.421 DEBUG:boto:Data: 2024-04-13T22:05:28.421 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.421 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.421 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.421 DEBUG:boto:Params: {} 2024-04-13T22:05:28.421 DEBUG:boto:Token: None 2024-04-13T22:05:28.422 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.422 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d883324db8ade31f-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.424 DEBUG:boto:b'vwnfns-50obj31000falseobj32024-04-13T22:04:24.768Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:05:28.424 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.424 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.424 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.424 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.425 DEBUG:boto:Method: GET 2024-04-13T22:05:28.425 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.425 DEBUG:boto:Data: 2024-04-13T22:05:28.425 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.425 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.425 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.425 DEBUG:boto:Params: {} 2024-04-13T22:05:28.425 DEBUG:boto:Token: None 2024-04-13T22:05:28.425 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.426 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4= 2024-04-13T22:05:28.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4=', 'Content-Length': '0'} 2024-04-13T22:05:28.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022861eef1c99e4cb-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.427 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.428 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.428 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.428 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.428 DEBUG:boto:Method: GET 2024-04-13T22:05:28.428 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.428 DEBUG:boto:Data: 2024-04-13T22:05:28.428 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.428 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.429 DEBUG:boto:Params: {} 2024-04-13T22:05:28.429 DEBUG:boto:Token: None 2024-04-13T22:05:28.429 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.429 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4= 2024-04-13T22:05:28.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4=', 'Content-Length': '0'} 2024-04-13T22:05:28.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a5c57477f5c8682-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.431 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.431 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.431 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.431 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.431 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.431 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.431 DEBUG:boto:Data: 2024-04-13T22:05:28.431 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.432 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.432 DEBUG:boto:Params: {} 2024-04-13T22:05:28.432 DEBUG:boto:Token: None 2024-04-13T22:05:28.432 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vwJEcFzvPh1XB7Z+r6mJwWJlnOE= 2024-04-13T22:05:28.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vwJEcFzvPh1XB7Z+r6mJwWJlnOE=', 'Content-Length': '0'} 2024-04-13T22:05:28.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cc727eebb034471-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.434 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.434 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.434 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.434 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.434 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.434 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.434 DEBUG:boto:Data: 2024-04-13T22:05:28.434 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.434 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.434 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.435 DEBUG:boto:Params: {} 2024-04-13T22:05:28.435 DEBUG:boto:Token: None 2024-04-13T22:05:28.435 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.435 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vwJEcFzvPh1XB7Z+r6mJwWJlnOE= 2024-04-13T22:05:28.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vwJEcFzvPh1XB7Z+r6mJwWJlnOE=', 'Content-Length': '0'} 2024-04-13T22:05:28.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084772719c375ad49-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:05:28.437 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.437 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.437 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.437 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.437 DEBUG:boto:Method: GET 2024-04-13T22:05:28.437 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.437 DEBUG:boto:Data: 2024-04-13T22:05:28.437 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.438 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.438 DEBUG:boto:Params: {} 2024-04-13T22:05:28.438 DEBUG:boto:Token: None 2024-04-13T22:05:28.438 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4= 2024-04-13T22:05:28.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4=', 'Content-Length': '0'} 2024-04-13T22:05:28.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093a2891ad926d5c8-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.440 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.440 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.440 DEBUG:boto:path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.440 DEBUG:boto:auth_path=/vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.441 DEBUG:boto:Method: GET 2024-04-13T22:05:28.441 DEBUG:boto:Path: /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.441 DEBUG:boto:Data: 2024-04-13T22:05:28.441 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.441 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.441 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.441 DEBUG:boto:Params: {} 2024-04-13T22:05:28.441 DEBUG:boto:Token: None 2024-04-13T22:05:28.442 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4?versionId=null 2024-04-13T22:05:28.442 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4= 2024-04-13T22:05:28.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Iq2N97BVtQ3d3Zqsdk3zOsgL2L4=', 'Content-Length': '0'} 2024-04-13T22:05:28.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c42203b419bd87b3-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.443 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.444 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.444 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.444 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.444 DEBUG:boto:Method: GET 2024-04-13T22:05:28.444 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.444 DEBUG:boto:Data: 2024-04-13T22:05:28.444 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.444 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.444 DEBUG:boto:Params: {} 2024-04-13T22:05:28.445 DEBUG:boto:Token: None 2024-04-13T22:05:28.445 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004521d5ac4c211bd-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.447 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.447 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.448 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.448 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.448 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.448 DEBUG:boto:Method: GET 2024-04-13T22:05:28.448 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.448 DEBUG:boto:Data: 2024-04-13T22:05:28.448 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.448 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.448 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.448 DEBUG:boto:Params: {} 2024-04-13T22:05:28.448 DEBUG:boto:Token: None 2024-04-13T22:05:28.449 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.449 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000172d39e557b38643-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.451 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.451 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.452 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.452 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.452 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:05:28.452 DEBUG:boto:Data: 2024-04-13T22:05:28.452 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.452 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.452 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.452 DEBUG:boto:Params: {} 2024-04-13T22:05:28.452 DEBUG:boto:Token: None 2024-04-13T22:05:28.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4 2024-04-13T22:05:28.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XuU3HSsonG5+LOF24vp2BPCBb20= 2024-04-13T22:05:28.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XuU3HSsonG5+LOF24vp2BPCBb20=', 'Content-Length': '0'} 2024-04-13T22:05:28.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000239bcb9e043b5d7e-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.455 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.455 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.455 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.455 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:05:28.455 DEBUG:boto:Data: 2024-04-13T22:05:28.455 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.455 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.455 DEBUG:boto:Params: {} 2024-04-13T22:05:28.455 DEBUG:boto:Token: None 2024-04-13T22:05:28.456 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4 2024-04-13T22:05:28.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XuU3HSsonG5+LOF24vp2BPCBb20= 2024-04-13T22:05:28.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XuU3HSsonG5+LOF24vp2BPCBb20=', 'Content-Length': '0'} 2024-04-13T22:05:28.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f513d506fdc1f68b-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:05:28.458 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.458 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.458 DEBUG:boto:Method: GET 2024-04-13T22:05:28.458 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:05:28.458 DEBUG:boto:Data: 2024-04-13T22:05:28.458 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.458 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.459 DEBUG:boto:Params: {} 2024-04-13T22:05:28.459 DEBUG:boto:Token: None 2024-04-13T22:05:28.459 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4 2024-04-13T22:05:28.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bi17mhR5fvnZ7xncNL1q2KemsX4= 2024-04-13T22:05:28.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bi17mhR5fvnZ7xncNL1q2KemsX4=', 'Content-Length': '0'} 2024-04-13T22:05:28.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1d73dfcecf2d3ae-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.502 DEBUG:boto:path=/vwnfns-50/obj4 2024-04-13T22:05:28.503 DEBUG:boto:auth_path=/vwnfns-50/obj4 2024-04-13T22:05:28.503 DEBUG:boto:Method: GET 2024-04-13T22:05:28.503 DEBUG:boto:Path: /vwnfns-50/obj4 2024-04-13T22:05:28.503 DEBUG:boto:Data: 2024-04-13T22:05:28.503 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.503 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.503 DEBUG:boto:Params: {} 2024-04-13T22:05:28.503 DEBUG:boto:Token: None 2024-04-13T22:05:28.504 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj4 2024-04-13T22:05:28.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bi17mhR5fvnZ7xncNL1q2KemsX4= 2024-04-13T22:05:28.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bi17mhR5fvnZ7xncNL1q2KemsX4=', 'Content-Length': '0'} 2024-04-13T22:05:28.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ac54254edf76b26-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.506 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.506 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.506 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.506 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.506 DEBUG:boto:Method: GET 2024-04-13T22:05:28.506 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.506 DEBUG:boto:Data: 2024-04-13T22:05:28.506 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.507 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.507 DEBUG:boto:Params: {} 2024-04-13T22:05:28.507 DEBUG:boto:Token: None 2024-04-13T22:05:28.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca0bb97f67533d17-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.509 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.510 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.510 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.510 DEBUG:boto:path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.510 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.510 DEBUG:boto:Method: GET 2024-04-13T22:05:28.510 DEBUG:boto:Path: /vwnfns-50/?prefix=obj4 2024-04-13T22:05:28.510 DEBUG:boto:Data: 2024-04-13T22:05:28.510 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.510 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.510 DEBUG:boto:Params: {} 2024-04-13T22:05:28.511 DEBUG:boto:Token: None 2024-04-13T22:05:28.511 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052acc050cc5a65d0-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.513 DEBUG:boto:b'vwnfns-50obj41000falseobj42024-04-13T22:04:24.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:05:28.513 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.514 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.514 DEBUG:boto:path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.514 DEBUG:boto:auth_path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.514 DEBUG:boto:Method: GET 2024-04-13T22:05:28.514 DEBUG:boto:Path: /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.514 DEBUG:boto:Data: 2024-04-13T22:05:28.514 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.514 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.514 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.514 DEBUG:boto:Params: {} 2024-04-13T22:05:28.514 DEBUG:boto:Token: None 2024-04-13T22:05:28.515 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes= 2024-04-13T22:05:28.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes=', 'Content-Length': '0'} 2024-04-13T22:05:28.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cc253fcc12823e0-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.517 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.517 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.517 DEBUG:boto:path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.517 DEBUG:boto:auth_path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.517 DEBUG:boto:Method: GET 2024-04-13T22:05:28.517 DEBUG:boto:Path: /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.517 DEBUG:boto:Data: 2024-04-13T22:05:28.517 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.517 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.517 DEBUG:boto:Params: {} 2024-04-13T22:05:28.518 DEBUG:boto:Token: None 2024-04-13T22:05:28.518 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes= 2024-04-13T22:05:28.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes=', 'Content-Length': '0'} 2024-04-13T22:05:28.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043c6ccc6d24210fe-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.520 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.520 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.520 DEBUG:boto:path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.520 DEBUG:boto:auth_path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.520 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.520 DEBUG:boto:Path: /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.520 DEBUG:boto:Data: 2024-04-13T22:05:28.520 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.520 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.520 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.521 DEBUG:boto:Params: {} 2024-04-13T22:05:28.521 DEBUG:boto:Token: None 2024-04-13T22:05:28.521 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UVEgxweBgVjeHES+9zFAJ/UYdi4= 2024-04-13T22:05:28.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UVEgxweBgVjeHES+9zFAJ/UYdi4=', 'Content-Length': '0'} 2024-04-13T22:05:28.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f52f029c6df0a45e-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.523 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.523 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.523 DEBUG:boto:path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.523 DEBUG:boto:auth_path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.523 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.523 DEBUG:boto:Path: /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.523 DEBUG:boto:Data: 2024-04-13T22:05:28.523 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.523 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.523 DEBUG:boto:Params: {} 2024-04-13T22:05:28.524 DEBUG:boto:Token: None 2024-04-13T22:05:28.524 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UVEgxweBgVjeHES+9zFAJ/UYdi4= 2024-04-13T22:05:28.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UVEgxweBgVjeHES+9zFAJ/UYdi4=', 'Content-Length': '0'} 2024-04-13T22:05:28.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a721469c56c2f5b6-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:05:28.526 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.526 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.526 DEBUG:boto:path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.526 DEBUG:boto:auth_path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.526 DEBUG:boto:Method: GET 2024-04-13T22:05:28.526 DEBUG:boto:Path: /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.526 DEBUG:boto:Data: 2024-04-13T22:05:28.527 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.527 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.527 DEBUG:boto:Params: {} 2024-04-13T22:05:28.527 DEBUG:boto:Token: None 2024-04-13T22:05:28.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes= 2024-04-13T22:05:28.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes=', 'Content-Length': '0'} 2024-04-13T22:05:28.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8527a2abc5ebfcf-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.529 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.529 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.529 DEBUG:boto:path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.529 DEBUG:boto:auth_path=/vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.529 DEBUG:boto:Method: GET 2024-04-13T22:05:28.529 DEBUG:boto:Path: /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.529 DEBUG:boto:Data: 2024-04-13T22:05:28.530 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.530 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.530 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.530 DEBUG:boto:Params: {} 2024-04-13T22:05:28.530 DEBUG:boto:Token: None 2024-04-13T22:05:28.530 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5?versionId=null 2024-04-13T22:05:28.530 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes= 2024-04-13T22:05:28.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YSOc84Mp8etLRaB0C2G9GEHsJes=', 'Content-Length': '0'} 2024-04-13T22:05:28.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097d624eaf8cd5437-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.532 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.532 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.532 DEBUG:boto:path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.532 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.532 DEBUG:boto:Method: GET 2024-04-13T22:05:28.532 DEBUG:boto:Path: /vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.533 DEBUG:boto:Data: 2024-04-13T22:05:28.533 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.533 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.533 DEBUG:boto:Params: {} 2024-04-13T22:05:28.533 DEBUG:boto:Token: None 2024-04-13T22:05:28.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003678543651e0f66c-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.535 DEBUG:boto:b'vwnfns-50obj51000falseobj52024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.536 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.536 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.536 DEBUG:boto:path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.536 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.536 DEBUG:boto:Method: GET 2024-04-13T22:05:28.536 DEBUG:boto:Path: /vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.536 DEBUG:boto:Data: 2024-04-13T22:05:28.536 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.536 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.537 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.537 DEBUG:boto:Params: {} 2024-04-13T22:05:28.537 DEBUG:boto:Token: None 2024-04-13T22:05:28.537 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.537 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1429a11d21a8237-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.539 DEBUG:boto:b'vwnfns-50obj51000falseobj52024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.539 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.540 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.540 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.540 DEBUG:boto:Path: /vwnfns-50/obj5 2024-04-13T22:05:28.540 DEBUG:boto:Data: 2024-04-13T22:05:28.540 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.540 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.540 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.540 DEBUG:boto:Params: {} 2024-04-13T22:05:28.540 DEBUG:boto:Token: None 2024-04-13T22:05:28.540 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5 2024-04-13T22:05:28.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tQ94ZXOP7PNXRGxObSiUuOtWWoo= 2024-04-13T22:05:28.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tQ94ZXOP7PNXRGxObSiUuOtWWoo=', 'Content-Length': '0'} 2024-04-13T22:05:28.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0fbcaaa61804574-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.542 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.543 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.543 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.543 DEBUG:boto:Path: /vwnfns-50/obj5 2024-04-13T22:05:28.543 DEBUG:boto:Data: 2024-04-13T22:05:28.543 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.543 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.543 DEBUG:boto:Params: {} 2024-04-13T22:05:28.543 DEBUG:boto:Token: None 2024-04-13T22:05:28.543 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5 2024-04-13T22:05:28.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tQ94ZXOP7PNXRGxObSiUuOtWWoo= 2024-04-13T22:05:28.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tQ94ZXOP7PNXRGxObSiUuOtWWoo=', 'Content-Length': '0'} 2024-04-13T22:05:28.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e55c658894adf05b-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:05:28.546 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.546 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.546 DEBUG:boto:Method: GET 2024-04-13T22:05:28.546 DEBUG:boto:Path: /vwnfns-50/obj5 2024-04-13T22:05:28.546 DEBUG:boto:Data: 2024-04-13T22:05:28.546 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.546 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.546 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.546 DEBUG:boto:Params: {} 2024-04-13T22:05:28.547 DEBUG:boto:Token: None 2024-04-13T22:05:28.547 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5 2024-04-13T22:05:28.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N7d02+W3YM8x51JoenDuZNR8DSY= 2024-04-13T22:05:28.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N7d02+W3YM8x51JoenDuZNR8DSY=', 'Content-Length': '0'} 2024-04-13T22:05:28.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f03a7d295f14685-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.590 DEBUG:boto:path=/vwnfns-50/obj5 2024-04-13T22:05:28.591 DEBUG:boto:auth_path=/vwnfns-50/obj5 2024-04-13T22:05:28.591 DEBUG:boto:Method: GET 2024-04-13T22:05:28.591 DEBUG:boto:Path: /vwnfns-50/obj5 2024-04-13T22:05:28.591 DEBUG:boto:Data: 2024-04-13T22:05:28.591 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.591 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.591 DEBUG:boto:Params: {} 2024-04-13T22:05:28.591 DEBUG:boto:Token: None 2024-04-13T22:05:28.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj5 2024-04-13T22:05:28.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N7d02+W3YM8x51JoenDuZNR8DSY= 2024-04-13T22:05:28.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N7d02+W3YM8x51JoenDuZNR8DSY=', 'Content-Length': '0'} 2024-04-13T22:05:28.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f37f311ef8eadbdc-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.594 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.594 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.594 DEBUG:boto:path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.594 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.594 DEBUG:boto:Method: GET 2024-04-13T22:05:28.594 DEBUG:boto:Path: /vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.594 DEBUG:boto:Data: 2024-04-13T22:05:28.595 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.595 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.595 DEBUG:boto:Params: {} 2024-04-13T22:05:28.595 DEBUG:boto:Token: None 2024-04-13T22:05:28.595 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.595 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0fbc044fc2cac44-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.600 DEBUG:boto:b'vwnfns-50obj51000falseobj52024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.600 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.600 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.600 DEBUG:boto:path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.601 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.601 DEBUG:boto:Method: GET 2024-04-13T22:05:28.601 DEBUG:boto:Path: /vwnfns-50/?prefix=obj5 2024-04-13T22:05:28.601 DEBUG:boto:Data: 2024-04-13T22:05:28.601 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.601 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.601 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.601 DEBUG:boto:Params: {} 2024-04-13T22:05:28.601 DEBUG:boto:Token: None 2024-04-13T22:05:28.602 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.602 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000364ab4b06c0f4781-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.604 DEBUG:boto:b'vwnfns-50obj51000falseobj52024-04-13T22:05:07.466Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:05:28.604 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.605 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.605 DEBUG:boto:path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.605 DEBUG:boto:auth_path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.605 DEBUG:boto:Method: GET 2024-04-13T22:05:28.605 DEBUG:boto:Path: /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.605 DEBUG:boto:Data: 2024-04-13T22:05:28.605 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.605 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.605 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.605 DEBUG:boto:Params: {} 2024-04-13T22:05:28.606 DEBUG:boto:Token: None 2024-04-13T22:05:28.606 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.606 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg= 2024-04-13T22:05:28.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg=', 'Content-Length': '0'} 2024-04-13T22:05:28.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036b000aadf386dd2-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.608 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.608 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.608 DEBUG:boto:path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.608 DEBUG:boto:auth_path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.609 DEBUG:boto:Method: GET 2024-04-13T22:05:28.609 DEBUG:boto:Path: /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.609 DEBUG:boto:Data: 2024-04-13T22:05:28.609 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.609 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.609 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.609 DEBUG:boto:Params: {} 2024-04-13T22:05:28.609 DEBUG:boto:Token: None 2024-04-13T22:05:28.609 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.609 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg= 2024-04-13T22:05:28.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg=', 'Content-Length': '0'} 2024-04-13T22:05:28.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060c16883f0652bff-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.611 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.611 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.611 DEBUG:boto:path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.612 DEBUG:boto:auth_path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.612 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.612 DEBUG:boto:Path: /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.612 DEBUG:boto:Data: 2024-04-13T22:05:28.612 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.612 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.612 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.612 DEBUG:boto:Params: {} 2024-04-13T22:05:28.612 DEBUG:boto:Token: None 2024-04-13T22:05:28.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.613 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aWQavyc1Wp5lGiZu1FBpsbq0714= 2024-04-13T22:05:28.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aWQavyc1Wp5lGiZu1FBpsbq0714=', 'Content-Length': '0'} 2024-04-13T22:05:28.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9bfee9ee89922ae-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.614 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.614 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.615 DEBUG:boto:path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.615 DEBUG:boto:auth_path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.615 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.615 DEBUG:boto:Path: /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.615 DEBUG:boto:Data: 2024-04-13T22:05:28.615 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.615 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.615 DEBUG:boto:Params: {} 2024-04-13T22:05:28.615 DEBUG:boto:Token: None 2024-04-13T22:05:28.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aWQavyc1Wp5lGiZu1FBpsbq0714= 2024-04-13T22:05:28.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aWQavyc1Wp5lGiZu1FBpsbq0714=', 'Content-Length': '0'} 2024-04-13T22:05:28.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9cb0a7a1b1082ad-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:05:28.618 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.618 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.618 DEBUG:boto:path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.618 DEBUG:boto:auth_path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.618 DEBUG:boto:Method: GET 2024-04-13T22:05:28.618 DEBUG:boto:Path: /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.618 DEBUG:boto:Data: 2024-04-13T22:05:28.618 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.618 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.618 DEBUG:boto:Params: {} 2024-04-13T22:05:28.619 DEBUG:boto:Token: None 2024-04-13T22:05:28.619 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg= 2024-04-13T22:05:28.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg=', 'Content-Length': '0'} 2024-04-13T22:05:28.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050f1db9aa94ae3c8-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.621 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.621 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.621 DEBUG:boto:path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.621 DEBUG:boto:auth_path=/vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.621 DEBUG:boto:Method: GET 2024-04-13T22:05:28.621 DEBUG:boto:Path: /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.621 DEBUG:boto:Data: 2024-04-13T22:05:28.621 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.621 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.622 DEBUG:boto:Params: {} 2024-04-13T22:05:28.622 DEBUG:boto:Token: None 2024-04-13T22:05:28.622 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6?versionId=null 2024-04-13T22:05:28.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg= 2024-04-13T22:05:28.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V0Zr5fSy7e1DTR2a2DgEeuECfCg=', 'Content-Length': '0'} 2024-04-13T22:05:28.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb3512b7c5ee5e67-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.624 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.624 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.624 DEBUG:boto:path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.624 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.624 DEBUG:boto:Method: GET 2024-04-13T22:05:28.624 DEBUG:boto:Path: /vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.624 DEBUG:boto:Data: 2024-04-13T22:05:28.625 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.625 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.625 DEBUG:boto:Params: {} 2024-04-13T22:05:28.625 DEBUG:boto:Token: None 2024-04-13T22:05:28.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002acf80448ea018c-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.628 DEBUG:boto:b'vwnfns-50obj61000falseobj62024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.628 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.628 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.628 DEBUG:boto:path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.628 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.628 DEBUG:boto:Method: GET 2024-04-13T22:05:28.629 DEBUG:boto:Path: /vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.629 DEBUG:boto:Data: 2024-04-13T22:05:28.629 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.629 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.629 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.629 DEBUG:boto:Params: {} 2024-04-13T22:05:28.629 DEBUG:boto:Token: None 2024-04-13T22:05:28.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a48a60385f3f2e26-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.632 DEBUG:boto:b'vwnfns-50obj61000falseobj62024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.632 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.632 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.632 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.632 DEBUG:boto:Path: /vwnfns-50/obj6 2024-04-13T22:05:28.632 DEBUG:boto:Data: 2024-04-13T22:05:28.633 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.633 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.633 DEBUG:boto:Params: {} 2024-04-13T22:05:28.633 DEBUG:boto:Token: None 2024-04-13T22:05:28.633 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6 2024-04-13T22:05:28.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ckd8I+WrGRZ0dfxsJaTORjwcF04= 2024-04-13T22:05:28.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ckd8I+WrGRZ0dfxsJaTORjwcF04=', 'Content-Length': '0'} 2024-04-13T22:05:28.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d65c31612404e67-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.635 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.635 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.635 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.635 DEBUG:boto:Path: /vwnfns-50/obj6 2024-04-13T22:05:28.635 DEBUG:boto:Data: 2024-04-13T22:05:28.635 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.635 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.636 DEBUG:boto:Params: {} 2024-04-13T22:05:28.636 DEBUG:boto:Token: None 2024-04-13T22:05:28.636 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6 2024-04-13T22:05:28.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ckd8I+WrGRZ0dfxsJaTORjwcF04= 2024-04-13T22:05:28.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ckd8I+WrGRZ0dfxsJaTORjwcF04=', 'Content-Length': '0'} 2024-04-13T22:05:28.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8052f69b8c6711f-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:05:28.638 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.638 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.638 DEBUG:boto:Method: GET 2024-04-13T22:05:28.638 DEBUG:boto:Path: /vwnfns-50/obj6 2024-04-13T22:05:28.638 DEBUG:boto:Data: 2024-04-13T22:05:28.638 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.638 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.639 DEBUG:boto:Params: {} 2024-04-13T22:05:28.639 DEBUG:boto:Token: None 2024-04-13T22:05:28.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6 2024-04-13T22:05:28.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+toj/fddhZDITjyU2xYuQbr2Ym0= 2024-04-13T22:05:28.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+toj/fddhZDITjyU2xYuQbr2Ym0=', 'Content-Length': '0'} 2024-04-13T22:05:28.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000def6b0ad2fbb4fd8-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.682 DEBUG:boto:path=/vwnfns-50/obj6 2024-04-13T22:05:28.683 DEBUG:boto:auth_path=/vwnfns-50/obj6 2024-04-13T22:05:28.683 DEBUG:boto:Method: GET 2024-04-13T22:05:28.683 DEBUG:boto:Path: /vwnfns-50/obj6 2024-04-13T22:05:28.683 DEBUG:boto:Data: 2024-04-13T22:05:28.683 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.683 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.683 DEBUG:boto:Params: {} 2024-04-13T22:05:28.684 DEBUG:boto:Token: None 2024-04-13T22:05:28.684 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj6 2024-04-13T22:05:28.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+toj/fddhZDITjyU2xYuQbr2Ym0= 2024-04-13T22:05:28.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+toj/fddhZDITjyU2xYuQbr2Ym0=', 'Content-Length': '0'} 2024-04-13T22:05:28.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d5c7cf370d0094-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.686 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.686 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.687 DEBUG:boto:path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.687 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.687 DEBUG:boto:Method: GET 2024-04-13T22:05:28.687 DEBUG:boto:Path: /vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.687 DEBUG:boto:Data: 2024-04-13T22:05:28.687 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.687 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.687 DEBUG:boto:Params: {} 2024-04-13T22:05:28.687 DEBUG:boto:Token: None 2024-04-13T22:05:28.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ae775475817416-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.690 DEBUG:boto:b'vwnfns-50obj61000falseobj62024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.690 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.690 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.690 DEBUG:boto:path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.690 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.691 DEBUG:boto:Method: GET 2024-04-13T22:05:28.691 DEBUG:boto:Path: /vwnfns-50/?prefix=obj6 2024-04-13T22:05:28.691 DEBUG:boto:Data: 2024-04-13T22:05:28.691 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.691 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.691 DEBUG:boto:Params: {} 2024-04-13T22:05:28.691 DEBUG:boto:Token: None 2024-04-13T22:05:28.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f064e2459cf6c14e-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.694 DEBUG:boto:b'vwnfns-50obj61000falseobj62024-04-13T22:05:07.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:05:28.694 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.694 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.694 DEBUG:boto:path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.695 DEBUG:boto:auth_path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.695 DEBUG:boto:Method: GET 2024-04-13T22:05:28.695 DEBUG:boto:Path: /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.695 DEBUG:boto:Data: 2024-04-13T22:05:28.695 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.695 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.695 DEBUG:boto:Params: {} 2024-04-13T22:05:28.695 DEBUG:boto:Token: None 2024-04-13T22:05:28.695 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY= 2024-04-13T22:05:28.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY=', 'Content-Length': '0'} 2024-04-13T22:05:28.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c95d1b6cd9f5d1e5-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.698 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.698 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.698 DEBUG:boto:path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.698 DEBUG:boto:auth_path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.698 DEBUG:boto:Method: GET 2024-04-13T22:05:28.698 DEBUG:boto:Path: /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.698 DEBUG:boto:Data: 2024-04-13T22:05:28.698 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.698 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.698 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.699 DEBUG:boto:Params: {} 2024-04-13T22:05:28.699 DEBUG:boto:Token: None 2024-04-13T22:05:28.699 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY= 2024-04-13T22:05:28.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY=', 'Content-Length': '0'} 2024-04-13T22:05:28.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7a9ba0d31481e1b-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.701 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.701 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.701 DEBUG:boto:path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.701 DEBUG:boto:auth_path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.701 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.701 DEBUG:boto:Path: /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.701 DEBUG:boto:Data: 2024-04-13T22:05:28.701 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.702 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.702 DEBUG:boto:Params: {} 2024-04-13T22:05:28.702 DEBUG:boto:Token: None 2024-04-13T22:05:28.702 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o7xVlnrrNzOO4EiGSgLQb5HrB9M= 2024-04-13T22:05:28.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o7xVlnrrNzOO4EiGSgLQb5HrB9M=', 'Content-Length': '0'} 2024-04-13T22:05:28.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006954c96bd595efdb-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.704 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.704 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.704 DEBUG:boto:path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.704 DEBUG:boto:auth_path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.704 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.704 DEBUG:boto:Path: /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.704 DEBUG:boto:Data: 2024-04-13T22:05:28.705 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.705 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.705 DEBUG:boto:Params: {} 2024-04-13T22:05:28.705 DEBUG:boto:Token: None 2024-04-13T22:05:28.705 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o7xVlnrrNzOO4EiGSgLQb5HrB9M= 2024-04-13T22:05:28.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o7xVlnrrNzOO4EiGSgLQb5HrB9M=', 'Content-Length': '0'} 2024-04-13T22:05:28.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055a6a233072c5a31-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:05:28.707 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.707 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.707 DEBUG:boto:path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.707 DEBUG:boto:auth_path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.707 DEBUG:boto:Method: GET 2024-04-13T22:05:28.708 DEBUG:boto:Path: /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.708 DEBUG:boto:Data: 2024-04-13T22:05:28.708 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.708 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.708 DEBUG:boto:Params: {} 2024-04-13T22:05:28.708 DEBUG:boto:Token: None 2024-04-13T22:05:28.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.709 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY= 2024-04-13T22:05:28.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY=', 'Content-Length': '0'} 2024-04-13T22:05:28.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9599f0270490988-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.711 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.711 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.711 DEBUG:boto:path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.711 DEBUG:boto:auth_path=/vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.711 DEBUG:boto:Method: GET 2024-04-13T22:05:28.711 DEBUG:boto:Path: /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.711 DEBUG:boto:Data: 2024-04-13T22:05:28.711 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.712 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.712 DEBUG:boto:Params: {} 2024-04-13T22:05:28.712 DEBUG:boto:Token: None 2024-04-13T22:05:28.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7?versionId=null 2024-04-13T22:05:28.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY= 2024-04-13T22:05:28.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DgrYfyHz+3ARyvPEbIkEe2fB0bY=', 'Content-Length': '0'} 2024-04-13T22:05:28.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc4bed01e83dea9f-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.754 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.754 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.755 DEBUG:boto:path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.755 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.755 DEBUG:boto:Method: GET 2024-04-13T22:05:28.755 DEBUG:boto:Path: /vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.755 DEBUG:boto:Data: 2024-04-13T22:05:28.755 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.755 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.755 DEBUG:boto:Params: {} 2024-04-13T22:05:28.755 DEBUG:boto:Token: None 2024-04-13T22:05:28.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5eff8534373fefb-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.758 DEBUG:boto:b'vwnfns-50obj71000falseobj72024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.758 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.758 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.758 DEBUG:boto:path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.758 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.759 DEBUG:boto:Method: GET 2024-04-13T22:05:28.759 DEBUG:boto:Path: /vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.759 DEBUG:boto:Data: 2024-04-13T22:05:28.759 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.759 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.759 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.759 DEBUG:boto:Params: {} 2024-04-13T22:05:28.759 DEBUG:boto:Token: None 2024-04-13T22:05:28.759 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f38204bbefa4e8-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.761 DEBUG:boto:b'vwnfns-50obj71000falseobj72024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.762 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.762 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.762 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.762 DEBUG:boto:Path: /vwnfns-50/obj7 2024-04-13T22:05:28.762 DEBUG:boto:Data: 2024-04-13T22:05:28.762 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.762 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.763 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.763 DEBUG:boto:Params: {} 2024-04-13T22:05:28.763 DEBUG:boto:Token: None 2024-04-13T22:05:28.763 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7 2024-04-13T22:05:28.763 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1Vj3hrSd1t8w3sVvMLldW3+NMvA= 2024-04-13T22:05:28.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Vj3hrSd1t8w3sVvMLldW3+NMvA=', 'Content-Length': '0'} 2024-04-13T22:05:28.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcd0cf22ee5d97fb-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.765 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.765 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.765 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.765 DEBUG:boto:Path: /vwnfns-50/obj7 2024-04-13T22:05:28.765 DEBUG:boto:Data: 2024-04-13T22:05:28.765 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.765 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.765 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.766 DEBUG:boto:Params: {} 2024-04-13T22:05:28.766 DEBUG:boto:Token: None 2024-04-13T22:05:28.766 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7 2024-04-13T22:05:28.766 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1Vj3hrSd1t8w3sVvMLldW3+NMvA= 2024-04-13T22:05:28.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Vj3hrSd1t8w3sVvMLldW3+NMvA=', 'Content-Length': '0'} 2024-04-13T22:05:28.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088ff36d732d83736-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:05:28.768 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.768 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.768 DEBUG:boto:Method: GET 2024-04-13T22:05:28.768 DEBUG:boto:Path: /vwnfns-50/obj7 2024-04-13T22:05:28.768 DEBUG:boto:Data: 2024-04-13T22:05:28.768 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.768 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.768 DEBUG:boto:Params: {} 2024-04-13T22:05:28.769 DEBUG:boto:Token: None 2024-04-13T22:05:28.769 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7 2024-04-13T22:05:28.769 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dmm+7jdZUDQvDOEdHVu3OrY08VM= 2024-04-13T22:05:28.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dmm+7jdZUDQvDOEdHVu3OrY08VM=', 'Content-Length': '0'} 2024-04-13T22:05:28.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc40df035ce2b3ec-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.771 DEBUG:boto:path=/vwnfns-50/obj7 2024-04-13T22:05:28.771 DEBUG:boto:auth_path=/vwnfns-50/obj7 2024-04-13T22:05:28.771 DEBUG:boto:Method: GET 2024-04-13T22:05:28.771 DEBUG:boto:Path: /vwnfns-50/obj7 2024-04-13T22:05:28.771 DEBUG:boto:Data: 2024-04-13T22:05:28.771 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.771 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.771 DEBUG:boto:Params: {} 2024-04-13T22:05:28.771 DEBUG:boto:Token: None 2024-04-13T22:05:28.772 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj7 2024-04-13T22:05:28.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Dmm+7jdZUDQvDOEdHVu3OrY08VM= 2024-04-13T22:05:28.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Dmm+7jdZUDQvDOEdHVu3OrY08VM=', 'Content-Length': '0'} 2024-04-13T22:05:28.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db111c7994881a76-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.774 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.774 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.774 DEBUG:boto:path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.774 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.774 DEBUG:boto:Method: GET 2024-04-13T22:05:28.774 DEBUG:boto:Path: /vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.774 DEBUG:boto:Data: 2024-04-13T22:05:28.774 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.774 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.774 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.775 DEBUG:boto:Params: {} 2024-04-13T22:05:28.775 DEBUG:boto:Token: None 2024-04-13T22:05:28.775 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.775 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d47679300bce5c-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.777 DEBUG:boto:b'vwnfns-50obj71000falseobj72024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.777 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.777 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.778 DEBUG:boto:path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.778 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.778 DEBUG:boto:Method: GET 2024-04-13T22:05:28.778 DEBUG:boto:Path: /vwnfns-50/?prefix=obj7 2024-04-13T22:05:28.778 DEBUG:boto:Data: 2024-04-13T22:05:28.778 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.778 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.778 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.778 DEBUG:boto:Params: {} 2024-04-13T22:05:28.779 DEBUG:boto:Token: None 2024-04-13T22:05:28.779 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b701f40266709921-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.781 DEBUG:boto:b'vwnfns-50obj71000falseobj72024-04-13T22:05:07.527Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:05:28.781 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.781 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.781 DEBUG:boto:path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.781 DEBUG:boto:auth_path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.782 DEBUG:boto:Method: GET 2024-04-13T22:05:28.782 DEBUG:boto:Path: /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.782 DEBUG:boto:Data: 2024-04-13T22:05:28.782 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.782 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.782 DEBUG:boto:Params: {} 2024-04-13T22:05:28.782 DEBUG:boto:Token: None 2024-04-13T22:05:28.782 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q= 2024-04-13T22:05:28.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4af52ca720f5b27-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.785 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.785 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.785 DEBUG:boto:path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.785 DEBUG:boto:auth_path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.785 DEBUG:boto:Method: GET 2024-04-13T22:05:28.785 DEBUG:boto:Path: /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.785 DEBUG:boto:Data: 2024-04-13T22:05:28.785 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.785 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.785 DEBUG:boto:Params: {} 2024-04-13T22:05:28.786 DEBUG:boto:Token: None 2024-04-13T22:05:28.786 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q= 2024-04-13T22:05:28.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004123e1f8b102c4be-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.788 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.788 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.788 DEBUG:boto:path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.788 DEBUG:boto:auth_path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.788 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.788 DEBUG:boto:Path: /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.788 DEBUG:boto:Data: 2024-04-13T22:05:28.788 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.788 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.788 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.789 DEBUG:boto:Params: {} 2024-04-13T22:05:28.789 DEBUG:boto:Token: None 2024-04-13T22:05:28.789 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hhaXxi3kogYW4Mjyq4AqBct/a9s= 2024-04-13T22:05:28.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hhaXxi3kogYW4Mjyq4AqBct/a9s=', 'Content-Length': '0'} 2024-04-13T22:05:28.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000853f5a886fa6592d-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.791 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.791 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.791 DEBUG:boto:path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.791 DEBUG:boto:auth_path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.791 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.791 DEBUG:boto:Path: /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.791 DEBUG:boto:Data: 2024-04-13T22:05:28.791 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.792 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.792 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.792 DEBUG:boto:Params: {} 2024-04-13T22:05:28.792 DEBUG:boto:Token: None 2024-04-13T22:05:28.792 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hhaXxi3kogYW4Mjyq4AqBct/a9s= 2024-04-13T22:05:28.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hhaXxi3kogYW4Mjyq4AqBct/a9s=', 'Content-Length': '0'} 2024-04-13T22:05:28.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1aefb0fe7108612-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:05:28.794 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.794 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.794 DEBUG:boto:path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.794 DEBUG:boto:auth_path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.794 DEBUG:boto:Method: GET 2024-04-13T22:05:28.795 DEBUG:boto:Path: /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.795 DEBUG:boto:Data: 2024-04-13T22:05:28.795 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.795 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.795 DEBUG:boto:Params: {} 2024-04-13T22:05:28.795 DEBUG:boto:Token: None 2024-04-13T22:05:28.795 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q= 2024-04-13T22:05:28.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002437fe229f3bc5eb-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.842 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.843 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.843 DEBUG:boto:path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.843 DEBUG:boto:auth_path=/vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.843 DEBUG:boto:Method: GET 2024-04-13T22:05:28.843 DEBUG:boto:Path: /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.843 DEBUG:boto:Data: 2024-04-13T22:05:28.843 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.844 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.844 DEBUG:boto:Params: {} 2024-04-13T22:05:28.844 DEBUG:boto:Token: None 2024-04-13T22:05:28.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8?versionId=null 2024-04-13T22:05:28.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q= 2024-04-13T22:05:28.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QbITtUHpIy5Qt0xII5VqKHf5m5Q=', 'Content-Length': '0'} 2024-04-13T22:05:28.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c67f19adeae4d264-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.846 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.846 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.846 DEBUG:boto:path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.846 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.847 DEBUG:boto:Method: GET 2024-04-13T22:05:28.847 DEBUG:boto:Path: /vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.847 DEBUG:boto:Data: 2024-04-13T22:05:28.847 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.847 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.847 DEBUG:boto:Params: {} 2024-04-13T22:05:28.847 DEBUG:boto:Token: None 2024-04-13T22:05:28.847 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1bbcb75d969aa24-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.850 DEBUG:boto:b'vwnfns-50obj81000falseobj82024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.850 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.850 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.850 DEBUG:boto:path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.850 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.850 DEBUG:boto:Method: GET 2024-04-13T22:05:28.851 DEBUG:boto:Path: /vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.851 DEBUG:boto:Data: 2024-04-13T22:05:28.851 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.851 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.851 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.851 DEBUG:boto:Params: {} 2024-04-13T22:05:28.851 DEBUG:boto:Token: None 2024-04-13T22:05:28.851 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000529c47b659e07996-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.853 DEBUG:boto:b'vwnfns-50obj81000falseobj82024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.854 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.854 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.854 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.854 DEBUG:boto:Path: /vwnfns-50/obj8 2024-04-13T22:05:28.854 DEBUG:boto:Data: 2024-04-13T22:05:28.854 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.854 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.854 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.855 DEBUG:boto:Params: {} 2024-04-13T22:05:28.855 DEBUG:boto:Token: None 2024-04-13T22:05:28.855 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8 2024-04-13T22:05:28.855 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rXcOYlviKJDglbF2cXBLUjDTmHU= 2024-04-13T22:05:28.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rXcOYlviKJDglbF2cXBLUjDTmHU=', 'Content-Length': '0'} 2024-04-13T22:05:28.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c654801e94a334-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.857 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.857 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.857 DEBUG:boto:Method: HEAD 2024-04-13T22:05:28.857 DEBUG:boto:Path: /vwnfns-50/obj8 2024-04-13T22:05:28.857 DEBUG:boto:Data: 2024-04-13T22:05:28.857 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.857 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.857 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.858 DEBUG:boto:Params: {} 2024-04-13T22:05:28.858 DEBUG:boto:Token: None 2024-04-13T22:05:28.858 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8 2024-04-13T22:05:28.858 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rXcOYlviKJDglbF2cXBLUjDTmHU= 2024-04-13T22:05:28.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rXcOYlviKJDglbF2cXBLUjDTmHU=', 'Content-Length': '0'} 2024-04-13T22:05:28.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f5a20d9eeddd14b-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:05:28.860 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.860 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.860 DEBUG:boto:Method: GET 2024-04-13T22:05:28.860 DEBUG:boto:Path: /vwnfns-50/obj8 2024-04-13T22:05:28.860 DEBUG:boto:Data: 2024-04-13T22:05:28.860 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.860 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.861 DEBUG:boto:Params: {} 2024-04-13T22:05:28.861 DEBUG:boto:Token: None 2024-04-13T22:05:28.861 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8 2024-04-13T22:05:28.861 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ucx52FpVwP2frWeyV4haXTxZEDI= 2024-04-13T22:05:28.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ucx52FpVwP2frWeyV4haXTxZEDI=', 'Content-Length': '0'} 2024-04-13T22:05:28.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-50:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029a95eaab6674ad0-00661b01a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.863 DEBUG:boto:path=/vwnfns-50/obj8 2024-04-13T22:05:28.863 DEBUG:boto:auth_path=/vwnfns-50/obj8 2024-04-13T22:05:28.863 DEBUG:boto:Method: GET 2024-04-13T22:05:28.863 DEBUG:boto:Path: /vwnfns-50/obj8 2024-04-13T22:05:28.863 DEBUG:boto:Data: 2024-04-13T22:05:28.863 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.863 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.863 DEBUG:boto:Params: {} 2024-04-13T22:05:28.864 DEBUG:boto:Token: None 2024-04-13T22:05:28.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/obj8 2024-04-13T22:05:28.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ucx52FpVwP2frWeyV4haXTxZEDI= 2024-04-13T22:05:28.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ucx52FpVwP2frWeyV4haXTxZEDI=', 'Content-Length': '0'} 2024-04-13T22:05:28.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aa26a642f65400e-00661b01a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.866 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.866 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.866 DEBUG:boto:path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.866 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.866 DEBUG:boto:Method: GET 2024-04-13T22:05:28.866 DEBUG:boto:Path: /vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.866 DEBUG:boto:Data: 2024-04-13T22:05:28.867 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:28.867 DEBUG:boto:Port: 8001 2024-04-13T22:05:28.867 DEBUG:boto:Params: {} 2024-04-13T22:05:28.867 DEBUG:boto:Token: None 2024-04-13T22:05:28.867 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008622675de0d90ea7-00661b01a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.870 DEBUG:boto:b'vwnfns-50obj81000falseobj82024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.870 DEBUG:boto:path=/vwnfns-50/ 2024-04-13T22:05:28.870 DEBUG:boto:auth_path=/vwnfns-50/ 2024-04-13T22:05:28.870 DEBUG:boto:path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.870 DEBUG:boto:auth_path=/vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.870 DEBUG:boto:Method: GET 2024-04-13T22:05:28.871 DEBUG:boto:Path: /vwnfns-50/?prefix=obj8 2024-04-13T22:05:28.871 DEBUG:boto:Data: 2024-04-13T22:05:28.871 DEBUG:boto:Headers: {} 2024-04-13T22:05:28.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:28.871 DEBUG:boto:Port: 8000 2024-04-13T22:05:28.871 DEBUG:boto:Params: {} 2024-04-13T22:05:28.871 DEBUG:boto:Token: None 2024-04-13T22:05:28.871 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:28 GMT /vwnfns-50/ 2024-04-13T22:05:28.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI= 2024-04-13T22:05:28.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGCVTQz54O8X7kTjo/K5WO4Q0HI=', 'Content-Length': '0'} 2024-04-13T22:05:28.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c0b72c6f70d6e20-00661b01a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:28.874 DEBUG:boto:b'vwnfns-50obj81000falseobj82024-04-13T22:05:07.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:28.874 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-50 zones={a2, a1} 2024-04-13T22:05:28.874 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-51 2024-04-13T22:05:28.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:28.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:28.875 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-51 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:05:28.930 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:28.930 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:28.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:28.965+0000 7fe87d86d780 1 waiting for incremental sync to catch up: 2024-04-13T22:05:28.965 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.35.1, 00000000002.45.1, 00000000003.23.1, 00000000002.42.1, 00000000003.20.1, 00000000002.33.1, 00000000002.24.1, 00000000003.46.1, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:28.965 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.38.6, 00000000002.45.1, 00000000004.26.6, 00000000002.42.1, 00000000004.22.6, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:33.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:33.967+0000 7fe87d86d780 1 waiting for incremental sync to catch up: 2024-04-13T22:05:33.967 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.35.1, 00000000002.45.1, 00000000003.23.1, 00000000002.42.1, 00000000003.20.1, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:33.967 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.38.6, 00000000002.45.1, 00000000004.26.6, 00000000002.42.1, 00000000004.22.6, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:38.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:38.967+0000 7fe87d86d780 1 waiting for incremental sync to catch up: 2024-04-13T22:05:38.969 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.35.1, 00000000002.45.1, 00000000003.23.1, 00000000002.42.1, 00000000003.20.1, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:38.969 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.38.6, 00000000002.45.1, 00000000004.26.6, 00000000002.42.1, 00000000004.22.6, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:43.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:43.969+0000 7fe87d86d780 1 waiting for incremental sync to catch up: 2024-04-13T22:05:43.970 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.35.1, 00000000002.45.1, 00000000003.23.1, 00000000002.42.1, 00000000003.20.1, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:43.970 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.38.6, 00000000002.45.1, 00000000004.26.6, 00000000002.42.1, 00000000004.22.6, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:48.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:48.970+0000 7fe87d86d780 1 bucket sync caught up with source: 2024-04-13T22:05:48.972 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.38.6, 00000000002.45.1, 00000000004.26.6, 00000000002.42.1, 00000000004.22.6, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:48.972 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.38.6, 00000000002.45.1, 00000000004.26.6, 00000000002.42.1, 00000000004.22.6, 00000000002.33.1, 00000000002.24.1, 00000000004.48.6, 00000000002.26.1, 00000000002.36.1, 00000000002.60.1] 2024-04-13T22:05:48.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:48.970+0000 7fe87d86d780 0 bucket checkpoint complete 2024-04-13T22:05:48.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-51 2024-04-13T22:05:48.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:48.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:48.987 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-51 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:05:49.042 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:49.042 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:49.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:49.076+0000 7fca72803780 1 bucket sync caught up with source: 2024-04-13T22:05:49.077 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.545.1, 00000000002.975.1, 00000000004.2015.1, 00000000002.936.1, 00000000004.811.1, 00000000002.1457.1, 00000000002.2017.1, 00000000004.1751.1, 00000000002.1102.1, 00000000002.546.1, 00000000002.2809.1] 2024-04-13T22:05:49.078 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.545.1, 00000000002.975.1, 00000000004.2015.1, 00000000002.936.1, 00000000004.811.1, 00000000002.1457.1, 00000000002.2017.1, 00000000004.1751.1, 00000000002.1102.1, 00000000002.546.1, 00000000002.2809.1] 2024-04-13T22:05:49.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:49.076+0000 7fca72803780 0 bucket checkpoint complete 2024-04-13T22:05:49.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-51 zones={a2, a1} 2024-04-13T22:05:49.090 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.091 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.091 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.091 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:05:49.091 DEBUG:boto:Data: 2024-04-13T22:05:49.091 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.091 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.091 DEBUG:boto:Params: {} 2024-04-13T22:05:49.092 DEBUG:boto:Token: None 2024-04-13T22:05:49.092 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yhtt84ofGM3PgGjHiEKmv/tyh68= 2024-04-13T22:05:49.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yhtt84ofGM3PgGjHiEKmv/tyh68=', 'Content-Length': '0'} 2024-04-13T22:05:49.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003599f59ec732173d-00661b01bd-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.095 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.095 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.095 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.095 DEBUG:boto:Path: /vwnfns-51/ 2024-04-13T22:05:49.095 DEBUG:boto:Data: 2024-04-13T22:05:49.095 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.095 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.095 DEBUG:boto:Params: {} 2024-04-13T22:05:49.096 DEBUG:boto:Token: None 2024-04-13T22:05:49.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yhtt84ofGM3PgGjHiEKmv/tyh68= 2024-04-13T22:05:49.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yhtt84ofGM3PgGjHiEKmv/tyh68=', 'Content-Length': '0'} 2024-04-13T22:05:49.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048eb89031b337542-00661b01bd-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:05:49.102 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.102 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.102 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.102 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.102 DEBUG:boto:Method: GET 2024-04-13T22:05:49.102 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.102 DEBUG:boto:Data: 2024-04-13T22:05:49.102 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.102 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.102 DEBUG:boto:Params: {} 2024-04-13T22:05:49.103 DEBUG:boto:Token: None 2024-04-13T22:05:49.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/?versions 2024-04-13T22:05:49.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ= 2024-04-13T22:05:49.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f53434a2980f4d1-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.106 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:05:49.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:05:49.107 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.107 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.107 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.108 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.108 DEBUG:boto:Method: GET 2024-04-13T22:05:49.108 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.108 DEBUG:boto:Data: 2024-04-13T22:05:49.108 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.108 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.108 DEBUG:boto:Params: {} 2024-04-13T22:05:49.108 DEBUG:boto:Token: None 2024-04-13T22:05:49.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/?versions 2024-04-13T22:05:49.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ= 2024-04-13T22:05:49.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb950a9bdd86a8e0-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.111 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-13T22:05:49.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-13T22:05:49.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-13T22:05:49.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-13T22:05:49.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-13T22:05:49.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-13T22:05:49.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-13T22:05:49.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-13T22:05:49.112 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.112 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.112 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.112 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.112 DEBUG:boto:Method: GET 2024-04-13T22:05:49.112 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.113 DEBUG:boto:Data: 2024-04-13T22:05:49.113 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.113 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.113 DEBUG:boto:Params: {} 2024-04-13T22:05:49.113 DEBUG:boto:Token: None 2024-04-13T22:05:49.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/?versions 2024-04-13T22:05:49.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ= 2024-04-13T22:05:49.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b117701031dbe1-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.116 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.116 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.116 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.116 DEBUG:boto:path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.116 DEBUG:boto:auth_path=/vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.117 DEBUG:boto:Method: GET 2024-04-13T22:05:49.117 DEBUG:boto:Path: /vwnfns-51/?versions&max-keys=999 2024-04-13T22:05:49.117 DEBUG:boto:Data: 2024-04-13T22:05:49.117 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.117 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.117 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.117 DEBUG:boto:Params: {} 2024-04-13T22:05:49.117 DEBUG:boto:Token: None 2024-04-13T22:05:49.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/?versions 2024-04-13T22:05:49.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ= 2024-04-13T22:05:49.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wp6ALGlpbp6aJbz/nZEWA/+PtqQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000881ccd9b158c16f6-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.120 DEBUG:boto:b'vwnfns-51999falseobj1nulltrue2024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:05:49.120 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.120 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.120 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.121 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.121 DEBUG:boto:Method: GET 2024-04-13T22:05:49.121 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.121 DEBUG:boto:Data: 2024-04-13T22:05:49.121 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.121 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.121 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.121 DEBUG:boto:Params: {} 2024-04-13T22:05:49.121 DEBUG:boto:Token: None 2024-04-13T22:05:49.121 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.122 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ= 2024-04-13T22:05:49.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ec47511ccb46b30-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.170 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.171 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.171 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.171 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.171 DEBUG:boto:Method: GET 2024-04-13T22:05:49.171 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.171 DEBUG:boto:Data: 2024-04-13T22:05:49.171 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.171 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.171 DEBUG:boto:Params: {} 2024-04-13T22:05:49.172 DEBUG:boto:Token: None 2024-04-13T22:05:49.172 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ= 2024-04-13T22:05:49.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db8671b741113735-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.174 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.174 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.174 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.174 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.174 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.175 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.175 DEBUG:boto:Data: 2024-04-13T22:05:49.175 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.175 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.175 DEBUG:boto:Params: {} 2024-04-13T22:05:49.175 DEBUG:boto:Token: None 2024-04-13T22:05:49.175 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Umh0qYjd7XuQMGCuGl8rxnSVRzs= 2024-04-13T22:05:49.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Umh0qYjd7XuQMGCuGl8rxnSVRzs=', 'Content-Length': '0'} 2024-04-13T22:05:49.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b76cadae224d07a1-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.177 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.177 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.177 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.177 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.177 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.177 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.178 DEBUG:boto:Data: 2024-04-13T22:05:49.178 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.178 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.178 DEBUG:boto:Params: {} 2024-04-13T22:05:49.178 DEBUG:boto:Token: None 2024-04-13T22:05:49.178 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Umh0qYjd7XuQMGCuGl8rxnSVRzs= 2024-04-13T22:05:49.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Umh0qYjd7XuQMGCuGl8rxnSVRzs=', 'Content-Length': '0'} 2024-04-13T22:05:49.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f78021e01c75f31-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:05:49.180 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.180 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.180 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.181 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.181 DEBUG:boto:Method: GET 2024-04-13T22:05:49.181 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.181 DEBUG:boto:Data: 2024-04-13T22:05:49.181 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.181 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.181 DEBUG:boto:Params: {} 2024-04-13T22:05:49.181 DEBUG:boto:Token: None 2024-04-13T22:05:49.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ= 2024-04-13T22:05:49.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034d9dc5d42be4509-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.184 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.184 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.184 DEBUG:boto:path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.184 DEBUG:boto:auth_path=/vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.184 DEBUG:boto:Method: GET 2024-04-13T22:05:49.184 DEBUG:boto:Path: /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.184 DEBUG:boto:Data: 2024-04-13T22:05:49.184 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.185 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.185 DEBUG:boto:Params: {} 2024-04-13T22:05:49.185 DEBUG:boto:Token: None 2024-04-13T22:05:49.185 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1?versionId=null 2024-04-13T22:05:49.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ= 2024-04-13T22:05:49.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SIgcUF8j/MHkUNM10kfcpzi1aKQ=', 'Content-Length': '0'} 2024-04-13T22:05:49.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d14fbaec4fcd22-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.187 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.187 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.187 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.187 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.187 DEBUG:boto:Method: GET 2024-04-13T22:05:49.188 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.188 DEBUG:boto:Data: 2024-04-13T22:05:49.188 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.188 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.188 DEBUG:boto:Params: {} 2024-04-13T22:05:49.188 DEBUG:boto:Token: None 2024-04-13T22:05:49.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000482e08d86bd6a8b7-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.191 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.192 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.192 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.192 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.192 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.192 DEBUG:boto:Method: GET 2024-04-13T22:05:49.192 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.192 DEBUG:boto:Data: 2024-04-13T22:05:49.192 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.192 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.193 DEBUG:boto:Params: {} 2024-04-13T22:05:49.193 DEBUG:boto:Token: None 2024-04-13T22:05:49.193 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008924865dd55307bd-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.242 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.243 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.243 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.243 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.243 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:05:49.243 DEBUG:boto:Data: 2024-04-13T22:05:49.244 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.244 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.244 DEBUG:boto:Params: {} 2024-04-13T22:05:49.244 DEBUG:boto:Token: None 2024-04-13T22:05:49.244 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1 2024-04-13T22:05:49.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0hoxx2fSqc5u23NMAxzme6/6hkA= 2024-04-13T22:05:49.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0hoxx2fSqc5u23NMAxzme6/6hkA=', 'Content-Length': '0'} 2024-04-13T22:05:49.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007484ad3a13008086-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.247 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.247 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.247 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.247 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:05:49.247 DEBUG:boto:Data: 2024-04-13T22:05:49.247 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.247 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.247 DEBUG:boto:Params: {} 2024-04-13T22:05:49.247 DEBUG:boto:Token: None 2024-04-13T22:05:49.248 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1 2024-04-13T22:05:49.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0hoxx2fSqc5u23NMAxzme6/6hkA= 2024-04-13T22:05:49.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0hoxx2fSqc5u23NMAxzme6/6hkA=', 'Content-Length': '0'} 2024-04-13T22:05:49.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bb935e62a1f9ac0-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-13T22:05:49.250 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.250 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.250 DEBUG:boto:Method: GET 2024-04-13T22:05:49.250 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:05:49.250 DEBUG:boto:Data: 2024-04-13T22:05:49.250 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.250 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.250 DEBUG:boto:Params: {} 2024-04-13T22:05:49.250 DEBUG:boto:Token: None 2024-04-13T22:05:49.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1 2024-04-13T22:05:49.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q4JN5XHUoc/4/pOEpBetli4hr+Q= 2024-04-13T22:05:49.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4JN5XHUoc/4/pOEpBetli4hr+Q=', 'Content-Length': '0'} 2024-04-13T22:05:49.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057a9154f5cc937fe-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.252 DEBUG:boto:path=/vwnfns-51/obj1 2024-04-13T22:05:49.252 DEBUG:boto:auth_path=/vwnfns-51/obj1 2024-04-13T22:05:49.252 DEBUG:boto:Method: GET 2024-04-13T22:05:49.253 DEBUG:boto:Path: /vwnfns-51/obj1 2024-04-13T22:05:49.253 DEBUG:boto:Data: 2024-04-13T22:05:49.253 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.253 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.253 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.253 DEBUG:boto:Params: {} 2024-04-13T22:05:49.253 DEBUG:boto:Token: None 2024-04-13T22:05:49.253 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj1 2024-04-13T22:05:49.253 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q4JN5XHUoc/4/pOEpBetli4hr+Q= 2024-04-13T22:05:49.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4JN5XHUoc/4/pOEpBetli4hr+Q=', 'Content-Length': '0'} 2024-04-13T22:05:49.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaefeb4faec77699-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.255 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.255 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.255 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.255 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.255 DEBUG:boto:Method: GET 2024-04-13T22:05:49.255 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.256 DEBUG:boto:Data: 2024-04-13T22:05:49.256 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.256 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.256 DEBUG:boto:Params: {} 2024-04-13T22:05:49.256 DEBUG:boto:Token: None 2024-04-13T22:05:49.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2c37e2716142702-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.258 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.259 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.259 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.259 DEBUG:boto:path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.259 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.259 DEBUG:boto:Method: GET 2024-04-13T22:05:49.259 DEBUG:boto:Path: /vwnfns-51/?prefix=obj1 2024-04-13T22:05:49.259 DEBUG:boto:Data: 2024-04-13T22:05:49.259 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.259 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.260 DEBUG:boto:Params: {} 2024-04-13T22:05:49.260 DEBUG:boto:Token: None 2024-04-13T22:05:49.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000463a296b1003088f-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.263 DEBUG:boto:b'vwnfns-51obj11000falseobj12024-04-13T22:04:24.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:05:49.264 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.264 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.264 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.264 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.264 DEBUG:boto:Method: GET 2024-04-13T22:05:49.264 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.264 DEBUG:boto:Data: 2024-04-13T22:05:49.264 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.264 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.264 DEBUG:boto:Params: {} 2024-04-13T22:05:49.265 DEBUG:boto:Token: None 2024-04-13T22:05:49.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg= 2024-04-13T22:05:49.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg=', 'Content-Length': '0'} 2024-04-13T22:05:49.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cf64395a3c713c8-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.267 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.267 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.267 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.267 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.267 DEBUG:boto:Method: GET 2024-04-13T22:05:49.267 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.267 DEBUG:boto:Data: 2024-04-13T22:05:49.267 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.267 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.268 DEBUG:boto:Params: {} 2024-04-13T22:05:49.268 DEBUG:boto:Token: None 2024-04-13T22:05:49.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg= 2024-04-13T22:05:49.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg=', 'Content-Length': '0'} 2024-04-13T22:05:49.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000594e5b10e941681d-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.310 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.311 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.311 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.311 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.311 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.311 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.311 DEBUG:boto:Data: 2024-04-13T22:05:49.311 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.311 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.312 DEBUG:boto:Params: {} 2024-04-13T22:05:49.312 DEBUG:boto:Token: None 2024-04-13T22:05:49.312 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JJu1x9At06qhYlKC0f+mq/9uzd4= 2024-04-13T22:05:49.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JJu1x9At06qhYlKC0f+mq/9uzd4=', 'Content-Length': '0'} 2024-04-13T22:05:49.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faeac78c2bfe67a0-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.314 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.314 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.315 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.315 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.315 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.315 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.315 DEBUG:boto:Data: 2024-04-13T22:05:49.315 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.315 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.315 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.315 DEBUG:boto:Params: {} 2024-04-13T22:05:49.315 DEBUG:boto:Token: None 2024-04-13T22:05:49.316 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JJu1x9At06qhYlKC0f+mq/9uzd4= 2024-04-13T22:05:49.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JJu1x9At06qhYlKC0f+mq/9uzd4=', 'Content-Length': '0'} 2024-04-13T22:05:49.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc787b98c21d959-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:05:49.318 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.318 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.318 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.318 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.318 DEBUG:boto:Method: GET 2024-04-13T22:05:49.318 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.318 DEBUG:boto:Data: 2024-04-13T22:05:49.319 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.319 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.319 DEBUG:boto:Params: {} 2024-04-13T22:05:49.319 DEBUG:boto:Token: None 2024-04-13T22:05:49.319 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg= 2024-04-13T22:05:49.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg=', 'Content-Length': '0'} 2024-04-13T22:05:49.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4a7aa8ec4c1e4f4-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.321 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.321 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.322 DEBUG:boto:path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.322 DEBUG:boto:auth_path=/vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.322 DEBUG:boto:Method: GET 2024-04-13T22:05:49.322 DEBUG:boto:Path: /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.322 DEBUG:boto:Data: 2024-04-13T22:05:49.322 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.322 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.322 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.322 DEBUG:boto:Params: {} 2024-04-13T22:05:49.322 DEBUG:boto:Token: None 2024-04-13T22:05:49.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2?versionId=null 2024-04-13T22:05:49.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg= 2024-04-13T22:05:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMQSzTdmsadJNjM4pDjnBtszCKg=', 'Content-Length': '0'} 2024-04-13T22:05:49.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f56bf3f7316995-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.325 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.325 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.325 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.325 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.325 DEBUG:boto:Method: GET 2024-04-13T22:05:49.325 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.325 DEBUG:boto:Data: 2024-04-13T22:05:49.326 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.326 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.326 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.326 DEBUG:boto:Params: {} 2024-04-13T22:05:49.326 DEBUG:boto:Token: None 2024-04-13T22:05:49.326 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14b1a5d8e362a29-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.329 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.329 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.329 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.329 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.330 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.330 DEBUG:boto:Method: GET 2024-04-13T22:05:49.330 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.330 DEBUG:boto:Data: 2024-04-13T22:05:49.330 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.330 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.330 DEBUG:boto:Params: {} 2024-04-13T22:05:49.330 DEBUG:boto:Token: None 2024-04-13T22:05:49.330 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.331 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a53a0cf67770722-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.333 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.333 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.333 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.333 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.334 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:05:49.334 DEBUG:boto:Data: 2024-04-13T22:05:49.334 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.334 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.334 DEBUG:boto:Params: {} 2024-04-13T22:05:49.334 DEBUG:boto:Token: None 2024-04-13T22:05:49.334 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2 2024-04-13T22:05:49.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1YhhpKgo6Ly7+ZHc7j8uZFfak5o= 2024-04-13T22:05:49.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1YhhpKgo6Ly7+ZHc7j8uZFfak5o=', 'Content-Length': '0'} 2024-04-13T22:05:49.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c67129c7452dc1d-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.336 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.336 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.336 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.337 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:05:49.337 DEBUG:boto:Data: 2024-04-13T22:05:49.337 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.337 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.337 DEBUG:boto:Params: {} 2024-04-13T22:05:49.337 DEBUG:boto:Token: None 2024-04-13T22:05:49.337 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2 2024-04-13T22:05:49.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1YhhpKgo6Ly7+ZHc7j8uZFfak5o= 2024-04-13T22:05:49.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1YhhpKgo6Ly7+ZHc7j8uZFfak5o=', 'Content-Length': '0'} 2024-04-13T22:05:49.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002484428dbc4a4e27-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-13T22:05:49.339 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.339 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.339 DEBUG:boto:Method: GET 2024-04-13T22:05:49.340 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:05:49.340 DEBUG:boto:Data: 2024-04-13T22:05:49.340 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.340 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.340 DEBUG:boto:Params: {} 2024-04-13T22:05:49.340 DEBUG:boto:Token: None 2024-04-13T22:05:49.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2 2024-04-13T22:05:49.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5bm//tYDY0PE6Q12ui29vBRSLXc= 2024-04-13T22:05:49.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5bm//tYDY0PE6Q12ui29vBRSLXc=', 'Content-Length': '0'} 2024-04-13T22:05:49.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e43bc9921171b672-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.342 DEBUG:boto:path=/vwnfns-51/obj2 2024-04-13T22:05:49.343 DEBUG:boto:auth_path=/vwnfns-51/obj2 2024-04-13T22:05:49.343 DEBUG:boto:Method: GET 2024-04-13T22:05:49.343 DEBUG:boto:Path: /vwnfns-51/obj2 2024-04-13T22:05:49.343 DEBUG:boto:Data: 2024-04-13T22:05:49.343 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.343 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.343 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.343 DEBUG:boto:Params: {} 2024-04-13T22:05:49.343 DEBUG:boto:Token: None 2024-04-13T22:05:49.343 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj2 2024-04-13T22:05:49.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5bm//tYDY0PE6Q12ui29vBRSLXc= 2024-04-13T22:05:49.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5bm//tYDY0PE6Q12ui29vBRSLXc=', 'Content-Length': '0'} 2024-04-13T22:05:49.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5739e789d863fa4-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.346 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.346 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.346 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.346 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.346 DEBUG:boto:Method: GET 2024-04-13T22:05:49.346 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.346 DEBUG:boto:Data: 2024-04-13T22:05:49.346 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.346 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.347 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.347 DEBUG:boto:Params: {} 2024-04-13T22:05:49.347 DEBUG:boto:Token: None 2024-04-13T22:05:49.347 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.347 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf062b017fcd97c6-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.390 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.391 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.391 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.391 DEBUG:boto:path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.391 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.391 DEBUG:boto:Method: GET 2024-04-13T22:05:49.391 DEBUG:boto:Path: /vwnfns-51/?prefix=obj2 2024-04-13T22:05:49.391 DEBUG:boto:Data: 2024-04-13T22:05:49.392 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.392 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.392 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.392 DEBUG:boto:Params: {} 2024-04-13T22:05:49.392 DEBUG:boto:Token: None 2024-04-13T22:05:49.392 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d4e53074d5d174-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.395 DEBUG:boto:b'vwnfns-51obj21000falseobj22024-04-13T22:04:24.882Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:05:49.395 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.395 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.396 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.396 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.396 DEBUG:boto:Method: GET 2024-04-13T22:05:49.396 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.396 DEBUG:boto:Data: 2024-04-13T22:05:49.396 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.396 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.396 DEBUG:boto:Params: {} 2024-04-13T22:05:49.396 DEBUG:boto:Token: None 2024-04-13T22:05:49.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA= 2024-04-13T22:05:49.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA=', 'Content-Length': '0'} 2024-04-13T22:05:49.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020a3da1a47f8fe4a-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.399 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.399 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.399 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.399 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.399 DEBUG:boto:Method: GET 2024-04-13T22:05:49.399 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.399 DEBUG:boto:Data: 2024-04-13T22:05:49.399 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.400 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.400 DEBUG:boto:Params: {} 2024-04-13T22:05:49.400 DEBUG:boto:Token: None 2024-04-13T22:05:49.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA= 2024-04-13T22:05:49.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA=', 'Content-Length': '0'} 2024-04-13T22:05:49.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aef60ed27cc9ef7-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.442 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.443 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.443 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.443 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.443 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.443 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.443 DEBUG:boto:Data: 2024-04-13T22:05:49.443 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.443 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.443 DEBUG:boto:Params: {} 2024-04-13T22:05:49.444 DEBUG:boto:Token: None 2024-04-13T22:05:49.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+5OTcR6HRp0htV4G1GSDeuFQcZ8= 2024-04-13T22:05:49.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+5OTcR6HRp0htV4G1GSDeuFQcZ8=', 'Content-Length': '0'} 2024-04-13T22:05:49.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cefd2b09624e017-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.446 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.446 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.446 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.446 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.446 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.446 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.446 DEBUG:boto:Data: 2024-04-13T22:05:49.447 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.447 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.447 DEBUG:boto:Params: {} 2024-04-13T22:05:49.447 DEBUG:boto:Token: None 2024-04-13T22:05:49.447 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+5OTcR6HRp0htV4G1GSDeuFQcZ8= 2024-04-13T22:05:49.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+5OTcR6HRp0htV4G1GSDeuFQcZ8=', 'Content-Length': '0'} 2024-04-13T22:05:49.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000780c2e1fef2178a1-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:05:49.449 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.449 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.449 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.449 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.449 DEBUG:boto:Method: GET 2024-04-13T22:05:49.449 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.450 DEBUG:boto:Data: 2024-04-13T22:05:49.450 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.450 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.450 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.450 DEBUG:boto:Params: {} 2024-04-13T22:05:49.450 DEBUG:boto:Token: None 2024-04-13T22:05:49.450 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.450 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA= 2024-04-13T22:05:49.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA=', 'Content-Length': '0'} 2024-04-13T22:05:49.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc360cd45033e54b-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.452 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.452 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.452 DEBUG:boto:path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.452 DEBUG:boto:auth_path=/vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.452 DEBUG:boto:Method: GET 2024-04-13T22:05:49.453 DEBUG:boto:Path: /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.453 DEBUG:boto:Data: 2024-04-13T22:05:49.453 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.453 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.453 DEBUG:boto:Params: {} 2024-04-13T22:05:49.453 DEBUG:boto:Token: None 2024-04-13T22:05:49.453 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3?versionId=null 2024-04-13T22:05:49.453 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA= 2024-04-13T22:05:49.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dm2p++QmvYfhuVFcBN/CrQNuglA=', 'Content-Length': '0'} 2024-04-13T22:05:49.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd57e1e4b427bb4-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.455 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.455 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.455 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.456 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.456 DEBUG:boto:Method: GET 2024-04-13T22:05:49.456 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.456 DEBUG:boto:Data: 2024-04-13T22:05:49.456 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.456 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.456 DEBUG:boto:Params: {} 2024-04-13T22:05:49.456 DEBUG:boto:Token: None 2024-04-13T22:05:49.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078f535d4716f113d-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.459 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.459 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.459 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.459 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.459 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.460 DEBUG:boto:Method: GET 2024-04-13T22:05:49.460 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.460 DEBUG:boto:Data: 2024-04-13T22:05:49.460 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.460 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.460 DEBUG:boto:Params: {} 2024-04-13T22:05:49.460 DEBUG:boto:Token: None 2024-04-13T22:05:49.460 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008339467b3da1a2b2-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.463 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.463 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.463 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.463 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.464 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:05:49.464 DEBUG:boto:Data: 2024-04-13T22:05:49.464 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.464 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.464 DEBUG:boto:Params: {} 2024-04-13T22:05:49.464 DEBUG:boto:Token: None 2024-04-13T22:05:49.464 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3 2024-04-13T22:05:49.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eFVwwZGUGXeosZGYY7WfRpyuojM= 2024-04-13T22:05:49.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eFVwwZGUGXeosZGYY7WfRpyuojM=', 'Content-Length': '0'} 2024-04-13T22:05:49.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f7dc9bf7899b1ce-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.466 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.466 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.466 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.467 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:05:49.467 DEBUG:boto:Data: 2024-04-13T22:05:49.467 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.467 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.467 DEBUG:boto:Params: {} 2024-04-13T22:05:49.467 DEBUG:boto:Token: None 2024-04-13T22:05:49.467 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3 2024-04-13T22:05:49.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eFVwwZGUGXeosZGYY7WfRpyuojM= 2024-04-13T22:05:49.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eFVwwZGUGXeosZGYY7WfRpyuojM=', 'Content-Length': '0'} 2024-04-13T22:05:49.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8de75e326a4fd9d-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-13T22:05:49.469 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.469 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.469 DEBUG:boto:Method: GET 2024-04-13T22:05:49.470 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:05:49.470 DEBUG:boto:Data: 2024-04-13T22:05:49.470 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.470 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.470 DEBUG:boto:Params: {} 2024-04-13T22:05:49.470 DEBUG:boto:Token: None 2024-04-13T22:05:49.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3 2024-04-13T22:05:49.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8GyRvBe6UaDXasJKeIQrK1ges7w= 2024-04-13T22:05:49.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8GyRvBe6UaDXasJKeIQrK1ges7w=', 'Content-Length': '0'} 2024-04-13T22:05:49.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe4fa30fc3b39ad-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.472 DEBUG:boto:path=/vwnfns-51/obj3 2024-04-13T22:05:49.473 DEBUG:boto:auth_path=/vwnfns-51/obj3 2024-04-13T22:05:49.473 DEBUG:boto:Method: GET 2024-04-13T22:05:49.473 DEBUG:boto:Path: /vwnfns-51/obj3 2024-04-13T22:05:49.473 DEBUG:boto:Data: 2024-04-13T22:05:49.473 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.473 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.473 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.473 DEBUG:boto:Params: {} 2024-04-13T22:05:49.473 DEBUG:boto:Token: None 2024-04-13T22:05:49.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj3 2024-04-13T22:05:49.474 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8GyRvBe6UaDXasJKeIQrK1ges7w= 2024-04-13T22:05:49.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8GyRvBe6UaDXasJKeIQrK1ges7w=', 'Content-Length': '0'} 2024-04-13T22:05:49.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347ba7aaf28bb18f-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.518 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.519 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.519 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.519 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.519 DEBUG:boto:Method: GET 2024-04-13T22:05:49.519 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.519 DEBUG:boto:Data: 2024-04-13T22:05:49.519 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.519 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.519 DEBUG:boto:Params: {} 2024-04-13T22:05:49.520 DEBUG:boto:Token: None 2024-04-13T22:05:49.520 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e29d68468ddff276-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.522 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.523 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.523 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.523 DEBUG:boto:path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.523 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.523 DEBUG:boto:Method: GET 2024-04-13T22:05:49.523 DEBUG:boto:Path: /vwnfns-51/?prefix=obj3 2024-04-13T22:05:49.524 DEBUG:boto:Data: 2024-04-13T22:05:49.524 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.524 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.524 DEBUG:boto:Params: {} 2024-04-13T22:05:49.524 DEBUG:boto:Token: None 2024-04-13T22:05:49.524 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000042c1988c5d080e2-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.526 DEBUG:boto:b'vwnfns-51obj31000falseobj32024-04-13T22:04:24.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:05:49.527 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.527 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.527 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.527 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.527 DEBUG:boto:Method: GET 2024-04-13T22:05:49.528 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.528 DEBUG:boto:Data: 2024-04-13T22:05:49.528 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.528 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.528 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.528 DEBUG:boto:Params: {} 2024-04-13T22:05:49.528 DEBUG:boto:Token: None 2024-04-13T22:05:49.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg= 2024-04-13T22:05:49.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg=', 'Content-Length': '0'} 2024-04-13T22:05:49.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000100562fde336e981-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.570 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.571 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.571 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.571 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.571 DEBUG:boto:Method: GET 2024-04-13T22:05:49.571 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.571 DEBUG:boto:Data: 2024-04-13T22:05:49.571 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.571 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.571 DEBUG:boto:Params: {} 2024-04-13T22:05:49.572 DEBUG:boto:Token: None 2024-04-13T22:05:49.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg= 2024-04-13T22:05:49.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg=', 'Content-Length': '0'} 2024-04-13T22:05:49.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad71c97c5d01b296-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.574 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.574 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.574 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.574 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.574 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.575 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.575 DEBUG:boto:Data: 2024-04-13T22:05:49.575 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.575 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.575 DEBUG:boto:Params: {} 2024-04-13T22:05:49.575 DEBUG:boto:Token: None 2024-04-13T22:05:49.575 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TfTJq4Nu+P1pvaBU6QO800CQnOk= 2024-04-13T22:05:49.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TfTJq4Nu+P1pvaBU6QO800CQnOk=', 'Content-Length': '0'} 2024-04-13T22:05:49.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003654881a7615a22f-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.577 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.577 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.577 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.577 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.578 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.578 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.578 DEBUG:boto:Data: 2024-04-13T22:05:49.578 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.578 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.578 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.578 DEBUG:boto:Params: {} 2024-04-13T22:05:49.578 DEBUG:boto:Token: None 2024-04-13T22:05:49.578 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.579 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TfTJq4Nu+P1pvaBU6QO800CQnOk= 2024-04-13T22:05:49.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TfTJq4Nu+P1pvaBU6QO800CQnOk=', 'Content-Length': '0'} 2024-04-13T22:05:49.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019da586034574990-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:05:49.580 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.580 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.581 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.581 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.581 DEBUG:boto:Method: GET 2024-04-13T22:05:49.581 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.581 DEBUG:boto:Data: 2024-04-13T22:05:49.581 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.581 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.581 DEBUG:boto:Params: {} 2024-04-13T22:05:49.581 DEBUG:boto:Token: None 2024-04-13T22:05:49.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg= 2024-04-13T22:05:49.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg=', 'Content-Length': '0'} 2024-04-13T22:05:49.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087f49627f58c368f-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.584 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.584 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.584 DEBUG:boto:path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.584 DEBUG:boto:auth_path=/vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.584 DEBUG:boto:Method: GET 2024-04-13T22:05:49.584 DEBUG:boto:Path: /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.584 DEBUG:boto:Data: 2024-04-13T22:05:49.584 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.584 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.585 DEBUG:boto:Params: {} 2024-04-13T22:05:49.585 DEBUG:boto:Token: None 2024-04-13T22:05:49.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4?versionId=null 2024-04-13T22:05:49.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg= 2024-04-13T22:05:49.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WYWjdSzL1TDDm1JbU/Qmo48jQSg=', 'Content-Length': '0'} 2024-04-13T22:05:49.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5eb1ddd264224cb-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.587 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.587 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.587 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.587 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.587 DEBUG:boto:Method: GET 2024-04-13T22:05:49.587 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.587 DEBUG:boto:Data: 2024-04-13T22:05:49.588 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.588 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.588 DEBUG:boto:Params: {} 2024-04-13T22:05:49.588 DEBUG:boto:Token: None 2024-04-13T22:05:49.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000140cdbfee04b6aed-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.590 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.591 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.591 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.591 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.591 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.591 DEBUG:boto:Method: GET 2024-04-13T22:05:49.591 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.591 DEBUG:boto:Data: 2024-04-13T22:05:49.592 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.592 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.592 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.592 DEBUG:boto:Params: {} 2024-04-13T22:05:49.592 DEBUG:boto:Token: None 2024-04-13T22:05:49.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c75db36a93ff8bf-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.595 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.595 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.595 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.595 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.595 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:05:49.595 DEBUG:boto:Data: 2024-04-13T22:05:49.596 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.596 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.596 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.596 DEBUG:boto:Params: {} 2024-04-13T22:05:49.596 DEBUG:boto:Token: None 2024-04-13T22:05:49.596 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4 2024-04-13T22:05:49.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Qsyw7eM/l2lk3hqAefgPamxCbU= 2024-04-13T22:05:49.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Qsyw7eM/l2lk3hqAefgPamxCbU=', 'Content-Length': '0'} 2024-04-13T22:05:49.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003958efb1965ddbed-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.598 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.598 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.598 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.598 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:05:49.598 DEBUG:boto:Data: 2024-04-13T22:05:49.598 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.599 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.599 DEBUG:boto:Params: {} 2024-04-13T22:05:49.599 DEBUG:boto:Token: None 2024-04-13T22:05:49.599 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4 2024-04-13T22:05:49.599 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Qsyw7eM/l2lk3hqAefgPamxCbU= 2024-04-13T22:05:49.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Qsyw7eM/l2lk3hqAefgPamxCbU=', 'Content-Length': '0'} 2024-04-13T22:05:49.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a7f500b6ec3b4b6-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-13T22:05:49.601 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.601 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.601 DEBUG:boto:Method: GET 2024-04-13T22:05:49.602 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:05:49.602 DEBUG:boto:Data: 2024-04-13T22:05:49.602 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.602 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.602 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.602 DEBUG:boto:Params: {} 2024-04-13T22:05:49.602 DEBUG:boto:Token: None 2024-04-13T22:05:49.602 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4 2024-04-13T22:05:49.602 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L7oTBkypTKCBaeJh+YaffV/v16w= 2024-04-13T22:05:49.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L7oTBkypTKCBaeJh+YaffV/v16w=', 'Content-Length': '0'} 2024-04-13T22:05:49.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e5d617c6fe23a77-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.604 DEBUG:boto:path=/vwnfns-51/obj4 2024-04-13T22:05:49.604 DEBUG:boto:auth_path=/vwnfns-51/obj4 2024-04-13T22:05:49.604 DEBUG:boto:Method: GET 2024-04-13T22:05:49.605 DEBUG:boto:Path: /vwnfns-51/obj4 2024-04-13T22:05:49.605 DEBUG:boto:Data: 2024-04-13T22:05:49.605 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.605 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.605 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.605 DEBUG:boto:Params: {} 2024-04-13T22:05:49.605 DEBUG:boto:Token: None 2024-04-13T22:05:49.605 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj4 2024-04-13T22:05:49.605 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L7oTBkypTKCBaeJh+YaffV/v16w= 2024-04-13T22:05:49.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L7oTBkypTKCBaeJh+YaffV/v16w=', 'Content-Length': '0'} 2024-04-13T22:05:49.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:04:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4831cc8a5a3a164-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.650 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.651 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.651 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.651 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.651 DEBUG:boto:Method: GET 2024-04-13T22:05:49.651 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.651 DEBUG:boto:Data: 2024-04-13T22:05:49.651 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.652 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.652 DEBUG:boto:Params: {} 2024-04-13T22:05:49.653 DEBUG:boto:Token: None 2024-04-13T22:05:49.653 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57fc5d4592db449-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.655 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.656 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.656 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.656 DEBUG:boto:path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.656 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.656 DEBUG:boto:Method: GET 2024-04-13T22:05:49.656 DEBUG:boto:Path: /vwnfns-51/?prefix=obj4 2024-04-13T22:05:49.656 DEBUG:boto:Data: 2024-04-13T22:05:49.656 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.657 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.657 DEBUG:boto:Params: {} 2024-04-13T22:05:49.657 DEBUG:boto:Token: None 2024-04-13T22:05:49.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a329d6ce13063619-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.659 DEBUG:boto:b'vwnfns-51obj41000falseobj42024-04-13T22:04:24.903Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:05:49.660 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.660 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.660 DEBUG:boto:path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.660 DEBUG:boto:auth_path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.660 DEBUG:boto:Method: GET 2024-04-13T22:05:49.660 DEBUG:boto:Path: /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.661 DEBUG:boto:Data: 2024-04-13T22:05:49.661 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.661 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.661 DEBUG:boto:Params: {} 2024-04-13T22:05:49.661 DEBUG:boto:Token: None 2024-04-13T22:05:49.661 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.661 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU= 2024-04-13T22:05:49.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU=', 'Content-Length': '0'} 2024-04-13T22:05:49.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000181f9b022575709e-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.706 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.707 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.707 DEBUG:boto:path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.707 DEBUG:boto:auth_path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.707 DEBUG:boto:Method: GET 2024-04-13T22:05:49.707 DEBUG:boto:Path: /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.707 DEBUG:boto:Data: 2024-04-13T22:05:49.707 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.707 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.708 DEBUG:boto:Params: {} 2024-04-13T22:05:49.708 DEBUG:boto:Token: None 2024-04-13T22:05:49.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU= 2024-04-13T22:05:49.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU=', 'Content-Length': '0'} 2024-04-13T22:05:49.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c49f9110fd5efc16-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.710 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.710 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.711 DEBUG:boto:path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.711 DEBUG:boto:auth_path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.711 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.711 DEBUG:boto:Path: /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.711 DEBUG:boto:Data: 2024-04-13T22:05:49.711 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.711 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.711 DEBUG:boto:Params: {} 2024-04-13T22:05:49.711 DEBUG:boto:Token: None 2024-04-13T22:05:49.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M7gbLUKwX0JfDJbOZCzCFD0YQl4= 2024-04-13T22:05:49.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M7gbLUKwX0JfDJbOZCzCFD0YQl4=', 'Content-Length': '0'} 2024-04-13T22:05:49.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd41f96777b5e454-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.713 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.713 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.714 DEBUG:boto:path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.714 DEBUG:boto:auth_path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.714 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.714 DEBUG:boto:Path: /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.714 DEBUG:boto:Data: 2024-04-13T22:05:49.714 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.714 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.714 DEBUG:boto:Params: {} 2024-04-13T22:05:49.714 DEBUG:boto:Token: None 2024-04-13T22:05:49.715 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M7gbLUKwX0JfDJbOZCzCFD0YQl4= 2024-04-13T22:05:49.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M7gbLUKwX0JfDJbOZCzCFD0YQl4=', 'Content-Length': '0'} 2024-04-13T22:05:49.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ba7236a056fd8a7-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:05:49.720 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.720 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.720 DEBUG:boto:path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.720 DEBUG:boto:auth_path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.720 DEBUG:boto:Method: GET 2024-04-13T22:05:49.720 DEBUG:boto:Path: /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.720 DEBUG:boto:Data: 2024-04-13T22:05:49.720 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.720 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.721 DEBUG:boto:Params: {} 2024-04-13T22:05:49.721 DEBUG:boto:Token: None 2024-04-13T22:05:49.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU= 2024-04-13T22:05:49.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU=', 'Content-Length': '0'} 2024-04-13T22:05:49.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009002475acb2f558-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.723 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.723 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.723 DEBUG:boto:path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.723 DEBUG:boto:auth_path=/vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.723 DEBUG:boto:Method: GET 2024-04-13T22:05:49.723 DEBUG:boto:Path: /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.723 DEBUG:boto:Data: 2024-04-13T22:05:49.723 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.724 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.724 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.724 DEBUG:boto:Params: {} 2024-04-13T22:05:49.724 DEBUG:boto:Token: None 2024-04-13T22:05:49.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5?versionId=null 2024-04-13T22:05:49.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU= 2024-04-13T22:05:49.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zkuIgD3i+i+GgYUJjmo9Fsz8mbU=', 'Content-Length': '0'} 2024-04-13T22:05:49.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb6dda5d648c488-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.726 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.726 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.726 DEBUG:boto:path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.726 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.726 DEBUG:boto:Method: GET 2024-04-13T22:05:49.727 DEBUG:boto:Path: /vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.727 DEBUG:boto:Data: 2024-04-13T22:05:49.727 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.727 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.727 DEBUG:boto:Params: {} 2024-04-13T22:05:49.727 DEBUG:boto:Token: None 2024-04-13T22:05:49.727 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc13a893ecf64b7b-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.730 DEBUG:boto:b'vwnfns-51obj51000falseobj52024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.730 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.730 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.731 DEBUG:boto:path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.731 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.731 DEBUG:boto:Method: GET 2024-04-13T22:05:49.731 DEBUG:boto:Path: /vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.731 DEBUG:boto:Data: 2024-04-13T22:05:49.731 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.731 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.731 DEBUG:boto:Params: {} 2024-04-13T22:05:49.731 DEBUG:boto:Token: None 2024-04-13T22:05:49.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c2c4ec69154f41-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.734 DEBUG:boto:b'vwnfns-51obj51000falseobj52024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.734 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.734 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.734 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.734 DEBUG:boto:Path: /vwnfns-51/obj5 2024-04-13T22:05:49.735 DEBUG:boto:Data: 2024-04-13T22:05:49.735 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.735 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.735 DEBUG:boto:Params: {} 2024-04-13T22:05:49.735 DEBUG:boto:Token: None 2024-04-13T22:05:49.735 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5 2024-04-13T22:05:49.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6mMlbr62RCA5bZiBfMq+AXyxkmg= 2024-04-13T22:05:49.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6mMlbr62RCA5bZiBfMq+AXyxkmg=', 'Content-Length': '0'} 2024-04-13T22:05:49.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a71b69a354fdaab6-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.737 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.737 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.737 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.737 DEBUG:boto:Path: /vwnfns-51/obj5 2024-04-13T22:05:49.738 DEBUG:boto:Data: 2024-04-13T22:05:49.738 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.738 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.738 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.738 DEBUG:boto:Params: {} 2024-04-13T22:05:49.738 DEBUG:boto:Token: None 2024-04-13T22:05:49.739 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5 2024-04-13T22:05:49.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6mMlbr62RCA5bZiBfMq+AXyxkmg= 2024-04-13T22:05:49.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6mMlbr62RCA5bZiBfMq+AXyxkmg=', 'Content-Length': '0'} 2024-04-13T22:05:49.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068a85d10684c0bdd-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-13T22:05:49.741 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.741 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.742 DEBUG:boto:Method: GET 2024-04-13T22:05:49.742 DEBUG:boto:Path: /vwnfns-51/obj5 2024-04-13T22:05:49.742 DEBUG:boto:Data: 2024-04-13T22:05:49.742 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.742 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.742 DEBUG:boto:Params: {} 2024-04-13T22:05:49.743 DEBUG:boto:Token: None 2024-04-13T22:05:49.743 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5 2024-04-13T22:05:49.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4SUlA3IAP5Z0HksC7hbcKnb+KIM= 2024-04-13T22:05:49.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4SUlA3IAP5Z0HksC7hbcKnb+KIM=', 'Content-Length': '0'} 2024-04-13T22:05:49.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dc607acf12e2c62-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.745 DEBUG:boto:path=/vwnfns-51/obj5 2024-04-13T22:05:49.745 DEBUG:boto:auth_path=/vwnfns-51/obj5 2024-04-13T22:05:49.745 DEBUG:boto:Method: GET 2024-04-13T22:05:49.745 DEBUG:boto:Path: /vwnfns-51/obj5 2024-04-13T22:05:49.745 DEBUG:boto:Data: 2024-04-13T22:05:49.745 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.746 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.746 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.746 DEBUG:boto:Params: {} 2024-04-13T22:05:49.746 DEBUG:boto:Token: None 2024-04-13T22:05:49.746 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj5 2024-04-13T22:05:49.747 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4SUlA3IAP5Z0HksC7hbcKnb+KIM= 2024-04-13T22:05:49.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4SUlA3IAP5Z0HksC7hbcKnb+KIM=', 'Content-Length': '0'} 2024-04-13T22:05:49.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d40796f1ac72a048-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.749 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.749 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.750 DEBUG:boto:path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.750 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.750 DEBUG:boto:Method: GET 2024-04-13T22:05:49.750 DEBUG:boto:Path: /vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.751 DEBUG:boto:Data: 2024-04-13T22:05:49.751 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.751 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.751 DEBUG:boto:Params: {} 2024-04-13T22:05:49.752 DEBUG:boto:Token: None 2024-04-13T22:05:49.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f469bbecc1c760-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.755 DEBUG:boto:b'vwnfns-51obj51000falseobj52024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.756 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.756 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.756 DEBUG:boto:path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.756 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.757 DEBUG:boto:Method: GET 2024-04-13T22:05:49.757 DEBUG:boto:Path: /vwnfns-51/?prefix=obj5 2024-04-13T22:05:49.757 DEBUG:boto:Data: 2024-04-13T22:05:49.757 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.758 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.758 DEBUG:boto:Params: {} 2024-04-13T22:05:49.758 DEBUG:boto:Token: None 2024-04-13T22:05:49.758 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040eb142e8f48ea20-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.761 DEBUG:boto:b'vwnfns-51obj51000falseobj52024-04-13T22:05:07.588Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:05:49.762 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.762 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.763 DEBUG:boto:path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.763 DEBUG:boto:auth_path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.763 DEBUG:boto:Method: GET 2024-04-13T22:05:49.763 DEBUG:boto:Path: /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.763 DEBUG:boto:Data: 2024-04-13T22:05:49.763 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.764 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.764 DEBUG:boto:Params: {} 2024-04-13T22:05:49.764 DEBUG:boto:Token: None 2024-04-13T22:05:49.764 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs= 2024-04-13T22:05:49.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs=', 'Content-Length': '0'} 2024-04-13T22:05:49.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eff2df6f248e2b58-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.810 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.811 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.811 DEBUG:boto:path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.811 DEBUG:boto:auth_path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.811 DEBUG:boto:Method: GET 2024-04-13T22:05:49.812 DEBUG:boto:Path: /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.812 DEBUG:boto:Data: 2024-04-13T22:05:49.812 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.812 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.812 DEBUG:boto:Params: {} 2024-04-13T22:05:49.812 DEBUG:boto:Token: None 2024-04-13T22:05:49.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs= 2024-04-13T22:05:49.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs=', 'Content-Length': '0'} 2024-04-13T22:05:49.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bf5c6e2b9d2b97b-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.815 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.816 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.816 DEBUG:boto:path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.816 DEBUG:boto:auth_path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.816 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.816 DEBUG:boto:Path: /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.816 DEBUG:boto:Data: 2024-04-13T22:05:49.817 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.817 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.817 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.817 DEBUG:boto:Params: {} 2024-04-13T22:05:49.817 DEBUG:boto:Token: None 2024-04-13T22:05:49.818 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.818 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qDL/AllYTNzOfxpa56drbv4EnT8= 2024-04-13T22:05:49.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qDL/AllYTNzOfxpa56drbv4EnT8=', 'Content-Length': '0'} 2024-04-13T22:05:49.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9a20a8ef441fc1e-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.820 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.820 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.821 DEBUG:boto:path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.821 DEBUG:boto:auth_path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.821 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.821 DEBUG:boto:Path: /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.821 DEBUG:boto:Data: 2024-04-13T22:05:49.821 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.821 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.821 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.821 DEBUG:boto:Params: {} 2024-04-13T22:05:49.822 DEBUG:boto:Token: None 2024-04-13T22:05:49.822 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.822 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qDL/AllYTNzOfxpa56drbv4EnT8= 2024-04-13T22:05:49.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qDL/AllYTNzOfxpa56drbv4EnT8=', 'Content-Length': '0'} 2024-04-13T22:05:49.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018e7422009afd871-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:05:49.825 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.825 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.825 DEBUG:boto:path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.826 DEBUG:boto:auth_path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.826 DEBUG:boto:Method: GET 2024-04-13T22:05:49.826 DEBUG:boto:Path: /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.826 DEBUG:boto:Data: 2024-04-13T22:05:49.826 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.826 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.826 DEBUG:boto:Params: {} 2024-04-13T22:05:49.826 DEBUG:boto:Token: None 2024-04-13T22:05:49.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs= 2024-04-13T22:05:49.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs=', 'Content-Length': '0'} 2024-04-13T22:05:49.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fab9cc038d1e96d3-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.828 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.829 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.829 DEBUG:boto:path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.829 DEBUG:boto:auth_path=/vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.829 DEBUG:boto:Method: GET 2024-04-13T22:05:49.829 DEBUG:boto:Path: /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.829 DEBUG:boto:Data: 2024-04-13T22:05:49.829 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.829 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.829 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.829 DEBUG:boto:Params: {} 2024-04-13T22:05:49.830 DEBUG:boto:Token: None 2024-04-13T22:05:49.830 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6?versionId=null 2024-04-13T22:05:49.830 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs= 2024-04-13T22:05:49.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNXZ7crgWkGbssOGp//6Z/1/HQs=', 'Content-Length': '0'} 2024-04-13T22:05:49.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000102e540c6267035c-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.878 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.879 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.879 DEBUG:boto:path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.879 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.879 DEBUG:boto:Method: GET 2024-04-13T22:05:49.879 DEBUG:boto:Path: /vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.879 DEBUG:boto:Data: 2024-04-13T22:05:49.879 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.879 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.879 DEBUG:boto:Params: {} 2024-04-13T22:05:49.880 DEBUG:boto:Token: None 2024-04-13T22:05:49.880 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d59cdf7805b66c49-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.882 DEBUG:boto:b'vwnfns-51obj61000falseobj62024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.883 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.883 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.883 DEBUG:boto:path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.883 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.883 DEBUG:boto:Method: GET 2024-04-13T22:05:49.883 DEBUG:boto:Path: /vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.883 DEBUG:boto:Data: 2024-04-13T22:05:49.884 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.884 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.884 DEBUG:boto:Params: {} 2024-04-13T22:05:49.884 DEBUG:boto:Token: None 2024-04-13T22:05:49.884 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058bf380700f328a6-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.886 DEBUG:boto:b'vwnfns-51obj61000falseobj62024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.887 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.887 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.887 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.887 DEBUG:boto:Path: /vwnfns-51/obj6 2024-04-13T22:05:49.887 DEBUG:boto:Data: 2024-04-13T22:05:49.887 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.888 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.888 DEBUG:boto:Params: {} 2024-04-13T22:05:49.888 DEBUG:boto:Token: None 2024-04-13T22:05:49.888 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6 2024-04-13T22:05:49.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uVamXawh4XJJVqlxelMjKV2+iqc= 2024-04-13T22:05:49.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uVamXawh4XJJVqlxelMjKV2+iqc=', 'Content-Length': '0'} 2024-04-13T22:05:49.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008323ca4e38e0b69a-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.890 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.890 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.890 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.890 DEBUG:boto:Path: /vwnfns-51/obj6 2024-04-13T22:05:49.890 DEBUG:boto:Data: 2024-04-13T22:05:49.890 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.890 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.890 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.890 DEBUG:boto:Params: {} 2024-04-13T22:05:49.891 DEBUG:boto:Token: None 2024-04-13T22:05:49.891 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6 2024-04-13T22:05:49.891 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uVamXawh4XJJVqlxelMjKV2+iqc= 2024-04-13T22:05:49.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uVamXawh4XJJVqlxelMjKV2+iqc=', 'Content-Length': '0'} 2024-04-13T22:05:49.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f918f725c7b3b4a1-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-13T22:05:49.893 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.893 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.893 DEBUG:boto:Method: GET 2024-04-13T22:05:49.893 DEBUG:boto:Path: /vwnfns-51/obj6 2024-04-13T22:05:49.893 DEBUG:boto:Data: 2024-04-13T22:05:49.893 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.893 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.893 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.893 DEBUG:boto:Params: {} 2024-04-13T22:05:49.894 DEBUG:boto:Token: None 2024-04-13T22:05:49.894 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6 2024-04-13T22:05:49.894 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FPmVKX1QzbBzeH82G+9MO5V7Upo= 2024-04-13T22:05:49.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FPmVKX1QzbBzeH82G+9MO5V7Upo=', 'Content-Length': '0'} 2024-04-13T22:05:49.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6d6eca0353a052a-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.896 DEBUG:boto:path=/vwnfns-51/obj6 2024-04-13T22:05:49.896 DEBUG:boto:auth_path=/vwnfns-51/obj6 2024-04-13T22:05:49.896 DEBUG:boto:Method: GET 2024-04-13T22:05:49.896 DEBUG:boto:Path: /vwnfns-51/obj6 2024-04-13T22:05:49.896 DEBUG:boto:Data: 2024-04-13T22:05:49.896 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.896 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.896 DEBUG:boto:Params: {} 2024-04-13T22:05:49.897 DEBUG:boto:Token: None 2024-04-13T22:05:49.897 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj6 2024-04-13T22:05:49.897 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FPmVKX1QzbBzeH82G+9MO5V7Upo= 2024-04-13T22:05:49.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FPmVKX1QzbBzeH82G+9MO5V7Upo=', 'Content-Length': '0'} 2024-04-13T22:05:49.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f418386cf3fe4cc-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.898 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.899 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.899 DEBUG:boto:path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.899 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.899 DEBUG:boto:Method: GET 2024-04-13T22:05:49.899 DEBUG:boto:Path: /vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.899 DEBUG:boto:Data: 2024-04-13T22:05:49.899 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.899 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.899 DEBUG:boto:Params: {} 2024-04-13T22:05:49.899 DEBUG:boto:Token: None 2024-04-13T22:05:49.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02732831a30f874-00661b01bd-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.902 DEBUG:boto:b'vwnfns-51obj61000falseobj62024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.902 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:49.903 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:49.903 DEBUG:boto:path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.903 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.903 DEBUG:boto:Method: GET 2024-04-13T22:05:49.903 DEBUG:boto:Path: /vwnfns-51/?prefix=obj6 2024-04-13T22:05:49.903 DEBUG:boto:Data: 2024-04-13T22:05:49.903 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.903 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.903 DEBUG:boto:Params: {} 2024-04-13T22:05:49.904 DEBUG:boto:Token: None 2024-04-13T22:05:49.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/ 2024-04-13T22:05:49.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I= 2024-04-13T22:05:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Egphfrzn8+TFapXnA0bNzXn8w6I=', 'Content-Length': '0'} 2024-04-13T22:05:49.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b1fc7c45aa1b1f-00661b01bd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.946 DEBUG:boto:b'vwnfns-51obj61000falseobj62024-04-13T22:05:07.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:49.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:05:49.947 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:49.947 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:49.948 DEBUG:boto:path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.948 DEBUG:boto:auth_path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.948 DEBUG:boto:Method: GET 2024-04-13T22:05:49.948 DEBUG:boto:Path: /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.948 DEBUG:boto:Data: 2024-04-13T22:05:49.948 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.948 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.948 DEBUG:boto:Params: {} 2024-04-13T22:05:49.948 DEBUG:boto:Token: None 2024-04-13T22:05:49.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yz7+4Gtt4aR1BXz/oUBaGKGDZnk= 2024-04-13T22:05:49.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yz7+4Gtt4aR1BXz/oUBaGKGDZnk=', 'Content-Length': '0'} 2024-04-13T22:05:49.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d93b12020563b7a-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.952 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:49.952 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:49.952 DEBUG:boto:path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.952 DEBUG:boto:auth_path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.952 DEBUG:boto:Method: GET 2024-04-13T22:05:49.952 DEBUG:boto:Path: /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.952 DEBUG:boto:Data: 2024-04-13T22:05:49.952 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.953 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.953 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.953 DEBUG:boto:Params: {} 2024-04-13T22:05:49.953 DEBUG:boto:Token: None 2024-04-13T22:05:49.953 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.953 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yz7+4Gtt4aR1BXz/oUBaGKGDZnk= 2024-04-13T22:05:49.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yz7+4Gtt4aR1BXz/oUBaGKGDZnk=', 'Content-Length': '0'} 2024-04-13T22:05:49.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcab5092b3c458cd-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.955 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:49.955 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:49.956 DEBUG:boto:path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.956 DEBUG:boto:auth_path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.956 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.956 DEBUG:boto:Path: /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.956 DEBUG:boto:Data: 2024-04-13T22:05:49.956 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.956 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.956 DEBUG:boto:Params: {} 2024-04-13T22:05:49.956 DEBUG:boto:Token: None 2024-04-13T22:05:49.957 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.957 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gKOioKZwkKKFk9aQFb7z/efA9cU= 2024-04-13T22:05:49.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gKOioKZwkKKFk9aQFb7z/efA9cU=', 'Content-Length': '0'} 2024-04-13T22:05:49.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c83213db7ec6ff2-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.958 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:49.958 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:49.959 DEBUG:boto:path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.959 DEBUG:boto:auth_path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.959 DEBUG:boto:Method: HEAD 2024-04-13T22:05:49.959 DEBUG:boto:Path: /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.959 DEBUG:boto:Data: 2024-04-13T22:05:49.959 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:49.959 DEBUG:boto:Port: 8000 2024-04-13T22:05:49.959 DEBUG:boto:Params: {} 2024-04-13T22:05:49.959 DEBUG:boto:Token: None 2024-04-13T22:05:49.959 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gKOioKZwkKKFk9aQFb7z/efA9cU= 2024-04-13T22:05:49.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gKOioKZwkKKFk9aQFb7z/efA9cU=', 'Content-Length': '0'} 2024-04-13T22:05:49.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b470f67c76cbbbc3-00661b01bd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:49.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:05:49.961 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:49.961 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:49.961 DEBUG:boto:path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.962 DEBUG:boto:auth_path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.962 DEBUG:boto:Method: GET 2024-04-13T22:05:49.962 DEBUG:boto:Path: /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.962 DEBUG:boto:Data: 2024-04-13T22:05:49.962 DEBUG:boto:Headers: {} 2024-04-13T22:05:49.962 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:49.962 DEBUG:boto:Port: 8001 2024-04-13T22:05:49.962 DEBUG:boto:Params: {} 2024-04-13T22:05:49.962 DEBUG:boto:Token: None 2024-04-13T22:05:49.963 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:49 GMT /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:49.963 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yz7+4Gtt4aR1BXz/oUBaGKGDZnk= 2024-04-13T22:05:49.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yz7+4Gtt4aR1BXz/oUBaGKGDZnk=', 'Content-Length': '0'} 2024-04-13T22:05:49.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d2178928c628036-00661b01bd-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.006 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:50.007 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:50.007 DEBUG:boto:path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:50.007 DEBUG:boto:auth_path=/vwnfns-51/obj7?versionId=null 2024-04-13T22:05:50.007 DEBUG:boto:Method: GET 2024-04-13T22:05:50.007 DEBUG:boto:Path: /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:50.007 DEBUG:boto:Data: 2024-04-13T22:05:50.007 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.007 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.007 DEBUG:boto:Params: {} 2024-04-13T22:05:50.008 DEBUG:boto:Token: None 2024-04-13T22:05:50.008 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj7?versionId=null 2024-04-13T22:05:50.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8yn5sSmnZFGzmvX01beFDRA1jKQ= 2024-04-13T22:05:50.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8yn5sSmnZFGzmvX01beFDRA1jKQ=', 'Content-Length': '0'} 2024-04-13T22:05:50.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058033ad1980995ee-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.010 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.010 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.011 DEBUG:boto:path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.011 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.011 DEBUG:boto:Method: GET 2024-04-13T22:05:50.011 DEBUG:boto:Path: /vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.011 DEBUG:boto:Data: 2024-04-13T22:05:50.011 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.011 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.011 DEBUG:boto:Params: {} 2024-04-13T22:05:50.012 DEBUG:boto:Token: None 2024-04-13T22:05:50.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c699596e611a7ffe-00661b01be-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.014 DEBUG:boto:b'vwnfns-51obj71000falseobj72024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.015 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.015 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.015 DEBUG:boto:path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.015 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.015 DEBUG:boto:Method: GET 2024-04-13T22:05:50.015 DEBUG:boto:Path: /vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.015 DEBUG:boto:Data: 2024-04-13T22:05:50.016 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.016 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.016 DEBUG:boto:Params: {} 2024-04-13T22:05:50.016 DEBUG:boto:Token: None 2024-04-13T22:05:50.016 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488da9ff76f2335f-00661b01be-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.018 DEBUG:boto:b'vwnfns-51obj71000falseobj72024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.019 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:50.019 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:50.019 DEBUG:boto:Method: HEAD 2024-04-13T22:05:50.019 DEBUG:boto:Path: /vwnfns-51/obj7 2024-04-13T22:05:50.019 DEBUG:boto:Data: 2024-04-13T22:05:50.019 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.020 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.020 DEBUG:boto:Params: {} 2024-04-13T22:05:50.020 DEBUG:boto:Token: None 2024-04-13T22:05:50.020 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj7 2024-04-13T22:05:50.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z+8q5bRdyarhFL0ivnC2vEIM8yQ= 2024-04-13T22:05:50.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z+8q5bRdyarhFL0ivnC2vEIM8yQ=', 'Content-Length': '0'} 2024-04-13T22:05:50.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f7b0eb8321b1d2f-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.022 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:50.022 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:50.022 DEBUG:boto:Method: HEAD 2024-04-13T22:05:50.023 DEBUG:boto:Path: /vwnfns-51/obj7 2024-04-13T22:05:50.023 DEBUG:boto:Data: 2024-04-13T22:05:50.023 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.023 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.023 DEBUG:boto:Params: {} 2024-04-13T22:05:50.023 DEBUG:boto:Token: None 2024-04-13T22:05:50.023 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj7 2024-04-13T22:05:50.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z+8q5bRdyarhFL0ivnC2vEIM8yQ= 2024-04-13T22:05:50.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z+8q5bRdyarhFL0ivnC2vEIM8yQ=', 'Content-Length': '0'} 2024-04-13T22:05:50.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003348ea8ca31a6a03-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-13T22:05:50.026 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:50.026 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:50.026 DEBUG:boto:Method: GET 2024-04-13T22:05:50.026 DEBUG:boto:Path: /vwnfns-51/obj7 2024-04-13T22:05:50.026 DEBUG:boto:Data: 2024-04-13T22:05:50.026 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.026 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.026 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.026 DEBUG:boto:Params: {} 2024-04-13T22:05:50.027 DEBUG:boto:Token: None 2024-04-13T22:05:50.027 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj7 2024-04-13T22:05:50.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pdr5hgsa+mlhP+yjx4Hc8rCqQs0= 2024-04-13T22:05:50.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pdr5hgsa+mlhP+yjx4Hc8rCqQs0=', 'Content-Length': '0'} 2024-04-13T22:05:50.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a21e9bb0869f629-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.029 DEBUG:boto:path=/vwnfns-51/obj7 2024-04-13T22:05:50.029 DEBUG:boto:auth_path=/vwnfns-51/obj7 2024-04-13T22:05:50.029 DEBUG:boto:Method: GET 2024-04-13T22:05:50.030 DEBUG:boto:Path: /vwnfns-51/obj7 2024-04-13T22:05:50.030 DEBUG:boto:Data: 2024-04-13T22:05:50.030 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.030 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.030 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.030 DEBUG:boto:Params: {} 2024-04-13T22:05:50.030 DEBUG:boto:Token: None 2024-04-13T22:05:50.030 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj7 2024-04-13T22:05:50.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pdr5hgsa+mlhP+yjx4Hc8rCqQs0= 2024-04-13T22:05:50.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pdr5hgsa+mlhP+yjx4Hc8rCqQs0=', 'Content-Length': '0'} 2024-04-13T22:05:50.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfc4aea2d43b5a22-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.074 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.075 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.075 DEBUG:boto:path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.075 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.075 DEBUG:boto:Method: GET 2024-04-13T22:05:50.075 DEBUG:boto:Path: /vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.075 DEBUG:boto:Data: 2024-04-13T22:05:50.075 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.075 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.075 DEBUG:boto:Params: {} 2024-04-13T22:05:50.076 DEBUG:boto:Token: None 2024-04-13T22:05:50.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004048280901597622-00661b01be-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.078 DEBUG:boto:b'vwnfns-51obj71000falseobj72024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.079 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.079 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.079 DEBUG:boto:path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.079 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.079 DEBUG:boto:Method: GET 2024-04-13T22:05:50.079 DEBUG:boto:Path: /vwnfns-51/?prefix=obj7 2024-04-13T22:05:50.080 DEBUG:boto:Data: 2024-04-13T22:05:50.080 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.080 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.080 DEBUG:boto:Params: {} 2024-04-13T22:05:50.080 DEBUG:boto:Token: None 2024-04-13T22:05:50.080 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf2dc3e6ae7ca710-00661b01be-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.082 DEBUG:boto:b'vwnfns-51obj71000falseobj72024-04-13T22:05:07.605Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:05:50.083 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.083 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.083 DEBUG:boto:path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.083 DEBUG:boto:auth_path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.083 DEBUG:boto:Method: GET 2024-04-13T22:05:50.083 DEBUG:boto:Path: /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.084 DEBUG:boto:Data: 2024-04-13T22:05:50.084 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.084 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.084 DEBUG:boto:Params: {} 2024-04-13T22:05:50.084 DEBUG:boto:Token: None 2024-04-13T22:05:50.084 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M= 2024-04-13T22:05:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M=', 'Content-Length': '0'} 2024-04-13T22:05:50.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025e62b9abcd95454-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.126 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.127 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.127 DEBUG:boto:path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.127 DEBUG:boto:auth_path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.127 DEBUG:boto:Method: GET 2024-04-13T22:05:50.127 DEBUG:boto:Path: /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.127 DEBUG:boto:Data: 2024-04-13T22:05:50.127 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.127 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.127 DEBUG:boto:Params: {} 2024-04-13T22:05:50.128 DEBUG:boto:Token: None 2024-04-13T22:05:50.128 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.128 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M= 2024-04-13T22:05:50.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M=', 'Content-Length': '0'} 2024-04-13T22:05:50.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055f38f66faca8788-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.130 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.131 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.131 DEBUG:boto:path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.131 DEBUG:boto:auth_path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.131 DEBUG:boto:Method: HEAD 2024-04-13T22:05:50.131 DEBUG:boto:Path: /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.131 DEBUG:boto:Data: 2024-04-13T22:05:50.131 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.131 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.131 DEBUG:boto:Params: {} 2024-04-13T22:05:50.132 DEBUG:boto:Token: None 2024-04-13T22:05:50.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:avUxNYKBNm2M2Ewd9OqOmreKXME= 2024-04-13T22:05:50.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:avUxNYKBNm2M2Ewd9OqOmreKXME=', 'Content-Length': '0'} 2024-04-13T22:05:50.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9af3127b2ce2728-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.134 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.134 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.134 DEBUG:boto:path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.134 DEBUG:boto:auth_path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.134 DEBUG:boto:Method: HEAD 2024-04-13T22:05:50.134 DEBUG:boto:Path: /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.135 DEBUG:boto:Data: 2024-04-13T22:05:50.135 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.135 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.135 DEBUG:boto:Params: {} 2024-04-13T22:05:50.135 DEBUG:boto:Token: None 2024-04-13T22:05:50.135 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:avUxNYKBNm2M2Ewd9OqOmreKXME= 2024-04-13T22:05:50.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:avUxNYKBNm2M2Ewd9OqOmreKXME=', 'Content-Length': '0'} 2024-04-13T22:05:50.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c74ca91156afeed4-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:05:50.137 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.137 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.137 DEBUG:boto:path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.137 DEBUG:boto:auth_path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.138 DEBUG:boto:Method: GET 2024-04-13T22:05:50.138 DEBUG:boto:Path: /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.138 DEBUG:boto:Data: 2024-04-13T22:05:50.138 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.138 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.138 DEBUG:boto:Params: {} 2024-04-13T22:05:50.138 DEBUG:boto:Token: None 2024-04-13T22:05:50.138 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M= 2024-04-13T22:05:50.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M=', 'Content-Length': '0'} 2024-04-13T22:05:50.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cb1f3141ad482fb-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.140 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.140 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.141 DEBUG:boto:path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.141 DEBUG:boto:auth_path=/vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.141 DEBUG:boto:Method: GET 2024-04-13T22:05:50.141 DEBUG:boto:Path: /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.141 DEBUG:boto:Data: 2024-04-13T22:05:50.141 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.141 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.141 DEBUG:boto:Params: {} 2024-04-13T22:05:50.141 DEBUG:boto:Token: None 2024-04-13T22:05:50.142 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8?versionId=null 2024-04-13T22:05:50.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M= 2024-04-13T22:05:50.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FTeGzw906f9BbdXWnJ/xxEMO94M=', 'Content-Length': '0'} 2024-04-13T22:05:50.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b903d4e4f10f8429-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.143 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.144 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.144 DEBUG:boto:path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.144 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.144 DEBUG:boto:Method: GET 2024-04-13T22:05:50.144 DEBUG:boto:Path: /vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.144 DEBUG:boto:Data: 2024-04-13T22:05:50.144 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.144 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.144 DEBUG:boto:Params: {} 2024-04-13T22:05:50.145 DEBUG:boto:Token: None 2024-04-13T22:05:50.145 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0e52cc8e839a3c-00661b01be-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.147 DEBUG:boto:b'vwnfns-51obj81000falseobj82024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.148 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.148 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.148 DEBUG:boto:path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.148 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.148 DEBUG:boto:Method: GET 2024-04-13T22:05:50.148 DEBUG:boto:Path: /vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.148 DEBUG:boto:Data: 2024-04-13T22:05:50.148 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.149 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.149 DEBUG:boto:Params: {} 2024-04-13T22:05:50.149 DEBUG:boto:Token: None 2024-04-13T22:05:50.149 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ef2c322cc090b2a-00661b01be-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.151 DEBUG:boto:b'vwnfns-51obj81000falseobj82024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.152 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.152 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.152 DEBUG:boto:Method: HEAD 2024-04-13T22:05:50.152 DEBUG:boto:Path: /vwnfns-51/obj8 2024-04-13T22:05:50.152 DEBUG:boto:Data: 2024-04-13T22:05:50.152 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.152 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.152 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.152 DEBUG:boto:Params: {} 2024-04-13T22:05:50.152 DEBUG:boto:Token: None 2024-04-13T22:05:50.153 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8 2024-04-13T22:05:50.153 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y/0lkYrK+kchGc6+ZUmE8o4kqQA= 2024-04-13T22:05:50.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y/0lkYrK+kchGc6+ZUmE8o4kqQA=', 'Content-Length': '0'} 2024-04-13T22:05:50.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006276471c7c95d0fa-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.154 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.154 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.155 DEBUG:boto:Method: HEAD 2024-04-13T22:05:50.155 DEBUG:boto:Path: /vwnfns-51/obj8 2024-04-13T22:05:50.155 DEBUG:boto:Data: 2024-04-13T22:05:50.155 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.155 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.155 DEBUG:boto:Params: {} 2024-04-13T22:05:50.155 DEBUG:boto:Token: None 2024-04-13T22:05:50.155 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8 2024-04-13T22:05:50.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y/0lkYrK+kchGc6+ZUmE8o4kqQA= 2024-04-13T22:05:50.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y/0lkYrK+kchGc6+ZUmE8o4kqQA=', 'Content-Length': '0'} 2024-04-13T22:05:50.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d964e66ef918d890-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-13T22:05:50.158 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.158 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.158 DEBUG:boto:Method: GET 2024-04-13T22:05:50.158 DEBUG:boto:Path: /vwnfns-51/obj8 2024-04-13T22:05:50.158 DEBUG:boto:Data: 2024-04-13T22:05:50.158 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.158 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.158 DEBUG:boto:Params: {} 2024-04-13T22:05:50.159 DEBUG:boto:Token: None 2024-04-13T22:05:50.159 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8 2024-04-13T22:05:50.159 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/ArLqWR0YDicaITc19HmV+4HIyM= 2024-04-13T22:05:50.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/ArLqWR0YDicaITc19HmV+4HIyM=', 'Content-Length': '0'} 2024-04-13T22:05:50.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c8496bb3776fa9b-00661b01be-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.161 DEBUG:boto:path=/vwnfns-51/obj8 2024-04-13T22:05:50.161 DEBUG:boto:auth_path=/vwnfns-51/obj8 2024-04-13T22:05:50.161 DEBUG:boto:Method: GET 2024-04-13T22:05:50.161 DEBUG:boto:Path: /vwnfns-51/obj8 2024-04-13T22:05:50.161 DEBUG:boto:Data: 2024-04-13T22:05:50.161 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.161 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.161 DEBUG:boto:Params: {} 2024-04-13T22:05:50.161 DEBUG:boto:Token: None 2024-04-13T22:05:50.162 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/obj8 2024-04-13T22:05:50.162 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/ArLqWR0YDicaITc19HmV+4HIyM= 2024-04-13T22:05:50.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/ArLqWR0YDicaITc19HmV+4HIyM=', 'Content-Length': '0'} 2024-04-13T22:05:50.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-51:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ad08a02f6e8d19d-00661b01be-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.206 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.207 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.207 DEBUG:boto:path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.207 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.207 DEBUG:boto:Method: GET 2024-04-13T22:05:50.207 DEBUG:boto:Path: /vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.207 DEBUG:boto:Data: 2024-04-13T22:05:50.208 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.208 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.208 DEBUG:boto:Params: {} 2024-04-13T22:05:50.208 DEBUG:boto:Token: None 2024-04-13T22:05:50.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60598a31abce3b9-00661b01be-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.211 DEBUG:boto:b'vwnfns-51obj81000falseobj82024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.211 DEBUG:boto:path=/vwnfns-51/ 2024-04-13T22:05:50.212 DEBUG:boto:auth_path=/vwnfns-51/ 2024-04-13T22:05:50.212 DEBUG:boto:path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.212 DEBUG:boto:auth_path=/vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.212 DEBUG:boto:Method: GET 2024-04-13T22:05:50.212 DEBUG:boto:Path: /vwnfns-51/?prefix=obj8 2024-04-13T22:05:50.212 DEBUG:boto:Data: 2024-04-13T22:05:50.212 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.213 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.213 DEBUG:boto:Params: {} 2024-04-13T22:05:50.213 DEBUG:boto:Token: None 2024-04-13T22:05:50.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-51/ 2024-04-13T22:05:50.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw= 2024-04-13T22:05:50.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H47N3Cii/ApT3fea5SgrGaImkPw=', 'Content-Length': '0'} 2024-04-13T22:05:50.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ca217f5b95698c-00661b01be-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.215 DEBUG:boto:b'vwnfns-51obj81000falseobj82024-04-13T22:05:07.614Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:05:50.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-51 zones={a2, a1} 2024-04-13T22:05:50.216 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-13T22:05:50.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.217 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:50.263 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:50.263 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:50.310 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:05:50.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.310 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000008966", 2024-04-13T22:05:50.310 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:07.831048Z" 2024-04-13T22:05:50.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007988", 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:27.607726Z" 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005723", 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:27.606472Z" 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.312 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000012345", 2024-04-13T22:05:50.312 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:27.607627Z" 2024-04-13T22:05:50.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.312 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:05:50.313 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000008966', 1: '00000000000000000000:00000000000000007988', 2: '00000000000000000000:00000000000000005723', 3: '00000000000000000000:00000000000000012345'} 2024-04-13T22:05:50.313 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:05:50.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.314 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:50.363 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:50.363 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:50.407 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 15685940596121292366 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.408 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000008966", 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:07.831048Z" 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007988", 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:27.607726Z" 2024-04-13T22:05:50.410 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000005723", 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.411 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:27.606472Z" 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000012345", 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.412 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:27.607627Z" 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.413 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:05:50Z" 2024-04-13T22:05:50.414 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:05:50.415 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15685940596121292366 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008966", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:07.831048Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007988", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:27.607726Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:27.606472Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:27.607627Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:05:50Z" } 2024-04-13T22:05:50.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008966', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:07.831048Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007988', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:27.607726Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:27.606472Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:27.607627Z'}}] 2024-04-13T22:05:50.415 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008966', 1: '00000000000000000000:00000000000000007988', 2: '00000000000000000000:00000000000000005723', 3: '00000000000000000000:00000000000000012345'} 2024-04-13T22:05:50.415 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008966', 1: '00000000000000000000:00000000000000007988', 2: '00000000000000000000:00000000000000005723', 3: '00000000000000000000:00000000000000012345'} 2024-04-13T22:05:50.416 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:05:50.416 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-13T22:05:50.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.417 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:50.464 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:50.464 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:50.513 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006712", 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:31.190084Z" 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006433", 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:31.191529Z" 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004736", 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:31.192910Z" 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000011358", 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:31.194423Z" 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.515 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:05:50.516 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006712', 1: '00000000000000000000:00000000000000006433', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000011358'} 2024-04-13T22:05:50.516 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:05:50.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:50.517 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:50.567 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:50.567 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:50.623 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006712", 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:31.190084Z" 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006433", 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:31.191529Z" 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000004736", 2024-04-13T22:05:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:31.192910Z" 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:05:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000011358", 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:31.194423Z" 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:05:50Z" 2024-04-13T22:05:50.630 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:05:50.631 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006712", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:31.190084Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006433", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:31.191529Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004736", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:31.192910Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:05:31.194423Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:05:50Z" } 2024-04-13T22:05:50.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006712', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:31.190084Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006433', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:31.191529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004736', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:31.192910Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:05:31.194423Z'}}] 2024-04-13T22:05:50.632 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006712', 1: '00000000000000000000:00000000000000006433', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000011358'} 2024-04-13T22:05:50.632 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006712', 1: '00000000000000000000:00000000000000006433', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000011358'} 2024-04-13T22:05:50.632 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:05:50.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2024-04-13T22:05:50.635 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-52 2024-04-13T22:05:50.635 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:05:50.635 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:05:50.635 DEBUG:boto:Method: PUT 2024-04-13T22:05:50.635 DEBUG:boto:Path: /vwnfns-52/ 2024-04-13T22:05:50.635 DEBUG:boto:Data: 2024-04-13T22:05:50.635 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.636 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.636 DEBUG:boto:Params: {} 2024-04-13T22:05:50.636 DEBUG:boto:Token: None 2024-04-13T22:05:50.636 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-52/ 2024-04-13T22:05:50.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tpggEq2vQqmP4cUdCs1O98jxIZM= 2024-04-13T22:05:50.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tpggEq2vQqmP4cUdCs1O98jxIZM=', 'Content-Length': '0'} 2024-04-13T22:05:50.650 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001bb3f2d09745817b-00661b01be-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.651 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-53 2024-04-13T22:05:50.651 DEBUG:boto:path=/vwnfns-53/ 2024-04-13T22:05:50.651 DEBUG:boto:auth_path=/vwnfns-53/ 2024-04-13T22:05:50.651 DEBUG:boto:Method: PUT 2024-04-13T22:05:50.651 DEBUG:boto:Path: /vwnfns-53/ 2024-04-13T22:05:50.651 DEBUG:boto:Data: 2024-04-13T22:05:50.651 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:05:50.651 DEBUG:boto:Port: 8001 2024-04-13T22:05:50.652 DEBUG:boto:Params: {} 2024-04-13T22:05:50.652 DEBUG:boto:Token: None 2024-04-13T22:05:50.652 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-53/ 2024-04-13T22:05:50.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:65SB3crVARTNw8FvrXlJof9Hzwg= 2024-04-13T22:05:50.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:65SB3crVARTNw8FvrXlJof9Hzwg=', 'Content-Length': '0'} 2024-04-13T22:05:50.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000960cb44c5ff1a911-00661b01be-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:05:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:50.678 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:50.678 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:50.679 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploads 2024-04-13T22:05:50.679 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploads 2024-04-13T22:05:50.679 DEBUG:boto:Method: POST 2024-04-13T22:05:50.679 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploads 2024-04-13T22:05:50.679 DEBUG:boto:Data: 2024-04-13T22:05:50.679 DEBUG:boto:Headers: {} 2024-04-13T22:05:50.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:50.680 DEBUG:boto:Port: 8000 2024-04-13T22:05:50.680 DEBUG:boto:Params: {} 2024-04-13T22:05:50.680 DEBUG:boto:Token: None 2024-04-13T22:05:50.680 DEBUG:boto:StringToSign: POST Sat, 13 Apr 2024 22:05:50 GMT /vwnfns-52/MULTIPART?uploads 2024-04-13T22:05:50.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZSELWaOuFiBsHOkhwZGU3vvw+5M= 2024-04-13T22:05:50.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZSELWaOuFiBsHOkhwZGU3vvw+5M=', 'Content-Length': '0'} 2024-04-13T22:05:52.396 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000817d2d407c596682-00661b01be-1a22-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.396 DEBUG:boto:b'vwnfns-52MULTIPART2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y' 2024-04-13T22:05:52.440 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.441 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.441 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=1 2024-04-13T22:05:52.441 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=1 2024-04-13T22:05:52.441 DEBUG:boto:Method: PUT 2024-04-13T22:05:52.441 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=1 2024-04-13T22:05:52.442 DEBUG:boto:Data: 2024-04-13T22:05:52.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-13T22:05:52.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:52.442 DEBUG:boto:Port: 8000 2024-04-13T22:05:52.442 DEBUG:boto:Params: {} 2024-04-13T22:05:52.443 DEBUG:boto:Token: None 2024-04-13T22:05:52.443 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Sat, 13 Apr 2024 22:05:52 GMT /vwnfns-52/MULTIPART?partNumber=1&uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:acELnGRZkLlptarcP8hYvM6wHBg= 2024-04-13T22:05:52.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:acELnGRZkLlptarcP8hYvM6wHBg='} 2024-04-13T22:05:52.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d715f366680c5e1-00661b01c0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.514 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.515 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.515 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=2 2024-04-13T22:05:52.515 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=2 2024-04-13T22:05:52.515 DEBUG:boto:Method: PUT 2024-04-13T22:05:52.515 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=2 2024-04-13T22:05:52.515 DEBUG:boto:Data: 2024-04-13T22:05:52.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-13T22:05:52.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:52.516 DEBUG:boto:Port: 8000 2024-04-13T22:05:52.516 DEBUG:boto:Params: {} 2024-04-13T22:05:52.516 DEBUG:boto:Token: None 2024-04-13T22:05:52.516 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Sat, 13 Apr 2024 22:05:52 GMT /vwnfns-52/MULTIPART?partNumber=2&uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BT2nkfRViYjbX28CT8PXN17KJkk= 2024-04-13T22:05:52.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BT2nkfRViYjbX28CT8PXN17KJkk='} 2024-04-13T22:05:52.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddeb2fe47f8139ab-00661b01c0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.564 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.564 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.564 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=3 2024-04-13T22:05:52.564 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=3 2024-04-13T22:05:52.564 DEBUG:boto:Method: PUT 2024-04-13T22:05:52.564 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=3 2024-04-13T22:05:52.564 DEBUG:boto:Data: 2024-04-13T22:05:52.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-13T22:05:52.565 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:52.565 DEBUG:boto:Port: 8000 2024-04-13T22:05:52.565 DEBUG:boto:Params: {} 2024-04-13T22:05:52.565 DEBUG:boto:Token: None 2024-04-13T22:05:52.565 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Sat, 13 Apr 2024 22:05:52 GMT /vwnfns-52/MULTIPART?partNumber=3&uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g9r6NtpDBeIdfP6MTSZzYcXazW8= 2024-04-13T22:05:52.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g9r6NtpDBeIdfP6MTSZzYcXazW8='} 2024-04-13T22:05:52.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f50eecc5797d4f17-00661b01c0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.614 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.615 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.615 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=4 2024-04-13T22:05:52.615 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=4 2024-04-13T22:05:52.615 DEBUG:boto:Method: PUT 2024-04-13T22:05:52.615 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y&partNumber=4 2024-04-13T22:05:52.615 DEBUG:boto:Data: 2024-04-13T22:05:52.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-13T22:05:52.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:52.616 DEBUG:boto:Port: 8000 2024-04-13T22:05:52.616 DEBUG:boto:Params: {} 2024-04-13T22:05:52.616 DEBUG:boto:Token: None 2024-04-13T22:05:52.616 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Sat, 13 Apr 2024 22:05:52 GMT /vwnfns-52/MULTIPART?partNumber=4&uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lsc9MMmkPDIPTKFvlUZY/n87Wmc= 2024-04-13T22:05:52.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:05:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lsc9MMmkPDIPTKFvlUZY/n87Wmc='} 2024-04-13T22:05:52.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000959a4d10b40c5ee9-00661b01c0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.650 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.650 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.650 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.650 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.650 DEBUG:boto:Method: GET 2024-04-13T22:05:52.651 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.651 DEBUG:boto:Data: 2024-04-13T22:05:52.651 DEBUG:boto:Headers: {} 2024-04-13T22:05:52.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:52.651 DEBUG:boto:Port: 8000 2024-04-13T22:05:52.651 DEBUG:boto:Params: {} 2024-04-13T22:05:52.651 DEBUG:boto:Token: None 2024-04-13T22:05:52.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:05:52 GMT /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NFv1/Poe/phgWOv1/OjCibiJvQE= 2024-04-13T22:05:52.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NFv1/Poe/phgWOv1/OjCibiJvQE=', 'Content-Length': '0'} 2024-04-13T22:05:52.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa4aac168fccfba-00661b01c0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.695 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.695 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:05:52.695 DEBUG:boto:path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.695 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.695 DEBUG:boto:Method: POST 2024-04-13T22:05:52.695 DEBUG:boto:Path: /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.696 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-04-13T22:05:52.696 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-04-13T22:05:52.696 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:05:52.696 DEBUG:boto:Port: 8000 2024-04-13T22:05:52.696 DEBUG:boto:Params: {} 2024-04-13T22:05:52.696 DEBUG:boto:Token: None 2024-04-13T22:05:52.696 DEBUG:boto:StringToSign: POST text/xml Sat, 13 Apr 2024 22:05:52 GMT /vwnfns-52/MULTIPART?uploadId=2~-uNHDt9uP5CtngzF0BAWXoIC3dLuS5Y 2024-04-13T22:05:52.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hIwtt3heYwvGm+x8dBIZKsH+PvU= 2024-04-13T22:05:52.696 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:05:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hIwtt3heYwvGm+x8dBIZKsH+PvU=', 'Content-Length': '460'} 2024-04-13T22:05:52.705 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000230cee943fd35a07-00661b01c0-1a22-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:05:52.706 DEBUG:boto:http://smithi132.front.sepia.ceph.com:8000/vwnfns-52/MULTIPARTvwnfns-52MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-13T22:05:52.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:52.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:52.706 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:52.755 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:52.755 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:52.784 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:05:52.796 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:05:52.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.796 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045950.668242_667.1", 2024-04-13T22:05:52.796 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:05:50.668242Z" 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:05:52.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.798 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:52.798 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:05:52.798 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:52.798 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:05:52.799 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045950.668242_667.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:52.799 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:05:52.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:52.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:05:52.800 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:05:52.847 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:52.847 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:52.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:52.882+0000 7f01ab945780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:05:52.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:52.883+0000 7f01ab945780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:05:52.897 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045950.668242_667.1", 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:50.668242Z", 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:52.899 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:52.900 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:05:52.901 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:05:52.902 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:05:52.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:05:52.903 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:05:52Z" 2024-04-13T22:05:52.903 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:05:52.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045950.668242_667.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:05:50.668242Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:05:52Z" } 2024-04-13T22:05:52.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045950.668242_667.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:05:50.668242Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:05:52.904 DEBUG:rgw_multi.tests:log_status={0: '1_1713045950.668242_667.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:52.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045950.668242_667.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:05:52.904 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:05:52.904 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-52 2024-04-13T22:05:52.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:52.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:05:52.905 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:05:52.959 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:05:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:05:52.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:52.993+0000 7efd2e47d780 1 waiting to reach incremental sync.. 2024-04-13T22:05:57.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:05:57.995+0000 7efd2e47d780 1 waiting to reach incremental sync.. 2024-04-13T22:06:02.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:02.996+0000 7efd2e47d780 1 waiting to reach incremental sync.. 2024-04-13T22:06:07.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:07.997+0000 7efd2e47d780 1 bucket sync caught up with source: 2024-04-13T22:06:07.998 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000011.48.6, , , , , ] 2024-04-13T22:06:07.998 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2024-04-13T22:06:07.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:07.997+0000 7efd2e47d780 0 bucket checkpoint complete 2024-04-13T22:06:08.016 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-52 2024-04-13T22:06:08.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:08.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:08.017 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-52 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:06:08.077 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:08.077 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:08.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:08.113+0000 7f2af97fb780 1 waiting to reach incremental sync.. 2024-04-13T22:06:13.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:13.112+0000 7f2af97fb780 1 waiting to reach incremental sync.. 2024-04-13T22:06:18.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:18.113+0000 7f2af97fb780 1 waiting to reach incremental sync.. 2024-04-13T22:06:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:23.115+0000 7f2af97fb780 1 bucket sync caught up with source: 2024-04-13T22:06:23.116 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1463.6, , , , , ] 2024-04-13T22:06:23.116 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1463.6, , , , , ] 2024-04-13T22:06:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:23.115+0000 7f2af97fb780 0 bucket checkpoint complete 2024-04-13T22:06:23.132 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-52 zones={a2, a1} 2024-04-13T22:06:23.133 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.133 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.133 DEBUG:boto:Method: HEAD 2024-04-13T22:06:23.133 DEBUG:boto:Path: /vwnfns-52/ 2024-04-13T22:06:23.133 DEBUG:boto:Data: 2024-04-13T22:06:23.133 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.133 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.133 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.133 DEBUG:boto:Params: {} 2024-04-13T22:06:23.134 DEBUG:boto:Token: None 2024-04-13T22:06:23.134 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/ 2024-04-13T22:06:23.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zCPYwJ3q0ZWZ5sOpQmTzVBmdPMM= 2024-04-13T22:06:23.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zCPYwJ3q0ZWZ5sOpQmTzVBmdPMM=', 'Content-Length': '0'} 2024-04-13T22:06:23.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad47c4add08a8a0-00661b01df-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.137 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.137 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.137 DEBUG:boto:Method: HEAD 2024-04-13T22:06:23.137 DEBUG:boto:Path: /vwnfns-52/ 2024-04-13T22:06:23.137 DEBUG:boto:Data: 2024-04-13T22:06:23.137 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.138 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.138 DEBUG:boto:Params: {} 2024-04-13T22:06:23.138 DEBUG:boto:Token: None 2024-04-13T22:06:23.138 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/ 2024-04-13T22:06:23.138 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zCPYwJ3q0ZWZ5sOpQmTzVBmdPMM= 2024-04-13T22:06:23.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zCPYwJ3q0ZWZ5sOpQmTzVBmdPMM=', 'Content-Length': '0'} 2024-04-13T22:06:23.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000255f37d8071a5611-00661b01df-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:06:23.141 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.141 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.141 DEBUG:boto:path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.141 DEBUG:boto:auth_path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.141 DEBUG:boto:Method: GET 2024-04-13T22:06:23.141 DEBUG:boto:Path: /vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.141 DEBUG:boto:Data: 2024-04-13T22:06:23.141 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.141 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.142 DEBUG:boto:Params: {} 2024-04-13T22:06:23.142 DEBUG:boto:Token: None 2024-04-13T22:06:23.142 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/?versions 2024-04-13T22:06:23.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ= 2024-04-13T22:06:23.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b459d0481abd4d-00661b01df-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.144 DEBUG:boto:b'vwnfns-52999falseMULTIPARTnulltrue2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.145 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-13T22:06:23.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:06:23.145 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.146 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.146 DEBUG:boto:path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.146 DEBUG:boto:auth_path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.146 DEBUG:boto:Method: GET 2024-04-13T22:06:23.146 DEBUG:boto:Path: /vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.146 DEBUG:boto:Data: 2024-04-13T22:06:23.146 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.146 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.146 DEBUG:boto:Params: {} 2024-04-13T22:06:23.146 DEBUG:boto:Token: None 2024-04-13T22:06:23.147 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/?versions 2024-04-13T22:06:23.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ= 2024-04-13T22:06:23.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053c7fdb98ea6a0a7-00661b01df-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.149 DEBUG:boto:b'vwnfns-52999falseMULTIPARTnulltrue2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.149 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-13T22:06:23.150 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.150 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.150 DEBUG:boto:path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.150 DEBUG:boto:auth_path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.150 DEBUG:boto:Method: GET 2024-04-13T22:06:23.150 DEBUG:boto:Path: /vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.150 DEBUG:boto:Data: 2024-04-13T22:06:23.150 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.150 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.150 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.150 DEBUG:boto:Params: {} 2024-04-13T22:06:23.151 DEBUG:boto:Token: None 2024-04-13T22:06:23.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/?versions 2024-04-13T22:06:23.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ= 2024-04-13T22:06:23.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047709846c47967c0-00661b01df-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.153 DEBUG:boto:b'vwnfns-52999falseMULTIPARTnulltrue2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.153 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.153 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.153 DEBUG:boto:path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.154 DEBUG:boto:auth_path=/vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.154 DEBUG:boto:Method: GET 2024-04-13T22:06:23.154 DEBUG:boto:Path: /vwnfns-52/?versions&max-keys=999 2024-04-13T22:06:23.154 DEBUG:boto:Data: 2024-04-13T22:06:23.154 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.154 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.154 DEBUG:boto:Params: {} 2024-04-13T22:06:23.154 DEBUG:boto:Token: None 2024-04-13T22:06:23.154 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/?versions 2024-04-13T22:06:23.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ= 2024-04-13T22:06:23.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnbgXhPTnWkPsjo7LLZ1twoe5XQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b4de6bc0197a678-00661b01df-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.157 DEBUG:boto:b'vwnfns-52999falseMULTIPARTnulltrue2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-13T22:06:23.157 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.157 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.158 DEBUG:boto:path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.158 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.158 DEBUG:boto:Method: GET 2024-04-13T22:06:23.158 DEBUG:boto:Path: /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.158 DEBUG:boto:Data: 2024-04-13T22:06:23.158 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.158 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.158 DEBUG:boto:Params: {} 2024-04-13T22:06:23.158 DEBUG:boto:Token: None 2024-04-13T22:06:23.159 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.159 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA= 2024-04-13T22:06:23.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA=', 'Content-Length': '0'} 2024-04-13T22:06:23.162 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-52:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001d6b47cfe6c291f3-00661b01df-1a1e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.239 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.239 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.240 DEBUG:boto:path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.240 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.240 DEBUG:boto:Method: GET 2024-04-13T22:06:23.240 DEBUG:boto:Path: /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.240 DEBUG:boto:Data: 2024-04-13T22:06:23.240 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.240 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.240 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.240 DEBUG:boto:Params: {} 2024-04-13T22:06:23.240 DEBUG:boto:Token: None 2024-04-13T22:06:23.241 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.241 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA= 2024-04-13T22:06:23.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA=', 'Content-Length': '0'} 2024-04-13T22:06:23.244 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b72a85ffb4df8512-00661b01df-1a22-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.332 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.332 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.333 DEBUG:boto:path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.333 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.333 DEBUG:boto:Method: HEAD 2024-04-13T22:06:23.333 DEBUG:boto:Path: /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.333 DEBUG:boto:Data: 2024-04-13T22:06:23.333 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.333 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.333 DEBUG:boto:Params: {} 2024-04-13T22:06:23.333 DEBUG:boto:Token: None 2024-04-13T22:06:23.334 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T+SI7hGljjg9/b4iEqH1Uq/yHfc= 2024-04-13T22:06:23.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T+SI7hGljjg9/b4iEqH1Uq/yHfc=', 'Content-Length': '0'} 2024-04-13T22:06:23.336 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-52:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001e6bbd6fed15733a-00661b01df-1a1e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.336 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.336 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.336 DEBUG:boto:path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.336 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.336 DEBUG:boto:Method: HEAD 2024-04-13T22:06:23.336 DEBUG:boto:Path: /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.337 DEBUG:boto:Data: 2024-04-13T22:06:23.337 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.337 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.337 DEBUG:boto:Params: {} 2024-04-13T22:06:23.337 DEBUG:boto:Token: None 2024-04-13T22:06:23.337 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T+SI7hGljjg9/b4iEqH1Uq/yHfc= 2024-04-13T22:06:23.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T+SI7hGljjg9/b4iEqH1Uq/yHfc=', 'Content-Length': '0'} 2024-04-13T22:06:23.339 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003fd9ecc379273fcd-00661b01df-1a22-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-13T22:06:23.339 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.339 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.339 DEBUG:boto:path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.340 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.340 DEBUG:boto:Method: GET 2024-04-13T22:06:23.340 DEBUG:boto:Path: /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.340 DEBUG:boto:Data: 2024-04-13T22:06:23.340 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.340 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.341 DEBUG:boto:Params: {} 2024-04-13T22:06:23.341 DEBUG:boto:Token: None 2024-04-13T22:06:23.341 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.341 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA= 2024-04-13T22:06:23.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA=', 'Content-Length': '0'} 2024-04-13T22:06:23.344 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-52:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000fca24efc1559dfc3-00661b01df-1a1e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.442 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.442 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.442 DEBUG:boto:path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.442 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.442 DEBUG:boto:Method: GET 2024-04-13T22:06:23.443 DEBUG:boto:Path: /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.443 DEBUG:boto:Data: 2024-04-13T22:06:23.443 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.443 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.443 DEBUG:boto:Params: {} 2024-04-13T22:06:23.443 DEBUG:boto:Token: None 2024-04-13T22:06:23.443 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART?versionId=null 2024-04-13T22:06:23.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA= 2024-04-13T22:06:23.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qC6fATHl7eQVcsNSouy8/gMJWcA=', 'Content-Length': '0'} 2024-04-13T22:06:23.447 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f4395a43f1c8b04d-00661b01df-1a22-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.541 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.542 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.542 DEBUG:boto:path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.542 DEBUG:boto:auth_path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.542 DEBUG:boto:Method: GET 2024-04-13T22:06:23.542 DEBUG:boto:Path: /vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.542 DEBUG:boto:Data: 2024-04-13T22:06:23.542 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.542 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.542 DEBUG:boto:Params: {} 2024-04-13T22:06:23.543 DEBUG:boto:Token: None 2024-04-13T22:06:23.543 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/ 2024-04-13T22:06:23.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ= 2024-04-13T22:06:23.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b38077e6a0a4069-00661b01df-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.546 DEBUG:boto:b'vwnfns-52MULTIPART1000falseMULTIPART2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.546 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.546 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.546 DEBUG:boto:path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.546 DEBUG:boto:auth_path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.547 DEBUG:boto:Method: GET 2024-04-13T22:06:23.547 DEBUG:boto:Path: /vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.547 DEBUG:boto:Data: 2024-04-13T22:06:23.547 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.547 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.547 DEBUG:boto:Params: {} 2024-04-13T22:06:23.547 DEBUG:boto:Token: None 2024-04-13T22:06:23.547 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/ 2024-04-13T22:06:23.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ= 2024-04-13T22:06:23.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e39c5c36ce6e7b9-00661b01df-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.550 DEBUG:boto:b'vwnfns-52MULTIPART1000falseMULTIPART2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.550 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.550 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.551 DEBUG:boto:Method: HEAD 2024-04-13T22:06:23.551 DEBUG:boto:Path: /vwnfns-52/MULTIPART 2024-04-13T22:06:23.551 DEBUG:boto:Data: 2024-04-13T22:06:23.551 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.551 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.551 DEBUG:boto:Params: {} 2024-04-13T22:06:23.551 DEBUG:boto:Token: None 2024-04-13T22:06:23.551 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART 2024-04-13T22:06:23.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rA7BC6c7XIJ7bVEtrgd/UJFtaew= 2024-04-13T22:06:23.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rA7BC6c7XIJ7bVEtrgd/UJFtaew=', 'Content-Length': '0'} 2024-04-13T22:06:23.553 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-52:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000088b51f89b380d203-00661b01df-1a1e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.553 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.554 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.554 DEBUG:boto:Method: HEAD 2024-04-13T22:06:23.554 DEBUG:boto:Path: /vwnfns-52/MULTIPART 2024-04-13T22:06:23.554 DEBUG:boto:Data: 2024-04-13T22:06:23.554 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.554 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.554 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.554 DEBUG:boto:Params: {} 2024-04-13T22:06:23.554 DEBUG:boto:Token: None 2024-04-13T22:06:23.555 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART 2024-04-13T22:06:23.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rA7BC6c7XIJ7bVEtrgd/UJFtaew= 2024-04-13T22:06:23.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rA7BC6c7XIJ7bVEtrgd/UJFtaew=', 'Content-Length': '0'} 2024-04-13T22:06:23.556 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004ccc4695bb2c3d15-00661b01df-1a22-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-13T22:06:23.557 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.557 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.557 DEBUG:boto:Method: GET 2024-04-13T22:06:23.557 DEBUG:boto:Path: /vwnfns-52/MULTIPART 2024-04-13T22:06:23.557 DEBUG:boto:Data: 2024-04-13T22:06:23.557 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.557 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.557 DEBUG:boto:Params: {} 2024-04-13T22:06:23.557 DEBUG:boto:Token: None 2024-04-13T22:06:23.558 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART 2024-04-13T22:06:23.558 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y5EQQEYrQVAVqFWnkgfptPBUstM= 2024-04-13T22:06:23.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y5EQQEYrQVAVqFWnkgfptPBUstM=', 'Content-Length': '0'} 2024-04-13T22:06:23.561 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-52:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000a79656aedfd31307-00661b01df-1a1e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.653 DEBUG:boto:path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.654 DEBUG:boto:auth_path=/vwnfns-52/MULTIPART 2024-04-13T22:06:23.654 DEBUG:boto:Method: GET 2024-04-13T22:06:23.654 DEBUG:boto:Path: /vwnfns-52/MULTIPART 2024-04-13T22:06:23.654 DEBUG:boto:Data: 2024-04-13T22:06:23.654 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.654 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.654 DEBUG:boto:Params: {} 2024-04-13T22:06:23.654 DEBUG:boto:Token: None 2024-04-13T22:06:23.655 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/MULTIPART 2024-04-13T22:06:23.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y5EQQEYrQVAVqFWnkgfptPBUstM= 2024-04-13T22:06:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y5EQQEYrQVAVqFWnkgfptPBUstM=', 'Content-Length': '0'} 2024-04-13T22:06:23.658 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:05:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d99b4bd6deb660d8-00661b01df-1a22-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.755 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.755 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.756 DEBUG:boto:path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.756 DEBUG:boto:auth_path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.756 DEBUG:boto:Method: GET 2024-04-13T22:06:23.756 DEBUG:boto:Path: /vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.756 DEBUG:boto:Data: 2024-04-13T22:06:23.756 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.756 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:23.756 DEBUG:boto:Port: 8001 2024-04-13T22:06:23.756 DEBUG:boto:Params: {} 2024-04-13T22:06:23.757 DEBUG:boto:Token: None 2024-04-13T22:06:23.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/ 2024-04-13T22:06:23.757 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ= 2024-04-13T22:06:23.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003845fb9b13fbc8d2-00661b01df-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.759 DEBUG:boto:b'vwnfns-52MULTIPART1000falseMULTIPART2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.760 DEBUG:boto:path=/vwnfns-52/ 2024-04-13T22:06:23.760 DEBUG:boto:auth_path=/vwnfns-52/ 2024-04-13T22:06:23.760 DEBUG:boto:path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.760 DEBUG:boto:auth_path=/vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.760 DEBUG:boto:Method: GET 2024-04-13T22:06:23.760 DEBUG:boto:Path: /vwnfns-52/?prefix=MULTIPART 2024-04-13T22:06:23.760 DEBUG:boto:Data: 2024-04-13T22:06:23.761 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.761 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.761 DEBUG:boto:Params: {} 2024-04-13T22:06:23.761 DEBUG:boto:Token: None 2024-04-13T22:06:23.761 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-52/ 2024-04-13T22:06:23.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ= 2024-04-13T22:06:23.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+d1+7PBSVbtPpd2iYc/IzRUPpJQ=', 'Content-Length': '0'} 2024-04-13T22:06:23.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0aaeca5c888579-00661b01df-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.764 DEBUG:boto:b'vwnfns-52MULTIPART1000falseMULTIPART2024-04-13T22:05:52.700Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:06:23.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-52 zones={a2, a1} 2024-04-13T22:06:23.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2024-04-13T22:06:23.766 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-54 2024-04-13T22:06:23.766 DEBUG:boto:path=/vwnfns-54/ 2024-04-13T22:06:23.766 DEBUG:boto:auth_path=/vwnfns-54/ 2024-04-13T22:06:23.766 DEBUG:boto:Method: PUT 2024-04-13T22:06:23.766 DEBUG:boto:Path: /vwnfns-54/ 2024-04-13T22:06:23.766 DEBUG:boto:Data: 2024-04-13T22:06:23.766 DEBUG:boto:Headers: {} 2024-04-13T22:06:23.766 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.766 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.767 DEBUG:boto:Params: {} 2024-04-13T22:06:23.767 DEBUG:boto:Token: None 2024-04-13T22:06:23.767 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:06:23 GMT /vwnfns-54/ 2024-04-13T22:06:23.767 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WobxVwDkpCARmtpOFvhuHxTnXUE= 2024-04-13T22:06:23.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WobxVwDkpCARmtpOFvhuHxTnXUE=', 'Content-Length': '0'} 2024-04-13T22:06:23.782 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fbe7de87f7e9bfd6-00661b01df-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.782 DEBUG:boto:path=/vwnfns-54/testobj-sse-c 2024-04-13T22:06:23.782 DEBUG:boto:auth_path=/vwnfns-54/testobj-sse-c 2024-04-13T22:06:23.782 DEBUG:boto:Method: PUT 2024-04-13T22:06:23.783 DEBUG:boto:Path: /vwnfns-54/testobj-sse-c 2024-04-13T22:06:23.783 DEBUG:boto:Data: 2024-04-13T22:06:23.783 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T22:06:23.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.783 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.783 DEBUG:boto:Params: {} 2024-04-13T22:06:23.783 DEBUG:boto:Token: None 2024-04-13T22:06:23.783 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 13 Apr 2024 22:06:23 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /vwnfns-54/testobj-sse-c 2024-04-13T22:06:23.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HltbyKFV9w0eSjZpfk1GHYp9MiU= 2024-04-13T22:06:23.784 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HltbyKFV9w0eSjZpfk1GHYp9MiU='} 2024-04-13T22:06:23.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000000b049a200a333a33-00661b01df-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.827 DEBUG:boto:path=/vwnfns-54/testobj-sse-kms 2024-04-13T22:06:23.827 DEBUG:boto:auth_path=/vwnfns-54/testobj-sse-kms 2024-04-13T22:06:23.827 DEBUG:boto:Method: PUT 2024-04-13T22:06:23.827 DEBUG:boto:Path: /vwnfns-54/testobj-sse-kms 2024-04-13T22:06:23.827 DEBUG:boto:Data: 2024-04-13T22:06:23.827 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-13T22:06:23.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:23.827 DEBUG:boto:Port: 8000 2024-04-13T22:06:23.828 DEBUG:boto:Params: {} 2024-04-13T22:06:23.828 DEBUG:boto:Token: None 2024-04-13T22:06:23.828 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 13 Apr 2024 22:06:23 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /vwnfns-54/testobj-sse-kms 2024-04-13T22:06:23.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gx56SL+m+Kg7sMz69uLmmxnAN8E= 2024-04-13T22:06:23.828 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:06:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gx56SL+m+Kg7sMz69uLmmxnAN8E='} 2024-04-13T22:06:23.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6dnduZm5zLTU0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000036dd2cea09eb7536-00661b01df-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:23.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:23.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:23.834 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:06:23.882 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:23.882 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:06:23.923 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:06:23.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:23.923 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045983.780381_672.1", 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:06:23.780381Z" 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:06:23.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:23.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:23.925 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:23.925 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:06:23.925 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:23.925 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:06:23.926 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045983.780381_672.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:23.926 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:06:23.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:23.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:23.927 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:06:23.974 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:23.975 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:24.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:24.002+0000 7ff8c389d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:06:24.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:24.003+0000 7ff8c389d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:24.017 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045950.668242_667.1", 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:05:50.668242Z", 2024-04-13T22:06:24.018 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:24.019 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:24.020 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:24.021 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:06:24.022 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:06:24.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:24.023 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:06:24Z" 2024-04-13T22:06:24.023 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:06:24.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045950.668242_667.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:05:50.668242Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:06:24Z" } 2024-04-13T22:06:24.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045950.668242_667.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:05:50.668242Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:06:24.024 DEBUG:rgw_multi.tests:log_status={0: '1_1713045983.780381_672.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:24.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045950.668242_667.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:24.024 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713045983.780381_672.1 target=1_1713045950.668242_667.1 2024-04-13T22:06:29.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:29.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:29.026 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:06:29.081 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:29.081 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:29.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.112+0000 7f386df68780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:06:29.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.113+0000 7f386df68780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045983.780381_672.1", 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.130 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:06:23.780381Z", 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:29.131 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.132 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.134 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:06:29Z" 2024-04-13T22:06:29.135 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:06:29.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045983.780381_672.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:06:23.780381Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:06:29Z" } 2024-04-13T22:06:29.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045983.780381_672.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:06:23.780381Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:06:29.136 DEBUG:rgw_multi.tests:log_status={0: '1_1713045983.780381_672.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:29.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045983.780381_672.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:29.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:06:29.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:29.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:29.137 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:06:29.193 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:29.193 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:29.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.228+0000 7fc8ba4e3780 1 bucket sync caught up with source: 2024-04-13T22:06:29.229 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.34.6, , , , 00000000001.29.6, , , , , ] 2024-04-13T22:06:29.229 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.34.6, , , , 00000000001.29.6, , , , , ] 2024-04-13T22:06:29.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.228+0000 7fc8ba4e3780 0 bucket checkpoint complete 2024-04-13T22:06:29.242 DEBUG:boto:path=/vwnfns-54/ 2024-04-13T22:06:29.243 DEBUG:boto:auth_path=/vwnfns-54/ 2024-04-13T22:06:29.243 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.243 DEBUG:boto:Path: /vwnfns-54/ 2024-04-13T22:06:29.243 DEBUG:boto:Data: 2024-04-13T22:06:29.243 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:29.243 DEBUG:boto:Port: 8001 2024-04-13T22:06:29.243 DEBUG:boto:Params: {} 2024-04-13T22:06:29.243 DEBUG:boto:Token: None 2024-04-13T22:06:29.244 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-54/ 2024-04-13T22:06:29.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zmbBXhfAO08crU3ahc6FBgfn/vQ= 2024-04-13T22:06:29.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zmbBXhfAO08crU3ahc6FBgfn/vQ=', 'Content-Length': '0'} 2024-04-13T22:06:29.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f613d850c971c8f6-00661b01e5-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.246 DEBUG:boto:path=/vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.246 DEBUG:boto:auth_path=/vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.247 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.247 DEBUG:boto:Path: /vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.247 DEBUG:boto:Data: 2024-04-13T22:06:29.247 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-13T22:06:29.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:29.247 DEBUG:boto:Port: 8001 2024-04-13T22:06:29.247 DEBUG:boto:Params: {} 2024-04-13T22:06:29.247 DEBUG:boto:Token: None 2024-04-13T22:06:29.247 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7dsFZByny+NryuyPCQ0VSIJADvI= 2024-04-13T22:06:29.247 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7dsFZByny+NryuyPCQ0VSIJADvI=', 'Content-Length': '0'} 2024-04-13T22:06:29.249 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-54:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000a82ea6957cbd616a-00661b01e5-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.249 DEBUG:boto:path=/vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.249 DEBUG:boto:auth_path=/vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.250 DEBUG:boto:Method: GET 2024-04-13T22:06:29.250 DEBUG:boto:Path: /vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.250 DEBUG:boto:Data: 2024-04-13T22:06:29.250 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-13T22:06:29.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:29.250 DEBUG:boto:Port: 8001 2024-04-13T22:06:29.250 DEBUG:boto:Params: {} 2024-04-13T22:06:29.250 DEBUG:boto:Token: None 2024-04-13T22:06:29.250 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:29 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /vwnfns-54/testobj-sse-c 2024-04-13T22:06:29.250 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bmmcHX8Rtjno+fa7CyXV+qqwRIM= 2024-04-13T22:06:29.251 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bmmcHX8Rtjno+fa7CyXV+qqwRIM=', 'Content-Length': '0'} 2024-04-13T22:06:29.253 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-54:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000004648de939af8681-00661b01e5-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.253 DEBUG:boto:path=/vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.253 DEBUG:boto:auth_path=/vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.253 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.253 DEBUG:boto:Path: /vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.253 DEBUG:boto:Data: 2024-04-13T22:06:29.253 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.253 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:29.253 DEBUG:boto:Port: 8001 2024-04-13T22:06:29.254 DEBUG:boto:Params: {} 2024-04-13T22:06:29.254 DEBUG:boto:Token: None 2024-04-13T22:06:29.254 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wNjbJEvagVTgMBaICkWZQ6Rn/Jk= 2024-04-13T22:06:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wNjbJEvagVTgMBaICkWZQ6Rn/Jk=', 'Content-Length': '0'} 2024-04-13T22:06:29.256 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-54:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000b935ee3e2283a4bc-00661b01e5-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.256 DEBUG:boto:path=/vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.256 DEBUG:boto:auth_path=/vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.256 DEBUG:boto:Method: GET 2024-04-13T22:06:29.256 DEBUG:boto:Path: /vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.257 DEBUG:boto:Data: 2024-04-13T22:06:29.257 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:06:29.257 DEBUG:boto:Port: 8001 2024-04-13T22:06:29.257 DEBUG:boto:Params: {} 2024-04-13T22:06:29.257 DEBUG:boto:Token: None 2024-04-13T22:06:29.257 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-54/testobj-sse-kms 2024-04-13T22:06:29.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BqdyfxLplE/3Jpbo8VIxIPfqqdc= 2024-04-13T22:06:29.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BqdyfxLplE/3Jpbo8VIxIPfqqdc=', 'Content-Length': '0'} 2024-04-13T22:06:29.259 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-54:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000029945caaf051059-00661b01e5-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2024-04-13T22:06:29.260 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-55 2024-04-13T22:06:29.260 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:06:29.260 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:06:29.260 DEBUG:boto:Method: PUT 2024-04-13T22:06:29.260 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:06:29.260 DEBUG:boto:Data: 2024-04-13T22:06:29.260 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.261 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.261 DEBUG:boto:Params: {} 2024-04-13T22:06:29.261 DEBUG:boto:Token: None 2024-04-13T22:06:29.261 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/ 2024-04-13T22:06:29.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0MbPuSmex0wXPl14ZBMuI/L6DcM= 2024-04-13T22:06:29.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0MbPuSmex0wXPl14ZBMuI/L6DcM=', 'Content-Length': '0'} 2024-04-13T22:06:29.276 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038c4660c68fea2a3-00661b01e5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.276 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:06:29.276 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:06:29.276 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.276 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:06:29.276 DEBUG:boto:Data: 2024-04-13T22:06:29.276 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.276 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.277 DEBUG:boto:Params: {} 2024-04-13T22:06:29.277 DEBUG:boto:Token: None 2024-04-13T22:06:29.277 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/ 2024-04-13T22:06:29.277 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg= 2024-04-13T22:06:29.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg=', 'Content-Length': '0'} 2024-04-13T22:06:29.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000507b6149667414e5-00661b01e5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.279 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:06:29.279 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:06:29.280 DEBUG:boto:Method: PUT 2024-04-13T22:06:29.280 DEBUG:boto:Path: /vwnfns-55/a 2024-04-13T22:06:29.280 DEBUG:boto:Data: 2024-04-13T22:06:29.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:06:29.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.280 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.280 DEBUG:boto:Params: {} 2024-04-13T22:06:29.280 DEBUG:boto:Token: None 2024-04-13T22:06:29.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/a 2024-04-13T22:06:29.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yj+u79Manw4uEk81hP95lBfOEtY= 2024-04-13T22:06:29.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yj+u79Manw4uEk81hP95lBfOEtY='} 2024-04-13T22:06:29.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8cf0fc760f070ec-00661b01e5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.323 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:06:29.323 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:06:29.323 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.323 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:06:29.323 DEBUG:boto:Data: 2024-04-13T22:06:29.323 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.323 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.323 DEBUG:boto:Params: {} 2024-04-13T22:06:29.323 DEBUG:boto:Token: None 2024-04-13T22:06:29.324 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/ 2024-04-13T22:06:29.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg= 2024-04-13T22:06:29.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg=', 'Content-Length': '0'} 2024-04-13T22:06:29.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085e87f63ae671228-00661b01e5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.326 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:06:29.326 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:06:29.327 DEBUG:boto:Method: PUT 2024-04-13T22:06:29.327 DEBUG:boto:Path: /vwnfns-55/b 2024-04-13T22:06:29.327 DEBUG:boto:Data: 2024-04-13T22:06:29.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:06:29.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.327 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.327 DEBUG:boto:Params: {} 2024-04-13T22:06:29.327 DEBUG:boto:Token: None 2024-04-13T22:06:29.327 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/b 2024-04-13T22:06:29.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:++KeYANvLaRXqqeOp3mMjX1nTbk= 2024-04-13T22:06:29.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:++KeYANvLaRXqqeOp3mMjX1nTbk='} 2024-04-13T22:06:29.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004984fa58cc8cf916-00661b01e5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.334 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:06:29.334 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:06:29.334 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.334 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:06:29.334 DEBUG:boto:Data: 2024-04-13T22:06:29.334 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.334 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.334 DEBUG:boto:Params: {} 2024-04-13T22:06:29.334 DEBUG:boto:Token: None 2024-04-13T22:06:29.335 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/ 2024-04-13T22:06:29.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg= 2024-04-13T22:06:29.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg=', 'Content-Length': '0'} 2024-04-13T22:06:29.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008abbc1a22e5e38ac-00661b01e5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.337 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:06:29.337 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:06:29.337 DEBUG:boto:Method: PUT 2024-04-13T22:06:29.337 DEBUG:boto:Path: /vwnfns-55/c 2024-04-13T22:06:29.337 DEBUG:boto:Data: 2024-04-13T22:06:29.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:06:29.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.338 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.338 DEBUG:boto:Params: {} 2024-04-13T22:06:29.338 DEBUG:boto:Token: None 2024-04-13T22:06:29.338 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/c 2024-04-13T22:06:29.338 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oqeADDcN5uPNOEByYP6PSTunOSI= 2024-04-13T22:06:29.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oqeADDcN5uPNOEByYP6PSTunOSI='} 2024-04-13T22:06:29.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8669a294e8e96d5-00661b01e5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.343 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:06:29.343 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:06:29.343 DEBUG:boto:Method: HEAD 2024-04-13T22:06:29.343 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:06:29.343 DEBUG:boto:Data: 2024-04-13T22:06:29.343 DEBUG:boto:Headers: {} 2024-04-13T22:06:29.343 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.343 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.343 DEBUG:boto:Params: {} 2024-04-13T22:06:29.343 DEBUG:boto:Token: None 2024-04-13T22:06:29.344 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/ 2024-04-13T22:06:29.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg= 2024-04-13T22:06:29.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lYltuBpC1n30yUn68ZOcRSDMMEg=', 'Content-Length': '0'} 2024-04-13T22:06:29.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbd20c5bc70d7470-00661b01e5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.346 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:06:29.346 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:06:29.346 DEBUG:boto:Method: PUT 2024-04-13T22:06:29.346 DEBUG:boto:Path: /vwnfns-55/d 2024-04-13T22:06:29.346 DEBUG:boto:Data: 2024-04-13T22:06:29.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:06:29.346 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:06:29.347 DEBUG:boto:Port: 8000 2024-04-13T22:06:29.347 DEBUG:boto:Params: {} 2024-04-13T22:06:29.347 DEBUG:boto:Token: None 2024-04-13T22:06:29.347 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:06:29 GMT /vwnfns-55/d 2024-04-13T22:06:29.347 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zNBGlFKpzUeAv/oF5ATLrT8Qkfg= 2024-04-13T22:06:29.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:06:29 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zNBGlFKpzUeAv/oF5ATLrT8Qkfg='} 2024-04-13T22:06:29.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e8170e36e5e08c-00661b01e5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:06:29.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:29.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:29.393 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:06:29.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:29.441 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:29.469 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045989.274510_676.1", 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:06:29.274510Z" 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.483 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:06:29.484 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713045989.274510_676.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:29.484 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:06:29.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:29.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:06:29.485 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:06:29.533 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:29.533 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:29.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.560+0000 7fcfcdd1c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:06:29.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.561+0000 7fcfcdd1c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:06:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713045989.274510_676.1", 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:06:29.274510Z", 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:06:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:06:29.579 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:06:29Z" 2024-04-13T22:06:29.580 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:06:29.581 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713045989.274510_676.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:06:29.274510Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:06:29Z" } 2024-04-13T22:06:29.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713045989.274510_676.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:06:29.274510Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:06:29.582 DEBUG:rgw_multi.tests:log_status={0: '1_1713045989.274510_676.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:29.582 DEBUG:rgw_multi.tests:sync_status={0: '1_1713045989.274510_676.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:06:29.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:06:29.582 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-55 2024-04-13T22:06:29.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:29.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:29.582 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:06:29.637 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:29.637 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:29.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:29.670+0000 7fdc45582780 1 waiting to reach incremental sync.. 2024-04-13T22:06:34.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:34.672+0000 7fdc45582780 1 waiting to reach incremental sync.. 2024-04-13T22:06:39.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:39.672+0000 7fdc45582780 1 waiting to reach incremental sync.. 2024-04-13T22:06:44.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:44.672+0000 7fdc45582780 1 waiting to reach incremental sync.. 2024-04-13T22:06:47.192 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:06:47.192+0000 7f5e93db2700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T22:06:49.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:49.673+0000 7fdc45582780 1 bucket sync caught up with source: 2024-04-13T22:06:49.674 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.29.6, , , , 00000000001.52.6, , , 00000000001.19.6, , 00000000001.65.6] 2024-04-13T22:06:49.674 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.29.6, , , , 00000000001.52.6, , , 00000000001.19.6, , 00000000001.65.6] 2024-04-13T22:06:49.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:49.673+0000 7fdc45582780 0 bucket checkpoint complete 2024-04-13T22:06:49.687 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-55 2024-04-13T22:06:49.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:49.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:06:49.688 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:06:49.742 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:06:49.743 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:06:49.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:49.775+0000 7ff8c3616780 1 waiting to reach incremental sync.. 2024-04-13T22:06:54.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:54.775+0000 7ff8c3616780 1 waiting to reach incremental sync.. 2024-04-13T22:06:59.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:06:59.776+0000 7ff8c3616780 1 waiting to reach incremental sync.. 2024-04-13T22:07:04.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:04.777+0000 7ff8c3616780 1 bucket sync caught up with source: 2024-04-13T22:07:04.779 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.450.6, , , , 00000000001.1001.6, , , 00000000001.1279.6, , 00000000001.2821.6] 2024-04-13T22:07:04.779 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.450.6, , , , 00000000001.1001.6, , , 00000000001.1279.6, , 00000000001.2821.6] 2024-04-13T22:07:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:04.777+0000 7ff8c3616780 0 bucket checkpoint complete 2024-04-13T22:07:04.796 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-55 zones={a2, a1} 2024-04-13T22:07:04.796 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.796 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.796 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.797 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:07:04.797 DEBUG:boto:Data: 2024-04-13T22:07:04.797 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.797 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.797 DEBUG:boto:Params: {} 2024-04-13T22:07:04.797 DEBUG:boto:Token: None 2024-04-13T22:07:04.798 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JeD+H4w0saiLjx49NIKoMijI6FQ= 2024-04-13T22:07:04.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JeD+H4w0saiLjx49NIKoMijI6FQ=', 'Content-Length': '0'} 2024-04-13T22:07:04.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d3cad93805e0d5e-00661b0208-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.801 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.801 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.801 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.801 DEBUG:boto:Path: /vwnfns-55/ 2024-04-13T22:07:04.801 DEBUG:boto:Data: 2024-04-13T22:07:04.801 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.801 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.801 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.802 DEBUG:boto:Params: {} 2024-04-13T22:07:04.802 DEBUG:boto:Token: None 2024-04-13T22:07:04.802 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.802 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JeD+H4w0saiLjx49NIKoMijI6FQ= 2024-04-13T22:07:04.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JeD+H4w0saiLjx49NIKoMijI6FQ=', 'Content-Length': '0'} 2024-04-13T22:07:04.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008220b8a45bf3de1-00661b0208-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.804 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:07:04.804 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.805 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.805 DEBUG:boto:path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.805 DEBUG:boto:auth_path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.805 DEBUG:boto:Method: GET 2024-04-13T22:07:04.805 DEBUG:boto:Path: /vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.805 DEBUG:boto:Data: 2024-04-13T22:07:04.805 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.805 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.805 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.805 DEBUG:boto:Params: {} 2024-04-13T22:07:04.805 DEBUG:boto:Token: None 2024-04-13T22:07:04.806 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/?versions 2024-04-13T22:07:04.806 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4= 2024-04-13T22:07:04.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4=', 'Content-Length': '0'} 2024-04-13T22:07:04.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aedc021c20afb0a4-00661b0208-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.808 DEBUG:boto:b'vwnfns-55999falseanulltrue2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.809 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:07:04.809 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:07:04.809 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:07:04.809 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:07:04.810 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:07:04.810 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.810 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.810 DEBUG:boto:path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.810 DEBUG:boto:auth_path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.810 DEBUG:boto:Method: GET 2024-04-13T22:07:04.810 DEBUG:boto:Path: /vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.810 DEBUG:boto:Data: 2024-04-13T22:07:04.810 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.811 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.811 DEBUG:boto:Params: {} 2024-04-13T22:07:04.811 DEBUG:boto:Token: None 2024-04-13T22:07:04.811 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/?versions 2024-04-13T22:07:04.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4= 2024-04-13T22:07:04.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4=', 'Content-Length': '0'} 2024-04-13T22:07:04.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb151bfd6170a375-00661b0208-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.814 DEBUG:boto:b'vwnfns-55999falseanulltrue2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.814 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:07:04.814 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:07:04.814 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:07:04.814 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:07:04.815 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.815 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.815 DEBUG:boto:path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.815 DEBUG:boto:auth_path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.815 DEBUG:boto:Method: GET 2024-04-13T22:07:04.815 DEBUG:boto:Path: /vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.815 DEBUG:boto:Data: 2024-04-13T22:07:04.815 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.815 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.815 DEBUG:boto:Params: {} 2024-04-13T22:07:04.816 DEBUG:boto:Token: None 2024-04-13T22:07:04.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/?versions 2024-04-13T22:07:04.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4= 2024-04-13T22:07:04.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4=', 'Content-Length': '0'} 2024-04-13T22:07:04.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001addccbf43e38b1-00661b0208-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.818 DEBUG:boto:b'vwnfns-55999falseanulltrue2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.819 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.819 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.819 DEBUG:boto:path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.819 DEBUG:boto:auth_path=/vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.819 DEBUG:boto:Method: GET 2024-04-13T22:07:04.819 DEBUG:boto:Path: /vwnfns-55/?versions&max-keys=999 2024-04-13T22:07:04.819 DEBUG:boto:Data: 2024-04-13T22:07:04.819 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.820 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.820 DEBUG:boto:Params: {} 2024-04-13T22:07:04.820 DEBUG:boto:Token: None 2024-04-13T22:07:04.820 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/?versions 2024-04-13T22:07:04.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4= 2024-04-13T22:07:04.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2p6NZjZrZ100Mii0xslktd8qZq4=', 'Content-Length': '0'} 2024-04-13T22:07:04.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a086bfaadb0a58a1-00661b0208-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.823 DEBUG:boto:b'vwnfns-55999falseanulltrue2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:07:04.824 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.824 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.824 DEBUG:boto:path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.824 DEBUG:boto:auth_path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.824 DEBUG:boto:Method: GET 2024-04-13T22:07:04.824 DEBUG:boto:Path: /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.824 DEBUG:boto:Data: 2024-04-13T22:07:04.824 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.824 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.824 DEBUG:boto:Params: {} 2024-04-13T22:07:04.825 DEBUG:boto:Token: None 2024-04-13T22:07:04.825 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.825 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8= 2024-04-13T22:07:04.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8=', 'Content-Length': '0'} 2024-04-13T22:07:04.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd39b4815a0fd4ee-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.827 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.827 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.827 DEBUG:boto:path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.827 DEBUG:boto:auth_path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.827 DEBUG:boto:Method: GET 2024-04-13T22:07:04.827 DEBUG:boto:Path: /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.828 DEBUG:boto:Data: 2024-04-13T22:07:04.828 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.828 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.828 DEBUG:boto:Params: {} 2024-04-13T22:07:04.828 DEBUG:boto:Token: None 2024-04-13T22:07:04.828 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8= 2024-04-13T22:07:04.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8=', 'Content-Length': '0'} 2024-04-13T22:07:04.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d3f2eeee95d555-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.830 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.830 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.831 DEBUG:boto:path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.831 DEBUG:boto:auth_path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.831 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.831 DEBUG:boto:Path: /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.831 DEBUG:boto:Data: 2024-04-13T22:07:04.831 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.831 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.831 DEBUG:boto:Params: {} 2024-04-13T22:07:04.831 DEBUG:boto:Token: None 2024-04-13T22:07:04.831 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.832 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DJzrINcb3VY9BNxWgxWe550o97E= 2024-04-13T22:07:04.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DJzrINcb3VY9BNxWgxWe550o97E=', 'Content-Length': '0'} 2024-04-13T22:07:04.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa0cbb87abdbd93-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.834 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.834 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.834 DEBUG:boto:path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.834 DEBUG:boto:auth_path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.834 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.834 DEBUG:boto:Path: /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.834 DEBUG:boto:Data: 2024-04-13T22:07:04.834 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.834 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.835 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.835 DEBUG:boto:Params: {} 2024-04-13T22:07:04.835 DEBUG:boto:Token: None 2024-04-13T22:07:04.835 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.835 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DJzrINcb3VY9BNxWgxWe550o97E= 2024-04-13T22:07:04.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DJzrINcb3VY9BNxWgxWe550o97E=', 'Content-Length': '0'} 2024-04-13T22:07:04.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a2d19b1e559cd7-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:07:04.837 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.837 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.837 DEBUG:boto:path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.837 DEBUG:boto:auth_path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.837 DEBUG:boto:Method: GET 2024-04-13T22:07:04.838 DEBUG:boto:Path: /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.838 DEBUG:boto:Data: 2024-04-13T22:07:04.838 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.838 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.838 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.838 DEBUG:boto:Params: {} 2024-04-13T22:07:04.838 DEBUG:boto:Token: None 2024-04-13T22:07:04.838 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.838 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8= 2024-04-13T22:07:04.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8=', 'Content-Length': '0'} 2024-04-13T22:07:04.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915135dc900bc7d6-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.840 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.841 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.841 DEBUG:boto:path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.841 DEBUG:boto:auth_path=/vwnfns-55/a?versionId=null 2024-04-13T22:07:04.841 DEBUG:boto:Method: GET 2024-04-13T22:07:04.841 DEBUG:boto:Path: /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.841 DEBUG:boto:Data: 2024-04-13T22:07:04.841 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.841 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.841 DEBUG:boto:Params: {} 2024-04-13T22:07:04.841 DEBUG:boto:Token: None 2024-04-13T22:07:04.842 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a?versionId=null 2024-04-13T22:07:04.842 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8= 2024-04-13T22:07:04.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iWyapJqQfx+u5Jbe+SIUT6DKuJ8=', 'Content-Length': '0'} 2024-04-13T22:07:04.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a836f476bfaa19-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.844 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.844 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.844 DEBUG:boto:path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.844 DEBUG:boto:auth_path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.844 DEBUG:boto:Method: GET 2024-04-13T22:07:04.844 DEBUG:boto:Path: /vwnfns-55/?prefix=a 2024-04-13T22:07:04.844 DEBUG:boto:Data: 2024-04-13T22:07:04.844 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.845 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.845 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.845 DEBUG:boto:Params: {} 2024-04-13T22:07:04.845 DEBUG:boto:Token: None 2024-04-13T22:07:04.845 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad64b24226560c1f-00661b0208-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.848 DEBUG:boto:b'vwnfns-55a1000falsea2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.848 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.848 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.848 DEBUG:boto:path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.848 DEBUG:boto:auth_path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.848 DEBUG:boto:Method: GET 2024-04-13T22:07:04.849 DEBUG:boto:Path: /vwnfns-55/?prefix=a 2024-04-13T22:07:04.849 DEBUG:boto:Data: 2024-04-13T22:07:04.849 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.849 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.849 DEBUG:boto:Params: {} 2024-04-13T22:07:04.849 DEBUG:boto:Token: None 2024-04-13T22:07:04.849 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b8dd5caeba1b4a-00661b0208-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.852 DEBUG:boto:b'vwnfns-55a1000falsea2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.852 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.852 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.852 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.852 DEBUG:boto:Path: /vwnfns-55/a 2024-04-13T22:07:04.852 DEBUG:boto:Data: 2024-04-13T22:07:04.852 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.853 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.853 DEBUG:boto:Params: {} 2024-04-13T22:07:04.853 DEBUG:boto:Token: None 2024-04-13T22:07:04.853 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a 2024-04-13T22:07:04.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+/YPfZY45QHbVj/eQpKURjN0ODc= 2024-04-13T22:07:04.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+/YPfZY45QHbVj/eQpKURjN0ODc=', 'Content-Length': '0'} 2024-04-13T22:07:04.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c1727fab828c715-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.855 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.855 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.855 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.855 DEBUG:boto:Path: /vwnfns-55/a 2024-04-13T22:07:04.855 DEBUG:boto:Data: 2024-04-13T22:07:04.855 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.855 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.855 DEBUG:boto:Params: {} 2024-04-13T22:07:04.856 DEBUG:boto:Token: None 2024-04-13T22:07:04.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a 2024-04-13T22:07:04.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+/YPfZY45QHbVj/eQpKURjN0ODc= 2024-04-13T22:07:04.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+/YPfZY45QHbVj/eQpKURjN0ODc=', 'Content-Length': '0'} 2024-04-13T22:07:04.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d072026c98359f29-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:07:04.858 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.858 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.858 DEBUG:boto:Method: GET 2024-04-13T22:07:04.858 DEBUG:boto:Path: /vwnfns-55/a 2024-04-13T22:07:04.858 DEBUG:boto:Data: 2024-04-13T22:07:04.858 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.858 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.858 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.858 DEBUG:boto:Params: {} 2024-04-13T22:07:04.858 DEBUG:boto:Token: None 2024-04-13T22:07:04.859 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a 2024-04-13T22:07:04.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ja+6P6eyqdlRiRukWW5W4Y6+FzE= 2024-04-13T22:07:04.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ja+6P6eyqdlRiRukWW5W4Y6+FzE=', 'Content-Length': '0'} 2024-04-13T22:07:04.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000250891f5172b391a-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.902 DEBUG:boto:path=/vwnfns-55/a 2024-04-13T22:07:04.902 DEBUG:boto:auth_path=/vwnfns-55/a 2024-04-13T22:07:04.903 DEBUG:boto:Method: GET 2024-04-13T22:07:04.903 DEBUG:boto:Path: /vwnfns-55/a 2024-04-13T22:07:04.903 DEBUG:boto:Data: 2024-04-13T22:07:04.903 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.903 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.903 DEBUG:boto:Params: {} 2024-04-13T22:07:04.903 DEBUG:boto:Token: None 2024-04-13T22:07:04.903 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/a 2024-04-13T22:07:04.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ja+6P6eyqdlRiRukWW5W4Y6+FzE= 2024-04-13T22:07:04.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ja+6P6eyqdlRiRukWW5W4Y6+FzE=', 'Content-Length': '0'} 2024-04-13T22:07:04.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008021c98353e497d6-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.906 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.906 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.906 DEBUG:boto:path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.906 DEBUG:boto:auth_path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.906 DEBUG:boto:Method: GET 2024-04-13T22:07:04.906 DEBUG:boto:Path: /vwnfns-55/?prefix=a 2024-04-13T22:07:04.906 DEBUG:boto:Data: 2024-04-13T22:07:04.906 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.906 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.907 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.907 DEBUG:boto:Params: {} 2024-04-13T22:07:04.907 DEBUG:boto:Token: None 2024-04-13T22:07:04.907 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2ca9d03dced756-00661b0208-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.909 DEBUG:boto:b'vwnfns-55a1000falsea2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.910 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.910 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.910 DEBUG:boto:path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.910 DEBUG:boto:auth_path=/vwnfns-55/?prefix=a 2024-04-13T22:07:04.910 DEBUG:boto:Method: GET 2024-04-13T22:07:04.910 DEBUG:boto:Path: /vwnfns-55/?prefix=a 2024-04-13T22:07:04.911 DEBUG:boto:Data: 2024-04-13T22:07:04.911 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.911 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.911 DEBUG:boto:Params: {} 2024-04-13T22:07:04.911 DEBUG:boto:Token: None 2024-04-13T22:07:04.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb534b1845851283-00661b0208-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.914 DEBUG:boto:b'vwnfns-55a1000falsea2024-04-13T22:06:29.283Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:07:04.914 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.914 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.915 DEBUG:boto:path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.915 DEBUG:boto:auth_path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.915 DEBUG:boto:Method: GET 2024-04-13T22:07:04.915 DEBUG:boto:Path: /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.915 DEBUG:boto:Data: 2024-04-13T22:07:04.915 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.915 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.915 DEBUG:boto:Params: {} 2024-04-13T22:07:04.915 DEBUG:boto:Token: None 2024-04-13T22:07:04.915 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE= 2024-04-13T22:07:04.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE=', 'Content-Length': '0'} 2024-04-13T22:07:04.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a17cf0cd9ee088-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.918 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.918 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.918 DEBUG:boto:path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.918 DEBUG:boto:auth_path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.918 DEBUG:boto:Method: GET 2024-04-13T22:07:04.918 DEBUG:boto:Path: /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.918 DEBUG:boto:Data: 2024-04-13T22:07:04.918 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.919 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.919 DEBUG:boto:Params: {} 2024-04-13T22:07:04.919 DEBUG:boto:Token: None 2024-04-13T22:07:04.919 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE= 2024-04-13T22:07:04.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE=', 'Content-Length': '0'} 2024-04-13T22:07:04.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2ae79dacef3b74-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.921 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.921 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.922 DEBUG:boto:path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.922 DEBUG:boto:auth_path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.922 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.922 DEBUG:boto:Path: /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.922 DEBUG:boto:Data: 2024-04-13T22:07:04.922 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.922 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.922 DEBUG:boto:Params: {} 2024-04-13T22:07:04.922 DEBUG:boto:Token: None 2024-04-13T22:07:04.923 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wCNmGKmh8UDwXkBuTRhaTFigy+4= 2024-04-13T22:07:04.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wCNmGKmh8UDwXkBuTRhaTFigy+4=', 'Content-Length': '0'} 2024-04-13T22:07:04.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6120e2f1e86cdf-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.925 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.925 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.925 DEBUG:boto:path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.925 DEBUG:boto:auth_path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.925 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.925 DEBUG:boto:Path: /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.925 DEBUG:boto:Data: 2024-04-13T22:07:04.925 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.925 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.925 DEBUG:boto:Params: {} 2024-04-13T22:07:04.926 DEBUG:boto:Token: None 2024-04-13T22:07:04.926 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wCNmGKmh8UDwXkBuTRhaTFigy+4= 2024-04-13T22:07:04.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wCNmGKmh8UDwXkBuTRhaTFigy+4=', 'Content-Length': '0'} 2024-04-13T22:07:04.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9ea07485f9b5a9-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:07:04.928 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.928 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.928 DEBUG:boto:path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.928 DEBUG:boto:auth_path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.928 DEBUG:boto:Method: GET 2024-04-13T22:07:04.928 DEBUG:boto:Path: /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.928 DEBUG:boto:Data: 2024-04-13T22:07:04.928 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.929 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.929 DEBUG:boto:Params: {} 2024-04-13T22:07:04.929 DEBUG:boto:Token: None 2024-04-13T22:07:04.929 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE= 2024-04-13T22:07:04.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE=', 'Content-Length': '0'} 2024-04-13T22:07:04.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18bed548ce82ef1-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.931 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.931 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.931 DEBUG:boto:path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.931 DEBUG:boto:auth_path=/vwnfns-55/b?versionId=null 2024-04-13T22:07:04.932 DEBUG:boto:Method: GET 2024-04-13T22:07:04.932 DEBUG:boto:Path: /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.932 DEBUG:boto:Data: 2024-04-13T22:07:04.932 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.932 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.932 DEBUG:boto:Params: {} 2024-04-13T22:07:04.932 DEBUG:boto:Token: None 2024-04-13T22:07:04.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b?versionId=null 2024-04-13T22:07:04.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE= 2024-04-13T22:07:04.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCWA+fDHm42iO5Exasd8p831kkE=', 'Content-Length': '0'} 2024-04-13T22:07:04.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002620d2645126a74f-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.935 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.935 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.935 DEBUG:boto:path=/vwnfns-55/?prefix=b 2024-04-13T22:07:04.935 DEBUG:boto:auth_path=/vwnfns-55/?prefix=b 2024-04-13T22:07:04.935 DEBUG:boto:Method: GET 2024-04-13T22:07:04.935 DEBUG:boto:Path: /vwnfns-55/?prefix=b 2024-04-13T22:07:04.935 DEBUG:boto:Data: 2024-04-13T22:07:04.936 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.936 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.936 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.936 DEBUG:boto:Params: {} 2024-04-13T22:07:04.936 DEBUG:boto:Token: None 2024-04-13T22:07:04.936 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.936 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f4b4ede35222092-00661b0208-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.938 DEBUG:boto:b'vwnfns-55b1000falseb2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.939 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.939 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.939 DEBUG:boto:path=/vwnfns-55/?prefix=b 2024-04-13T22:07:04.939 DEBUG:boto:auth_path=/vwnfns-55/?prefix=b 2024-04-13T22:07:04.939 DEBUG:boto:Method: GET 2024-04-13T22:07:04.939 DEBUG:boto:Path: /vwnfns-55/?prefix=b 2024-04-13T22:07:04.939 DEBUG:boto:Data: 2024-04-13T22:07:04.939 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.939 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.940 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.940 DEBUG:boto:Params: {} 2024-04-13T22:07:04.940 DEBUG:boto:Token: None 2024-04-13T22:07:04.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682b404abe686820-00661b0208-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.982 DEBUG:boto:b'vwnfns-55b1000falseb2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:04.983 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.983 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.983 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.983 DEBUG:boto:Path: /vwnfns-55/b 2024-04-13T22:07:04.983 DEBUG:boto:Data: 2024-04-13T22:07:04.983 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.984 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.984 DEBUG:boto:Params: {} 2024-04-13T22:07:04.984 DEBUG:boto:Token: None 2024-04-13T22:07:04.984 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b 2024-04-13T22:07:04.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OV0QUpyM+mP92uFIPsu/w3RZ63Y= 2024-04-13T22:07:04.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OV0QUpyM+mP92uFIPsu/w3RZ63Y=', 'Content-Length': '0'} 2024-04-13T22:07:04.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e667d043bcd1b5-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.986 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.986 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.987 DEBUG:boto:Method: HEAD 2024-04-13T22:07:04.987 DEBUG:boto:Path: /vwnfns-55/b 2024-04-13T22:07:04.987 DEBUG:boto:Data: 2024-04-13T22:07:04.987 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.987 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.987 DEBUG:boto:Params: {} 2024-04-13T22:07:04.987 DEBUG:boto:Token: None 2024-04-13T22:07:04.987 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b 2024-04-13T22:07:04.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OV0QUpyM+mP92uFIPsu/w3RZ63Y= 2024-04-13T22:07:04.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OV0QUpyM+mP92uFIPsu/w3RZ63Y=', 'Content-Length': '0'} 2024-04-13T22:07:04.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008425a5d376bf7cba-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:07:04.989 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.989 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.989 DEBUG:boto:Method: GET 2024-04-13T22:07:04.989 DEBUG:boto:Path: /vwnfns-55/b 2024-04-13T22:07:04.990 DEBUG:boto:Data: 2024-04-13T22:07:04.990 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.990 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.990 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.990 DEBUG:boto:Params: {} 2024-04-13T22:07:04.990 DEBUG:boto:Token: None 2024-04-13T22:07:04.990 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b 2024-04-13T22:07:04.990 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wvgBNMvT5QoN4+TVCxwal3z3yqM= 2024-04-13T22:07:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wvgBNMvT5QoN4+TVCxwal3z3yqM=', 'Content-Length': '0'} 2024-04-13T22:07:04.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a231a1e4484edd7-00661b0208-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.992 DEBUG:boto:path=/vwnfns-55/b 2024-04-13T22:07:04.993 DEBUG:boto:auth_path=/vwnfns-55/b 2024-04-13T22:07:04.993 DEBUG:boto:Method: GET 2024-04-13T22:07:04.993 DEBUG:boto:Path: /vwnfns-55/b 2024-04-13T22:07:04.993 DEBUG:boto:Data: 2024-04-13T22:07:04.993 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.993 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:04.993 DEBUG:boto:Port: 8000 2024-04-13T22:07:04.993 DEBUG:boto:Params: {} 2024-04-13T22:07:04.993 DEBUG:boto:Token: None 2024-04-13T22:07:04.993 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/b 2024-04-13T22:07:04.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wvgBNMvT5QoN4+TVCxwal3z3yqM= 2024-04-13T22:07:04.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wvgBNMvT5QoN4+TVCxwal3z3yqM=', 'Content-Length': '0'} 2024-04-13T22:07:04.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4e004458af42705-00661b0208-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.996 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:04.996 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:04.996 DEBUG:boto:path=/vwnfns-55/?prefix=b 2024-04-13T22:07:04.996 DEBUG:boto:auth_path=/vwnfns-55/?prefix=b 2024-04-13T22:07:04.996 DEBUG:boto:Method: GET 2024-04-13T22:07:04.996 DEBUG:boto:Path: /vwnfns-55/?prefix=b 2024-04-13T22:07:04.996 DEBUG:boto:Data: 2024-04-13T22:07:04.996 DEBUG:boto:Headers: {} 2024-04-13T22:07:04.996 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:04.997 DEBUG:boto:Port: 8001 2024-04-13T22:07:04.997 DEBUG:boto:Params: {} 2024-04-13T22:07:04.997 DEBUG:boto:Token: None 2024-04-13T22:07:04.997 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:04 GMT /vwnfns-55/ 2024-04-13T22:07:04.997 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU= 2024-04-13T22:07:04.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9FyvgrhUb+ACR1k3JxF+dOtniyU=', 'Content-Length': '0'} 2024-04-13T22:07:04.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000360782e3f6273c9f-00661b0208-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:04.999 DEBUG:boto:b'vwnfns-55b1000falseb2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.000 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.000 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.000 DEBUG:boto:path=/vwnfns-55/?prefix=b 2024-04-13T22:07:05.000 DEBUG:boto:auth_path=/vwnfns-55/?prefix=b 2024-04-13T22:07:05.000 DEBUG:boto:Method: GET 2024-04-13T22:07:05.000 DEBUG:boto:Path: /vwnfns-55/?prefix=b 2024-04-13T22:07:05.000 DEBUG:boto:Data: 2024-04-13T22:07:05.000 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.000 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.000 DEBUG:boto:Params: {} 2024-04-13T22:07:05.001 DEBUG:boto:Token: None 2024-04-13T22:07:05.001 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9913ffdf7bf4b8c-00661b0209-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.003 DEBUG:boto:b'vwnfns-55b1000falseb2024-04-13T22:06:29.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:07:05.003 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.003 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.003 DEBUG:boto:path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.004 DEBUG:boto:auth_path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.004 DEBUG:boto:Method: GET 2024-04-13T22:07:05.004 DEBUG:boto:Path: /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.004 DEBUG:boto:Data: 2024-04-13T22:07:05.004 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.004 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.004 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.004 DEBUG:boto:Params: {} 2024-04-13T22:07:05.004 DEBUG:boto:Token: None 2024-04-13T22:07:05.005 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.005 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc= 2024-04-13T22:07:05.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc=', 'Content-Length': '0'} 2024-04-13T22:07:05.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a4cf5b8dc39006-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.050 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.050 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.051 DEBUG:boto:path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.051 DEBUG:boto:auth_path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.051 DEBUG:boto:Method: GET 2024-04-13T22:07:05.051 DEBUG:boto:Path: /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.051 DEBUG:boto:Data: 2024-04-13T22:07:05.051 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.051 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.051 DEBUG:boto:Params: {} 2024-04-13T22:07:05.051 DEBUG:boto:Token: None 2024-04-13T22:07:05.051 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc= 2024-04-13T22:07:05.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc=', 'Content-Length': '0'} 2024-04-13T22:07:05.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c752793151de781a-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.054 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.054 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.054 DEBUG:boto:path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.054 DEBUG:boto:auth_path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.054 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.054 DEBUG:boto:Path: /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.054 DEBUG:boto:Data: 2024-04-13T22:07:05.055 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.055 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.055 DEBUG:boto:Params: {} 2024-04-13T22:07:05.055 DEBUG:boto:Token: None 2024-04-13T22:07:05.055 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7UG2i8XBjr+gqQ9MIrtS/QolkjY= 2024-04-13T22:07:05.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7UG2i8XBjr+gqQ9MIrtS/QolkjY=', 'Content-Length': '0'} 2024-04-13T22:07:05.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69c52982777669e-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.057 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.057 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.057 DEBUG:boto:path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.057 DEBUG:boto:auth_path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.057 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.057 DEBUG:boto:Path: /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.058 DEBUG:boto:Data: 2024-04-13T22:07:05.058 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.058 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.058 DEBUG:boto:Params: {} 2024-04-13T22:07:05.058 DEBUG:boto:Token: None 2024-04-13T22:07:05.058 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.058 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7UG2i8XBjr+gqQ9MIrtS/QolkjY= 2024-04-13T22:07:05.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7UG2i8XBjr+gqQ9MIrtS/QolkjY=', 'Content-Length': '0'} 2024-04-13T22:07:05.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998e8706524dc46e-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:07:05.060 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.060 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.060 DEBUG:boto:path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.061 DEBUG:boto:auth_path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.061 DEBUG:boto:Method: GET 2024-04-13T22:07:05.061 DEBUG:boto:Path: /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.061 DEBUG:boto:Data: 2024-04-13T22:07:05.061 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.061 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.061 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.061 DEBUG:boto:Params: {} 2024-04-13T22:07:05.061 DEBUG:boto:Token: None 2024-04-13T22:07:05.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.062 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc= 2024-04-13T22:07:05.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc=', 'Content-Length': '0'} 2024-04-13T22:07:05.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cda4b8bebc4d67d-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.064 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.064 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.064 DEBUG:boto:path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.064 DEBUG:boto:auth_path=/vwnfns-55/c?versionId=null 2024-04-13T22:07:05.064 DEBUG:boto:Method: GET 2024-04-13T22:07:05.064 DEBUG:boto:Path: /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.065 DEBUG:boto:Data: 2024-04-13T22:07:05.065 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.065 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.065 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.065 DEBUG:boto:Params: {} 2024-04-13T22:07:05.065 DEBUG:boto:Token: None 2024-04-13T22:07:05.065 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c?versionId=null 2024-04-13T22:07:05.065 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc= 2024-04-13T22:07:05.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R+r2VhjnvsTDjD9R9iIBFb/+Lkc=', 'Content-Length': '0'} 2024-04-13T22:07:05.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a10d4e725fbf60-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.068 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.068 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.068 DEBUG:boto:path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.068 DEBUG:boto:auth_path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.068 DEBUG:boto:Method: GET 2024-04-13T22:07:05.068 DEBUG:boto:Path: /vwnfns-55/?prefix=c 2024-04-13T22:07:05.068 DEBUG:boto:Data: 2024-04-13T22:07:05.069 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.069 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.069 DEBUG:boto:Params: {} 2024-04-13T22:07:05.069 DEBUG:boto:Token: None 2024-04-13T22:07:05.069 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.069 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf8d346d20a801d3-00661b0209-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.072 DEBUG:boto:b'vwnfns-55c1000falsec2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.072 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.073 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.073 DEBUG:boto:path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.073 DEBUG:boto:auth_path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.073 DEBUG:boto:Method: GET 2024-04-13T22:07:05.073 DEBUG:boto:Path: /vwnfns-55/?prefix=c 2024-04-13T22:07:05.073 DEBUG:boto:Data: 2024-04-13T22:07:05.073 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.074 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.074 DEBUG:boto:Params: {} 2024-04-13T22:07:05.074 DEBUG:boto:Token: None 2024-04-13T22:07:05.074 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061176d108d96a41a-00661b0209-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.077 DEBUG:boto:b'vwnfns-55c1000falsec2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.077 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.077 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.078 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.078 DEBUG:boto:Path: /vwnfns-55/c 2024-04-13T22:07:05.078 DEBUG:boto:Data: 2024-04-13T22:07:05.078 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.078 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.078 DEBUG:boto:Params: {} 2024-04-13T22:07:05.078 DEBUG:boto:Token: None 2024-04-13T22:07:05.078 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c 2024-04-13T22:07:05.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xuKP+x4ZdFrEuws6Ny8Jx1k0Dag= 2024-04-13T22:07:05.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xuKP+x4ZdFrEuws6Ny8Jx1k0Dag=', 'Content-Length': '0'} 2024-04-13T22:07:05.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2901dddbcf3ae76-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.081 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.081 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.081 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.081 DEBUG:boto:Path: /vwnfns-55/c 2024-04-13T22:07:05.081 DEBUG:boto:Data: 2024-04-13T22:07:05.081 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.081 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.081 DEBUG:boto:Params: {} 2024-04-13T22:07:05.081 DEBUG:boto:Token: None 2024-04-13T22:07:05.082 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c 2024-04-13T22:07:05.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xuKP+x4ZdFrEuws6Ny8Jx1k0Dag= 2024-04-13T22:07:05.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xuKP+x4ZdFrEuws6Ny8Jx1k0Dag=', 'Content-Length': '0'} 2024-04-13T22:07:05.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea101ebde870558-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:07:05.084 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.084 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.084 DEBUG:boto:Method: GET 2024-04-13T22:07:05.084 DEBUG:boto:Path: /vwnfns-55/c 2024-04-13T22:07:05.084 DEBUG:boto:Data: 2024-04-13T22:07:05.084 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.085 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.085 DEBUG:boto:Params: {} 2024-04-13T22:07:05.085 DEBUG:boto:Token: None 2024-04-13T22:07:05.085 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c 2024-04-13T22:07:05.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fGNyiL7pgqSvetSiTKbwbf+5hN4= 2024-04-13T22:07:05.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fGNyiL7pgqSvetSiTKbwbf+5hN4=', 'Content-Length': '0'} 2024-04-13T22:07:05.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c47bce26264dfa1-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.131 DEBUG:boto:path=/vwnfns-55/c 2024-04-13T22:07:05.131 DEBUG:boto:auth_path=/vwnfns-55/c 2024-04-13T22:07:05.132 DEBUG:boto:Method: GET 2024-04-13T22:07:05.132 DEBUG:boto:Path: /vwnfns-55/c 2024-04-13T22:07:05.132 DEBUG:boto:Data: 2024-04-13T22:07:05.132 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.132 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.132 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.133 DEBUG:boto:Params: {} 2024-04-13T22:07:05.133 DEBUG:boto:Token: None 2024-04-13T22:07:05.134 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/c 2024-04-13T22:07:05.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fGNyiL7pgqSvetSiTKbwbf+5hN4= 2024-04-13T22:07:05.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fGNyiL7pgqSvetSiTKbwbf+5hN4=', 'Content-Length': '0'} 2024-04-13T22:07:05.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6dfbbd339e3525-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.136 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.136 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.136 DEBUG:boto:path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.136 DEBUG:boto:auth_path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.136 DEBUG:boto:Method: GET 2024-04-13T22:07:05.136 DEBUG:boto:Path: /vwnfns-55/?prefix=c 2024-04-13T22:07:05.136 DEBUG:boto:Data: 2024-04-13T22:07:05.137 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.137 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.137 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.137 DEBUG:boto:Params: {} 2024-04-13T22:07:05.137 DEBUG:boto:Token: None 2024-04-13T22:07:05.137 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e425f2bdde32df6e-00661b0209-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.140 DEBUG:boto:b'vwnfns-55c1000falsec2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.140 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.140 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.140 DEBUG:boto:path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.141 DEBUG:boto:auth_path=/vwnfns-55/?prefix=c 2024-04-13T22:07:05.141 DEBUG:boto:Method: GET 2024-04-13T22:07:05.141 DEBUG:boto:Path: /vwnfns-55/?prefix=c 2024-04-13T22:07:05.141 DEBUG:boto:Data: 2024-04-13T22:07:05.141 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.141 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.141 DEBUG:boto:Params: {} 2024-04-13T22:07:05.141 DEBUG:boto:Token: None 2024-04-13T22:07:05.142 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007be725cd992b1ee2-00661b0209-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.144 DEBUG:boto:b'vwnfns-55c1000falsec2024-04-13T22:06:29.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:07:05.144 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.144 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.145 DEBUG:boto:path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.145 DEBUG:boto:auth_path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.145 DEBUG:boto:Method: GET 2024-04-13T22:07:05.145 DEBUG:boto:Path: /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.145 DEBUG:boto:Data: 2024-04-13T22:07:05.145 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.145 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.145 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.145 DEBUG:boto:Params: {} 2024-04-13T22:07:05.145 DEBUG:boto:Token: None 2024-04-13T22:07:05.145 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.146 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s= 2024-04-13T22:07:05.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s=', 'Content-Length': '0'} 2024-04-13T22:07:05.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f80b53317ef345b-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.147 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.148 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.148 DEBUG:boto:path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.148 DEBUG:boto:auth_path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.148 DEBUG:boto:Method: GET 2024-04-13T22:07:05.148 DEBUG:boto:Path: /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.148 DEBUG:boto:Data: 2024-04-13T22:07:05.148 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.148 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.149 DEBUG:boto:Params: {} 2024-04-13T22:07:05.149 DEBUG:boto:Token: None 2024-04-13T22:07:05.149 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s= 2024-04-13T22:07:05.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s=', 'Content-Length': '0'} 2024-04-13T22:07:05.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006acb78e7fdb160d9-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.154 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.154 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.154 DEBUG:boto:path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.154 DEBUG:boto:auth_path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.154 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.155 DEBUG:boto:Path: /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.155 DEBUG:boto:Data: 2024-04-13T22:07:05.155 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.155 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.155 DEBUG:boto:Params: {} 2024-04-13T22:07:05.155 DEBUG:boto:Token: None 2024-04-13T22:07:05.155 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wQLDe0/DMAH/iT3zHZXF765UTeU= 2024-04-13T22:07:05.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wQLDe0/DMAH/iT3zHZXF765UTeU=', 'Content-Length': '0'} 2024-04-13T22:07:05.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e8378e5590c06b-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.158 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.158 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.158 DEBUG:boto:path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.158 DEBUG:boto:auth_path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.158 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.158 DEBUG:boto:Path: /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.158 DEBUG:boto:Data: 2024-04-13T22:07:05.158 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.158 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.158 DEBUG:boto:Params: {} 2024-04-13T22:07:05.159 DEBUG:boto:Token: None 2024-04-13T22:07:05.159 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.159 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wQLDe0/DMAH/iT3zHZXF765UTeU= 2024-04-13T22:07:05.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wQLDe0/DMAH/iT3zHZXF765UTeU=', 'Content-Length': '0'} 2024-04-13T22:07:05.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001102449793c1ad63-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:07:05.161 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.161 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.161 DEBUG:boto:path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.161 DEBUG:boto:auth_path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.161 DEBUG:boto:Method: GET 2024-04-13T22:07:05.161 DEBUG:boto:Path: /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.161 DEBUG:boto:Data: 2024-04-13T22:07:05.161 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.162 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.162 DEBUG:boto:Params: {} 2024-04-13T22:07:05.162 DEBUG:boto:Token: None 2024-04-13T22:07:05.162 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.162 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s= 2024-04-13T22:07:05.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s=', 'Content-Length': '0'} 2024-04-13T22:07:05.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3abaea92a03acec-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.164 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.164 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.164 DEBUG:boto:path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.165 DEBUG:boto:auth_path=/vwnfns-55/d?versionId=null 2024-04-13T22:07:05.165 DEBUG:boto:Method: GET 2024-04-13T22:07:05.165 DEBUG:boto:Path: /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.165 DEBUG:boto:Data: 2024-04-13T22:07:05.165 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.165 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.165 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.165 DEBUG:boto:Params: {} 2024-04-13T22:07:05.165 DEBUG:boto:Token: None 2024-04-13T22:07:05.165 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d?versionId=null 2024-04-13T22:07:05.166 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s= 2024-04-13T22:07:05.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5DlWYJ2kPDlr47DLJ8RdWS/DY3s=', 'Content-Length': '0'} 2024-04-13T22:07:05.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e69e13978ddf527-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.167 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.168 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.168 DEBUG:boto:path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.168 DEBUG:boto:auth_path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.168 DEBUG:boto:Method: GET 2024-04-13T22:07:05.168 DEBUG:boto:Path: /vwnfns-55/?prefix=d 2024-04-13T22:07:05.168 DEBUG:boto:Data: 2024-04-13T22:07:05.168 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.168 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.168 DEBUG:boto:Params: {} 2024-04-13T22:07:05.169 DEBUG:boto:Token: None 2024-04-13T22:07:05.169 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9b1d09eeb4f018-00661b0209-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.171 DEBUG:boto:b'vwnfns-55d1000falsed2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.172 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.172 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.172 DEBUG:boto:path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.172 DEBUG:boto:auth_path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.172 DEBUG:boto:Method: GET 2024-04-13T22:07:05.172 DEBUG:boto:Path: /vwnfns-55/?prefix=d 2024-04-13T22:07:05.172 DEBUG:boto:Data: 2024-04-13T22:07:05.172 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.172 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.172 DEBUG:boto:Params: {} 2024-04-13T22:07:05.173 DEBUG:boto:Token: None 2024-04-13T22:07:05.173 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d60072b816fb1a1-00661b0209-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.175 DEBUG:boto:b'vwnfns-55d1000falsed2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.175 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.175 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.175 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.175 DEBUG:boto:Path: /vwnfns-55/d 2024-04-13T22:07:05.175 DEBUG:boto:Data: 2024-04-13T22:07:05.176 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.176 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.176 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.176 DEBUG:boto:Params: {} 2024-04-13T22:07:05.176 DEBUG:boto:Token: None 2024-04-13T22:07:05.176 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d 2024-04-13T22:07:05.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:npPlbStYRWDc/8j03U4qt9fiPj4= 2024-04-13T22:07:05.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:npPlbStYRWDc/8j03U4qt9fiPj4=', 'Content-Length': '0'} 2024-04-13T22:07:05.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f02ab11d4845af-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.178 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.178 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.178 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.178 DEBUG:boto:Path: /vwnfns-55/d 2024-04-13T22:07:05.178 DEBUG:boto:Data: 2024-04-13T22:07:05.178 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.179 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.179 DEBUG:boto:Params: {} 2024-04-13T22:07:05.179 DEBUG:boto:Token: None 2024-04-13T22:07:05.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d 2024-04-13T22:07:05.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:npPlbStYRWDc/8j03U4qt9fiPj4= 2024-04-13T22:07:05.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:npPlbStYRWDc/8j03U4qt9fiPj4=', 'Content-Length': '0'} 2024-04-13T22:07:05.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000650f829aa6528010-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:07:05.181 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.181 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.181 DEBUG:boto:Method: GET 2024-04-13T22:07:05.181 DEBUG:boto:Path: /vwnfns-55/d 2024-04-13T22:07:05.181 DEBUG:boto:Data: 2024-04-13T22:07:05.181 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.181 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.182 DEBUG:boto:Params: {} 2024-04-13T22:07:05.182 DEBUG:boto:Token: None 2024-04-13T22:07:05.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d 2024-04-13T22:07:05.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xmxpq5+feVHYSU2au8jTxMKj/mQ= 2024-04-13T22:07:05.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xmxpq5+feVHYSU2au8jTxMKj/mQ=', 'Content-Length': '0'} 2024-04-13T22:07:05.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007312bad2d905b845-00661b0209-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.226 DEBUG:boto:path=/vwnfns-55/d 2024-04-13T22:07:05.226 DEBUG:boto:auth_path=/vwnfns-55/d 2024-04-13T22:07:05.227 DEBUG:boto:Method: GET 2024-04-13T22:07:05.227 DEBUG:boto:Path: /vwnfns-55/d 2024-04-13T22:07:05.227 DEBUG:boto:Data: 2024-04-13T22:07:05.227 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.227 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.227 DEBUG:boto:Params: {} 2024-04-13T22:07:05.227 DEBUG:boto:Token: None 2024-04-13T22:07:05.227 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/d 2024-04-13T22:07:05.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xmxpq5+feVHYSU2au8jTxMKj/mQ= 2024-04-13T22:07:05.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xmxpq5+feVHYSU2au8jTxMKj/mQ=', 'Content-Length': '0'} 2024-04-13T22:07:05.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:06:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000796a5c778611a5d3-00661b0209-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.230 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.230 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.230 DEBUG:boto:path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.230 DEBUG:boto:auth_path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.230 DEBUG:boto:Method: GET 2024-04-13T22:07:05.230 DEBUG:boto:Path: /vwnfns-55/?prefix=d 2024-04-13T22:07:05.231 DEBUG:boto:Data: 2024-04-13T22:07:05.231 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:07:05.231 DEBUG:boto:Port: 8001 2024-04-13T22:07:05.231 DEBUG:boto:Params: {} 2024-04-13T22:07:05.231 DEBUG:boto:Token: None 2024-04-13T22:07:05.231 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.231 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000440874477df37f8c-00661b0209-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.234 DEBUG:boto:b'vwnfns-55d1000falsed2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.234 DEBUG:boto:path=/vwnfns-55/ 2024-04-13T22:07:05.235 DEBUG:boto:auth_path=/vwnfns-55/ 2024-04-13T22:07:05.235 DEBUG:boto:path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.235 DEBUG:boto:auth_path=/vwnfns-55/?prefix=d 2024-04-13T22:07:05.235 DEBUG:boto:Method: GET 2024-04-13T22:07:05.235 DEBUG:boto:Path: /vwnfns-55/?prefix=d 2024-04-13T22:07:05.235 DEBUG:boto:Data: 2024-04-13T22:07:05.235 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.235 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.235 DEBUG:boto:Params: {} 2024-04-13T22:07:05.235 DEBUG:boto:Token: None 2024-04-13T22:07:05.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-55/ 2024-04-13T22:07:05.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw= 2024-04-13T22:07:05.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ypzp/0s4XDGqQs4ELsmPbkPxJdw=', 'Content-Length': '0'} 2024-04-13T22:07:05.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013b3007f4affb936-00661b0209-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.238 DEBUG:boto:b'vwnfns-55d1000falsed2024-04-13T22:06:29.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:07:05.238 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-55 zones={a2, a1} 2024-04-13T22:07:05.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:07:05.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:07:05.239 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:07:05.283 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:07:05.283 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:07:05.327 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-56 2024-04-13T22:07:05.328 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:07:05.328 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:07:05.328 DEBUG:boto:Method: PUT 2024-04-13T22:07:05.328 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:07:05.328 DEBUG:boto:Data: 2024-04-13T22:07:05.328 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.328 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.328 DEBUG:boto:Params: {} 2024-04-13T22:07:05.329 DEBUG:boto:Token: None 2024-04-13T22:07:05.329 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/ 2024-04-13T22:07:05.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuMOSYwg7XQ/UZPgsq9mjPoj688= 2024-04-13T22:07:05.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuMOSYwg7XQ/UZPgsq9mjPoj688=', 'Content-Length': '0'} 2024-04-13T22:07:05.346 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000390aa4faef80df5c-00661b0209-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.346 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:07:05.346 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:07:05.346 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.346 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:07:05.347 DEBUG:boto:Data: 2024-04-13T22:07:05.347 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.347 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.347 DEBUG:boto:Params: {} 2024-04-13T22:07:05.347 DEBUG:boto:Token: None 2024-04-13T22:07:05.347 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/ 2024-04-13T22:07:05.347 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic= 2024-04-13T22:07:05.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic=', 'Content-Length': '0'} 2024-04-13T22:07:05.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dcc0e5608549442-00661b0209-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.350 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:07:05.350 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:07:05.350 DEBUG:boto:Method: PUT 2024-04-13T22:07:05.350 DEBUG:boto:Path: /vwnfns-56/a 2024-04-13T22:07:05.350 DEBUG:boto:Data: 2024-04-13T22:07:05.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:07:05.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.350 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.351 DEBUG:boto:Params: {} 2024-04-13T22:07:05.351 DEBUG:boto:Token: None 2024-04-13T22:07:05.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/a 2024-04-13T22:07:05.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WyHOIAdOIYVuXGACjzl2cki4O+A= 2024-04-13T22:07:05.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WyHOIAdOIYVuXGACjzl2cki4O+A='} 2024-04-13T22:07:05.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041ae9c57567b001d-00661b0209-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.356 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:07:05.357 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:07:05.357 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.357 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:07:05.357 DEBUG:boto:Data: 2024-04-13T22:07:05.357 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.357 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.357 DEBUG:boto:Params: {} 2024-04-13T22:07:05.357 DEBUG:boto:Token: None 2024-04-13T22:07:05.357 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/ 2024-04-13T22:07:05.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic= 2024-04-13T22:07:05.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic=', 'Content-Length': '0'} 2024-04-13T22:07:05.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85ae7ecaef8d055-00661b0209-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.360 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:07:05.360 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:07:05.360 DEBUG:boto:Method: PUT 2024-04-13T22:07:05.360 DEBUG:boto:Path: /vwnfns-56/b 2024-04-13T22:07:05.360 DEBUG:boto:Data: 2024-04-13T22:07:05.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:07:05.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.361 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.361 DEBUG:boto:Params: {} 2024-04-13T22:07:05.361 DEBUG:boto:Token: None 2024-04-13T22:07:05.361 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/b 2024-04-13T22:07:05.361 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qJ/3F9O/e0M7qmKdUlNFnjIvzCY= 2024-04-13T22:07:05.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qJ/3F9O/e0M7qmKdUlNFnjIvzCY='} 2024-04-13T22:07:05.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039906e640c3dd33f-00661b0209-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.403 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:07:05.403 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:07:05.403 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.403 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:07:05.403 DEBUG:boto:Data: 2024-04-13T22:07:05.403 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.404 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.404 DEBUG:boto:Params: {} 2024-04-13T22:07:05.404 DEBUG:boto:Token: None 2024-04-13T22:07:05.404 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/ 2024-04-13T22:07:05.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic= 2024-04-13T22:07:05.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic=', 'Content-Length': '0'} 2024-04-13T22:07:05.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9c80a3771111c7a-00661b0209-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.407 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:07:05.407 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:07:05.407 DEBUG:boto:Method: PUT 2024-04-13T22:07:05.407 DEBUG:boto:Path: /vwnfns-56/c 2024-04-13T22:07:05.407 DEBUG:boto:Data: 2024-04-13T22:07:05.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:07:05.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.407 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.407 DEBUG:boto:Params: {} 2024-04-13T22:07:05.407 DEBUG:boto:Token: None 2024-04-13T22:07:05.408 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/c 2024-04-13T22:07:05.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g07FfTjPdQlvjBTRPqCF6ZFghP8= 2024-04-13T22:07:05.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g07FfTjPdQlvjBTRPqCF6ZFghP8='} 2024-04-13T22:07:05.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004778fd7192aba737-00661b0209-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.413 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:07:05.413 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:07:05.413 DEBUG:boto:Method: HEAD 2024-04-13T22:07:05.413 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:07:05.413 DEBUG:boto:Data: 2024-04-13T22:07:05.413 DEBUG:boto:Headers: {} 2024-04-13T22:07:05.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.413 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.413 DEBUG:boto:Params: {} 2024-04-13T22:07:05.414 DEBUG:boto:Token: None 2024-04-13T22:07:05.414 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/ 2024-04-13T22:07:05.414 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic= 2024-04-13T22:07:05.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AWl4rjFR4CD5iEpOwnxTYb9Knic=', 'Content-Length': '0'} 2024-04-13T22:07:05.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d3db56407ab07bc-00661b0209-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.416 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:07:05.416 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:07:05.416 DEBUG:boto:Method: PUT 2024-04-13T22:07:05.416 DEBUG:boto:Path: /vwnfns-56/d 2024-04-13T22:07:05.416 DEBUG:boto:Data: 2024-04-13T22:07:05.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:07:05.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:07:05.417 DEBUG:boto:Port: 8000 2024-04-13T22:07:05.417 DEBUG:boto:Params: {} 2024-04-13T22:07:05.417 DEBUG:boto:Token: None 2024-04-13T22:07:05.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:07:05 GMT /vwnfns-56/d 2024-04-13T22:07:05.417 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L4AF0oFP4R3x6bkwcy7o5A1RQH8= 2024-04-13T22:07:05.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L4AF0oFP4R3x6bkwcy7o5A1RQH8='} 2024-04-13T22:07:05.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a03028c2f2d0edb-00661b0209-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:07:05.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:07:05.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:07:05.459 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:07:05.508 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:07:05.508 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:07:05.542 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:07:05.554 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:07:05.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.554 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046025.344683_686.1", 2024-04-13T22:07:05.554 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:07:05.344683Z" 2024-04-13T22:07:05.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:07:05.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:07:05.556 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:07:05.557 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046025.344683_686.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:07:05.557 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:07:05.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:07:05.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:07:05.557 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:07:05.604 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:07:05.605 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:07:05.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:05.633+0000 7f858ec5c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:07:05.635 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:05.634+0000 7f858ec5c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:07:05.649 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:07:05.649 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:07:05.649 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:07:05.650 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046025.344683_686.1", 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:07:05.344683Z", 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:07:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:07:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:07:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:07:05Z" 2024-04-13T22:07:05.655 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:07:05.656 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046025.344683_686.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:07:05.344683Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:07:05Z" } 2024-04-13T22:07:05.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046025.344683_686.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:07:05.344683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:07:05.656 DEBUG:rgw_multi.tests:log_status={0: '1_1713046025.344683_686.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:07:05.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046025.344683_686.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:07:05.657 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:07:05.657 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-56 2024-04-13T22:07:05.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:07:05.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:07:05.657 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:07:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:07:05.712 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:07:05.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:05.744+0000 7f81f5d3e780 1 waiting to reach incremental sync.. 2024-04-13T22:07:10.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:10.746+0000 7f81f5d3e780 1 bucket sync caught up with source: 2024-04-13T22:07:10.748 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.46.6, , , , 00000000001.21.6, , , 00000000001.26.6, , 00000000001.26.6] 2024-04-13T22:07:10.748 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.46.6, , , , 00000000001.21.6, , , 00000000001.26.6, , 00000000001.26.6] 2024-04-13T22:07:10.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:10.746+0000 7f81f5d3e780 0 bucket checkpoint complete 2024-04-13T22:07:10.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-56 2024-04-13T22:07:10.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:07:10.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:07:10.764 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:07:10.821 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:07:10.821 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:07:10.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:10.854+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:15.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:15.856+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:20.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:20.856+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:25.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:25.857+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:30.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:30.856+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:35.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:35.857+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:40.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:40.858+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:45.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:45.859+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:50.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:50.860+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:07:55.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:07:55.861+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:00.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:00.862+0000 7f25fcddd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:05.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:05.863+0000 7f25fcddd780 1 bucket sync caught up with source: 2024-04-13T22:08:05.864 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.1116.6, , , , 00000000001.457.6, , , 00000000001.1824.6, , 00000000001.817.6] 2024-04-13T22:08:05.865 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.1116.6, , , , 00000000001.457.6, , , 00000000001.1824.6, , 00000000001.817.6] 2024-04-13T22:08:05.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:05.863+0000 7f25fcddd780 0 bucket checkpoint complete 2024-04-13T22:08:05.881 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-56 zones={a2, a1} 2024-04-13T22:08:05.881 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.881 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.881 DEBUG:boto:Method: HEAD 2024-04-13T22:08:05.882 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:08:05.882 DEBUG:boto:Data: 2024-04-13T22:08:05.882 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.882 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.882 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.882 DEBUG:boto:Params: {} 2024-04-13T22:08:05.882 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:08:05.883 DEBUG:boto:Token: None 2024-04-13T22:08:05.883 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/ 2024-04-13T22:08:05.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0+KONDBpRhx3XLc66QXLll/H0TY= 2024-04-13T22:08:05.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0+KONDBpRhx3XLc66QXLll/H0TY=', 'Content-Length': '0'} 2024-04-13T22:08:05.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042451a82048b7a26-00661b0245-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.888 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.888 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.888 DEBUG:boto:Method: HEAD 2024-04-13T22:08:05.888 DEBUG:boto:Path: /vwnfns-56/ 2024-04-13T22:08:05.888 DEBUG:boto:Data: 2024-04-13T22:08:05.888 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.888 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.888 DEBUG:boto:Params: {} 2024-04-13T22:08:05.889 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:08:05.889 DEBUG:boto:Token: None 2024-04-13T22:08:05.889 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/ 2024-04-13T22:08:05.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0+KONDBpRhx3XLc66QXLll/H0TY= 2024-04-13T22:08:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0+KONDBpRhx3XLc66QXLll/H0TY=', 'Content-Length': '0'} 2024-04-13T22:08:05.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b07047fb754d435e-00661b0245-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:08:05.893 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.893 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.893 DEBUG:boto:path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.893 DEBUG:boto:auth_path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.893 DEBUG:boto:Method: GET 2024-04-13T22:08:05.893 DEBUG:boto:Path: /vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.893 DEBUG:boto:Data: 2024-04-13T22:08:05.893 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.893 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.894 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.894 DEBUG:boto:Params: {} 2024-04-13T22:08:05.894 DEBUG:boto:Token: None 2024-04-13T22:08:05.894 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/?versions 2024-04-13T22:08:05.894 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0= 2024-04-13T22:08:05.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0=', 'Content-Length': '0'} 2024-04-13T22:08:05.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4a02c7cad38455-00661b0245-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.896 DEBUG:boto:b'vwnfns-56999falseanulltrue2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:05.897 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:08:05.897 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:08:05.897 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:08:05.898 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:08:05.898 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:08:05.898 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.898 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.898 DEBUG:boto:path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.898 DEBUG:boto:auth_path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.898 DEBUG:boto:Method: GET 2024-04-13T22:08:05.898 DEBUG:boto:Path: /vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.898 DEBUG:boto:Data: 2024-04-13T22:08:05.898 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.899 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.899 DEBUG:boto:Params: {} 2024-04-13T22:08:05.899 DEBUG:boto:Token: None 2024-04-13T22:08:05.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/?versions 2024-04-13T22:08:05.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0= 2024-04-13T22:08:05.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0=', 'Content-Length': '0'} 2024-04-13T22:08:05.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4b242cab3d44c5-00661b0245-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.902 DEBUG:boto:b'vwnfns-56999falseanulltrue2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:05.902 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:08:05.902 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:08:05.902 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:08:05.903 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:08:05.903 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.903 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.903 DEBUG:boto:path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.903 DEBUG:boto:auth_path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.903 DEBUG:boto:Method: GET 2024-04-13T22:08:05.903 DEBUG:boto:Path: /vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.903 DEBUG:boto:Data: 2024-04-13T22:08:05.903 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.903 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.904 DEBUG:boto:Params: {} 2024-04-13T22:08:05.904 DEBUG:boto:Token: None 2024-04-13T22:08:05.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/?versions 2024-04-13T22:08:05.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0= 2024-04-13T22:08:05.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0=', 'Content-Length': '0'} 2024-04-13T22:08:05.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33888e6fc6930eb-00661b0245-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.906 DEBUG:boto:b'vwnfns-56999falseanulltrue2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:05.907 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.907 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.907 DEBUG:boto:path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.907 DEBUG:boto:auth_path=/vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.910 DEBUG:boto:Method: GET 2024-04-13T22:08:05.911 DEBUG:boto:Path: /vwnfns-56/?versions&max-keys=999 2024-04-13T22:08:05.911 DEBUG:boto:Data: 2024-04-13T22:08:05.911 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.911 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.911 DEBUG:boto:Params: {} 2024-04-13T22:08:05.912 DEBUG:boto:Token: None 2024-04-13T22:08:05.912 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/?versions 2024-04-13T22:08:05.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0= 2024-04-13T22:08:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HPS3uzUYUEqVlZgSORhO/WAbPQ0=', 'Content-Length': '0'} 2024-04-13T22:08:05.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057033a744dd9e306-00661b0245-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.914 DEBUG:boto:b'vwnfns-56999falseanulltrue2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:05.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:08:05.915 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.915 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.915 DEBUG:boto:path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.915 DEBUG:boto:auth_path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.915 DEBUG:boto:Method: GET 2024-04-13T22:08:05.915 DEBUG:boto:Path: /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.915 DEBUG:boto:Data: 2024-04-13T22:08:05.915 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.916 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.916 DEBUG:boto:Params: {} 2024-04-13T22:08:05.916 DEBUG:boto:Token: None 2024-04-13T22:08:05.916 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U= 2024-04-13T22:08:05.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U=', 'Content-Length': '0'} 2024-04-13T22:08:05.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040766b823b3fe24b-00661b0245-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.918 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.919 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.919 DEBUG:boto:path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.919 DEBUG:boto:auth_path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.919 DEBUG:boto:Method: GET 2024-04-13T22:08:05.919 DEBUG:boto:Path: /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.919 DEBUG:boto:Data: 2024-04-13T22:08:05.919 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.919 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.919 DEBUG:boto:Params: {} 2024-04-13T22:08:05.919 DEBUG:boto:Token: None 2024-04-13T22:08:05.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U= 2024-04-13T22:08:05.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U=', 'Content-Length': '0'} 2024-04-13T22:08:05.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a21d1a83c0fde9a-00661b0245-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.922 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.922 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.922 DEBUG:boto:path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.922 DEBUG:boto:auth_path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.922 DEBUG:boto:Method: HEAD 2024-04-13T22:08:05.922 DEBUG:boto:Path: /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.922 DEBUG:boto:Data: 2024-04-13T22:08:05.922 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.923 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.923 DEBUG:boto:Params: {} 2024-04-13T22:08:05.923 DEBUG:boto:Token: None 2024-04-13T22:08:05.923 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NgO71j8xInNzVTetFPtf6obkbxQ= 2024-04-13T22:08:05.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NgO71j8xInNzVTetFPtf6obkbxQ=', 'Content-Length': '0'} 2024-04-13T22:08:05.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f418660d0b645d0-00661b0245-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.925 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.925 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.925 DEBUG:boto:path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.925 DEBUG:boto:auth_path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.925 DEBUG:boto:Method: HEAD 2024-04-13T22:08:05.925 DEBUG:boto:Path: /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.925 DEBUG:boto:Data: 2024-04-13T22:08:05.926 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.926 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.926 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.926 DEBUG:boto:Params: {} 2024-04-13T22:08:05.926 DEBUG:boto:Token: None 2024-04-13T22:08:05.926 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NgO71j8xInNzVTetFPtf6obkbxQ= 2024-04-13T22:08:05.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NgO71j8xInNzVTetFPtf6obkbxQ=', 'Content-Length': '0'} 2024-04-13T22:08:05.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e1c859f0cd65f1-00661b0245-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:08:05.928 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.928 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.928 DEBUG:boto:path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.928 DEBUG:boto:auth_path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.929 DEBUG:boto:Method: GET 2024-04-13T22:08:05.929 DEBUG:boto:Path: /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.929 DEBUG:boto:Data: 2024-04-13T22:08:05.929 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.929 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.929 DEBUG:boto:Params: {} 2024-04-13T22:08:05.929 DEBUG:boto:Token: None 2024-04-13T22:08:05.929 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U= 2024-04-13T22:08:05.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U=', 'Content-Length': '0'} 2024-04-13T22:08:05.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f130e1fa3bc84b72-00661b0245-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.974 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.975 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.975 DEBUG:boto:path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.975 DEBUG:boto:auth_path=/vwnfns-56/a?versionId=null 2024-04-13T22:08:05.975 DEBUG:boto:Method: GET 2024-04-13T22:08:05.975 DEBUG:boto:Path: /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.975 DEBUG:boto:Data: 2024-04-13T22:08:05.975 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.975 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.975 DEBUG:boto:Params: {} 2024-04-13T22:08:05.976 DEBUG:boto:Token: None 2024-04-13T22:08:05.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a?versionId=null 2024-04-13T22:08:05.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U= 2024-04-13T22:08:05.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MUvKQ2c2i2Zu1AAk7ZyIfk6HR1U=', 'Content-Length': '0'} 2024-04-13T22:08:05.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158b8c0722fe66c5-00661b0245-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.978 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.978 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.979 DEBUG:boto:path=/vwnfns-56/?prefix=a 2024-04-13T22:08:05.979 DEBUG:boto:auth_path=/vwnfns-56/?prefix=a 2024-04-13T22:08:05.979 DEBUG:boto:Method: GET 2024-04-13T22:08:05.979 DEBUG:boto:Path: /vwnfns-56/?prefix=a 2024-04-13T22:08:05.979 DEBUG:boto:Data: 2024-04-13T22:08:05.979 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.979 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.979 DEBUG:boto:Params: {} 2024-04-13T22:08:05.980 DEBUG:boto:Token: None 2024-04-13T22:08:05.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/ 2024-04-13T22:08:05.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gwyn7FBv53XM1Ym+2DL4KLuSnJg= 2024-04-13T22:08:05.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gwyn7FBv53XM1Ym+2DL4KLuSnJg=', 'Content-Length': '0'} 2024-04-13T22:08:05.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8cad6b07681cf61-00661b0245-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.982 DEBUG:boto:b'vwnfns-56a1000falsea2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:05.983 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:05.983 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:05.983 DEBUG:boto:path=/vwnfns-56/?prefix=a 2024-04-13T22:08:05.983 DEBUG:boto:auth_path=/vwnfns-56/?prefix=a 2024-04-13T22:08:05.983 DEBUG:boto:Method: GET 2024-04-13T22:08:05.983 DEBUG:boto:Path: /vwnfns-56/?prefix=a 2024-04-13T22:08:05.984 DEBUG:boto:Data: 2024-04-13T22:08:05.984 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.984 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.984 DEBUG:boto:Params: {} 2024-04-13T22:08:05.984 DEBUG:boto:Token: None 2024-04-13T22:08:05.984 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/ 2024-04-13T22:08:05.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gwyn7FBv53XM1Ym+2DL4KLuSnJg= 2024-04-13T22:08:05.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gwyn7FBv53XM1Ym+2DL4KLuSnJg=', 'Content-Length': '0'} 2024-04-13T22:08:05.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6c0a282a70cac2-00661b0245-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.987 DEBUG:boto:b'vwnfns-56a1000falsea2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:05.987 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.988 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.988 DEBUG:boto:Method: HEAD 2024-04-13T22:08:05.988 DEBUG:boto:Path: /vwnfns-56/a 2024-04-13T22:08:05.988 DEBUG:boto:Data: 2024-04-13T22:08:05.988 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.988 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.988 DEBUG:boto:Params: {} 2024-04-13T22:08:05.988 DEBUG:boto:Token: None 2024-04-13T22:08:05.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a 2024-04-13T22:08:05.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c0NFGNUEeFsY7+mzsV0POifBZfQ= 2024-04-13T22:08:05.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c0NFGNUEeFsY7+mzsV0POifBZfQ=', 'Content-Length': '0'} 2024-04-13T22:08:05.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d062977bb007626f-00661b0245-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.991 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.991 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.991 DEBUG:boto:Method: HEAD 2024-04-13T22:08:05.991 DEBUG:boto:Path: /vwnfns-56/a 2024-04-13T22:08:05.991 DEBUG:boto:Data: 2024-04-13T22:08:05.991 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:05.991 DEBUG:boto:Port: 8000 2024-04-13T22:08:05.991 DEBUG:boto:Params: {} 2024-04-13T22:08:05.991 DEBUG:boto:Token: None 2024-04-13T22:08:05.992 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a 2024-04-13T22:08:05.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c0NFGNUEeFsY7+mzsV0POifBZfQ= 2024-04-13T22:08:05.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c0NFGNUEeFsY7+mzsV0POifBZfQ=', 'Content-Length': '0'} 2024-04-13T22:08:05.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5360c47aa14440-00661b0245-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:05.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:08:05.994 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:05.994 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:05.994 DEBUG:boto:Method: GET 2024-04-13T22:08:05.994 DEBUG:boto:Path: /vwnfns-56/a 2024-04-13T22:08:05.994 DEBUG:boto:Data: 2024-04-13T22:08:05.994 DEBUG:boto:Headers: {} 2024-04-13T22:08:05.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:05.994 DEBUG:boto:Port: 8001 2024-04-13T22:08:05.994 DEBUG:boto:Params: {} 2024-04-13T22:08:05.995 DEBUG:boto:Token: None 2024-04-13T22:08:05.995 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:05 GMT /vwnfns-56/a 2024-04-13T22:08:05.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CDGkkJoivvKbNmHA+4RgEex3/kU= 2024-04-13T22:08:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CDGkkJoivvKbNmHA+4RgEex3/kU=', 'Content-Length': '0'} 2024-04-13T22:08:05.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001050974974b6aa21-00661b0245-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.038 DEBUG:boto:path=/vwnfns-56/a 2024-04-13T22:08:06.039 DEBUG:boto:auth_path=/vwnfns-56/a 2024-04-13T22:08:06.039 DEBUG:boto:Method: GET 2024-04-13T22:08:06.039 DEBUG:boto:Path: /vwnfns-56/a 2024-04-13T22:08:06.039 DEBUG:boto:Data: 2024-04-13T22:08:06.039 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.039 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.039 DEBUG:boto:Params: {} 2024-04-13T22:08:06.039 DEBUG:boto:Token: None 2024-04-13T22:08:06.040 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/a 2024-04-13T22:08:06.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/f3yeG1xdiUj8b5b9WSn8evPpJw= 2024-04-13T22:08:06.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/f3yeG1xdiUj8b5b9WSn8evPpJw=', 'Content-Length': '0'} 2024-04-13T22:08:06.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff2aa2ca93c037e-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.042 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.042 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.042 DEBUG:boto:path=/vwnfns-56/?prefix=a 2024-04-13T22:08:06.042 DEBUG:boto:auth_path=/vwnfns-56/?prefix=a 2024-04-13T22:08:06.042 DEBUG:boto:Method: GET 2024-04-13T22:08:06.042 DEBUG:boto:Path: /vwnfns-56/?prefix=a 2024-04-13T22:08:06.043 DEBUG:boto:Data: 2024-04-13T22:08:06.043 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.043 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.043 DEBUG:boto:Params: {} 2024-04-13T22:08:06.043 DEBUG:boto:Token: None 2024-04-13T22:08:06.043 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c9e63b5d2d160cd-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.046 DEBUG:boto:b'vwnfns-56a1000falsea2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.046 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.046 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.047 DEBUG:boto:path=/vwnfns-56/?prefix=a 2024-04-13T22:08:06.047 DEBUG:boto:auth_path=/vwnfns-56/?prefix=a 2024-04-13T22:08:06.047 DEBUG:boto:Method: GET 2024-04-13T22:08:06.047 DEBUG:boto:Path: /vwnfns-56/?prefix=a 2024-04-13T22:08:06.047 DEBUG:boto:Data: 2024-04-13T22:08:06.047 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.047 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.047 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.047 DEBUG:boto:Params: {} 2024-04-13T22:08:06.047 DEBUG:boto:Token: None 2024-04-13T22:08:06.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d8a7b26e718ba0-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.050 DEBUG:boto:b'vwnfns-56a1000falsea2024-04-13T22:07:05.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:08:06.051 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.051 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.051 DEBUG:boto:path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.051 DEBUG:boto:auth_path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.051 DEBUG:boto:Method: GET 2024-04-13T22:08:06.051 DEBUG:boto:Path: /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.051 DEBUG:boto:Data: 2024-04-13T22:08:06.051 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.052 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.052 DEBUG:boto:Params: {} 2024-04-13T22:08:06.052 DEBUG:boto:Token: None 2024-04-13T22:08:06.052 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo= 2024-04-13T22:08:06.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo=', 'Content-Length': '0'} 2024-04-13T22:08:06.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f143f71b87cd15-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.054 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.054 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.055 DEBUG:boto:path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.055 DEBUG:boto:auth_path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.055 DEBUG:boto:Method: GET 2024-04-13T22:08:06.055 DEBUG:boto:Path: /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.055 DEBUG:boto:Data: 2024-04-13T22:08:06.055 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.055 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.055 DEBUG:boto:Params: {} 2024-04-13T22:08:06.055 DEBUG:boto:Token: None 2024-04-13T22:08:06.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo= 2024-04-13T22:08:06.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo=', 'Content-Length': '0'} 2024-04-13T22:08:06.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b12ff920c45f39f-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.058 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.058 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.058 DEBUG:boto:path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.058 DEBUG:boto:auth_path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.058 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.058 DEBUG:boto:Path: /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.058 DEBUG:boto:Data: 2024-04-13T22:08:06.058 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.059 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.059 DEBUG:boto:Params: {} 2024-04-13T22:08:06.059 DEBUG:boto:Token: None 2024-04-13T22:08:06.059 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lswFjP2ln0u2poR2IPACG2HLjag= 2024-04-13T22:08:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lswFjP2ln0u2poR2IPACG2HLjag=', 'Content-Length': '0'} 2024-04-13T22:08:06.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019eb70c1f8c23a50-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.061 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.061 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.061 DEBUG:boto:path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.061 DEBUG:boto:auth_path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.062 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.062 DEBUG:boto:Path: /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.062 DEBUG:boto:Data: 2024-04-13T22:08:06.062 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.062 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.062 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.062 DEBUG:boto:Params: {} 2024-04-13T22:08:06.062 DEBUG:boto:Token: None 2024-04-13T22:08:06.062 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lswFjP2ln0u2poR2IPACG2HLjag= 2024-04-13T22:08:06.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lswFjP2ln0u2poR2IPACG2HLjag=', 'Content-Length': '0'} 2024-04-13T22:08:06.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb909f5d595e4ad2-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:08:06.065 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.065 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.065 DEBUG:boto:path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.065 DEBUG:boto:auth_path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.065 DEBUG:boto:Method: GET 2024-04-13T22:08:06.065 DEBUG:boto:Path: /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.065 DEBUG:boto:Data: 2024-04-13T22:08:06.065 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.065 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.065 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.065 DEBUG:boto:Params: {} 2024-04-13T22:08:06.065 DEBUG:boto:Token: None 2024-04-13T22:08:06.066 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.066 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo= 2024-04-13T22:08:06.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo=', 'Content-Length': '0'} 2024-04-13T22:08:06.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4cfbbe2716b514-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.068 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.068 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.068 DEBUG:boto:path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.068 DEBUG:boto:auth_path=/vwnfns-56/b?versionId=null 2024-04-13T22:08:06.068 DEBUG:boto:Method: GET 2024-04-13T22:08:06.068 DEBUG:boto:Path: /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.069 DEBUG:boto:Data: 2024-04-13T22:08:06.069 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.069 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.069 DEBUG:boto:Params: {} 2024-04-13T22:08:06.069 DEBUG:boto:Token: None 2024-04-13T22:08:06.069 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b?versionId=null 2024-04-13T22:08:06.069 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo= 2024-04-13T22:08:06.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3bBa01EWfIb2s93xmVAWH7JYHXo=', 'Content-Length': '0'} 2024-04-13T22:08:06.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008089e43b37a87971-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.072 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.072 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.072 DEBUG:boto:path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.072 DEBUG:boto:auth_path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.072 DEBUG:boto:Method: GET 2024-04-13T22:08:06.072 DEBUG:boto:Path: /vwnfns-56/?prefix=b 2024-04-13T22:08:06.072 DEBUG:boto:Data: 2024-04-13T22:08:06.072 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.072 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.073 DEBUG:boto:Params: {} 2024-04-13T22:08:06.073 DEBUG:boto:Token: None 2024-04-13T22:08:06.073 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.073 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e35e713b0d4165c-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.075 DEBUG:boto:b'vwnfns-56b1000falseb2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.076 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.076 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.076 DEBUG:boto:path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.076 DEBUG:boto:auth_path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.076 DEBUG:boto:Method: GET 2024-04-13T22:08:06.076 DEBUG:boto:Path: /vwnfns-56/?prefix=b 2024-04-13T22:08:06.076 DEBUG:boto:Data: 2024-04-13T22:08:06.076 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.077 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.077 DEBUG:boto:Params: {} 2024-04-13T22:08:06.077 DEBUG:boto:Token: None 2024-04-13T22:08:06.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003055d0e4e2b44612-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.079 DEBUG:boto:b'vwnfns-56b1000falseb2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.080 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.080 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.080 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.080 DEBUG:boto:Path: /vwnfns-56/b 2024-04-13T22:08:06.080 DEBUG:boto:Data: 2024-04-13T22:08:06.080 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.080 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.081 DEBUG:boto:Params: {} 2024-04-13T22:08:06.081 DEBUG:boto:Token: None 2024-04-13T22:08:06.081 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b 2024-04-13T22:08:06.081 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fpSs+AFg8DkQ9tSIvOLE32DTJ84= 2024-04-13T22:08:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fpSs+AFg8DkQ9tSIvOLE32DTJ84=', 'Content-Length': '0'} 2024-04-13T22:08:06.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003522667876620221-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.083 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.083 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.083 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.083 DEBUG:boto:Path: /vwnfns-56/b 2024-04-13T22:08:06.083 DEBUG:boto:Data: 2024-04-13T22:08:06.083 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.083 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.083 DEBUG:boto:Params: {} 2024-04-13T22:08:06.084 DEBUG:boto:Token: None 2024-04-13T22:08:06.084 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b 2024-04-13T22:08:06.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fpSs+AFg8DkQ9tSIvOLE32DTJ84= 2024-04-13T22:08:06.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fpSs+AFg8DkQ9tSIvOLE32DTJ84=', 'Content-Length': '0'} 2024-04-13T22:08:06.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334b11ea3ddee532-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:08:06.086 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.086 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.086 DEBUG:boto:Method: GET 2024-04-13T22:08:06.086 DEBUG:boto:Path: /vwnfns-56/b 2024-04-13T22:08:06.086 DEBUG:boto:Data: 2024-04-13T22:08:06.086 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.086 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.086 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.086 DEBUG:boto:Params: {} 2024-04-13T22:08:06.087 DEBUG:boto:Token: None 2024-04-13T22:08:06.087 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b 2024-04-13T22:08:06.087 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H1lUJO/9J9/AscVSQV2QerltYU8= 2024-04-13T22:08:06.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H1lUJO/9J9/AscVSQV2QerltYU8=', 'Content-Length': '0'} 2024-04-13T22:08:06.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd3d83c3ce4dccd-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.130 DEBUG:boto:path=/vwnfns-56/b 2024-04-13T22:08:06.131 DEBUG:boto:auth_path=/vwnfns-56/b 2024-04-13T22:08:06.131 DEBUG:boto:Method: GET 2024-04-13T22:08:06.131 DEBUG:boto:Path: /vwnfns-56/b 2024-04-13T22:08:06.131 DEBUG:boto:Data: 2024-04-13T22:08:06.131 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.131 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.131 DEBUG:boto:Params: {} 2024-04-13T22:08:06.131 DEBUG:boto:Token: None 2024-04-13T22:08:06.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/b 2024-04-13T22:08:06.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H1lUJO/9J9/AscVSQV2QerltYU8= 2024-04-13T22:08:06.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H1lUJO/9J9/AscVSQV2QerltYU8=', 'Content-Length': '0'} 2024-04-13T22:08:06.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b57b6f64a17db6-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.134 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.134 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.134 DEBUG:boto:path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.134 DEBUG:boto:auth_path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.134 DEBUG:boto:Method: GET 2024-04-13T22:08:06.134 DEBUG:boto:Path: /vwnfns-56/?prefix=b 2024-04-13T22:08:06.134 DEBUG:boto:Data: 2024-04-13T22:08:06.135 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.135 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.135 DEBUG:boto:Params: {} 2024-04-13T22:08:06.135 DEBUG:boto:Token: None 2024-04-13T22:08:06.135 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004592beb634039091-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.138 DEBUG:boto:b'vwnfns-56b1000falseb2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.138 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.138 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.138 DEBUG:boto:path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.138 DEBUG:boto:auth_path=/vwnfns-56/?prefix=b 2024-04-13T22:08:06.139 DEBUG:boto:Method: GET 2024-04-13T22:08:06.139 DEBUG:boto:Path: /vwnfns-56/?prefix=b 2024-04-13T22:08:06.139 DEBUG:boto:Data: 2024-04-13T22:08:06.139 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.139 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.139 DEBUG:boto:Params: {} 2024-04-13T22:08:06.139 DEBUG:boto:Token: None 2024-04-13T22:08:06.139 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e20488efa9c186-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.142 DEBUG:boto:b'vwnfns-56b1000falseb2024-04-13T22:07:05.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:08:06.143 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.143 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.143 DEBUG:boto:path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.143 DEBUG:boto:auth_path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.143 DEBUG:boto:Method: GET 2024-04-13T22:08:06.143 DEBUG:boto:Path: /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.143 DEBUG:boto:Data: 2024-04-13T22:08:06.143 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.143 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.143 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.143 DEBUG:boto:Params: {} 2024-04-13T22:08:06.144 DEBUG:boto:Token: None 2024-04-13T22:08:06.144 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE= 2024-04-13T22:08:06.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE=', 'Content-Length': '0'} 2024-04-13T22:08:06.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1be9ecdcc60c997-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.146 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.146 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.146 DEBUG:boto:path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.146 DEBUG:boto:auth_path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.147 DEBUG:boto:Method: GET 2024-04-13T22:08:06.147 DEBUG:boto:Path: /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.147 DEBUG:boto:Data: 2024-04-13T22:08:06.147 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.147 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.147 DEBUG:boto:Params: {} 2024-04-13T22:08:06.147 DEBUG:boto:Token: None 2024-04-13T22:08:06.147 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE= 2024-04-13T22:08:06.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE=', 'Content-Length': '0'} 2024-04-13T22:08:06.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c1525f7508f8bc-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.149 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.150 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.150 DEBUG:boto:path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.150 DEBUG:boto:auth_path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.150 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.150 DEBUG:boto:Path: /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.150 DEBUG:boto:Data: 2024-04-13T22:08:06.150 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.150 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.150 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.150 DEBUG:boto:Params: {} 2024-04-13T22:08:06.151 DEBUG:boto:Token: None 2024-04-13T22:08:06.151 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uyCtq4SrxoalnzPiYDO9WUeSyLw= 2024-04-13T22:08:06.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uyCtq4SrxoalnzPiYDO9WUeSyLw=', 'Content-Length': '0'} 2024-04-13T22:08:06.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286debd56f7518fb-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.153 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.153 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.153 DEBUG:boto:path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.153 DEBUG:boto:auth_path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.153 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.153 DEBUG:boto:Path: /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.154 DEBUG:boto:Data: 2024-04-13T22:08:06.154 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.154 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.154 DEBUG:boto:Params: {} 2024-04-13T22:08:06.154 DEBUG:boto:Token: None 2024-04-13T22:08:06.154 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.154 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uyCtq4SrxoalnzPiYDO9WUeSyLw= 2024-04-13T22:08:06.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uyCtq4SrxoalnzPiYDO9WUeSyLw=', 'Content-Length': '0'} 2024-04-13T22:08:06.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8bb74b2b2efb34-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:08:06.156 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.157 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.157 DEBUG:boto:path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.157 DEBUG:boto:auth_path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.157 DEBUG:boto:Method: GET 2024-04-13T22:08:06.157 DEBUG:boto:Path: /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.157 DEBUG:boto:Data: 2024-04-13T22:08:06.157 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.157 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.157 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.157 DEBUG:boto:Params: {} 2024-04-13T22:08:06.157 DEBUG:boto:Token: None 2024-04-13T22:08:06.158 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.158 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE= 2024-04-13T22:08:06.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE=', 'Content-Length': '0'} 2024-04-13T22:08:06.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b430cfd911edb5-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.160 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.160 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.160 DEBUG:boto:path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.160 DEBUG:boto:auth_path=/vwnfns-56/c?versionId=null 2024-04-13T22:08:06.160 DEBUG:boto:Method: GET 2024-04-13T22:08:06.160 DEBUG:boto:Path: /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.160 DEBUG:boto:Data: 2024-04-13T22:08:06.161 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.161 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.161 DEBUG:boto:Params: {} 2024-04-13T22:08:06.161 DEBUG:boto:Token: None 2024-04-13T22:08:06.161 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c?versionId=null 2024-04-13T22:08:06.161 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE= 2024-04-13T22:08:06.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FFI29Yn0CAdQHIwB87owa6W+4dE=', 'Content-Length': '0'} 2024-04-13T22:08:06.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b7d7fcc8aadeaa4-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.163 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.163 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.163 DEBUG:boto:path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.164 DEBUG:boto:auth_path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.164 DEBUG:boto:Method: GET 2024-04-13T22:08:06.164 DEBUG:boto:Path: /vwnfns-56/?prefix=c 2024-04-13T22:08:06.164 DEBUG:boto:Data: 2024-04-13T22:08:06.164 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.164 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.164 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.164 DEBUG:boto:Params: {} 2024-04-13T22:08:06.164 DEBUG:boto:Token: None 2024-04-13T22:08:06.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd77d14d94c38f8-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.167 DEBUG:boto:b'vwnfns-56c1000falsec2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.167 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.168 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.168 DEBUG:boto:path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.168 DEBUG:boto:auth_path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.168 DEBUG:boto:Method: GET 2024-04-13T22:08:06.168 DEBUG:boto:Path: /vwnfns-56/?prefix=c 2024-04-13T22:08:06.168 DEBUG:boto:Data: 2024-04-13T22:08:06.168 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.168 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.168 DEBUG:boto:Params: {} 2024-04-13T22:08:06.168 DEBUG:boto:Token: None 2024-04-13T22:08:06.169 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e2897a1fd552201-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.172 DEBUG:boto:b'vwnfns-56c1000falsec2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.172 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.172 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.172 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.172 DEBUG:boto:Path: /vwnfns-56/c 2024-04-13T22:08:06.172 DEBUG:boto:Data: 2024-04-13T22:08:06.172 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.173 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.173 DEBUG:boto:Params: {} 2024-04-13T22:08:06.173 DEBUG:boto:Token: None 2024-04-13T22:08:06.173 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c 2024-04-13T22:08:06.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rREbCugAZfPH+SkzRWyukefECiM= 2024-04-13T22:08:06.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rREbCugAZfPH+SkzRWyukefECiM=', 'Content-Length': '0'} 2024-04-13T22:08:06.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072d5f9f8b0c6295d-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.175 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.175 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.175 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.175 DEBUG:boto:Path: /vwnfns-56/c 2024-04-13T22:08:06.175 DEBUG:boto:Data: 2024-04-13T22:08:06.175 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.176 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.176 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.176 DEBUG:boto:Params: {} 2024-04-13T22:08:06.176 DEBUG:boto:Token: None 2024-04-13T22:08:06.176 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c 2024-04-13T22:08:06.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rREbCugAZfPH+SkzRWyukefECiM= 2024-04-13T22:08:06.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rREbCugAZfPH+SkzRWyukefECiM=', 'Content-Length': '0'} 2024-04-13T22:08:06.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c15e876544c4166-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:08:06.178 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.178 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.178 DEBUG:boto:Method: GET 2024-04-13T22:08:06.179 DEBUG:boto:Path: /vwnfns-56/c 2024-04-13T22:08:06.179 DEBUG:boto:Data: 2024-04-13T22:08:06.179 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.179 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.179 DEBUG:boto:Params: {} 2024-04-13T22:08:06.179 DEBUG:boto:Token: None 2024-04-13T22:08:06.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c 2024-04-13T22:08:06.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kApCWrKiY8iOvQe4HD+vCcBfh+A= 2024-04-13T22:08:06.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kApCWrKiY8iOvQe4HD+vCcBfh+A=', 'Content-Length': '0'} 2024-04-13T22:08:06.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a26b4c2ddb16c6f-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.222 DEBUG:boto:path=/vwnfns-56/c 2024-04-13T22:08:06.222 DEBUG:boto:auth_path=/vwnfns-56/c 2024-04-13T22:08:06.223 DEBUG:boto:Method: GET 2024-04-13T22:08:06.223 DEBUG:boto:Path: /vwnfns-56/c 2024-04-13T22:08:06.223 DEBUG:boto:Data: 2024-04-13T22:08:06.223 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.223 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.223 DEBUG:boto:Params: {} 2024-04-13T22:08:06.223 DEBUG:boto:Token: None 2024-04-13T22:08:06.223 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/c 2024-04-13T22:08:06.223 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kApCWrKiY8iOvQe4HD+vCcBfh+A= 2024-04-13T22:08:06.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kApCWrKiY8iOvQe4HD+vCcBfh+A=', 'Content-Length': '0'} 2024-04-13T22:08:06.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040dbf30a4cf3a803-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.226 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.226 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.226 DEBUG:boto:path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.226 DEBUG:boto:auth_path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.226 DEBUG:boto:Method: GET 2024-04-13T22:08:06.226 DEBUG:boto:Path: /vwnfns-56/?prefix=c 2024-04-13T22:08:06.226 DEBUG:boto:Data: 2024-04-13T22:08:06.227 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.227 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.227 DEBUG:boto:Params: {} 2024-04-13T22:08:06.227 DEBUG:boto:Token: None 2024-04-13T22:08:06.227 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.227 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004744b21d4101a314-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.230 DEBUG:boto:b'vwnfns-56c1000falsec2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.230 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.230 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.230 DEBUG:boto:path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.230 DEBUG:boto:auth_path=/vwnfns-56/?prefix=c 2024-04-13T22:08:06.230 DEBUG:boto:Method: GET 2024-04-13T22:08:06.230 DEBUG:boto:Path: /vwnfns-56/?prefix=c 2024-04-13T22:08:06.231 DEBUG:boto:Data: 2024-04-13T22:08:06.231 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.231 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.231 DEBUG:boto:Params: {} 2024-04-13T22:08:06.231 DEBUG:boto:Token: None 2024-04-13T22:08:06.231 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.231 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3253c8fb6c3a483-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.234 DEBUG:boto:b'vwnfns-56c1000falsec2024-04-13T22:07:05.409Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:08:06.234 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.234 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.234 DEBUG:boto:path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.235 DEBUG:boto:auth_path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.235 DEBUG:boto:Method: GET 2024-04-13T22:08:06.235 DEBUG:boto:Path: /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.235 DEBUG:boto:Data: 2024-04-13T22:08:06.235 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.235 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.235 DEBUG:boto:Params: {} 2024-04-13T22:08:06.235 DEBUG:boto:Token: None 2024-04-13T22:08:06.235 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4= 2024-04-13T22:08:06.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4=', 'Content-Length': '0'} 2024-04-13T22:08:06.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1066f7e1720d7e-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.238 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.238 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.238 DEBUG:boto:path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.238 DEBUG:boto:auth_path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.238 DEBUG:boto:Method: GET 2024-04-13T22:08:06.238 DEBUG:boto:Path: /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.238 DEBUG:boto:Data: 2024-04-13T22:08:06.238 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.239 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.239 DEBUG:boto:Params: {} 2024-04-13T22:08:06.239 DEBUG:boto:Token: None 2024-04-13T22:08:06.239 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4= 2024-04-13T22:08:06.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4=', 'Content-Length': '0'} 2024-04-13T22:08:06.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2bd1835ce12698-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.241 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.241 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.241 DEBUG:boto:path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.241 DEBUG:boto:auth_path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.241 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.242 DEBUG:boto:Path: /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.242 DEBUG:boto:Data: 2024-04-13T22:08:06.242 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.242 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.242 DEBUG:boto:Params: {} 2024-04-13T22:08:06.242 DEBUG:boto:Token: None 2024-04-13T22:08:06.242 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RNC6di1DAxFOYaBBlSdYz3yBRp0= 2024-04-13T22:08:06.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RNC6di1DAxFOYaBBlSdYz3yBRp0=', 'Content-Length': '0'} 2024-04-13T22:08:06.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cef46b86e5b8c38-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.244 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.244 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.244 DEBUG:boto:path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.244 DEBUG:boto:auth_path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.245 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.245 DEBUG:boto:Path: /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.245 DEBUG:boto:Data: 2024-04-13T22:08:06.245 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.245 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.245 DEBUG:boto:Params: {} 2024-04-13T22:08:06.245 DEBUG:boto:Token: None 2024-04-13T22:08:06.245 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RNC6di1DAxFOYaBBlSdYz3yBRp0= 2024-04-13T22:08:06.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RNC6di1DAxFOYaBBlSdYz3yBRp0=', 'Content-Length': '0'} 2024-04-13T22:08:06.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c82965c70b11ab8-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:08:06.247 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.247 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.248 DEBUG:boto:path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.248 DEBUG:boto:auth_path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.248 DEBUG:boto:Method: GET 2024-04-13T22:08:06.248 DEBUG:boto:Path: /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.248 DEBUG:boto:Data: 2024-04-13T22:08:06.248 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.248 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.248 DEBUG:boto:Params: {} 2024-04-13T22:08:06.248 DEBUG:boto:Token: None 2024-04-13T22:08:06.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.249 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4= 2024-04-13T22:08:06.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4=', 'Content-Length': '0'} 2024-04-13T22:08:06.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f0a4f07dfa1777-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.250 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.251 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.251 DEBUG:boto:path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.251 DEBUG:boto:auth_path=/vwnfns-56/d?versionId=null 2024-04-13T22:08:06.251 DEBUG:boto:Method: GET 2024-04-13T22:08:06.251 DEBUG:boto:Path: /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.251 DEBUG:boto:Data: 2024-04-13T22:08:06.251 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.251 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.251 DEBUG:boto:Params: {} 2024-04-13T22:08:06.252 DEBUG:boto:Token: None 2024-04-13T22:08:06.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d?versionId=null 2024-04-13T22:08:06.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4= 2024-04-13T22:08:06.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kZltoXaqbQkCrW1B+Xe8z+Uzyk4=', 'Content-Length': '0'} 2024-04-13T22:08:06.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080513079a3487ec8-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.294 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.295 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.295 DEBUG:boto:path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.295 DEBUG:boto:auth_path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.295 DEBUG:boto:Method: GET 2024-04-13T22:08:06.295 DEBUG:boto:Path: /vwnfns-56/?prefix=d 2024-04-13T22:08:06.295 DEBUG:boto:Data: 2024-04-13T22:08:06.295 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.295 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.295 DEBUG:boto:Params: {} 2024-04-13T22:08:06.296 DEBUG:boto:Token: None 2024-04-13T22:08:06.296 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5de1ca0badd7147-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.298 DEBUG:boto:b'vwnfns-56d1000falsed2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.299 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.299 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.299 DEBUG:boto:path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.299 DEBUG:boto:auth_path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.299 DEBUG:boto:Method: GET 2024-04-13T22:08:06.299 DEBUG:boto:Path: /vwnfns-56/?prefix=d 2024-04-13T22:08:06.299 DEBUG:boto:Data: 2024-04-13T22:08:06.299 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.300 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.300 DEBUG:boto:Params: {} 2024-04-13T22:08:06.300 DEBUG:boto:Token: None 2024-04-13T22:08:06.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38aa8bf5a718b35-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.303 DEBUG:boto:b'vwnfns-56d1000falsed2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.303 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.303 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.303 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.303 DEBUG:boto:Path: /vwnfns-56/d 2024-04-13T22:08:06.303 DEBUG:boto:Data: 2024-04-13T22:08:06.304 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.304 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.304 DEBUG:boto:Params: {} 2024-04-13T22:08:06.304 DEBUG:boto:Token: None 2024-04-13T22:08:06.304 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d 2024-04-13T22:08:06.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rexwcD9RpV9GE7J+TYILbNKC4I0= 2024-04-13T22:08:06.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rexwcD9RpV9GE7J+TYILbNKC4I0=', 'Content-Length': '0'} 2024-04-13T22:08:06.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007379abb4240a7cb8-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.306 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.306 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.306 DEBUG:boto:Method: HEAD 2024-04-13T22:08:06.306 DEBUG:boto:Path: /vwnfns-56/d 2024-04-13T22:08:06.306 DEBUG:boto:Data: 2024-04-13T22:08:06.307 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.307 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.307 DEBUG:boto:Params: {} 2024-04-13T22:08:06.307 DEBUG:boto:Token: None 2024-04-13T22:08:06.307 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d 2024-04-13T22:08:06.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rexwcD9RpV9GE7J+TYILbNKC4I0= 2024-04-13T22:08:06.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rexwcD9RpV9GE7J+TYILbNKC4I0=', 'Content-Length': '0'} 2024-04-13T22:08:06.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c903ca770bb6ca13-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:08:06.309 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.309 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.309 DEBUG:boto:Method: GET 2024-04-13T22:08:06.309 DEBUG:boto:Path: /vwnfns-56/d 2024-04-13T22:08:06.310 DEBUG:boto:Data: 2024-04-13T22:08:06.310 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.310 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.310 DEBUG:boto:Params: {} 2024-04-13T22:08:06.310 DEBUG:boto:Token: None 2024-04-13T22:08:06.310 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d 2024-04-13T22:08:06.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7d9bgBJfJQ8wU033QSU1xcMidzc= 2024-04-13T22:08:06.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7d9bgBJfJQ8wU033QSU1xcMidzc=', 'Content-Length': '0'} 2024-04-13T22:08:06.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-56:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c536c9428ad68f80-00661b0246-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.312 DEBUG:boto:path=/vwnfns-56/d 2024-04-13T22:08:06.313 DEBUG:boto:auth_path=/vwnfns-56/d 2024-04-13T22:08:06.313 DEBUG:boto:Method: GET 2024-04-13T22:08:06.313 DEBUG:boto:Path: /vwnfns-56/d 2024-04-13T22:08:06.313 DEBUG:boto:Data: 2024-04-13T22:08:06.313 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.313 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.313 DEBUG:boto:Params: {} 2024-04-13T22:08:06.313 DEBUG:boto:Token: None 2024-04-13T22:08:06.314 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/d 2024-04-13T22:08:06.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7d9bgBJfJQ8wU033QSU1xcMidzc= 2024-04-13T22:08:06.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7d9bgBJfJQ8wU033QSU1xcMidzc=', 'Content-Length': '0'} 2024-04-13T22:08:06.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:07:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de5606404a26fd0-00661b0246-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.316 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.316 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.316 DEBUG:boto:path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.316 DEBUG:boto:auth_path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.316 DEBUG:boto:Method: GET 2024-04-13T22:08:06.316 DEBUG:boto:Path: /vwnfns-56/?prefix=d 2024-04-13T22:08:06.316 DEBUG:boto:Data: 2024-04-13T22:08:06.317 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.317 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:08:06.317 DEBUG:boto:Port: 8001 2024-04-13T22:08:06.317 DEBUG:boto:Params: {} 2024-04-13T22:08:06.317 DEBUG:boto:Token: None 2024-04-13T22:08:06.317 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.317 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d2f9a3e4bb7dda-00661b0246-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.319 DEBUG:boto:b'vwnfns-56d1000falsed2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.320 DEBUG:boto:path=/vwnfns-56/ 2024-04-13T22:08:06.320 DEBUG:boto:auth_path=/vwnfns-56/ 2024-04-13T22:08:06.320 DEBUG:boto:path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.320 DEBUG:boto:auth_path=/vwnfns-56/?prefix=d 2024-04-13T22:08:06.320 DEBUG:boto:Method: GET 2024-04-13T22:08:06.320 DEBUG:boto:Path: /vwnfns-56/?prefix=d 2024-04-13T22:08:06.320 DEBUG:boto:Data: 2024-04-13T22:08:06.320 DEBUG:boto:Headers: {} 2024-04-13T22:08:06.321 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:06.321 DEBUG:boto:Port: 8000 2024-04-13T22:08:06.321 DEBUG:boto:Params: {} 2024-04-13T22:08:06.321 DEBUG:boto:Token: None 2024-04-13T22:08:06.321 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:08:06 GMT /vwnfns-56/ 2024-04-13T22:08:06.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q= 2024-04-13T22:08:06.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dCXWgvXncXtlCCaqb8wRhyS0b8Q=', 'Content-Length': '0'} 2024-04-13T22:08:06.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa5b8f3018f4001-00661b0246-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:06.323 DEBUG:boto:b'vwnfns-56d1000falsed2024-04-13T22:07:05.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:08:06.324 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-56 zones={a2, a1} 2024-04-13T22:08:06.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:08:06.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:08:06.325 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:08:06.374 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:06.374 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:06.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-56', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:06.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-56', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:06.427 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-56 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:08:06.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:06.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:06.519 INFO:teuthology.orchestra.run.smithi132.stdout:[] 2024-04-13T22:08:06.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:06.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:06.520 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:08:06.567 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:06.567 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "1#00000000001.29.6", 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13316879972157124844", 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "a", 2024-04-13T22:08:06.610 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:06:29.283265923Z", 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:08:06.611 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18" 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "5#00000000001.52.6", 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13323924847393628300", 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "d", 2024-04-13T22:08:06.612 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:06:29.349416489Z", 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:08:06.613 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18" 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:06.614 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "8#00000000001.19.6", 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13287477252805531349", 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "c", 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:08:06.615 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:08:06.616 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:06:29.340370003Z", 2024-04-13T22:08:06.616 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:08:06.616 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:08:06.616 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:08:06.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:08:06.617 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:08:06.617 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:08:06.617 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18" 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "10#00000000001.65.6", 2024-04-13T22:08:06.618 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.5297634320990402838", 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "b", 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:06:29.330230993Z", 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:08:06.619 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-55:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.18" 2024-04-13T22:08:06.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:06.621 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:08:06.621 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:06.621 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:08:06.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:08:06.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:08:06.622 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:08:06.667 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:06.667 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:06.709 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.708+0000 7f5daf727700 -1 req 11792442542196979435 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-30:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.30,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-30:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.30,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.709 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.708+0000 7f5d8dee4700 -1 req 10368520623306036476 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-22:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.22,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-22:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.22,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.709 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.708+0000 7f5dbff48700 -1 req 8923039270903439341 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-33:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.33,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-33:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.33,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.709 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.709+0000 7f5e12fee700 -1 req 12848011600426443517 0.002999922s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-32:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.32,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-32:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.32,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.712 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.711+0000 7f5e46855700 -1 req 5117776802909726224 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-25:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.25,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-25:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.25,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.713 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.712+0000 7f5e3182b700 -1 req 15879915960423402491 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-40:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.3,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-40:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.3,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.756 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.756+0000 7f5dd5f74700 -1 req 10462353284765119016 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-39:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.2,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-39:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.2,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.759 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.759+0000 7f5d90eea700 -1 req 4328109876386700429 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-23:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.23,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-23:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.23,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.760 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.760+0000 7f5de3f90700 -1 req 9106530729120905155 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-24:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.24,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-24:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.24,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.802 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.802+0000 7f5e27016700 -1 req 18099817370534331758 0.002999922s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-5:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.5,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-5:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.5,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.803 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.802+0000 7f5deb79f700 -1 req 3129558455562364431 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-42:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.5,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-42:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.5,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.846 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.846+0000 7f5e3b03e700 -1 req 8848163330389302341 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-38:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.1,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-38:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.1,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.889 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.888+0000 7f5db9f3c700 -1 req 973725350630105850 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-29:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.29,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-29:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.29,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:06.976 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:06.976+0000 7f5e34030700 -1 req 7974090393216284199 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-34:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.34,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-34:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.34,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.020 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.020+0000 7f5debfa0700 -1 req 2138640164764487737 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-3:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.3,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-3:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.3,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.037 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.036+0000 7f5d86ed6700 -1 req 3517275042192177285 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-6:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.6,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-6:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.6,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.039 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.039+0000 7f5e48859700 -1 req 4101133917718832097 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-31:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.31,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-31:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.31,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.064 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.063+0000 7f5e36034700 -1 req 6351597974453617327 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-1:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.1,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-1:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.1,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.080 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.080+0000 7f5df07a9700 -1 req 3678748605616262960 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-21:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.21,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-21:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.21,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.082 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.082+0000 7f5da9f1c700 -1 req 5832072592494106424 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-27:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.27,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-27:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.27,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.082 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.082+0000 7f5e3f046700 -1 req 15994504583132068397 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-43:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.6,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-43:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.6,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.153 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.152+0000 7f5e1a7fd700 -1 req 11877850116993345477 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-4:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.4,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-4:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.4,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.168 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.168+0000 7f5e44050700 -1 req 18404052187759148016 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-2:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.2,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-2:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.2,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.172 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.172+0000 7f5e3d042700 -1 req 10201529072500117546 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-28:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.28,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-28:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.28,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.196 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.195+0000 7f5e5286d700 -1 req 11651309853954900856 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-45:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.8,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-45:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.8,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.215 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.215+0000 7f5e58078700 -1 req 18389198852697534735 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-44:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.7,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-44:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.7,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.239 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.238+0000 7f5e20008700 -1 req 17278410449591563217 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-26:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.26,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-26:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.26,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.259 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.258+0000 7f5de7f98700 -1 req 14039220266226457236 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-41:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.4,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-41:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.4,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.266 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:08:07.265+0000 7f5e01fcc700 -1 req 4170478708820876211 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-53:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.16,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-53:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.16,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:08:07.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:07.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:07.323 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:08:07.370 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:07.371 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:07.413 INFO:teuthology.orchestra.run.smithi132.stdout:[] 2024-04-13T22:08:07.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2024-04-13T22:08:07.416 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-57 2024-04-13T22:08:07.416 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:08:07.416 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:08:07.416 DEBUG:boto:Method: PUT 2024-04-13T22:08:07.416 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:08:07.417 DEBUG:boto:Data: 2024-04-13T22:08:07.417 DEBUG:boto:Headers: {} 2024-04-13T22:08:07.417 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.417 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.417 DEBUG:boto:Params: {} 2024-04-13T22:08:07.417 DEBUG:boto:Token: None 2024-04-13T22:08:07.417 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/ 2024-04-13T22:08:07.417 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ov85PUqSEDPlxopFG5ylrDiB8jg= 2024-04-13T22:08:07.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ov85PUqSEDPlxopFG5ylrDiB8jg=', 'Content-Length': '0'} 2024-04-13T22:08:07.432 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd8b2510be33dda6-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.432 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:08:07.433 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:08:07.433 DEBUG:boto:Method: HEAD 2024-04-13T22:08:07.433 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:08:07.433 DEBUG:boto:Data: 2024-04-13T22:08:07.433 DEBUG:boto:Headers: {} 2024-04-13T22:08:07.433 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.433 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.433 DEBUG:boto:Params: {} 2024-04-13T22:08:07.433 DEBUG:boto:Token: None 2024-04-13T22:08:07.434 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/ 2024-04-13T22:08:07.434 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T22:08:07.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T22:08:07.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bc7da754c8fa49c-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.436 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:08:07.436 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:08:07.436 DEBUG:boto:Method: PUT 2024-04-13T22:08:07.437 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:08:07.437 DEBUG:boto:Data: 2024-04-13T22:08:07.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:08:07.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.437 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.437 DEBUG:boto:Params: {} 2024-04-13T22:08:07.437 DEBUG:boto:Token: None 2024-04-13T22:08:07.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/a 2024-04-13T22:08:07.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h7YSzn94GEN8azPqxooSz+jzoxs= 2024-04-13T22:08:07.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h7YSzn94GEN8azPqxooSz+jzoxs='} 2024-04-13T22:08:07.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ae220d1fc2a5685-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.483 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:08:07.483 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:08:07.483 DEBUG:boto:Method: HEAD 2024-04-13T22:08:07.483 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:08:07.483 DEBUG:boto:Data: 2024-04-13T22:08:07.483 DEBUG:boto:Headers: {} 2024-04-13T22:08:07.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.484 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.484 DEBUG:boto:Params: {} 2024-04-13T22:08:07.484 DEBUG:boto:Token: None 2024-04-13T22:08:07.484 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/ 2024-04-13T22:08:07.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T22:08:07.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T22:08:07.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe7eaa3461ee382-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.486 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:08:07.487 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:08:07.487 DEBUG:boto:Method: PUT 2024-04-13T22:08:07.487 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:08:07.487 DEBUG:boto:Data: 2024-04-13T22:08:07.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:08:07.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.487 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.487 DEBUG:boto:Params: {} 2024-04-13T22:08:07.487 DEBUG:boto:Token: None 2024-04-13T22:08:07.488 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/b 2024-04-13T22:08:07.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:asnqkugpRzoWoJrYPKYh1DnxrXo= 2024-04-13T22:08:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:asnqkugpRzoWoJrYPKYh1DnxrXo='} 2024-04-13T22:08:07.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cf2fbb8750ceea5-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.492 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:08:07.493 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:08:07.493 DEBUG:boto:Method: HEAD 2024-04-13T22:08:07.493 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:08:07.493 DEBUG:boto:Data: 2024-04-13T22:08:07.493 DEBUG:boto:Headers: {} 2024-04-13T22:08:07.493 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.493 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.493 DEBUG:boto:Params: {} 2024-04-13T22:08:07.493 DEBUG:boto:Token: None 2024-04-13T22:08:07.494 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/ 2024-04-13T22:08:07.494 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T22:08:07.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T22:08:07.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e837cb57ad324e1f-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.496 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:08:07.496 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:08:07.496 DEBUG:boto:Method: PUT 2024-04-13T22:08:07.496 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:08:07.496 DEBUG:boto:Data: 2024-04-13T22:08:07.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:08:07.496 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.496 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.496 DEBUG:boto:Params: {} 2024-04-13T22:08:07.497 DEBUG:boto:Token: None 2024-04-13T22:08:07.497 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/c 2024-04-13T22:08:07.497 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ER4Vanx0ImZEQZuWLSGy+M5kx8c= 2024-04-13T22:08:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ER4Vanx0ImZEQZuWLSGy+M5kx8c='} 2024-04-13T22:08:07.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfeebfd1e72f5ee3-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.542 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:08:07.542 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:08:07.542 DEBUG:boto:Method: HEAD 2024-04-13T22:08:07.542 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:08:07.542 DEBUG:boto:Data: 2024-04-13T22:08:07.542 DEBUG:boto:Headers: {} 2024-04-13T22:08:07.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.543 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.543 DEBUG:boto:Params: {} 2024-04-13T22:08:07.543 DEBUG:boto:Token: None 2024-04-13T22:08:07.543 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/ 2024-04-13T22:08:07.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T22:08:07.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T22:08:07.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bbc43acdb0d751b-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.546 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:08:07.546 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:08:07.546 DEBUG:boto:Method: PUT 2024-04-13T22:08:07.546 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:08:07.546 DEBUG:boto:Data: 2024-04-13T22:08:07.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:08:07.546 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:08:07.546 DEBUG:boto:Port: 8000 2024-04-13T22:08:07.546 DEBUG:boto:Params: {} 2024-04-13T22:08:07.547 DEBUG:boto:Token: None 2024-04-13T22:08:07.547 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:08:07 GMT /vwnfns-57/d 2024-04-13T22:08:07.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZdgvwTWPT8kl0WyNUEProoE1FO8= 2024-04-13T22:08:07.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZdgvwTWPT8kl0WyNUEProoE1FO8='} 2024-04-13T22:08:07.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55c38057d55b8e5-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:08:07.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:07.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:07.552 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:08:07.601 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:07.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:07.632 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:08:07.645 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:08:07.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.645 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046087.430279_694.1", 2024-04-13T22:08:07.645 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:08:07.430279Z" 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:08:07.646 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:08:07.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.647 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:08:07.647 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:08:07.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:07.647 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:08:07.648 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:08:07.648 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:08:07.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:07.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:07.649 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:08:07.696 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:07.696 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:07.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:07.724+0000 7fe270341780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:08:07.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:07.725+0000 7fe270341780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:08:07.739 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:08:07.739 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:08:07.739 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:08:07.739 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:08:07.739 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046025.344683_686.1", 2024-04-13T22:08:07.740 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:07:05.344683Z", 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:07.741 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:08:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:07.743 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:07.744 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:08:07Z" 2024-04-13T22:08:07.745 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:08:07.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046025.344683_686.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:07:05.344683Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:08:07Z" } 2024-04-13T22:08:07.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046025.344683_686.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:07:05.344683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:08:07.746 DEBUG:rgw_multi.tests:log_status={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:08:07.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046025.344683_686.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:08:07.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046087.430279_694.1 target=1_1713046025.344683_686.1 2024-04-13T22:08:12.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:12.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:08:12.748 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:08:12.804 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:12.804 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:12.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:12.835+0000 7fbeca613780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:08:12.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:12.836+0000 7fbeca613780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:08:12.851 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:08:12.851 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:12.852 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046087.430279_694.1", 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:08:07.430279Z", 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:12.853 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:12.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:12.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:08:12.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:08:12Z" 2024-04-13T22:08:12.857 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:08:12.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046087.430279_694.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:08:07.430279Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:08:12Z" } 2024-04-13T22:08:12.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046087.430279_694.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:08:07.430279Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:08:12.858 DEBUG:rgw_multi.tests:log_status={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:08:12.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:08:12.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:08:12.859 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-57 2024-04-13T22:08:12.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:08:12.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:08:12.859 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:08:12.914 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:12.915 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:12.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:12.949+0000 7f398aca4780 1 bucket sync caught up with source: 2024-04-13T22:08:12.950 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.47.6, , , , 00000000001.34.6, , , 00000000001.28.6, , 00000000001.23.6] 2024-04-13T22:08:12.950 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.34.6, , , 00000000001.28.6, , 00000000001.23.6] 2024-04-13T22:08:12.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:12.949+0000 7f398aca4780 0 bucket checkpoint complete 2024-04-13T22:08:12.964 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-57 2024-04-13T22:08:12.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:08:12.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:08:12.964 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:08:13.018 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:08:13.018 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:08:13.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:13.051+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:18.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:18.052+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:23.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:23.052+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:28.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:28.053+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:33.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:33.055+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:38.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:38.055+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:43.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:43.056+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:48.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:48.055+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:53.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:53.056+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:08:58.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:08:58.056+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:09:03.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:03.057+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:09:08.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:08.059+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:09:13.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:13.059+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:09:18.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:18.060+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T22:09:23.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:23.062+0000 7fd75bbdd780 1 bucket sync caught up with source: 2024-04-13T22:09:23.063 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T22:09:23.063 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T22:09:23.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:23.062+0000 7fd75bbdd780 0 bucket checkpoint complete 2024-04-13T22:09:23.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-57 zones={a2, a1} 2024-04-13T22:09:23.080 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.080 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.081 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.081 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:23.081 DEBUG:boto:Data: 2024-04-13T22:09:23.081 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.081 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.081 DEBUG:boto:Params: {} 2024-04-13T22:09:23.082 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:09:23.082 DEBUG:boto:Token: None 2024-04-13T22:09:23.082 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE= 2024-04-13T22:09:23.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE=', 'Content-Length': '0'} 2024-04-13T22:09:23.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018f6e83d1036b3e3-00661b0293-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.089 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.089 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.089 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.089 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:23.089 DEBUG:boto:Data: 2024-04-13T22:09:23.089 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.089 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.090 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.090 DEBUG:boto:Params: {} 2024-04-13T22:09:23.090 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:09:23.090 DEBUG:boto:Token: None 2024-04-13T22:09:23.090 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.090 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE= 2024-04-13T22:09:23.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE=', 'Content-Length': '0'} 2024-04-13T22:09:23.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2b93d2473d92528-00661b0293-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:09:23.094 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.095 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.095 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.095 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.095 DEBUG:boto:Method: GET 2024-04-13T22:09:23.095 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.095 DEBUG:boto:Data: 2024-04-13T22:09:23.095 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.095 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.095 DEBUG:boto:Params: {} 2024-04-13T22:09:23.095 DEBUG:boto:Token: None 2024-04-13T22:09:23.096 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/?versions 2024-04-13T22:09:23.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T22:09:23.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T22:09:23.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e64cc12fd926ff8-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.098 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.099 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:09:23.100 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:09:23.100 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:09:23.100 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:09:23.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:09:23.100 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.100 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.100 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.100 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.100 DEBUG:boto:Method: GET 2024-04-13T22:09:23.100 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.101 DEBUG:boto:Data: 2024-04-13T22:09:23.101 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.101 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.101 DEBUG:boto:Params: {} 2024-04-13T22:09:23.101 DEBUG:boto:Token: None 2024-04-13T22:09:23.101 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/?versions 2024-04-13T22:09:23.101 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T22:09:23.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T22:09:23.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c739d51e2ecce89-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.104 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.104 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:09:23.105 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:09:23.105 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:09:23.105 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:09:23.105 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.105 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.105 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.105 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.105 DEBUG:boto:Method: GET 2024-04-13T22:09:23.105 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.105 DEBUG:boto:Data: 2024-04-13T22:09:23.106 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.106 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.106 DEBUG:boto:Params: {} 2024-04-13T22:09:23.106 DEBUG:boto:Token: None 2024-04-13T22:09:23.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/?versions 2024-04-13T22:09:23.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T22:09:23.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T22:09:23.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8aed40b590df873-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.109 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.109 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.110 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.110 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.110 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.110 DEBUG:boto:Method: GET 2024-04-13T22:09:23.110 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:23.110 DEBUG:boto:Data: 2024-04-13T22:09:23.110 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.110 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.111 DEBUG:boto:Params: {} 2024-04-13T22:09:23.111 DEBUG:boto:Token: None 2024-04-13T22:09:23.111 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/?versions 2024-04-13T22:09:23.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T22:09:23.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T22:09:23.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177ad64d252bf4a3-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.114 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:23.114 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.114 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.115 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.115 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.115 DEBUG:boto:Method: GET 2024-04-13T22:09:23.115 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.115 DEBUG:boto:Data: 2024-04-13T22:09:23.115 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.115 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.115 DEBUG:boto:Params: {} 2024-04-13T22:09:23.115 DEBUG:boto:Token: None 2024-04-13T22:09:23.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T22:09:23.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T22:09:23.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa592f9d0128ed4f-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.118 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.118 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.118 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.118 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.118 DEBUG:boto:Method: GET 2024-04-13T22:09:23.118 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.118 DEBUG:boto:Data: 2024-04-13T22:09:23.119 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.119 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.119 DEBUG:boto:Params: {} 2024-04-13T22:09:23.119 DEBUG:boto:Token: None 2024-04-13T22:09:23.119 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.119 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T22:09:23.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T22:09:23.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000641b0d96c7cb370-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.121 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.121 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.122 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.122 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.122 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.122 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.122 DEBUG:boto:Data: 2024-04-13T22:09:23.122 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.122 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.122 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.122 DEBUG:boto:Params: {} 2024-04-13T22:09:23.122 DEBUG:boto:Token: None 2024-04-13T22:09:23.123 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8= 2024-04-13T22:09:23.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8=', 'Content-Length': '0'} 2024-04-13T22:09:23.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a3316ecf787d4a-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.124 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.125 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.125 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.125 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.125 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.125 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.125 DEBUG:boto:Data: 2024-04-13T22:09:23.125 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.125 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.125 DEBUG:boto:Params: {} 2024-04-13T22:09:23.126 DEBUG:boto:Token: None 2024-04-13T22:09:23.126 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8= 2024-04-13T22:09:23.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8=', 'Content-Length': '0'} 2024-04-13T22:09:23.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100751f6e10f687d-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:23.128 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.128 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.128 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.128 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.128 DEBUG:boto:Method: GET 2024-04-13T22:09:23.128 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.128 DEBUG:boto:Data: 2024-04-13T22:09:23.128 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.129 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.129 DEBUG:boto:Params: {} 2024-04-13T22:09:23.129 DEBUG:boto:Token: None 2024-04-13T22:09:23.129 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T22:09:23.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T22:09:23.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a59c3296cd4483-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.131 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.132 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.132 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.132 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:23.132 DEBUG:boto:Method: GET 2024-04-13T22:09:23.132 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.132 DEBUG:boto:Data: 2024-04-13T22:09:23.132 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.132 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.132 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.132 DEBUG:boto:Params: {} 2024-04-13T22:09:23.132 DEBUG:boto:Token: None 2024-04-13T22:09:23.133 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:23.133 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T22:09:23.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T22:09:23.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f917cc4e2e710bd5-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.135 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.135 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.135 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.135 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.135 DEBUG:boto:Method: GET 2024-04-13T22:09:23.135 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:23.135 DEBUG:boto:Data: 2024-04-13T22:09:23.135 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.136 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.136 DEBUG:boto:Params: {} 2024-04-13T22:09:23.136 DEBUG:boto:Token: None 2024-04-13T22:09:23.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd8f70eeb243c1b7-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.139 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.139 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.139 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.139 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.139 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.140 DEBUG:boto:Method: GET 2024-04-13T22:09:23.140 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:23.140 DEBUG:boto:Data: 2024-04-13T22:09:23.140 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.140 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.140 DEBUG:boto:Params: {} 2024-04-13T22:09:23.140 DEBUG:boto:Token: None 2024-04-13T22:09:23.140 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bd124721035a6e-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.143 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.143 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.144 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.144 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.144 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:23.144 DEBUG:boto:Data: 2024-04-13T22:09:23.144 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.144 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.144 DEBUG:boto:Params: {} 2024-04-13T22:09:23.144 DEBUG:boto:Token: None 2024-04-13T22:09:23.144 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a 2024-04-13T22:09:23.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw= 2024-04-13T22:09:23.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw=', 'Content-Length': '0'} 2024-04-13T22:09:23.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bfdc05510c6ce95-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.146 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.146 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.147 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.147 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:23.147 DEBUG:boto:Data: 2024-04-13T22:09:23.147 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.147 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.147 DEBUG:boto:Params: {} 2024-04-13T22:09:23.147 DEBUG:boto:Token: None 2024-04-13T22:09:23.147 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a 2024-04-13T22:09:23.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw= 2024-04-13T22:09:23.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw=', 'Content-Length': '0'} 2024-04-13T22:09:23.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c797a048c8dccbac-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:23.150 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.150 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.150 DEBUG:boto:Method: GET 2024-04-13T22:09:23.150 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:23.150 DEBUG:boto:Data: 2024-04-13T22:09:23.150 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.150 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.151 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.151 DEBUG:boto:Params: {} 2024-04-13T22:09:23.151 DEBUG:boto:Token: None 2024-04-13T22:09:23.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a 2024-04-13T22:09:23.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE= 2024-04-13T22:09:23.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE=', 'Content-Length': '0'} 2024-04-13T22:09:23.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9a1ca71f39852d-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.194 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:23.194 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:23.195 DEBUG:boto:Method: GET 2024-04-13T22:09:23.195 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:23.195 DEBUG:boto:Data: 2024-04-13T22:09:23.195 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.195 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.195 DEBUG:boto:Params: {} 2024-04-13T22:09:23.195 DEBUG:boto:Token: None 2024-04-13T22:09:23.195 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/a 2024-04-13T22:09:23.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE= 2024-04-13T22:09:23.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE=', 'Content-Length': '0'} 2024-04-13T22:09:23.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb1fa909cacbe34-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.198 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.198 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.198 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.198 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.198 DEBUG:boto:Method: GET 2024-04-13T22:09:23.199 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:23.199 DEBUG:boto:Data: 2024-04-13T22:09:23.199 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.199 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.199 DEBUG:boto:Params: {} 2024-04-13T22:09:23.199 DEBUG:boto:Token: None 2024-04-13T22:09:23.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bba58ed5f5f9b3e-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.202 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.202 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.203 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.203 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.203 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:23.203 DEBUG:boto:Method: GET 2024-04-13T22:09:23.203 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:23.203 DEBUG:boto:Data: 2024-04-13T22:09:23.203 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.203 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.203 DEBUG:boto:Params: {} 2024-04-13T22:09:23.204 DEBUG:boto:Token: None 2024-04-13T22:09:23.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d789aa53c00cad0-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.206 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:23.207 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.207 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.207 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.207 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.207 DEBUG:boto:Method: GET 2024-04-13T22:09:23.207 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.207 DEBUG:boto:Data: 2024-04-13T22:09:23.208 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.208 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.208 DEBUG:boto:Params: {} 2024-04-13T22:09:23.208 DEBUG:boto:Token: None 2024-04-13T22:09:23.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T22:09:23.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T22:09:23.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e11f5df7a02a8918-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.210 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.210 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.210 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.211 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.211 DEBUG:boto:Method: GET 2024-04-13T22:09:23.211 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.211 DEBUG:boto:Data: 2024-04-13T22:09:23.211 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.211 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.211 DEBUG:boto:Params: {} 2024-04-13T22:09:23.211 DEBUG:boto:Token: None 2024-04-13T22:09:23.211 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T22:09:23.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T22:09:23.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e8a693d07eea4f-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.213 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.214 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.214 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.214 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.214 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.214 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.214 DEBUG:boto:Data: 2024-04-13T22:09:23.214 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.214 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.214 DEBUG:boto:Params: {} 2024-04-13T22:09:23.215 DEBUG:boto:Token: None 2024-04-13T22:09:23.215 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY= 2024-04-13T22:09:23.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY=', 'Content-Length': '0'} 2024-04-13T22:09:23.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000217c270935dd893e-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.217 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.217 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.217 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.217 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.217 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.217 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.218 DEBUG:boto:Data: 2024-04-13T22:09:23.218 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.218 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.218 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.218 DEBUG:boto:Params: {} 2024-04-13T22:09:23.218 DEBUG:boto:Token: None 2024-04-13T22:09:23.218 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY= 2024-04-13T22:09:23.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY=', 'Content-Length': '0'} 2024-04-13T22:09:23.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec64f0ddfc72fec9-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:23.220 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.220 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.221 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.221 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.221 DEBUG:boto:Method: GET 2024-04-13T22:09:23.221 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.221 DEBUG:boto:Data: 2024-04-13T22:09:23.221 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.221 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.221 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.221 DEBUG:boto:Params: {} 2024-04-13T22:09:23.221 DEBUG:boto:Token: None 2024-04-13T22:09:23.221 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.222 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T22:09:23.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T22:09:23.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d17aba8c5b0e268-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.223 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.224 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.224 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.224 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:23.224 DEBUG:boto:Method: GET 2024-04-13T22:09:23.224 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.224 DEBUG:boto:Data: 2024-04-13T22:09:23.224 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.224 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.224 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.224 DEBUG:boto:Params: {} 2024-04-13T22:09:23.225 DEBUG:boto:Token: None 2024-04-13T22:09:23.225 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:23.225 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T22:09:23.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T22:09:23.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ed97308b0e3ce8-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.227 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.227 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.227 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.227 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.227 DEBUG:boto:Method: GET 2024-04-13T22:09:23.227 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:23.228 DEBUG:boto:Data: 2024-04-13T22:09:23.228 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.228 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.228 DEBUG:boto:Params: {} 2024-04-13T22:09:23.228 DEBUG:boto:Token: None 2024-04-13T22:09:23.228 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2363773e224c26-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.231 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.231 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.231 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.232 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.232 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.232 DEBUG:boto:Method: GET 2024-04-13T22:09:23.232 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:23.232 DEBUG:boto:Data: 2024-04-13T22:09:23.232 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.232 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.232 DEBUG:boto:Params: {} 2024-04-13T22:09:23.232 DEBUG:boto:Token: None 2024-04-13T22:09:23.233 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.233 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b4203cfc924aa2-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.235 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.235 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.235 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.236 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.236 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:23.236 DEBUG:boto:Data: 2024-04-13T22:09:23.236 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.236 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.236 DEBUG:boto:Params: {} 2024-04-13T22:09:23.236 DEBUG:boto:Token: None 2024-04-13T22:09:23.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b 2024-04-13T22:09:23.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY= 2024-04-13T22:09:23.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY=', 'Content-Length': '0'} 2024-04-13T22:09:23.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6130b7ee07f7ad-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.238 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.239 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.239 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.239 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:23.239 DEBUG:boto:Data: 2024-04-13T22:09:23.239 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.239 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.239 DEBUG:boto:Params: {} 2024-04-13T22:09:23.242 DEBUG:boto:Token: None 2024-04-13T22:09:23.242 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b 2024-04-13T22:09:23.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY= 2024-04-13T22:09:23.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY=', 'Content-Length': '0'} 2024-04-13T22:09:23.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5991af13725eea-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:23.244 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.244 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.244 DEBUG:boto:Method: GET 2024-04-13T22:09:23.244 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:23.244 DEBUG:boto:Data: 2024-04-13T22:09:23.244 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.245 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.245 DEBUG:boto:Params: {} 2024-04-13T22:09:23.245 DEBUG:boto:Token: None 2024-04-13T22:09:23.245 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b 2024-04-13T22:09:23.245 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34= 2024-04-13T22:09:23.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34=', 'Content-Length': '0'} 2024-04-13T22:09:23.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075187101dae670ee-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.247 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:23.247 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:23.247 DEBUG:boto:Method: GET 2024-04-13T22:09:23.247 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:23.247 DEBUG:boto:Data: 2024-04-13T22:09:23.247 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.248 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.248 DEBUG:boto:Params: {} 2024-04-13T22:09:23.248 DEBUG:boto:Token: None 2024-04-13T22:09:23.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/b 2024-04-13T22:09:23.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34= 2024-04-13T22:09:23.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34=', 'Content-Length': '0'} 2024-04-13T22:09:23.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0f347b6442c9d1-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.290 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.291 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.291 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.291 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.291 DEBUG:boto:Method: GET 2024-04-13T22:09:23.291 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:23.291 DEBUG:boto:Data: 2024-04-13T22:09:23.291 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.291 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.291 DEBUG:boto:Params: {} 2024-04-13T22:09:23.291 DEBUG:boto:Token: None 2024-04-13T22:09:23.292 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076441db18ae42b5c-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.295 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.295 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.295 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.296 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.296 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:23.296 DEBUG:boto:Method: GET 2024-04-13T22:09:23.296 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:23.296 DEBUG:boto:Data: 2024-04-13T22:09:23.296 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.296 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.296 DEBUG:boto:Params: {} 2024-04-13T22:09:23.297 DEBUG:boto:Token: None 2024-04-13T22:09:23.297 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.297 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8474c9a20d1ac50-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.299 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:23.300 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.300 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.300 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.300 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.300 DEBUG:boto:Method: GET 2024-04-13T22:09:23.300 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.300 DEBUG:boto:Data: 2024-04-13T22:09:23.300 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.301 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.301 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.301 DEBUG:boto:Params: {} 2024-04-13T22:09:23.301 DEBUG:boto:Token: None 2024-04-13T22:09:23.301 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T22:09:23.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T22:09:23.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dfb3a39196a4f77-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.350 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.351 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.351 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.351 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.351 DEBUG:boto:Method: GET 2024-04-13T22:09:23.351 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.351 DEBUG:boto:Data: 2024-04-13T22:09:23.351 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.351 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.351 DEBUG:boto:Params: {} 2024-04-13T22:09:23.352 DEBUG:boto:Token: None 2024-04-13T22:09:23.352 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.352 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T22:09:23.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T22:09:23.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e345c9a06880055-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.354 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.354 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.354 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.355 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.355 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.355 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.355 DEBUG:boto:Data: 2024-04-13T22:09:23.355 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.355 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.355 DEBUG:boto:Params: {} 2024-04-13T22:09:23.355 DEBUG:boto:Token: None 2024-04-13T22:09:23.355 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI= 2024-04-13T22:09:23.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI=', 'Content-Length': '0'} 2024-04-13T22:09:23.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002adc289a8c6b7b6b-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.358 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.358 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.358 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.358 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.358 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.358 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.358 DEBUG:boto:Data: 2024-04-13T22:09:23.358 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.358 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.358 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.359 DEBUG:boto:Params: {} 2024-04-13T22:09:23.359 DEBUG:boto:Token: None 2024-04-13T22:09:23.359 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.359 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI= 2024-04-13T22:09:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI=', 'Content-Length': '0'} 2024-04-13T22:09:23.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f4fdb66ca5c78b9-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:23.361 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.361 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.361 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.361 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.361 DEBUG:boto:Method: GET 2024-04-13T22:09:23.362 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.362 DEBUG:boto:Data: 2024-04-13T22:09:23.362 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.362 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.362 DEBUG:boto:Params: {} 2024-04-13T22:09:23.362 DEBUG:boto:Token: None 2024-04-13T22:09:23.362 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T22:09:23.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T22:09:23.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117ddc3f76167c72-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.365 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.365 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.365 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.365 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:23.366 DEBUG:boto:Method: GET 2024-04-13T22:09:23.366 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.366 DEBUG:boto:Data: 2024-04-13T22:09:23.366 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.366 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.366 DEBUG:boto:Params: {} 2024-04-13T22:09:23.366 DEBUG:boto:Token: None 2024-04-13T22:09:23.367 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:23.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T22:09:23.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T22:09:23.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f97194d6ba02dd26-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.369 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.369 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.369 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.370 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.370 DEBUG:boto:Method: GET 2024-04-13T22:09:23.370 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:23.370 DEBUG:boto:Data: 2024-04-13T22:09:23.370 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.370 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.370 DEBUG:boto:Params: {} 2024-04-13T22:09:23.370 DEBUG:boto:Token: None 2024-04-13T22:09:23.370 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e575a97689be5d41-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.373 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.374 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.374 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.374 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.374 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.374 DEBUG:boto:Method: GET 2024-04-13T22:09:23.374 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:23.374 DEBUG:boto:Data: 2024-04-13T22:09:23.374 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.374 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.375 DEBUG:boto:Params: {} 2024-04-13T22:09:23.375 DEBUG:boto:Token: None 2024-04-13T22:09:23.375 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e866a6b334fd8942-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.377 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.378 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.378 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.378 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.378 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:23.378 DEBUG:boto:Data: 2024-04-13T22:09:23.378 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.378 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.378 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.378 DEBUG:boto:Params: {} 2024-04-13T22:09:23.379 DEBUG:boto:Token: None 2024-04-13T22:09:23.379 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c 2024-04-13T22:09:23.379 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0= 2024-04-13T22:09:23.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0=', 'Content-Length': '0'} 2024-04-13T22:09:23.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adbc740c27de6edd-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.381 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.381 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.381 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.381 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:23.381 DEBUG:boto:Data: 2024-04-13T22:09:23.381 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.381 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.381 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.382 DEBUG:boto:Params: {} 2024-04-13T22:09:23.382 DEBUG:boto:Token: None 2024-04-13T22:09:23.382 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c 2024-04-13T22:09:23.382 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0= 2024-04-13T22:09:23.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0=', 'Content-Length': '0'} 2024-04-13T22:09:23.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c2f93e71ead2b6-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:23.384 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.384 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.384 DEBUG:boto:Method: GET 2024-04-13T22:09:23.384 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:23.384 DEBUG:boto:Data: 2024-04-13T22:09:23.384 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.385 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.385 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.385 DEBUG:boto:Params: {} 2024-04-13T22:09:23.385 DEBUG:boto:Token: None 2024-04-13T22:09:23.385 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c 2024-04-13T22:09:23.385 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc= 2024-04-13T22:09:23.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc=', 'Content-Length': '0'} 2024-04-13T22:09:23.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fcca34f88ba9e66-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.387 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:23.387 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:23.387 DEBUG:boto:Method: GET 2024-04-13T22:09:23.387 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:23.388 DEBUG:boto:Data: 2024-04-13T22:09:23.388 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.388 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.388 DEBUG:boto:Params: {} 2024-04-13T22:09:23.388 DEBUG:boto:Token: None 2024-04-13T22:09:23.388 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/c 2024-04-13T22:09:23.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc= 2024-04-13T22:09:23.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc=', 'Content-Length': '0'} 2024-04-13T22:09:23.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ec2557176001f6-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.390 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.390 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.390 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.390 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.391 DEBUG:boto:Method: GET 2024-04-13T22:09:23.391 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:23.391 DEBUG:boto:Data: 2024-04-13T22:09:23.391 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.391 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.391 DEBUG:boto:Params: {} 2024-04-13T22:09:23.391 DEBUG:boto:Token: None 2024-04-13T22:09:23.391 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b427db4c78d0777-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.394 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.395 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.395 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.395 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.395 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:23.395 DEBUG:boto:Method: GET 2024-04-13T22:09:23.395 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:23.395 DEBUG:boto:Data: 2024-04-13T22:09:23.395 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.395 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.396 DEBUG:boto:Params: {} 2024-04-13T22:09:23.396 DEBUG:boto:Token: None 2024-04-13T22:09:23.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbcc7104e6f20edc-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.398 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:23.399 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.399 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.399 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.399 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.399 DEBUG:boto:Method: GET 2024-04-13T22:09:23.399 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.399 DEBUG:boto:Data: 2024-04-13T22:09:23.399 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.400 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.400 DEBUG:boto:Params: {} 2024-04-13T22:09:23.400 DEBUG:boto:Token: None 2024-04-13T22:09:23.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T22:09:23.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T22:09:23.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003461cbdd2fa92e-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.442 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.442 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.445 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.445 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.445 DEBUG:boto:Method: GET 2024-04-13T22:09:23.446 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.446 DEBUG:boto:Data: 2024-04-13T22:09:23.446 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.446 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.446 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.446 DEBUG:boto:Params: {} 2024-04-13T22:09:23.446 DEBUG:boto:Token: None 2024-04-13T22:09:23.446 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T22:09:23.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T22:09:23.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512eeeacfbb4dee0-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.449 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.449 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.449 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.449 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.449 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.449 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.449 DEBUG:boto:Data: 2024-04-13T22:09:23.449 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.449 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.450 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.450 DEBUG:boto:Params: {} 2024-04-13T22:09:23.450 DEBUG:boto:Token: None 2024-04-13T22:09:23.450 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.450 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM= 2024-04-13T22:09:23.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM=', 'Content-Length': '0'} 2024-04-13T22:09:23.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da326be6dfe04cc-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.452 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.452 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.452 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.452 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.452 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.452 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.453 DEBUG:boto:Data: 2024-04-13T22:09:23.453 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.453 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.453 DEBUG:boto:Params: {} 2024-04-13T22:09:23.453 DEBUG:boto:Token: None 2024-04-13T22:09:23.453 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.453 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM= 2024-04-13T22:09:23.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM=', 'Content-Length': '0'} 2024-04-13T22:09:23.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b718339a4fae3b-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:23.455 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.456 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.456 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.456 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.456 DEBUG:boto:Method: GET 2024-04-13T22:09:23.456 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.456 DEBUG:boto:Data: 2024-04-13T22:09:23.456 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.456 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.456 DEBUG:boto:Params: {} 2024-04-13T22:09:23.456 DEBUG:boto:Token: None 2024-04-13T22:09:23.457 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.457 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T22:09:23.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T22:09:23.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227dcefcd949b129-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.459 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.459 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.459 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.459 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:23.459 DEBUG:boto:Method: GET 2024-04-13T22:09:23.459 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.459 DEBUG:boto:Data: 2024-04-13T22:09:23.459 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.459 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.460 DEBUG:boto:Params: {} 2024-04-13T22:09:23.460 DEBUG:boto:Token: None 2024-04-13T22:09:23.460 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:23.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T22:09:23.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T22:09:23.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3d619212ae97a7-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.462 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.463 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.463 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.463 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.463 DEBUG:boto:Method: GET 2024-04-13T22:09:23.463 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:23.463 DEBUG:boto:Data: 2024-04-13T22:09:23.463 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.463 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.463 DEBUG:boto:Params: {} 2024-04-13T22:09:23.463 DEBUG:boto:Token: None 2024-04-13T22:09:23.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ac377a1a2ddb4e-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.466 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.467 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.467 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.467 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.467 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.467 DEBUG:boto:Method: GET 2024-04-13T22:09:23.467 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:23.467 DEBUG:boto:Data: 2024-04-13T22:09:23.468 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.468 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.468 DEBUG:boto:Params: {} 2024-04-13T22:09:23.468 DEBUG:boto:Token: None 2024-04-13T22:09:23.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9f67f8624c0d9e-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.518 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.519 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.519 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.519 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.519 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:23.519 DEBUG:boto:Data: 2024-04-13T22:09:23.519 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.519 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.520 DEBUG:boto:Params: {} 2024-04-13T22:09:23.520 DEBUG:boto:Token: None 2024-04-13T22:09:23.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d 2024-04-13T22:09:23.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM= 2024-04-13T22:09:23.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM=', 'Content-Length': '0'} 2024-04-13T22:09:23.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27aa161780ae2a8-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.523 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.523 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.523 DEBUG:boto:Method: HEAD 2024-04-13T22:09:23.523 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:23.523 DEBUG:boto:Data: 2024-04-13T22:09:23.523 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.523 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.523 DEBUG:boto:Params: {} 2024-04-13T22:09:23.524 DEBUG:boto:Token: None 2024-04-13T22:09:23.524 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d 2024-04-13T22:09:23.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM= 2024-04-13T22:09:23.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM=', 'Content-Length': '0'} 2024-04-13T22:09:23.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56bbcf0f2d6a788-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:23.526 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.526 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.526 DEBUG:boto:Method: GET 2024-04-13T22:09:23.526 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:23.526 DEBUG:boto:Data: 2024-04-13T22:09:23.527 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.527 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.527 DEBUG:boto:Params: {} 2024-04-13T22:09:23.527 DEBUG:boto:Token: None 2024-04-13T22:09:23.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d 2024-04-13T22:09:23.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0= 2024-04-13T22:09:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0=', 'Content-Length': '0'} 2024-04-13T22:09:23.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b3803fee8694f0-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.529 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:23.530 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:23.530 DEBUG:boto:Method: GET 2024-04-13T22:09:23.530 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:23.530 DEBUG:boto:Data: 2024-04-13T22:09:23.530 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.531 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.531 DEBUG:boto:Params: {} 2024-04-13T22:09:23.531 DEBUG:boto:Token: None 2024-04-13T22:09:23.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/d 2024-04-13T22:09:23.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0= 2024-04-13T22:09:23.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0=', 'Content-Length': '0'} 2024-04-13T22:09:23.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f06752da5d478b6-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.534 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.534 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.535 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.535 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.535 DEBUG:boto:Method: GET 2024-04-13T22:09:23.535 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:23.535 DEBUG:boto:Data: 2024-04-13T22:09:23.535 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.536 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:23.536 DEBUG:boto:Port: 8001 2024-04-13T22:09:23.536 DEBUG:boto:Params: {} 2024-04-13T22:09:23.536 DEBUG:boto:Token: None 2024-04-13T22:09:23.537 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.537 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fade15e8298dfbc-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.540 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.541 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:23.541 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:23.541 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.541 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:23.541 DEBUG:boto:Method: GET 2024-04-13T22:09:23.541 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:23.541 DEBUG:boto:Data: 2024-04-13T22:09:23.541 DEBUG:boto:Headers: {} 2024-04-13T22:09:23.541 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:23.541 DEBUG:boto:Port: 8000 2024-04-13T22:09:23.542 DEBUG:boto:Params: {} 2024-04-13T22:09:23.542 DEBUG:boto:Token: None 2024-04-13T22:09:23.542 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:23 GMT /vwnfns-57/ 2024-04-13T22:09:23.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T22:09:23.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T22:09:23.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb9c52f019a48d2f-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:23.545 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:23.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-57 zones={a2, a1} 2024-04-13T22:09:23.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:23.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:23.546 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:23.595 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:23.595 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:23.637 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:09:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding": "None", 2024-04-13T22:09:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "current_index": { 2024-04-13T22:09:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:23.639 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.639 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "Normal", 2024-04-13T22:09:23.639 INFO:teuthology.orchestra.run.smithi132.stdout: "normal": { 2024-04-13T22:09:23.639 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 11, 2024-04-13T22:09:23.640 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:23.640 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.640 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.640 INFO:teuthology.orchestra.run.smithi132.stdout: "logs": [ 2024-04-13T22:09:23.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.641 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:23.641 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.641 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:23.642 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:23.642 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:23.642 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.642 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 11, 2024-04-13T22:09:23.642 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:23.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.644 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:23.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.644 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:09:23.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:23.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:23.649 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-57 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:09:23.698 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:23.698 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:23.746 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:09:23.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.746 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "1#00000000001.47.6", 2024-04-13T22:09:23.746 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.6548832894401009285", 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "a", 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:08:07.439673791Z", 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 36 2024-04-13T22:09:23.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.748 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "5#00000000001.34.6", 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.11915460310344251621", 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "d", 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:09:23.749 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:08:07.549016590Z", 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:09:23.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "8#00000000001.28.6", 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16136045423162777315", 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "c", 2024-04-13T22:09:23.751 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:08:07.498897639Z", 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:09:23.752 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "op_id": "10#00000000001.23.6", 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4391849356295990949", 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "op": "write", 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "object": "b", 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:09:23.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "complete", 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "index_ver": 1, 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:08:07.489738979Z", 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "bilog_flags": 0, 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned": false, 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "", 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "", 2024-04-13T22:09:23.754 INFO:teuthology.orchestra.run.smithi132.stdout: "zones_trace": [ 2024-04-13T22:09:23.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.755 INFO:teuthology.orchestra.run.smithi132.stdout: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T22:09:23.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.755 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:23.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.755 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:09:23.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:23.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:23.756 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:23.812 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:23.812 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:23.854 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:09:23.854 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-57 2024-04-13T22:09:23.855 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 4 2024-04-13T22:09:23.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:23.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:23.891 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:23.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:23.936 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:23.977 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:09:23.977 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.977 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding": "None", 2024-04-13T22:09:23.977 INFO:teuthology.orchestra.run.smithi132.stdout: "current_index": { 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "Normal", 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "normal": { 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: "logs": [ 2024-04-13T22:09:23.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 11, 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.979 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.981 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:23.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:23.981 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:09:23.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:23.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:23.982 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-57 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:09:24.030 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:24.030 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:24.071 INFO:teuthology.orchestra.run.smithi132.stdout:[] 2024-04-13T22:09:24.073 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:24.073 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:24.073 DEBUG:boto:Method: HEAD 2024-04-13T22:09:24.073 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:24.073 DEBUG:boto:Data: 2024-04-13T22:09:24.073 DEBUG:boto:Headers: {} 2024-04-13T22:09:24.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.074 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.074 DEBUG:boto:Params: {} 2024-04-13T22:09:24.074 DEBUG:boto:Token: None 2024-04-13T22:09:24.074 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/ 2024-04-13T22:09:24.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T22:09:24.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T22:09:24.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004eb49a91ab4e79e2-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.077 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:24.077 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:24.077 DEBUG:boto:Method: PUT 2024-04-13T22:09:24.077 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:24.077 DEBUG:boto:Data: 2024-04-13T22:09:24.077 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:09:24.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.077 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.078 DEBUG:boto:Params: {} 2024-04-13T22:09:24.078 DEBUG:boto:Token: None 2024-04-13T22:09:24.078 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/e 2024-04-13T22:09:24.078 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OUAB/fpetK/jnb8oBKCCCQpOdZo= 2024-04-13T22:09:24.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OUAB/fpetK/jnb8oBKCCCQpOdZo='} 2024-04-13T22:09:24.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001b07c0f4c7fccb7-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.083 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:24.083 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:24.084 DEBUG:boto:Method: HEAD 2024-04-13T22:09:24.084 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:24.084 DEBUG:boto:Data: 2024-04-13T22:09:24.084 DEBUG:boto:Headers: {} 2024-04-13T22:09:24.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.084 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.084 DEBUG:boto:Params: {} 2024-04-13T22:09:24.084 DEBUG:boto:Token: None 2024-04-13T22:09:24.084 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/ 2024-04-13T22:09:24.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T22:09:24.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T22:09:24.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001edcd9ce6f4c027f-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.086 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:24.086 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:24.087 DEBUG:boto:Method: PUT 2024-04-13T22:09:24.087 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:24.087 DEBUG:boto:Data: 2024-04-13T22:09:24.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:09:24.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.087 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.087 DEBUG:boto:Params: {} 2024-04-13T22:09:24.087 DEBUG:boto:Token: None 2024-04-13T22:09:24.087 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/f 2024-04-13T22:09:24.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2rUIYQUJoLvt2CZMxbRs6ZJwMeY= 2024-04-13T22:09:24.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2rUIYQUJoLvt2CZMxbRs6ZJwMeY='} 2024-04-13T22:09:24.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000618153d25d485cf5-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.092 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:24.092 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:24.092 DEBUG:boto:Method: HEAD 2024-04-13T22:09:24.092 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:24.093 DEBUG:boto:Data: 2024-04-13T22:09:24.093 DEBUG:boto:Headers: {} 2024-04-13T22:09:24.093 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.093 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.093 DEBUG:boto:Params: {} 2024-04-13T22:09:24.093 DEBUG:boto:Token: None 2024-04-13T22:09:24.093 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/ 2024-04-13T22:09:24.093 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T22:09:24.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T22:09:24.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8101fb934e8863c-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.095 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:24.095 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:24.095 DEBUG:boto:Method: PUT 2024-04-13T22:09:24.095 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:24.096 DEBUG:boto:Data: 2024-04-13T22:09:24.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:09:24.096 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.096 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.096 DEBUG:boto:Params: {} 2024-04-13T22:09:24.096 DEBUG:boto:Token: None 2024-04-13T22:09:24.096 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/g 2024-04-13T22:09:24.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkcWU3NNbWfVvTiA5hEtgScz9Zo= 2024-04-13T22:09:24.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkcWU3NNbWfVvTiA5hEtgScz9Zo='} 2024-04-13T22:09:24.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054c6dc90c45b5278-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.139 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:24.139 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:24.139 DEBUG:boto:Method: HEAD 2024-04-13T22:09:24.139 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:24.139 DEBUG:boto:Data: 2024-04-13T22:09:24.139 DEBUG:boto:Headers: {} 2024-04-13T22:09:24.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.139 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.139 DEBUG:boto:Params: {} 2024-04-13T22:09:24.140 DEBUG:boto:Token: None 2024-04-13T22:09:24.140 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/ 2024-04-13T22:09:24.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T22:09:24.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T22:09:24.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000280614d52e6bb0eb-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.142 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:24.142 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:24.142 DEBUG:boto:Method: PUT 2024-04-13T22:09:24.142 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:24.142 DEBUG:boto:Data: 2024-04-13T22:09:24.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:09:24.142 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:24.143 DEBUG:boto:Port: 8000 2024-04-13T22:09:24.143 DEBUG:boto:Params: {} 2024-04-13T22:09:24.143 DEBUG:boto:Token: None 2024-04-13T22:09:24.143 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:09:24 GMT /vwnfns-57/h 2024-04-13T22:09:24.143 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YRCkaFSDh4OcbCyNd6wGUSfMKh0= 2024-04-13T22:09:24.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YRCkaFSDh4OcbCyNd6wGUSfMKh0='} 2024-04-13T22:09:24.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4217c2e143d2b6f-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:24.147 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-57 2024-04-13T22:09:24.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:24.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:24.147 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:09:24.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:24.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:24.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:24.236+0000 7fd125eb8780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T22:09:29.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:29.236+0000 7fd125eb8780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T22:09:34.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:34.237+0000 7fd125eb8780 1 bucket sync caught up with source: 2024-04-13T22:09:34.239 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.39.6, 00000000002.76.6, 00000000002.26.6] 2024-04-13T22:09:34.239 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.39.6, 00000000002.76.6, 00000000002.26.6] 2024-04-13T22:09:34.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:34.237+0000 7fd125eb8780 0 bucket checkpoint complete 2024-04-13T22:09:34.256 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-57 2024-04-13T22:09:34.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:34.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:34.257 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:09:34.317 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:34.317 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:34.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:34.352+0000 7f9ea556a780 1 waiting for incremental sync to catch up: 2024-04-13T22:09:34.354 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T22:09:34.354 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T22:09:39.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:39.353+0000 7f9ea556a780 1 waiting for incremental sync to catch up: 2024-04-13T22:09:39.356 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T22:09:39.356 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T22:09:44.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:44.355+0000 7f9ea556a780 1 bucket sync caught up with source: 2024-04-13T22:09:44.357 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T22:09:44.357 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T22:09:44.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:44.355+0000 7f9ea556a780 0 bucket checkpoint complete 2024-04-13T22:09:44.374 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-57 zones={a2, a1} 2024-04-13T22:09:44.374 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.374 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.374 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.374 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:44.374 DEBUG:boto:Data: 2024-04-13T22:09:44.374 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.375 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.375 DEBUG:boto:Params: {} 2024-04-13T22:09:44.375 DEBUG:boto:Token: None 2024-04-13T22:09:44.375 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE= 2024-04-13T22:09:44.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE=', 'Content-Length': '0'} 2024-04-13T22:09:44.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090bc245c1cfff698-00661b02a8-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.379 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.379 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.380 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.380 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:44.380 DEBUG:boto:Data: 2024-04-13T22:09:44.380 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.380 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.380 DEBUG:boto:Params: {} 2024-04-13T22:09:44.380 DEBUG:boto:Token: None 2024-04-13T22:09:44.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE= 2024-04-13T22:09:44.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE=', 'Content-Length': '0'} 2024-04-13T22:09:44.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924adc1b1847d267-00661b02a8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:09:44.383 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.383 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.383 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.383 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.383 DEBUG:boto:Method: GET 2024-04-13T22:09:44.383 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.383 DEBUG:boto:Data: 2024-04-13T22:09:44.384 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.384 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.384 DEBUG:boto:Params: {} 2024-04-13T22:09:44.384 DEBUG:boto:Token: None 2024-04-13T22:09:44.384 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/?versions 2024-04-13T22:09:44.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T22:09:44.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T22:09:44.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7fb875f8500294-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.389 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:09:44.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:09:44.391 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.391 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.391 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.391 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.391 DEBUG:boto:Method: GET 2024-04-13T22:09:44.391 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.391 DEBUG:boto:Data: 2024-04-13T22:09:44.391 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.391 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.391 DEBUG:boto:Params: {} 2024-04-13T22:09:44.392 DEBUG:boto:Token: None 2024-04-13T22:09:44.392 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/?versions 2024-04-13T22:09:44.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T22:09:44.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T22:09:44.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b02a4e7a926c06b-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.397 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.397 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:09:44.397 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:09:44.398 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:09:44.398 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:09:44.398 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:09:44.398 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:09:44.398 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:09:44.398 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:09:44.398 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.399 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.399 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.399 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.399 DEBUG:boto:Method: GET 2024-04-13T22:09:44.399 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.399 DEBUG:boto:Data: 2024-04-13T22:09:44.399 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.399 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.399 DEBUG:boto:Params: {} 2024-04-13T22:09:44.400 DEBUG:boto:Token: None 2024-04-13T22:09:44.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/?versions 2024-04-13T22:09:44.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T22:09:44.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T22:09:44.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009748040814e3a907-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.406 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.407 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.407 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.407 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.407 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.407 DEBUG:boto:Method: GET 2024-04-13T22:09:44.408 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:44.408 DEBUG:boto:Data: 2024-04-13T22:09:44.408 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.408 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.408 DEBUG:boto:Params: {} 2024-04-13T22:09:44.408 DEBUG:boto:Token: None 2024-04-13T22:09:44.408 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/?versions 2024-04-13T22:09:44.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T22:09:44.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T22:09:44.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e0ee2df8e70b65-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.411 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:44.412 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.412 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.412 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.412 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.412 DEBUG:boto:Method: GET 2024-04-13T22:09:44.412 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.413 DEBUG:boto:Data: 2024-04-13T22:09:44.413 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.413 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.413 DEBUG:boto:Params: {} 2024-04-13T22:09:44.413 DEBUG:boto:Token: None 2024-04-13T22:09:44.413 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T22:09:44.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T22:09:44.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd2785b0671236f-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.459 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.459 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.459 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.459 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.459 DEBUG:boto:Method: GET 2024-04-13T22:09:44.459 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.459 DEBUG:boto:Data: 2024-04-13T22:09:44.460 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.460 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.461 DEBUG:boto:Params: {} 2024-04-13T22:09:44.461 DEBUG:boto:Token: None 2024-04-13T22:09:44.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T22:09:44.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T22:09:44.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ce41814f24ed65-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.464 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.464 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.464 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.464 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.464 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.464 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.464 DEBUG:boto:Data: 2024-04-13T22:09:44.464 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.464 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.465 DEBUG:boto:Params: {} 2024-04-13T22:09:44.465 DEBUG:boto:Token: None 2024-04-13T22:09:44.465 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk= 2024-04-13T22:09:44.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk=', 'Content-Length': '0'} 2024-04-13T22:09:44.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a24ec7033e5a41-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.467 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.467 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.467 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.467 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.468 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.468 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.468 DEBUG:boto:Data: 2024-04-13T22:09:44.468 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.468 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.468 DEBUG:boto:Params: {} 2024-04-13T22:09:44.468 DEBUG:boto:Token: None 2024-04-13T22:09:44.468 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk= 2024-04-13T22:09:44.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk=', 'Content-Length': '0'} 2024-04-13T22:09:44.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a34d7daddbd3be6-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:44.471 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.472 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.472 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.472 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.472 DEBUG:boto:Method: GET 2024-04-13T22:09:44.472 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.472 DEBUG:boto:Data: 2024-04-13T22:09:44.472 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.472 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.472 DEBUG:boto:Params: {} 2024-04-13T22:09:44.472 DEBUG:boto:Token: None 2024-04-13T22:09:44.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T22:09:44.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T22:09:44.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075b14febc129642-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.475 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.475 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.475 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.475 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:44.475 DEBUG:boto:Method: GET 2024-04-13T22:09:44.475 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.475 DEBUG:boto:Data: 2024-04-13T22:09:44.475 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.476 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.476 DEBUG:boto:Params: {} 2024-04-13T22:09:44.476 DEBUG:boto:Token: None 2024-04-13T22:09:44.476 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:44.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T22:09:44.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T22:09:44.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc9dd875eb92e81-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.481 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.481 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.481 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.482 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.482 DEBUG:boto:Method: GET 2024-04-13T22:09:44.482 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:44.482 DEBUG:boto:Data: 2024-04-13T22:09:44.482 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.482 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.482 DEBUG:boto:Params: {} 2024-04-13T22:09:44.482 DEBUG:boto:Token: None 2024-04-13T22:09:44.483 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e10d966b020a864-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.526 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.527 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.527 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.527 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.527 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.527 DEBUG:boto:Method: GET 2024-04-13T22:09:44.527 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:44.527 DEBUG:boto:Data: 2024-04-13T22:09:44.527 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.528 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.528 DEBUG:boto:Params: {} 2024-04-13T22:09:44.528 DEBUG:boto:Token: None 2024-04-13T22:09:44.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da0875d75acbc09-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.530 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.531 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.531 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.531 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.531 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:44.531 DEBUG:boto:Data: 2024-04-13T22:09:44.531 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.531 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.531 DEBUG:boto:Params: {} 2024-04-13T22:09:44.531 DEBUG:boto:Token: None 2024-04-13T22:09:44.532 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a 2024-04-13T22:09:44.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8= 2024-04-13T22:09:44.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8=', 'Content-Length': '0'} 2024-04-13T22:09:44.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066de601c0777cdda-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.534 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.534 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.534 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.534 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:44.534 DEBUG:boto:Data: 2024-04-13T22:09:44.534 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.535 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.535 DEBUG:boto:Params: {} 2024-04-13T22:09:44.535 DEBUG:boto:Token: None 2024-04-13T22:09:44.535 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a 2024-04-13T22:09:44.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8= 2024-04-13T22:09:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8=', 'Content-Length': '0'} 2024-04-13T22:09:44.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000048c6f23ad779113-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:44.537 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.537 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.537 DEBUG:boto:Method: GET 2024-04-13T22:09:44.537 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:44.537 DEBUG:boto:Data: 2024-04-13T22:09:44.537 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.537 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.538 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.538 DEBUG:boto:Params: {} 2024-04-13T22:09:44.538 DEBUG:boto:Token: None 2024-04-13T22:09:44.538 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a 2024-04-13T22:09:44.538 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I= 2024-04-13T22:09:44.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I=', 'Content-Length': '0'} 2024-04-13T22:09:44.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eaeccc3d14fddc7-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.540 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:44.540 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:44.540 DEBUG:boto:Method: GET 2024-04-13T22:09:44.540 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:44.540 DEBUG:boto:Data: 2024-04-13T22:09:44.540 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.541 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.541 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.541 DEBUG:boto:Params: {} 2024-04-13T22:09:44.541 DEBUG:boto:Token: None 2024-04-13T22:09:44.541 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/a 2024-04-13T22:09:44.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I= 2024-04-13T22:09:44.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I=', 'Content-Length': '0'} 2024-04-13T22:09:44.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2c9f413065f18d-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.590 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.590 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.591 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.591 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.591 DEBUG:boto:Method: GET 2024-04-13T22:09:44.591 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:44.591 DEBUG:boto:Data: 2024-04-13T22:09:44.591 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.591 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.591 DEBUG:boto:Params: {} 2024-04-13T22:09:44.591 DEBUG:boto:Token: None 2024-04-13T22:09:44.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b874d5c89b2e2fd9-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.594 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.595 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.595 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.595 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.595 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:44.595 DEBUG:boto:Method: GET 2024-04-13T22:09:44.595 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:44.595 DEBUG:boto:Data: 2024-04-13T22:09:44.595 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.596 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.596 DEBUG:boto:Params: {} 2024-04-13T22:09:44.596 DEBUG:boto:Token: None 2024-04-13T22:09:44.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008649a75081eae40e-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.598 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:44.599 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.599 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.599 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.599 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.599 DEBUG:boto:Method: GET 2024-04-13T22:09:44.599 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.599 DEBUG:boto:Data: 2024-04-13T22:09:44.599 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.599 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.599 DEBUG:boto:Params: {} 2024-04-13T22:09:44.600 DEBUG:boto:Token: None 2024-04-13T22:09:44.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T22:09:44.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T22:09:44.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008166c99d80e4304d-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.602 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.602 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.602 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.602 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.603 DEBUG:boto:Method: GET 2024-04-13T22:09:44.603 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.603 DEBUG:boto:Data: 2024-04-13T22:09:44.603 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.603 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.603 DEBUG:boto:Params: {} 2024-04-13T22:09:44.603 DEBUG:boto:Token: None 2024-04-13T22:09:44.603 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T22:09:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T22:09:44.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb9f03cd1af0ad3-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.606 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.606 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.606 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.606 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.606 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.606 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.606 DEBUG:boto:Data: 2024-04-13T22:09:44.606 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.607 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.607 DEBUG:boto:Params: {} 2024-04-13T22:09:44.607 DEBUG:boto:Token: None 2024-04-13T22:09:44.607 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8= 2024-04-13T22:09:44.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8=', 'Content-Length': '0'} 2024-04-13T22:09:44.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef510ee3a841391-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.609 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.610 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.610 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.610 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.610 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.610 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.610 DEBUG:boto:Data: 2024-04-13T22:09:44.610 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.610 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.610 DEBUG:boto:Params: {} 2024-04-13T22:09:44.611 DEBUG:boto:Token: None 2024-04-13T22:09:44.611 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8= 2024-04-13T22:09:44.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8=', 'Content-Length': '0'} 2024-04-13T22:09:44.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003409b020ffee8536-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:44.613 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.613 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.613 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.614 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.614 DEBUG:boto:Method: GET 2024-04-13T22:09:44.614 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.614 DEBUG:boto:Data: 2024-04-13T22:09:44.614 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.614 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.614 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.614 DEBUG:boto:Params: {} 2024-04-13T22:09:44.614 DEBUG:boto:Token: None 2024-04-13T22:09:44.614 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T22:09:44.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T22:09:44.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f7ff236b4140bf-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.617 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.617 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.617 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.617 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:44.617 DEBUG:boto:Method: GET 2024-04-13T22:09:44.617 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.617 DEBUG:boto:Data: 2024-04-13T22:09:44.617 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.617 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.617 DEBUG:boto:Params: {} 2024-04-13T22:09:44.618 DEBUG:boto:Token: None 2024-04-13T22:09:44.618 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:44.618 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T22:09:44.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T22:09:44.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70e9dcea4fdf7eb-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.620 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.620 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.620 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.621 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.621 DEBUG:boto:Method: GET 2024-04-13T22:09:44.621 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:44.621 DEBUG:boto:Data: 2024-04-13T22:09:44.621 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.621 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.621 DEBUG:boto:Params: {} 2024-04-13T22:09:44.621 DEBUG:boto:Token: None 2024-04-13T22:09:44.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f9ed0d9e8dbcc6-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.624 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.624 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.624 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.625 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.625 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.625 DEBUG:boto:Method: GET 2024-04-13T22:09:44.625 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:44.625 DEBUG:boto:Data: 2024-04-13T22:09:44.625 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.625 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.625 DEBUG:boto:Params: {} 2024-04-13T22:09:44.625 DEBUG:boto:Token: None 2024-04-13T22:09:44.626 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.626 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f8567a6209aee15-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.628 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.628 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.628 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.628 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.629 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:44.629 DEBUG:boto:Data: 2024-04-13T22:09:44.629 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.629 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.629 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.629 DEBUG:boto:Params: {} 2024-04-13T22:09:44.629 DEBUG:boto:Token: None 2024-04-13T22:09:44.629 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b 2024-04-13T22:09:44.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE= 2024-04-13T22:09:44.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE=', 'Content-Length': '0'} 2024-04-13T22:09:44.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06dc4fb2b8a23b6-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.631 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.631 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.632 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.632 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:44.632 DEBUG:boto:Data: 2024-04-13T22:09:44.632 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.632 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.632 DEBUG:boto:Params: {} 2024-04-13T22:09:44.632 DEBUG:boto:Token: None 2024-04-13T22:09:44.632 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b 2024-04-13T22:09:44.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE= 2024-04-13T22:09:44.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE=', 'Content-Length': '0'} 2024-04-13T22:09:44.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db504cbb7c92daf-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:44.634 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.635 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.635 DEBUG:boto:Method: GET 2024-04-13T22:09:44.635 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:44.635 DEBUG:boto:Data: 2024-04-13T22:09:44.635 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.635 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.635 DEBUG:boto:Params: {} 2024-04-13T22:09:44.635 DEBUG:boto:Token: None 2024-04-13T22:09:44.635 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b 2024-04-13T22:09:44.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0= 2024-04-13T22:09:44.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0=', 'Content-Length': '0'} 2024-04-13T22:09:44.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009080dc5b3604a311-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.638 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:44.638 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:44.638 DEBUG:boto:Method: GET 2024-04-13T22:09:44.638 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:44.638 DEBUG:boto:Data: 2024-04-13T22:09:44.638 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.638 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.638 DEBUG:boto:Params: {} 2024-04-13T22:09:44.639 DEBUG:boto:Token: None 2024-04-13T22:09:44.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/b 2024-04-13T22:09:44.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0= 2024-04-13T22:09:44.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0=', 'Content-Length': '0'} 2024-04-13T22:09:44.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e5bbbdad7debd7-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.641 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.641 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.641 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.641 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.641 DEBUG:boto:Method: GET 2024-04-13T22:09:44.641 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:44.641 DEBUG:boto:Data: 2024-04-13T22:09:44.642 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.642 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.642 DEBUG:boto:Params: {} 2024-04-13T22:09:44.642 DEBUG:boto:Token: None 2024-04-13T22:09:44.642 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbafd9673d179c20-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.645 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.645 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.645 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.645 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.645 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:44.645 DEBUG:boto:Method: GET 2024-04-13T22:09:44.646 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:44.646 DEBUG:boto:Data: 2024-04-13T22:09:44.646 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.646 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.646 DEBUG:boto:Params: {} 2024-04-13T22:09:44.646 DEBUG:boto:Token: None 2024-04-13T22:09:44.646 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.646 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14e4dbf926edf09-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.648 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:44.649 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.649 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.649 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.649 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.649 DEBUG:boto:Method: GET 2024-04-13T22:09:44.649 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.649 DEBUG:boto:Data: 2024-04-13T22:09:44.649 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.649 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.650 DEBUG:boto:Params: {} 2024-04-13T22:09:44.650 DEBUG:boto:Token: None 2024-04-13T22:09:44.650 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.650 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T22:09:44.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T22:09:44.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e73044c8873baad-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.652 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.652 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.652 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.652 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.652 DEBUG:boto:Method: GET 2024-04-13T22:09:44.652 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.652 DEBUG:boto:Data: 2024-04-13T22:09:44.652 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.653 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.653 DEBUG:boto:Params: {} 2024-04-13T22:09:44.653 DEBUG:boto:Token: None 2024-04-13T22:09:44.653 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T22:09:44.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T22:09:44.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e92524ad668ddbf-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.655 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.655 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.655 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.655 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.655 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.655 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.655 DEBUG:boto:Data: 2024-04-13T22:09:44.655 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.656 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.656 DEBUG:boto:Params: {} 2024-04-13T22:09:44.656 DEBUG:boto:Token: None 2024-04-13T22:09:44.656 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg= 2024-04-13T22:09:44.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg=', 'Content-Length': '0'} 2024-04-13T22:09:44.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396162906b972d6c-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.658 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.658 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.658 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.658 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.659 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.659 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.659 DEBUG:boto:Data: 2024-04-13T22:09:44.659 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.659 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.659 DEBUG:boto:Params: {} 2024-04-13T22:09:44.659 DEBUG:boto:Token: None 2024-04-13T22:09:44.659 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg= 2024-04-13T22:09:44.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg=', 'Content-Length': '0'} 2024-04-13T22:09:44.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d6fb3306b26f17-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:44.661 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.662 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.662 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.662 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.662 DEBUG:boto:Method: GET 2024-04-13T22:09:44.662 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.662 DEBUG:boto:Data: 2024-04-13T22:09:44.662 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.662 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.662 DEBUG:boto:Params: {} 2024-04-13T22:09:44.663 DEBUG:boto:Token: None 2024-04-13T22:09:44.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T22:09:44.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T22:09:44.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f14101ac01dd45-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.665 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.665 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.665 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.665 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:44.665 DEBUG:boto:Method: GET 2024-04-13T22:09:44.665 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.665 DEBUG:boto:Data: 2024-04-13T22:09:44.665 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.665 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.666 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.666 DEBUG:boto:Params: {} 2024-04-13T22:09:44.666 DEBUG:boto:Token: None 2024-04-13T22:09:44.666 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:44.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T22:09:44.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T22:09:44.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc07adc8e8569af-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.668 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.668 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.668 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.668 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.668 DEBUG:boto:Method: GET 2024-04-13T22:09:44.668 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:44.669 DEBUG:boto:Data: 2024-04-13T22:09:44.669 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.669 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.669 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.669 DEBUG:boto:Params: {} 2024-04-13T22:09:44.669 DEBUG:boto:Token: None 2024-04-13T22:09:44.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c82fb13645945af-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.671 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.672 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.672 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.672 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.672 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.672 DEBUG:boto:Method: GET 2024-04-13T22:09:44.672 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:44.672 DEBUG:boto:Data: 2024-04-13T22:09:44.672 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.672 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.673 DEBUG:boto:Params: {} 2024-04-13T22:09:44.673 DEBUG:boto:Token: None 2024-04-13T22:09:44.673 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.673 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e936ebd884a0068-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.675 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.675 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.675 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.675 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.675 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:44.676 DEBUG:boto:Data: 2024-04-13T22:09:44.676 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.676 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.676 DEBUG:boto:Params: {} 2024-04-13T22:09:44.676 DEBUG:boto:Token: None 2024-04-13T22:09:44.676 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c 2024-04-13T22:09:44.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg= 2024-04-13T22:09:44.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg=', 'Content-Length': '0'} 2024-04-13T22:09:44.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2a230fa81f50beb-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.678 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.678 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.679 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.679 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:44.679 DEBUG:boto:Data: 2024-04-13T22:09:44.679 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.679 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.679 DEBUG:boto:Params: {} 2024-04-13T22:09:44.679 DEBUG:boto:Token: None 2024-04-13T22:09:44.679 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c 2024-04-13T22:09:44.679 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg= 2024-04-13T22:09:44.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg=', 'Content-Length': '0'} 2024-04-13T22:09:44.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d43c104215b7878-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:44.682 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.682 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.682 DEBUG:boto:Method: GET 2024-04-13T22:09:44.682 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:44.682 DEBUG:boto:Data: 2024-04-13T22:09:44.682 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.682 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.682 DEBUG:boto:Params: {} 2024-04-13T22:09:44.682 DEBUG:boto:Token: None 2024-04-13T22:09:44.683 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c 2024-04-13T22:09:44.683 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs= 2024-04-13T22:09:44.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs=', 'Content-Length': '0'} 2024-04-13T22:09:44.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446a80ec46c354fd-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.726 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:44.726 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:44.727 DEBUG:boto:Method: GET 2024-04-13T22:09:44.727 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:44.727 DEBUG:boto:Data: 2024-04-13T22:09:44.727 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.727 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.727 DEBUG:boto:Params: {} 2024-04-13T22:09:44.727 DEBUG:boto:Token: None 2024-04-13T22:09:44.727 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/c 2024-04-13T22:09:44.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs= 2024-04-13T22:09:44.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs=', 'Content-Length': '0'} 2024-04-13T22:09:44.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999439e6be7c5ad9-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.729 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.729 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.730 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.730 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.730 DEBUG:boto:Method: GET 2024-04-13T22:09:44.730 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:44.730 DEBUG:boto:Data: 2024-04-13T22:09:44.730 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.730 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.730 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.730 DEBUG:boto:Params: {} 2024-04-13T22:09:44.730 DEBUG:boto:Token: None 2024-04-13T22:09:44.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.731 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aad8350605c808f-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.733 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.733 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.733 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.733 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.733 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:44.734 DEBUG:boto:Method: GET 2024-04-13T22:09:44.734 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:44.734 DEBUG:boto:Data: 2024-04-13T22:09:44.734 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.734 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.734 DEBUG:boto:Params: {} 2024-04-13T22:09:44.734 DEBUG:boto:Token: None 2024-04-13T22:09:44.734 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf6e8916661cf58d-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.736 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:44.737 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.737 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.737 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.737 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.737 DEBUG:boto:Method: GET 2024-04-13T22:09:44.737 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.737 DEBUG:boto:Data: 2024-04-13T22:09:44.737 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.737 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.738 DEBUG:boto:Params: {} 2024-04-13T22:09:44.738 DEBUG:boto:Token: None 2024-04-13T22:09:44.738 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.738 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T22:09:44.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T22:09:44.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ffe126ec2e0838d-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.740 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.740 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.740 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.740 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.740 DEBUG:boto:Method: GET 2024-04-13T22:09:44.741 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.741 DEBUG:boto:Data: 2024-04-13T22:09:44.741 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.741 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.741 DEBUG:boto:Params: {} 2024-04-13T22:09:44.741 DEBUG:boto:Token: None 2024-04-13T22:09:44.741 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T22:09:44.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T22:09:44.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001325b5d654f3a8cb-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.743 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.743 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.744 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.744 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.744 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.744 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.744 DEBUG:boto:Data: 2024-04-13T22:09:44.744 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.744 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.744 DEBUG:boto:Params: {} 2024-04-13T22:09:44.744 DEBUG:boto:Token: None 2024-04-13T22:09:44.745 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.745 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY= 2024-04-13T22:09:44.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY=', 'Content-Length': '0'} 2024-04-13T22:09:44.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b93b9009089ab3-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.746 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.747 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.747 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.747 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.747 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.747 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.747 DEBUG:boto:Data: 2024-04-13T22:09:44.747 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.747 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.747 DEBUG:boto:Params: {} 2024-04-13T22:09:44.747 DEBUG:boto:Token: None 2024-04-13T22:09:44.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY= 2024-04-13T22:09:44.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY=', 'Content-Length': '0'} 2024-04-13T22:09:44.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233f33e507d3d6a5-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:44.750 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.750 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.750 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.750 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.750 DEBUG:boto:Method: GET 2024-04-13T22:09:44.750 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.750 DEBUG:boto:Data: 2024-04-13T22:09:44.750 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.751 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.751 DEBUG:boto:Params: {} 2024-04-13T22:09:44.751 DEBUG:boto:Token: None 2024-04-13T22:09:44.751 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T22:09:44.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T22:09:44.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098739a39b958aaaf-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.753 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.753 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.753 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.753 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:44.753 DEBUG:boto:Method: GET 2024-04-13T22:09:44.754 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.754 DEBUG:boto:Data: 2024-04-13T22:09:44.754 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.754 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.754 DEBUG:boto:Params: {} 2024-04-13T22:09:44.754 DEBUG:boto:Token: None 2024-04-13T22:09:44.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:44.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T22:09:44.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T22:09:44.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5cd6604a8b723d-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.756 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.756 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.757 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.757 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.757 DEBUG:boto:Method: GET 2024-04-13T22:09:44.757 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:44.757 DEBUG:boto:Data: 2024-04-13T22:09:44.757 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.757 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.757 DEBUG:boto:Params: {} 2024-04-13T22:09:44.757 DEBUG:boto:Token: None 2024-04-13T22:09:44.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c517869882b04c3c-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.760 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.760 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.761 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.761 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.761 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.761 DEBUG:boto:Method: GET 2024-04-13T22:09:44.761 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:44.761 DEBUG:boto:Data: 2024-04-13T22:09:44.761 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.761 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.761 DEBUG:boto:Params: {} 2024-04-13T22:09:44.761 DEBUG:boto:Token: None 2024-04-13T22:09:44.762 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.762 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ff5bc4b5c7c009-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.764 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.764 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.764 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.765 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.765 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:44.765 DEBUG:boto:Data: 2024-04-13T22:09:44.765 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.765 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.765 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.765 DEBUG:boto:Params: {} 2024-04-13T22:09:44.765 DEBUG:boto:Token: None 2024-04-13T22:09:44.765 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d 2024-04-13T22:09:44.766 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0= 2024-04-13T22:09:44.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0=', 'Content-Length': '0'} 2024-04-13T22:09:44.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de995c8f52b5a4cc-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.767 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.767 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.768 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.768 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:44.768 DEBUG:boto:Data: 2024-04-13T22:09:44.768 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.768 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.768 DEBUG:boto:Params: {} 2024-04-13T22:09:44.768 DEBUG:boto:Token: None 2024-04-13T22:09:44.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d 2024-04-13T22:09:44.769 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0= 2024-04-13T22:09:44.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0=', 'Content-Length': '0'} 2024-04-13T22:09:44.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897157f2bac6dc1e-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:44.770 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.771 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.771 DEBUG:boto:Method: GET 2024-04-13T22:09:44.771 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:44.771 DEBUG:boto:Data: 2024-04-13T22:09:44.771 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.771 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.772 DEBUG:boto:Params: {} 2024-04-13T22:09:44.772 DEBUG:boto:Token: None 2024-04-13T22:09:44.772 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d 2024-04-13T22:09:44.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ= 2024-04-13T22:09:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ=', 'Content-Length': '0'} 2024-04-13T22:09:44.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4531d4d08079500-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.774 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:44.774 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:44.774 DEBUG:boto:Method: GET 2024-04-13T22:09:44.774 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:44.775 DEBUG:boto:Data: 2024-04-13T22:09:44.775 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.775 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.775 DEBUG:boto:Params: {} 2024-04-13T22:09:44.775 DEBUG:boto:Token: None 2024-04-13T22:09:44.775 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/d 2024-04-13T22:09:44.775 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ= 2024-04-13T22:09:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ=', 'Content-Length': '0'} 2024-04-13T22:09:44.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf19b52b852705fe-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.777 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.777 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.777 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.778 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.778 DEBUG:boto:Method: GET 2024-04-13T22:09:44.778 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:44.778 DEBUG:boto:Data: 2024-04-13T22:09:44.778 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.778 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.778 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.778 DEBUG:boto:Params: {} 2024-04-13T22:09:44.778 DEBUG:boto:Token: None 2024-04-13T22:09:44.779 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000032385b495b0488b-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.781 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.781 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.782 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.782 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.782 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:44.782 DEBUG:boto:Method: GET 2024-04-13T22:09:44.782 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:44.782 DEBUG:boto:Data: 2024-04-13T22:09:44.782 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.782 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.783 DEBUG:boto:Params: {} 2024-04-13T22:09:44.783 DEBUG:boto:Token: None 2024-04-13T22:09:44.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed952c1a02b935f-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.785 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:09:44.786 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.786 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.786 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.786 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.786 DEBUG:boto:Method: GET 2024-04-13T22:09:44.786 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.786 DEBUG:boto:Data: 2024-04-13T22:09:44.786 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.786 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.787 DEBUG:boto:Params: {} 2024-04-13T22:09:44.787 DEBUG:boto:Token: None 2024-04-13T22:09:44.787 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.787 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T22:09:44.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T22:09:44.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e781cfa458df768-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.789 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.789 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.789 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.790 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.790 DEBUG:boto:Method: GET 2024-04-13T22:09:44.790 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.790 DEBUG:boto:Data: 2024-04-13T22:09:44.790 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.790 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.790 DEBUG:boto:Params: {} 2024-04-13T22:09:44.790 DEBUG:boto:Token: None 2024-04-13T22:09:44.791 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T22:09:44.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T22:09:44.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4b679857cf5c4e-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.793 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.793 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.793 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.793 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.793 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.794 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.794 DEBUG:boto:Data: 2024-04-13T22:09:44.794 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.794 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.794 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.794 DEBUG:boto:Params: {} 2024-04-13T22:09:44.794 DEBUG:boto:Token: None 2024-04-13T22:09:44.794 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.795 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w= 2024-04-13T22:09:44.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w=', 'Content-Length': '0'} 2024-04-13T22:09:44.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5dec2760fe95fd9-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.796 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.796 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.797 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.797 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.797 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.797 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.797 DEBUG:boto:Data: 2024-04-13T22:09:44.797 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.797 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.797 DEBUG:boto:Params: {} 2024-04-13T22:09:44.797 DEBUG:boto:Token: None 2024-04-13T22:09:44.798 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w= 2024-04-13T22:09:44.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w=', 'Content-Length': '0'} 2024-04-13T22:09:44.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c667af3068a319c-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:09:44.800 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.800 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.800 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.800 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.800 DEBUG:boto:Method: GET 2024-04-13T22:09:44.800 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.800 DEBUG:boto:Data: 2024-04-13T22:09:44.801 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.801 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.801 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.801 DEBUG:boto:Params: {} 2024-04-13T22:09:44.801 DEBUG:boto:Token: None 2024-04-13T22:09:44.801 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.801 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T22:09:44.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T22:09:44.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b027c2b74448134-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.803 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.803 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.803 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.803 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:44.804 DEBUG:boto:Method: GET 2024-04-13T22:09:44.804 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.804 DEBUG:boto:Data: 2024-04-13T22:09:44.804 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.804 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.804 DEBUG:boto:Params: {} 2024-04-13T22:09:44.804 DEBUG:boto:Token: None 2024-04-13T22:09:44.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:44.805 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T22:09:44.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T22:09:44.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000875d8ad47f173551-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.806 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.807 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.807 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.807 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.807 DEBUG:boto:Method: GET 2024-04-13T22:09:44.807 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:44.807 DEBUG:boto:Data: 2024-04-13T22:09:44.807 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.807 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.807 DEBUG:boto:Params: {} 2024-04-13T22:09:44.808 DEBUG:boto:Token: None 2024-04-13T22:09:44.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d952ece3c1349550-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.850 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.851 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.851 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.851 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.851 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.851 DEBUG:boto:Method: GET 2024-04-13T22:09:44.851 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:44.851 DEBUG:boto:Data: 2024-04-13T22:09:44.852 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.852 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.852 DEBUG:boto:Params: {} 2024-04-13T22:09:44.852 DEBUG:boto:Token: None 2024-04-13T22:09:44.852 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083163c6d455b9082-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.854 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.855 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.855 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.855 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.855 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:44.855 DEBUG:boto:Data: 2024-04-13T22:09:44.855 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.855 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.855 DEBUG:boto:Params: {} 2024-04-13T22:09:44.856 DEBUG:boto:Token: None 2024-04-13T22:09:44.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e 2024-04-13T22:09:44.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA= 2024-04-13T22:09:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA=', 'Content-Length': '0'} 2024-04-13T22:09:44.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf014a041f7ee81-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.858 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.858 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.858 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.858 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:44.858 DEBUG:boto:Data: 2024-04-13T22:09:44.858 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.858 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.858 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.858 DEBUG:boto:Params: {} 2024-04-13T22:09:44.858 DEBUG:boto:Token: None 2024-04-13T22:09:44.859 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e 2024-04-13T22:09:44.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA= 2024-04-13T22:09:44.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA=', 'Content-Length': '0'} 2024-04-13T22:09:44.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd2e7245f3f42d0-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:09:44.861 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.861 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.861 DEBUG:boto:Method: GET 2024-04-13T22:09:44.861 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:44.861 DEBUG:boto:Data: 2024-04-13T22:09:44.861 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.861 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.861 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.861 DEBUG:boto:Params: {} 2024-04-13T22:09:44.861 DEBUG:boto:Token: None 2024-04-13T22:09:44.862 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e 2024-04-13T22:09:44.862 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q= 2024-04-13T22:09:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q=', 'Content-Length': '0'} 2024-04-13T22:09:44.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af7800c52127af5-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.864 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:44.864 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:44.864 DEBUG:boto:Method: GET 2024-04-13T22:09:44.864 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:44.864 DEBUG:boto:Data: 2024-04-13T22:09:44.864 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.864 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.864 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.864 DEBUG:boto:Params: {} 2024-04-13T22:09:44.865 DEBUG:boto:Token: None 2024-04-13T22:09:44.865 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/e 2024-04-13T22:09:44.865 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q= 2024-04-13T22:09:44.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q=', 'Content-Length': '0'} 2024-04-13T22:09:44.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53d327d348a9e77-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.867 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.867 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.867 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.867 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.867 DEBUG:boto:Method: GET 2024-04-13T22:09:44.867 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:44.867 DEBUG:boto:Data: 2024-04-13T22:09:44.867 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.868 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.868 DEBUG:boto:Params: {} 2024-04-13T22:09:44.868 DEBUG:boto:Token: None 2024-04-13T22:09:44.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2344ae7a597f2d-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.870 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.871 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.871 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.871 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.871 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:44.871 DEBUG:boto:Method: GET 2024-04-13T22:09:44.871 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:44.871 DEBUG:boto:Data: 2024-04-13T22:09:44.871 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.871 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.872 DEBUG:boto:Params: {} 2024-04-13T22:09:44.872 DEBUG:boto:Token: None 2024-04-13T22:09:44.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077cde880ff834a4-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.874 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:09:44.874 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.875 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.875 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.875 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.875 DEBUG:boto:Method: GET 2024-04-13T22:09:44.875 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.875 DEBUG:boto:Data: 2024-04-13T22:09:44.875 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.875 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.875 DEBUG:boto:Params: {} 2024-04-13T22:09:44.876 DEBUG:boto:Token: None 2024-04-13T22:09:44.876 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T22:09:44.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T22:09:44.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15b7bf9f4e53551-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.878 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.878 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.878 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.878 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.878 DEBUG:boto:Method: GET 2024-04-13T22:09:44.878 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.879 DEBUG:boto:Data: 2024-04-13T22:09:44.879 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.879 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.879 DEBUG:boto:Params: {} 2024-04-13T22:09:44.879 DEBUG:boto:Token: None 2024-04-13T22:09:44.879 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.879 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T22:09:44.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T22:09:44.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000641402082ad45d35-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.881 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.881 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.882 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.882 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.882 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.882 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.882 DEBUG:boto:Data: 2024-04-13T22:09:44.882 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.882 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.882 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.882 DEBUG:boto:Params: {} 2024-04-13T22:09:44.882 DEBUG:boto:Token: None 2024-04-13T22:09:44.883 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8= 2024-04-13T22:09:44.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8=', 'Content-Length': '0'} 2024-04-13T22:09:44.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478f91c3a7187d02-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.885 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.885 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.885 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.885 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.885 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.885 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.886 DEBUG:boto:Data: 2024-04-13T22:09:44.886 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.886 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.886 DEBUG:boto:Params: {} 2024-04-13T22:09:44.886 DEBUG:boto:Token: None 2024-04-13T22:09:44.886 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.886 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8= 2024-04-13T22:09:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8=', 'Content-Length': '0'} 2024-04-13T22:09:44.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1c73d81b84b64c-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:09:44.888 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.888 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.889 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.889 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.889 DEBUG:boto:Method: GET 2024-04-13T22:09:44.889 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.889 DEBUG:boto:Data: 2024-04-13T22:09:44.889 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.889 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.889 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.889 DEBUG:boto:Params: {} 2024-04-13T22:09:44.889 DEBUG:boto:Token: None 2024-04-13T22:09:44.889 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.890 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T22:09:44.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T22:09:44.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da45824da9c904ed-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.892 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.892 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.892 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.892 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:44.892 DEBUG:boto:Method: GET 2024-04-13T22:09:44.892 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.892 DEBUG:boto:Data: 2024-04-13T22:09:44.892 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.892 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.892 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.892 DEBUG:boto:Params: {} 2024-04-13T22:09:44.893 DEBUG:boto:Token: None 2024-04-13T22:09:44.893 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:44.893 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T22:09:44.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T22:09:44.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e510c661cc82501-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.895 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.895 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.895 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.895 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.895 DEBUG:boto:Method: GET 2024-04-13T22:09:44.895 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:44.896 DEBUG:boto:Data: 2024-04-13T22:09:44.896 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.896 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.896 DEBUG:boto:Params: {} 2024-04-13T22:09:44.896 DEBUG:boto:Token: None 2024-04-13T22:09:44.896 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b5aed782fce0e83-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.898 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.899 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.899 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.899 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.899 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.899 DEBUG:boto:Method: GET 2024-04-13T22:09:44.899 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:44.899 DEBUG:boto:Data: 2024-04-13T22:09:44.900 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.900 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.900 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.900 DEBUG:boto:Params: {} 2024-04-13T22:09:44.900 DEBUG:boto:Token: None 2024-04-13T22:09:44.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193c3d3f1ec18e64-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.942 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.943 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.943 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.943 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.943 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:44.943 DEBUG:boto:Data: 2024-04-13T22:09:44.943 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.943 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.944 DEBUG:boto:Params: {} 2024-04-13T22:09:44.944 DEBUG:boto:Token: None 2024-04-13T22:09:44.944 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f 2024-04-13T22:09:44.944 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE= 2024-04-13T22:09:44.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE=', 'Content-Length': '0'} 2024-04-13T22:09:44.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b656c6091cb0cb-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.946 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.946 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.946 DEBUG:boto:Method: HEAD 2024-04-13T22:09:44.946 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:44.946 DEBUG:boto:Data: 2024-04-13T22:09:44.947 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.947 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.947 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.947 DEBUG:boto:Params: {} 2024-04-13T22:09:44.947 DEBUG:boto:Token: None 2024-04-13T22:09:44.947 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f 2024-04-13T22:09:44.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE= 2024-04-13T22:09:44.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE=', 'Content-Length': '0'} 2024-04-13T22:09:44.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c2690f7ff558045-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:09:44.949 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.949 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.949 DEBUG:boto:Method: GET 2024-04-13T22:09:44.949 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:44.949 DEBUG:boto:Data: 2024-04-13T22:09:44.950 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.950 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.950 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.950 DEBUG:boto:Params: {} 2024-04-13T22:09:44.950 DEBUG:boto:Token: None 2024-04-13T22:09:44.950 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f 2024-04-13T22:09:44.950 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM= 2024-04-13T22:09:44.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM=', 'Content-Length': '0'} 2024-04-13T22:09:44.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031157ea5b00601fb-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.952 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:44.952 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:44.953 DEBUG:boto:Method: GET 2024-04-13T22:09:44.953 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:44.953 DEBUG:boto:Data: 2024-04-13T22:09:44.953 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.953 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.953 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.953 DEBUG:boto:Params: {} 2024-04-13T22:09:44.953 DEBUG:boto:Token: None 2024-04-13T22:09:44.953 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/f 2024-04-13T22:09:44.954 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM= 2024-04-13T22:09:44.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM=', 'Content-Length': '0'} 2024-04-13T22:09:44.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7593fd4eb79dff3-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.956 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.956 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.956 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.956 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.956 DEBUG:boto:Method: GET 2024-04-13T22:09:44.956 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:44.956 DEBUG:boto:Data: 2024-04-13T22:09:44.956 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.956 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.956 DEBUG:boto:Params: {} 2024-04-13T22:09:44.957 DEBUG:boto:Token: None 2024-04-13T22:09:44.957 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.957 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b624457cb60dd527-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.959 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.959 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:44.959 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:44.960 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.960 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:44.960 DEBUG:boto:Method: GET 2024-04-13T22:09:44.960 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:44.960 DEBUG:boto:Data: 2024-04-13T22:09:44.960 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.960 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:44.960 DEBUG:boto:Port: 8000 2024-04-13T22:09:44.960 DEBUG:boto:Params: {} 2024-04-13T22:09:44.960 DEBUG:boto:Token: None 2024-04-13T22:09:44.961 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/ 2024-04-13T22:09:44.961 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T22:09:44.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T22:09:44.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4ca600f11e553e-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:44.963 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:44.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:09:44.963 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:44.963 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:44.963 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:44.963 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:44.964 DEBUG:boto:Method: GET 2024-04-13T22:09:44.964 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:44.964 DEBUG:boto:Data: 2024-04-13T22:09:44.964 DEBUG:boto:Headers: {} 2024-04-13T22:09:44.964 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:44.964 DEBUG:boto:Port: 8001 2024-04-13T22:09:44.964 DEBUG:boto:Params: {} 2024-04-13T22:09:44.964 DEBUG:boto:Token: None 2024-04-13T22:09:44.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:44 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:44.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uzsHpYEprVPgB6gSO++E8AisZic= 2024-04-13T22:09:44.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uzsHpYEprVPgB6gSO++E8AisZic=', 'Content-Length': '0'} 2024-04-13T22:09:44.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078d9e3eeeea0f44a-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.010 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.011 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.011 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.011 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.011 DEBUG:boto:Method: GET 2024-04-13T22:09:45.011 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.011 DEBUG:boto:Data: 2024-04-13T22:09:45.011 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.011 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.011 DEBUG:boto:Params: {} 2024-04-13T22:09:45.012 DEBUG:boto:Token: None 2024-04-13T22:09:45.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys= 2024-04-13T22:09:45.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys=', 'Content-Length': '0'} 2024-04-13T22:09:45.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce1eff5971aaecc-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.014 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.015 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.015 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.015 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.015 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.015 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.015 DEBUG:boto:Data: 2024-04-13T22:09:45.015 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.015 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.015 DEBUG:boto:Params: {} 2024-04-13T22:09:45.015 DEBUG:boto:Token: None 2024-04-13T22:09:45.016 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI= 2024-04-13T22:09:45.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI=', 'Content-Length': '0'} 2024-04-13T22:09:45.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b30aa6e6348c0b-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.017 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.018 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.018 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.018 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.018 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.018 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.018 DEBUG:boto:Data: 2024-04-13T22:09:45.018 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.018 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.018 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.018 DEBUG:boto:Params: {} 2024-04-13T22:09:45.019 DEBUG:boto:Token: None 2024-04-13T22:09:45.019 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.019 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI= 2024-04-13T22:09:45.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI=', 'Content-Length': '0'} 2024-04-13T22:09:45.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f570f531c415e4-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:09:45.021 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.021 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.021 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.021 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.021 DEBUG:boto:Method: GET 2024-04-13T22:09:45.021 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.021 DEBUG:boto:Data: 2024-04-13T22:09:45.021 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.021 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.022 DEBUG:boto:Params: {} 2024-04-13T22:09:45.022 DEBUG:boto:Token: None 2024-04-13T22:09:45.022 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.022 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys= 2024-04-13T22:09:45.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys=', 'Content-Length': '0'} 2024-04-13T22:09:45.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe892cfec5b155ef-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.024 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.024 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.024 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.024 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:45.024 DEBUG:boto:Method: GET 2024-04-13T22:09:45.024 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.025 DEBUG:boto:Data: 2024-04-13T22:09:45.025 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.025 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.025 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.025 DEBUG:boto:Params: {} 2024-04-13T22:09:45.025 DEBUG:boto:Token: None 2024-04-13T22:09:45.025 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:45.025 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys= 2024-04-13T22:09:45.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys=', 'Content-Length': '0'} 2024-04-13T22:09:45.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2d15076acca3ec-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.027 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.027 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.027 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.027 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.028 DEBUG:boto:Method: GET 2024-04-13T22:09:45.028 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:45.028 DEBUG:boto:Data: 2024-04-13T22:09:45.028 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.028 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.028 DEBUG:boto:Params: {} 2024-04-13T22:09:45.028 DEBUG:boto:Token: None 2024-04-13T22:09:45.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eef93cc01e2596c2-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.031 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.031 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.031 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.031 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.031 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.032 DEBUG:boto:Method: GET 2024-04-13T22:09:45.032 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:45.032 DEBUG:boto:Data: 2024-04-13T22:09:45.032 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.032 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.032 DEBUG:boto:Params: {} 2024-04-13T22:09:45.032 DEBUG:boto:Token: None 2024-04-13T22:09:45.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005279c85fa8be254c-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.074 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.075 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.075 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.075 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.075 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:45.075 DEBUG:boto:Data: 2024-04-13T22:09:45.075 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.075 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.075 DEBUG:boto:Params: {} 2024-04-13T22:09:45.075 DEBUG:boto:Token: None 2024-04-13T22:09:45.076 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g 2024-04-13T22:09:45.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8= 2024-04-13T22:09:45.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8=', 'Content-Length': '0'} 2024-04-13T22:09:45.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091790a52fd489ab-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.077 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.078 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.078 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.078 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:45.078 DEBUG:boto:Data: 2024-04-13T22:09:45.078 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.078 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.078 DEBUG:boto:Params: {} 2024-04-13T22:09:45.078 DEBUG:boto:Token: None 2024-04-13T22:09:45.078 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g 2024-04-13T22:09:45.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8= 2024-04-13T22:09:45.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8=', 'Content-Length': '0'} 2024-04-13T22:09:45.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4437c0e840a3f4-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:09:45.080 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.081 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.081 DEBUG:boto:Method: GET 2024-04-13T22:09:45.081 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:45.081 DEBUG:boto:Data: 2024-04-13T22:09:45.081 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.081 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.081 DEBUG:boto:Params: {} 2024-04-13T22:09:45.081 DEBUG:boto:Token: None 2024-04-13T22:09:45.082 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g 2024-04-13T22:09:45.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE= 2024-04-13T22:09:45.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE=', 'Content-Length': '0'} 2024-04-13T22:09:45.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012412aa29fa7ced8-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.083 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:45.084 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:45.084 DEBUG:boto:Method: GET 2024-04-13T22:09:45.084 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:45.084 DEBUG:boto:Data: 2024-04-13T22:09:45.084 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.084 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.084 DEBUG:boto:Params: {} 2024-04-13T22:09:45.084 DEBUG:boto:Token: None 2024-04-13T22:09:45.084 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/g 2024-04-13T22:09:45.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE= 2024-04-13T22:09:45.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE=', 'Content-Length': '0'} 2024-04-13T22:09:45.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020e29cbd77b6aaa8-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.086 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.086 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.087 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.087 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.087 DEBUG:boto:Method: GET 2024-04-13T22:09:45.087 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:45.087 DEBUG:boto:Data: 2024-04-13T22:09:45.087 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.087 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.087 DEBUG:boto:Params: {} 2024-04-13T22:09:45.087 DEBUG:boto:Token: None 2024-04-13T22:09:45.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2dda049bd2f998-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.090 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.090 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.090 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.091 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.091 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:45.091 DEBUG:boto:Method: GET 2024-04-13T22:09:45.091 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:45.091 DEBUG:boto:Data: 2024-04-13T22:09:45.091 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.091 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.091 DEBUG:boto:Params: {} 2024-04-13T22:09:45.091 DEBUG:boto:Token: None 2024-04-13T22:09:45.092 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9879d59c6bbf9a2-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.094 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:09:45.094 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.094 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.095 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.095 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.095 DEBUG:boto:Method: GET 2024-04-13T22:09:45.095 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.095 DEBUG:boto:Data: 2024-04-13T22:09:45.095 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.095 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.095 DEBUG:boto:Params: {} 2024-04-13T22:09:45.095 DEBUG:boto:Token: None 2024-04-13T22:09:45.096 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T22:09:45.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T22:09:45.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55d5abf126eaf05-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.098 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.098 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.098 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.098 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.098 DEBUG:boto:Method: GET 2024-04-13T22:09:45.098 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.098 DEBUG:boto:Data: 2024-04-13T22:09:45.098 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.099 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.099 DEBUG:boto:Params: {} 2024-04-13T22:09:45.099 DEBUG:boto:Token: None 2024-04-13T22:09:45.099 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T22:09:45.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T22:09:45.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037de119bc934a7b2-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.101 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.101 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.101 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.101 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.101 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.101 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.102 DEBUG:boto:Data: 2024-04-13T22:09:45.102 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.102 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.102 DEBUG:boto:Params: {} 2024-04-13T22:09:45.102 DEBUG:boto:Token: None 2024-04-13T22:09:45.102 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c= 2024-04-13T22:09:45.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c=', 'Content-Length': '0'} 2024-04-13T22:09:45.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f645beb6e0c494a5-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.104 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.104 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.104 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.104 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.104 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.104 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.105 DEBUG:boto:Data: 2024-04-13T22:09:45.105 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.105 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.105 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.105 DEBUG:boto:Params: {} 2024-04-13T22:09:45.105 DEBUG:boto:Token: None 2024-04-13T22:09:45.105 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.105 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c= 2024-04-13T22:09:45.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c=', 'Content-Length': '0'} 2024-04-13T22:09:45.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b4444cdd118641-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:09:45.107 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.107 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.108 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.108 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.108 DEBUG:boto:Method: GET 2024-04-13T22:09:45.108 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.108 DEBUG:boto:Data: 2024-04-13T22:09:45.108 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.108 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.109 DEBUG:boto:Params: {} 2024-04-13T22:09:45.109 DEBUG:boto:Token: None 2024-04-13T22:09:45.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T22:09:45.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T22:09:45.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6a99b8397dacdb-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.111 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.111 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.111 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.111 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:45.111 DEBUG:boto:Method: GET 2024-04-13T22:09:45.112 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.112 DEBUG:boto:Data: 2024-04-13T22:09:45.112 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.112 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.112 DEBUG:boto:Params: {} 2024-04-13T22:09:45.112 DEBUG:boto:Token: None 2024-04-13T22:09:45.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:45.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T22:09:45.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T22:09:45.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325760e443d7a67c-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.154 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.154 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.155 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.155 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.155 DEBUG:boto:Method: GET 2024-04-13T22:09:45.155 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:45.155 DEBUG:boto:Data: 2024-04-13T22:09:45.155 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.155 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.155 DEBUG:boto:Params: {} 2024-04-13T22:09:45.155 DEBUG:boto:Token: None 2024-04-13T22:09:45.156 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005093f81802bb582f-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.158 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.158 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.159 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.159 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.159 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.159 DEBUG:boto:Method: GET 2024-04-13T22:09:45.159 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:45.159 DEBUG:boto:Data: 2024-04-13T22:09:45.159 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.159 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.159 DEBUG:boto:Params: {} 2024-04-13T22:09:45.160 DEBUG:boto:Token: None 2024-04-13T22:09:45.160 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c627d667aa9e76-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.162 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.162 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.162 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.163 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.163 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:45.163 DEBUG:boto:Data: 2024-04-13T22:09:45.163 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.163 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.163 DEBUG:boto:Params: {} 2024-04-13T22:09:45.163 DEBUG:boto:Token: None 2024-04-13T22:09:45.163 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h 2024-04-13T22:09:45.163 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ= 2024-04-13T22:09:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ=', 'Content-Length': '0'} 2024-04-13T22:09:45.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000913c9f7c0125300-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.165 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.165 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.165 DEBUG:boto:Method: HEAD 2024-04-13T22:09:45.165 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:45.166 DEBUG:boto:Data: 2024-04-13T22:09:45.166 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.166 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.166 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.166 DEBUG:boto:Params: {} 2024-04-13T22:09:45.166 DEBUG:boto:Token: None 2024-04-13T22:09:45.166 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h 2024-04-13T22:09:45.166 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ= 2024-04-13T22:09:45.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ=', 'Content-Length': '0'} 2024-04-13T22:09:45.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f196ba81fad0d8-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:09:45.168 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.169 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.169 DEBUG:boto:Method: GET 2024-04-13T22:09:45.169 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:45.169 DEBUG:boto:Data: 2024-04-13T22:09:45.169 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.169 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.169 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.169 DEBUG:boto:Params: {} 2024-04-13T22:09:45.169 DEBUG:boto:Token: None 2024-04-13T22:09:45.169 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h 2024-04-13T22:09:45.170 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM= 2024-04-13T22:09:45.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM=', 'Content-Length': '0'} 2024-04-13T22:09:45.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ec52875a8b36b8-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.171 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:45.171 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:45.172 DEBUG:boto:Method: GET 2024-04-13T22:09:45.172 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:45.172 DEBUG:boto:Data: 2024-04-13T22:09:45.172 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.172 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.172 DEBUG:boto:Params: {} 2024-04-13T22:09:45.172 DEBUG:boto:Token: None 2024-04-13T22:09:45.172 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/h 2024-04-13T22:09:45.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM= 2024-04-13T22:09:45.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM=', 'Content-Length': '0'} 2024-04-13T22:09:45.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b43c86ab14a817-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.174 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.174 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.175 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.175 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.175 DEBUG:boto:Method: GET 2024-04-13T22:09:45.175 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:45.175 DEBUG:boto:Data: 2024-04-13T22:09:45.175 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:45.175 DEBUG:boto:Port: 8001 2024-04-13T22:09:45.175 DEBUG:boto:Params: {} 2024-04-13T22:09:45.175 DEBUG:boto:Token: None 2024-04-13T22:09:45.175 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d23354c1073fd90-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.178 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.178 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:45.178 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:45.178 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.178 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:45.179 DEBUG:boto:Method: GET 2024-04-13T22:09:45.179 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:45.179 DEBUG:boto:Data: 2024-04-13T22:09:45.179 DEBUG:boto:Headers: {} 2024-04-13T22:09:45.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:45.179 DEBUG:boto:Port: 8000 2024-04-13T22:09:45.179 DEBUG:boto:Params: {} 2024-04-13T22:09:45.179 DEBUG:boto:Token: None 2024-04-13T22:09:45.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:45 GMT /vwnfns-57/ 2024-04-13T22:09:45.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T22:09:45.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T22:09:45.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c67e729c156f3f-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:45.181 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:45.182 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-57 zones={a2, a1} 2024-04-13T22:09:45.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:45.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:45.182 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:45.230 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:45.230 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:45.266 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:09:45.266 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-57 2024-04-13T22:09:45.267 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 8 2024-04-13T22:09:45.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:45.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:45.294 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:45.338 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:45.339 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:45.383 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:09:45.383 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.383 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding": "None", 2024-04-13T22:09:45.383 INFO:teuthology.orchestra.run.smithi132.stdout: "current_index": { 2024-04-13T22:09:45.383 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:45.383 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "Normal", 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "normal": { 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "logs": [ 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:45.384 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 0, 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 11, 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:45.385 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.386 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:45.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.388 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:45.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:45.388 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:09:45.389 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-57 2024-04-13T22:09:45.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:45.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:45.390 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:09:45.444 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:45.445 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:45.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:45.477+0000 7f966cc2d780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T22:09:50.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:50.478+0000 7f966cc2d780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T22:09:55.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:55.479+0000 7f966cc2d780 1 bucket sync caught up with empty source 2024-04-13T22:09:55.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:55.479+0000 7f966cc2d780 0 bucket checkpoint complete 2024-04-13T22:09:55.500 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-57 2024-04-13T22:09:55.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:55.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:09:55.501 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:09:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:55.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:55.598+0000 7fe3eb0ea780 1 bucket sync caught up with source: 2024-04-13T22:09:55.599 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T22:09:55.599 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T22:09:55.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:55.598+0000 7fe3eb0ea780 0 bucket checkpoint complete 2024-04-13T22:09:55.612 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-57 zones={a2, a1} 2024-04-13T22:09:55.613 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.613 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.613 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.613 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:55.613 DEBUG:boto:Data: 2024-04-13T22:09:55.613 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.613 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.613 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.613 DEBUG:boto:Params: {} 2024-04-13T22:09:55.614 DEBUG:boto:Token: None 2024-04-13T22:09:55.614 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.614 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk= 2024-04-13T22:09:55.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk=', 'Content-Length': '0'} 2024-04-13T22:09:55.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000344b7eaf3fa7262b-00661b02b3-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.617 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.617 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.617 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.617 DEBUG:boto:Path: /vwnfns-57/ 2024-04-13T22:09:55.617 DEBUG:boto:Data: 2024-04-13T22:09:55.617 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.618 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.618 DEBUG:boto:Params: {} 2024-04-13T22:09:55.618 DEBUG:boto:Token: None 2024-04-13T22:09:55.618 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.618 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk= 2024-04-13T22:09:55.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk=', 'Content-Length': '0'} 2024-04-13T22:09:55.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a0f7ab814464fa0-00661b02b3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:09:55.620 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.620 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.620 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.620 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.620 DEBUG:boto:Method: GET 2024-04-13T22:09:55.620 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.621 DEBUG:boto:Data: 2024-04-13T22:09:55.621 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.621 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.621 DEBUG:boto:Params: {} 2024-04-13T22:09:55.621 DEBUG:boto:Token: None 2024-04-13T22:09:55.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/?versions 2024-04-13T22:09:55.621 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T22:09:55.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T22:09:55.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a1737686c47379-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.624 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.624 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:09:55.625 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:09:55.625 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:09:55.625 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:09:55.625 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:09:55.625 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:09:55.626 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:09:55.626 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:09:55.626 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:09:55.626 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.626 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.626 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.626 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.626 DEBUG:boto:Method: GET 2024-04-13T22:09:55.626 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.627 DEBUG:boto:Data: 2024-04-13T22:09:55.627 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.627 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.627 DEBUG:boto:Params: {} 2024-04-13T22:09:55.627 DEBUG:boto:Token: None 2024-04-13T22:09:55.627 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/?versions 2024-04-13T22:09:55.627 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T22:09:55.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T22:09:55.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000985c1302285a2e0c-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.630 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.630 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:09:55.631 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:09:55.631 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.632 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.632 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.632 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.632 DEBUG:boto:Method: GET 2024-04-13T22:09:55.632 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.632 DEBUG:boto:Data: 2024-04-13T22:09:55.632 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.632 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.633 DEBUG:boto:Params: {} 2024-04-13T22:09:55.633 DEBUG:boto:Token: None 2024-04-13T22:09:55.633 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/?versions 2024-04-13T22:09:55.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T22:09:55.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T22:09:55.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d389abce9b976a-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.635 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.636 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.636 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.636 DEBUG:boto:path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.636 DEBUG:boto:auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.637 DEBUG:boto:Method: GET 2024-04-13T22:09:55.637 DEBUG:boto:Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T22:09:55.637 DEBUG:boto:Data: 2024-04-13T22:09:55.637 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.637 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.637 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.637 DEBUG:boto:Params: {} 2024-04-13T22:09:55.638 DEBUG:boto:Token: None 2024-04-13T22:09:55.638 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/?versions 2024-04-13T22:09:55.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T22:09:55.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T22:09:55.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f6d7aaffb781a4-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.640 DEBUG:boto:b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:55.641 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.641 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.641 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.642 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.642 DEBUG:boto:Method: GET 2024-04-13T22:09:55.642 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.642 DEBUG:boto:Data: 2024-04-13T22:09:55.642 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.642 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.642 DEBUG:boto:Params: {} 2024-04-13T22:09:55.642 DEBUG:boto:Token: None 2024-04-13T22:09:55.642 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T22:09:55.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T22:09:55.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe43b39aa47e12c0-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.690 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.691 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.691 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.691 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.691 DEBUG:boto:Method: GET 2024-04-13T22:09:55.691 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.691 DEBUG:boto:Data: 2024-04-13T22:09:55.691 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.691 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.691 DEBUG:boto:Params: {} 2024-04-13T22:09:55.692 DEBUG:boto:Token: None 2024-04-13T22:09:55.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T22:09:55.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T22:09:55.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d415cc6c75f791-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.694 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.694 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.694 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.694 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.694 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.694 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.695 DEBUG:boto:Data: 2024-04-13T22:09:55.695 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.695 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.695 DEBUG:boto:Params: {} 2024-04-13T22:09:55.695 DEBUG:boto:Token: None 2024-04-13T22:09:55.695 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.695 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps= 2024-04-13T22:09:55.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps=', 'Content-Length': '0'} 2024-04-13T22:09:55.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068ba66a428694b12-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.697 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.697 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.697 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.697 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.697 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.698 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.698 DEBUG:boto:Data: 2024-04-13T22:09:55.698 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.698 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.698 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.698 DEBUG:boto:Params: {} 2024-04-13T22:09:55.698 DEBUG:boto:Token: None 2024-04-13T22:09:55.698 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.698 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps= 2024-04-13T22:09:55.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps=', 'Content-Length': '0'} 2024-04-13T22:09:55.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006238674cc12bae9a-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:55.700 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.700 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.701 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.701 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.701 DEBUG:boto:Method: GET 2024-04-13T22:09:55.701 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.701 DEBUG:boto:Data: 2024-04-13T22:09:55.701 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.701 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.701 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.701 DEBUG:boto:Params: {} 2024-04-13T22:09:55.701 DEBUG:boto:Token: None 2024-04-13T22:09:55.702 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T22:09:55.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T22:09:55.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036179e73b8e9e450-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.703 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.703 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.704 DEBUG:boto:path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.704 DEBUG:boto:auth_path=/vwnfns-57/a?versionId=null 2024-04-13T22:09:55.704 DEBUG:boto:Method: GET 2024-04-13T22:09:55.704 DEBUG:boto:Path: /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.704 DEBUG:boto:Data: 2024-04-13T22:09:55.704 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.704 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.704 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.704 DEBUG:boto:Params: {} 2024-04-13T22:09:55.704 DEBUG:boto:Token: None 2024-04-13T22:09:55.705 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a?versionId=null 2024-04-13T22:09:55.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T22:09:55.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T22:09:55.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8eaf0c62d22fe57-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.706 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.706 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.707 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.707 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.707 DEBUG:boto:Method: GET 2024-04-13T22:09:55.707 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:55.707 DEBUG:boto:Data: 2024-04-13T22:09:55.707 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.707 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.707 DEBUG:boto:Params: {} 2024-04-13T22:09:55.707 DEBUG:boto:Token: None 2024-04-13T22:09:55.707 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009831e6b9878005c6-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.710 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.710 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.710 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.711 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.711 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.711 DEBUG:boto:Method: GET 2024-04-13T22:09:55.711 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:55.711 DEBUG:boto:Data: 2024-04-13T22:09:55.711 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.711 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.711 DEBUG:boto:Params: {} 2024-04-13T22:09:55.711 DEBUG:boto:Token: None 2024-04-13T22:09:55.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056547d283079465b-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.713 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.713 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.714 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.714 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.714 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:55.714 DEBUG:boto:Data: 2024-04-13T22:09:55.714 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.714 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.714 DEBUG:boto:Params: {} 2024-04-13T22:09:55.714 DEBUG:boto:Token: None 2024-04-13T22:09:55.715 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a 2024-04-13T22:09:55.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk= 2024-04-13T22:09:55.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk=', 'Content-Length': '0'} 2024-04-13T22:09:55.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090bbc178ea4937fa-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.717 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.717 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.717 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.717 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:55.718 DEBUG:boto:Data: 2024-04-13T22:09:55.718 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.718 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.718 DEBUG:boto:Params: {} 2024-04-13T22:09:55.718 DEBUG:boto:Token: None 2024-04-13T22:09:55.718 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a 2024-04-13T22:09:55.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk= 2024-04-13T22:09:55.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk=', 'Content-Length': '0'} 2024-04-13T22:09:55.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007bc8388b3cd288a-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:09:55.720 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.720 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.720 DEBUG:boto:Method: GET 2024-04-13T22:09:55.720 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:55.721 DEBUG:boto:Data: 2024-04-13T22:09:55.721 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.721 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.721 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.721 DEBUG:boto:Params: {} 2024-04-13T22:09:55.721 DEBUG:boto:Token: None 2024-04-13T22:09:55.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a 2024-04-13T22:09:55.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc= 2024-04-13T22:09:55.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc=', 'Content-Length': '0'} 2024-04-13T22:09:55.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a852e2ef13e092-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.723 DEBUG:boto:path=/vwnfns-57/a 2024-04-13T22:09:55.723 DEBUG:boto:auth_path=/vwnfns-57/a 2024-04-13T22:09:55.723 DEBUG:boto:Method: GET 2024-04-13T22:09:55.723 DEBUG:boto:Path: /vwnfns-57/a 2024-04-13T22:09:55.723 DEBUG:boto:Data: 2024-04-13T22:09:55.724 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.724 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.724 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.724 DEBUG:boto:Params: {} 2024-04-13T22:09:55.724 DEBUG:boto:Token: None 2024-04-13T22:09:55.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/a 2024-04-13T22:09:55.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc= 2024-04-13T22:09:55.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc=', 'Content-Length': '0'} 2024-04-13T22:09:55.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7da2b87c95afdae-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.726 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.726 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.726 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.726 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.726 DEBUG:boto:Method: GET 2024-04-13T22:09:55.727 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:55.727 DEBUG:boto:Data: 2024-04-13T22:09:55.727 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.727 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.727 DEBUG:boto:Params: {} 2024-04-13T22:09:55.727 DEBUG:boto:Token: None 2024-04-13T22:09:55.727 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c0128f0b053040-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.730 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.730 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.730 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.731 DEBUG:boto:path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.731 DEBUG:boto:auth_path=/vwnfns-57/?prefix=a 2024-04-13T22:09:55.731 DEBUG:boto:Method: GET 2024-04-13T22:09:55.731 DEBUG:boto:Path: /vwnfns-57/?prefix=a 2024-04-13T22:09:55.731 DEBUG:boto:Data: 2024-04-13T22:09:55.731 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.731 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.731 DEBUG:boto:Params: {} 2024-04-13T22:09:55.731 DEBUG:boto:Token: None 2024-04-13T22:09:55.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf19b66a65ca0b7e-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.734 DEBUG:boto:b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:55.734 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.734 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.734 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.734 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.735 DEBUG:boto:Method: GET 2024-04-13T22:09:55.735 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.735 DEBUG:boto:Data: 2024-04-13T22:09:55.735 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.735 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.735 DEBUG:boto:Params: {} 2024-04-13T22:09:55.735 DEBUG:boto:Token: None 2024-04-13T22:09:55.735 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T22:09:55.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T22:09:55.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ae9c1f9e5e9e16-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.737 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.738 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.738 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.738 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.738 DEBUG:boto:Method: GET 2024-04-13T22:09:55.738 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.738 DEBUG:boto:Data: 2024-04-13T22:09:55.739 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.739 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.739 DEBUG:boto:Params: {} 2024-04-13T22:09:55.739 DEBUG:boto:Token: None 2024-04-13T22:09:55.739 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T22:09:55.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T22:09:55.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fe022c368c804b-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.741 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.742 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.742 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.742 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.742 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.742 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.742 DEBUG:boto:Data: 2024-04-13T22:09:55.742 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.742 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.742 DEBUG:boto:Params: {} 2024-04-13T22:09:55.742 DEBUG:boto:Token: None 2024-04-13T22:09:55.743 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI= 2024-04-13T22:09:55.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI=', 'Content-Length': '0'} 2024-04-13T22:09:55.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe268aa4b01a5986-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.744 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.745 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.745 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.745 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.745 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.745 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.745 DEBUG:boto:Data: 2024-04-13T22:09:55.745 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.745 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.745 DEBUG:boto:Params: {} 2024-04-13T22:09:55.745 DEBUG:boto:Token: None 2024-04-13T22:09:55.746 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.746 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI= 2024-04-13T22:09:55.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI=', 'Content-Length': '0'} 2024-04-13T22:09:55.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df3841854fcdb68b-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:55.748 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.748 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.748 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.748 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.748 DEBUG:boto:Method: GET 2024-04-13T22:09:55.748 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.748 DEBUG:boto:Data: 2024-04-13T22:09:55.748 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.749 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.749 DEBUG:boto:Params: {} 2024-04-13T22:09:55.749 DEBUG:boto:Token: None 2024-04-13T22:09:55.749 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T22:09:55.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T22:09:55.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007458315542bec2cc-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.751 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.751 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.751 DEBUG:boto:path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.751 DEBUG:boto:auth_path=/vwnfns-57/b?versionId=null 2024-04-13T22:09:55.751 DEBUG:boto:Method: GET 2024-04-13T22:09:55.752 DEBUG:boto:Path: /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.752 DEBUG:boto:Data: 2024-04-13T22:09:55.752 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.752 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.752 DEBUG:boto:Params: {} 2024-04-13T22:09:55.752 DEBUG:boto:Token: None 2024-04-13T22:09:55.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b?versionId=null 2024-04-13T22:09:55.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T22:09:55.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T22:09:55.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b477d8d37eb7a19-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.754 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.754 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.754 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.754 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.755 DEBUG:boto:Method: GET 2024-04-13T22:09:55.755 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:55.755 DEBUG:boto:Data: 2024-04-13T22:09:55.755 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.755 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.755 DEBUG:boto:Params: {} 2024-04-13T22:09:55.755 DEBUG:boto:Token: None 2024-04-13T22:09:55.755 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.755 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee5e43d52413caf-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.798 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.799 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.799 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.799 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.799 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.799 DEBUG:boto:Method: GET 2024-04-13T22:09:55.800 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:55.800 DEBUG:boto:Data: 2024-04-13T22:09:55.800 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.800 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.800 DEBUG:boto:Params: {} 2024-04-13T22:09:55.800 DEBUG:boto:Token: None 2024-04-13T22:09:55.801 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.801 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f58a20112859c5e8-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.803 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.803 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.803 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.803 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.804 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:55.804 DEBUG:boto:Data: 2024-04-13T22:09:55.804 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.804 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.804 DEBUG:boto:Params: {} 2024-04-13T22:09:55.804 DEBUG:boto:Token: None 2024-04-13T22:09:55.804 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b 2024-04-13T22:09:55.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4= 2024-04-13T22:09:55.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4=', 'Content-Length': '0'} 2024-04-13T22:09:55.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587599133f549b78-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.806 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.806 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.806 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.806 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:55.807 DEBUG:boto:Data: 2024-04-13T22:09:55.807 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.807 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.807 DEBUG:boto:Params: {} 2024-04-13T22:09:55.807 DEBUG:boto:Token: None 2024-04-13T22:09:55.807 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b 2024-04-13T22:09:55.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4= 2024-04-13T22:09:55.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4=', 'Content-Length': '0'} 2024-04-13T22:09:55.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d07ea4ff167680-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:09:55.809 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.809 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.810 DEBUG:boto:Method: GET 2024-04-13T22:09:55.810 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:55.810 DEBUG:boto:Data: 2024-04-13T22:09:55.810 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.810 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.810 DEBUG:boto:Params: {} 2024-04-13T22:09:55.810 DEBUG:boto:Token: None 2024-04-13T22:09:55.810 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b 2024-04-13T22:09:55.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI= 2024-04-13T22:09:55.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI=', 'Content-Length': '0'} 2024-04-13T22:09:55.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3011ab0aefcbce8-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.812 DEBUG:boto:path=/vwnfns-57/b 2024-04-13T22:09:55.813 DEBUG:boto:auth_path=/vwnfns-57/b 2024-04-13T22:09:55.813 DEBUG:boto:Method: GET 2024-04-13T22:09:55.813 DEBUG:boto:Path: /vwnfns-57/b 2024-04-13T22:09:55.813 DEBUG:boto:Data: 2024-04-13T22:09:55.813 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.813 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.813 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.813 DEBUG:boto:Params: {} 2024-04-13T22:09:55.813 DEBUG:boto:Token: None 2024-04-13T22:09:55.813 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/b 2024-04-13T22:09:55.814 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI= 2024-04-13T22:09:55.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI=', 'Content-Length': '0'} 2024-04-13T22:09:55.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3cc0332bec5323-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.862 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.863 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.863 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.863 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.863 DEBUG:boto:Method: GET 2024-04-13T22:09:55.863 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:55.863 DEBUG:boto:Data: 2024-04-13T22:09:55.863 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.863 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.863 DEBUG:boto:Params: {} 2024-04-13T22:09:55.864 DEBUG:boto:Token: None 2024-04-13T22:09:55.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f47dc70089349490-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.866 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.867 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.867 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.867 DEBUG:boto:path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.867 DEBUG:boto:auth_path=/vwnfns-57/?prefix=b 2024-04-13T22:09:55.867 DEBUG:boto:Method: GET 2024-04-13T22:09:55.867 DEBUG:boto:Path: /vwnfns-57/?prefix=b 2024-04-13T22:09:55.867 DEBUG:boto:Data: 2024-04-13T22:09:55.868 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.868 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.868 DEBUG:boto:Params: {} 2024-04-13T22:09:55.868 DEBUG:boto:Token: None 2024-04-13T22:09:55.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fa1fd158955cc75-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.870 DEBUG:boto:b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:55.870 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.870 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.871 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.871 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.871 DEBUG:boto:Method: GET 2024-04-13T22:09:55.871 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.871 DEBUG:boto:Data: 2024-04-13T22:09:55.871 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.871 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.871 DEBUG:boto:Params: {} 2024-04-13T22:09:55.871 DEBUG:boto:Token: None 2024-04-13T22:09:55.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T22:09:55.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T22:09:55.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e4674e3ce418ef-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.874 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.874 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.874 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.874 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.874 DEBUG:boto:Method: GET 2024-04-13T22:09:55.874 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.875 DEBUG:boto:Data: 2024-04-13T22:09:55.875 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.875 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.875 DEBUG:boto:Params: {} 2024-04-13T22:09:55.875 DEBUG:boto:Token: None 2024-04-13T22:09:55.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T22:09:55.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T22:09:55.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088358d4d46cb9683-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.877 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.877 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.877 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.878 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.878 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.878 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.878 DEBUG:boto:Data: 2024-04-13T22:09:55.878 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.878 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.878 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.878 DEBUG:boto:Params: {} 2024-04-13T22:09:55.878 DEBUG:boto:Token: None 2024-04-13T22:09:55.878 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.879 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs= 2024-04-13T22:09:55.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs=', 'Content-Length': '0'} 2024-04-13T22:09:55.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f8489f50ef063a-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.880 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.881 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.881 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.881 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.881 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.881 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.881 DEBUG:boto:Data: 2024-04-13T22:09:55.881 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.881 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.881 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.881 DEBUG:boto:Params: {} 2024-04-13T22:09:55.881 DEBUG:boto:Token: None 2024-04-13T22:09:55.882 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.882 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs= 2024-04-13T22:09:55.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs=', 'Content-Length': '0'} 2024-04-13T22:09:55.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a366dea6dfd2f02-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:55.884 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.884 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.884 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.884 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.885 DEBUG:boto:Method: GET 2024-04-13T22:09:55.885 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.885 DEBUG:boto:Data: 2024-04-13T22:09:55.885 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.885 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.885 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.885 DEBUG:boto:Params: {} 2024-04-13T22:09:55.885 DEBUG:boto:Token: None 2024-04-13T22:09:55.885 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.885 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T22:09:55.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T22:09:55.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a9ee4c260fefadc-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.887 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.887 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.887 DEBUG:boto:path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.887 DEBUG:boto:auth_path=/vwnfns-57/c?versionId=null 2024-04-13T22:09:55.888 DEBUG:boto:Method: GET 2024-04-13T22:09:55.888 DEBUG:boto:Path: /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.888 DEBUG:boto:Data: 2024-04-13T22:09:55.888 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.888 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.888 DEBUG:boto:Params: {} 2024-04-13T22:09:55.888 DEBUG:boto:Token: None 2024-04-13T22:09:55.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c?versionId=null 2024-04-13T22:09:55.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T22:09:55.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T22:09:55.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4c5963ffd0c40d-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.891 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.891 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.891 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.891 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.891 DEBUG:boto:Method: GET 2024-04-13T22:09:55.891 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:55.891 DEBUG:boto:Data: 2024-04-13T22:09:55.891 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.891 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.891 DEBUG:boto:Params: {} 2024-04-13T22:09:55.892 DEBUG:boto:Token: None 2024-04-13T22:09:55.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca55ed68755de98-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.938 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.939 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.939 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.939 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.939 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.939 DEBUG:boto:Method: GET 2024-04-13T22:09:55.939 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:55.939 DEBUG:boto:Data: 2024-04-13T22:09:55.939 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.940 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.940 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.940 DEBUG:boto:Params: {} 2024-04-13T22:09:55.940 DEBUG:boto:Token: None 2024-04-13T22:09:55.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c4038c387b5b18-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.942 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.942 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.942 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.943 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.943 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:55.943 DEBUG:boto:Data: 2024-04-13T22:09:55.943 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.943 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.943 DEBUG:boto:Params: {} 2024-04-13T22:09:55.943 DEBUG:boto:Token: None 2024-04-13T22:09:55.943 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c 2024-04-13T22:09:55.944 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0= 2024-04-13T22:09:55.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0=', 'Content-Length': '0'} 2024-04-13T22:09:55.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fe0833efa45595-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.945 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.945 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.946 DEBUG:boto:Method: HEAD 2024-04-13T22:09:55.946 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:55.946 DEBUG:boto:Data: 2024-04-13T22:09:55.946 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.946 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.946 DEBUG:boto:Params: {} 2024-04-13T22:09:55.946 DEBUG:boto:Token: None 2024-04-13T22:09:55.947 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c 2024-04-13T22:09:55.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0= 2024-04-13T22:09:55.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0=', 'Content-Length': '0'} 2024-04-13T22:09:55.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8ccf720cdca525-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:09:55.948 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.948 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.948 DEBUG:boto:Method: GET 2024-04-13T22:09:55.949 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:55.949 DEBUG:boto:Data: 2024-04-13T22:09:55.949 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.949 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.949 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.949 DEBUG:boto:Params: {} 2024-04-13T22:09:55.949 DEBUG:boto:Token: None 2024-04-13T22:09:55.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c 2024-04-13T22:09:55.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8= 2024-04-13T22:09:55.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8=', 'Content-Length': '0'} 2024-04-13T22:09:55.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de861fd2f517c84-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.952 DEBUG:boto:path=/vwnfns-57/c 2024-04-13T22:09:55.952 DEBUG:boto:auth_path=/vwnfns-57/c 2024-04-13T22:09:55.952 DEBUG:boto:Method: GET 2024-04-13T22:09:55.952 DEBUG:boto:Path: /vwnfns-57/c 2024-04-13T22:09:55.952 DEBUG:boto:Data: 2024-04-13T22:09:55.952 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.952 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.952 DEBUG:boto:Params: {} 2024-04-13T22:09:55.952 DEBUG:boto:Token: None 2024-04-13T22:09:55.953 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/c 2024-04-13T22:09:55.953 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8= 2024-04-13T22:09:55.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8=', 'Content-Length': '0'} 2024-04-13T22:09:55.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89c7bb4308c8e5a-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.955 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.955 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.955 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.955 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.955 DEBUG:boto:Method: GET 2024-04-13T22:09:55.955 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:55.955 DEBUG:boto:Data: 2024-04-13T22:09:55.955 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.955 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.956 DEBUG:boto:Params: {} 2024-04-13T22:09:55.956 DEBUG:boto:Token: None 2024-04-13T22:09:55.956 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007438af9c1a41568c-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.958 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.959 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:55.959 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:55.959 DEBUG:boto:path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.959 DEBUG:boto:auth_path=/vwnfns-57/?prefix=c 2024-04-13T22:09:55.959 DEBUG:boto:Method: GET 2024-04-13T22:09:55.959 DEBUG:boto:Path: /vwnfns-57/?prefix=c 2024-04-13T22:09:55.960 DEBUG:boto:Data: 2024-04-13T22:09:55.960 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.960 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:55.960 DEBUG:boto:Port: 8000 2024-04-13T22:09:55.960 DEBUG:boto:Params: {} 2024-04-13T22:09:55.960 DEBUG:boto:Token: None 2024-04-13T22:09:55.960 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/ 2024-04-13T22:09:55.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T22:09:55.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T22:09:55.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9109cedff1ef868-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:55.962 DEBUG:boto:b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:55.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:55.962 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:55.963 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:55.963 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:55.963 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:55.963 DEBUG:boto:Method: GET 2024-04-13T22:09:55.963 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:55.963 DEBUG:boto:Data: 2024-04-13T22:09:55.963 DEBUG:boto:Headers: {} 2024-04-13T22:09:55.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:55.963 DEBUG:boto:Port: 8001 2024-04-13T22:09:55.963 DEBUG:boto:Params: {} 2024-04-13T22:09:55.963 DEBUG:boto:Token: None 2024-04-13T22:09:55.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:55 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:55.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fvWfpMZMBR5yP0w4Wo0TDGJwnoM= 2024-04-13T22:09:55.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fvWfpMZMBR5yP0w4Wo0TDGJwnoM=', 'Content-Length': '0'} 2024-04-13T22:09:55.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cfe47a8402db8a9-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.006 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.007 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.007 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.007 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.007 DEBUG:boto:Method: GET 2024-04-13T22:09:56.007 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.007 DEBUG:boto:Data: 2024-04-13T22:09:56.007 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.007 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.007 DEBUG:boto:Params: {} 2024-04-13T22:09:56.007 DEBUG:boto:Token: None 2024-04-13T22:09:56.008 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M= 2024-04-13T22:09:56.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M=', 'Content-Length': '0'} 2024-04-13T22:09:56.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d846999ea75709-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.010 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.010 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.010 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.010 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.010 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.010 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.011 DEBUG:boto:Data: 2024-04-13T22:09:56.011 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.011 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.011 DEBUG:boto:Params: {} 2024-04-13T22:09:56.011 DEBUG:boto:Token: None 2024-04-13T22:09:56.011 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc= 2024-04-13T22:09:56.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc=', 'Content-Length': '0'} 2024-04-13T22:09:56.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c5bfd17851bf718-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.013 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.013 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.013 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.013 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.013 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.014 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.014 DEBUG:boto:Data: 2024-04-13T22:09:56.014 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.014 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.014 DEBUG:boto:Params: {} 2024-04-13T22:09:56.014 DEBUG:boto:Token: None 2024-04-13T22:09:56.014 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.015 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc= 2024-04-13T22:09:56.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc=', 'Content-Length': '0'} 2024-04-13T22:09:56.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b671ed01095b2a8-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:56.017 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.017 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.017 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.017 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.017 DEBUG:boto:Method: GET 2024-04-13T22:09:56.017 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.017 DEBUG:boto:Data: 2024-04-13T22:09:56.017 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.017 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.018 DEBUG:boto:Params: {} 2024-04-13T22:09:56.018 DEBUG:boto:Token: None 2024-04-13T22:09:56.018 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.018 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M= 2024-04-13T22:09:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M=', 'Content-Length': '0'} 2024-04-13T22:09:56.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002504882484fab151-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.020 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.020 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.020 DEBUG:boto:path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.020 DEBUG:boto:auth_path=/vwnfns-57/d?versionId=null 2024-04-13T22:09:56.020 DEBUG:boto:Method: GET 2024-04-13T22:09:56.020 DEBUG:boto:Path: /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.020 DEBUG:boto:Data: 2024-04-13T22:09:56.021 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.021 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.021 DEBUG:boto:Params: {} 2024-04-13T22:09:56.021 DEBUG:boto:Token: None 2024-04-13T22:09:56.021 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d?versionId=null 2024-04-13T22:09:56.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M= 2024-04-13T22:09:56.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M=', 'Content-Length': '0'} 2024-04-13T22:09:56.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b23efec3c2bca8a-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.023 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.023 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.023 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.023 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.024 DEBUG:boto:Method: GET 2024-04-13T22:09:56.024 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:56.024 DEBUG:boto:Data: 2024-04-13T22:09:56.024 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.024 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.024 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.024 DEBUG:boto:Params: {} 2024-04-13T22:09:56.024 DEBUG:boto:Token: None 2024-04-13T22:09:56.024 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6c471dabb36306-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.027 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.027 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.027 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.027 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.028 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.028 DEBUG:boto:Method: GET 2024-04-13T22:09:56.028 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:56.028 DEBUG:boto:Data: 2024-04-13T22:09:56.028 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.028 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.028 DEBUG:boto:Params: {} 2024-04-13T22:09:56.028 DEBUG:boto:Token: None 2024-04-13T22:09:56.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.029 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1724680251e93ca-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.070 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.071 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.071 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.071 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.071 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:56.071 DEBUG:boto:Data: 2024-04-13T22:09:56.071 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.072 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.072 DEBUG:boto:Params: {} 2024-04-13T22:09:56.072 DEBUG:boto:Token: None 2024-04-13T22:09:56.072 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d 2024-04-13T22:09:56.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU= 2024-04-13T22:09:56.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU=', 'Content-Length': '0'} 2024-04-13T22:09:56.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec792180d72c76fc-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.074 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.074 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.074 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.074 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:56.074 DEBUG:boto:Data: 2024-04-13T22:09:56.074 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.074 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.074 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.075 DEBUG:boto:Params: {} 2024-04-13T22:09:56.075 DEBUG:boto:Token: None 2024-04-13T22:09:56.075 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d 2024-04-13T22:09:56.075 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU= 2024-04-13T22:09:56.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU=', 'Content-Length': '0'} 2024-04-13T22:09:56.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc388f77f9df5d9-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:09:56.077 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.077 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.077 DEBUG:boto:Method: GET 2024-04-13T22:09:56.077 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:56.077 DEBUG:boto:Data: 2024-04-13T22:09:56.077 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.077 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.077 DEBUG:boto:Params: {} 2024-04-13T22:09:56.078 DEBUG:boto:Token: None 2024-04-13T22:09:56.078 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d 2024-04-13T22:09:56.078 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM= 2024-04-13T22:09:56.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM=', 'Content-Length': '0'} 2024-04-13T22:09:56.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006392e09a716568e4-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.080 DEBUG:boto:path=/vwnfns-57/d 2024-04-13T22:09:56.080 DEBUG:boto:auth_path=/vwnfns-57/d 2024-04-13T22:09:56.080 DEBUG:boto:Method: GET 2024-04-13T22:09:56.080 DEBUG:boto:Path: /vwnfns-57/d 2024-04-13T22:09:56.080 DEBUG:boto:Data: 2024-04-13T22:09:56.080 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.081 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.081 DEBUG:boto:Params: {} 2024-04-13T22:09:56.081 DEBUG:boto:Token: None 2024-04-13T22:09:56.081 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/d 2024-04-13T22:09:56.081 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM= 2024-04-13T22:09:56.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM=', 'Content-Length': '0'} 2024-04-13T22:09:56.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004be018731b4b3eaa-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.083 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.083 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.083 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.083 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.083 DEBUG:boto:Method: GET 2024-04-13T22:09:56.083 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:56.083 DEBUG:boto:Data: 2024-04-13T22:09:56.083 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.083 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.084 DEBUG:boto:Params: {} 2024-04-13T22:09:56.084 DEBUG:boto:Token: None 2024-04-13T22:09:56.084 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002def083a184ec12d-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.086 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.086 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.086 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.086 DEBUG:boto:path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.087 DEBUG:boto:auth_path=/vwnfns-57/?prefix=d 2024-04-13T22:09:56.087 DEBUG:boto:Method: GET 2024-04-13T22:09:56.087 DEBUG:boto:Path: /vwnfns-57/?prefix=d 2024-04-13T22:09:56.087 DEBUG:boto:Data: 2024-04-13T22:09:56.087 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.087 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.087 DEBUG:boto:Params: {} 2024-04-13T22:09:56.087 DEBUG:boto:Token: None 2024-04-13T22:09:56.087 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff2beded3d9a8a8-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.089 DEBUG:boto:b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:09:56.090 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.090 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.090 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.090 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.090 DEBUG:boto:Method: GET 2024-04-13T22:09:56.090 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.090 DEBUG:boto:Data: 2024-04-13T22:09:56.090 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.090 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.091 DEBUG:boto:Params: {} 2024-04-13T22:09:56.091 DEBUG:boto:Token: None 2024-04-13T22:09:56.091 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T22:09:56.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T22:09:56.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000076ccc8a521e0727-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.093 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.093 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.093 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.093 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.094 DEBUG:boto:Method: GET 2024-04-13T22:09:56.094 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.094 DEBUG:boto:Data: 2024-04-13T22:09:56.094 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.094 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.094 DEBUG:boto:Params: {} 2024-04-13T22:09:56.094 DEBUG:boto:Token: None 2024-04-13T22:09:56.094 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.094 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T22:09:56.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T22:09:56.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a91283875c72a1-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.096 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.096 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.096 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.096 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.096 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.097 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.097 DEBUG:boto:Data: 2024-04-13T22:09:56.097 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.097 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.097 DEBUG:boto:Params: {} 2024-04-13T22:09:56.097 DEBUG:boto:Token: None 2024-04-13T22:09:56.097 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.097 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY= 2024-04-13T22:09:56.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY=', 'Content-Length': '0'} 2024-04-13T22:09:56.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e183d8c7ab8e79-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.099 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.099 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.099 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.099 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.099 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.099 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.099 DEBUG:boto:Data: 2024-04-13T22:09:56.100 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.100 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.100 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.100 DEBUG:boto:Params: {} 2024-04-13T22:09:56.100 DEBUG:boto:Token: None 2024-04-13T22:09:56.100 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY= 2024-04-13T22:09:56.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY=', 'Content-Length': '0'} 2024-04-13T22:09:56.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b290907997bab3f8-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:09:56.102 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.102 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.102 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.102 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.102 DEBUG:boto:Method: GET 2024-04-13T22:09:56.102 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.103 DEBUG:boto:Data: 2024-04-13T22:09:56.103 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.103 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.103 DEBUG:boto:Params: {} 2024-04-13T22:09:56.103 DEBUG:boto:Token: None 2024-04-13T22:09:56.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T22:09:56.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T22:09:56.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ecde0ee01ba097-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.105 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.105 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.105 DEBUG:boto:path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.105 DEBUG:boto:auth_path=/vwnfns-57/e?versionId=null 2024-04-13T22:09:56.106 DEBUG:boto:Method: GET 2024-04-13T22:09:56.106 DEBUG:boto:Path: /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.106 DEBUG:boto:Data: 2024-04-13T22:09:56.106 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.106 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.106 DEBUG:boto:Params: {} 2024-04-13T22:09:56.106 DEBUG:boto:Token: None 2024-04-13T22:09:56.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e?versionId=null 2024-04-13T22:09:56.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T22:09:56.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T22:09:56.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33a274fb0f06438-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.108 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.109 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.109 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.109 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.109 DEBUG:boto:Method: GET 2024-04-13T22:09:56.109 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:56.109 DEBUG:boto:Data: 2024-04-13T22:09:56.109 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.109 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.109 DEBUG:boto:Params: {} 2024-04-13T22:09:56.109 DEBUG:boto:Token: None 2024-04-13T22:09:56.110 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e7e2ef3e2aef82-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.112 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.112 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.112 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.112 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.113 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.113 DEBUG:boto:Method: GET 2024-04-13T22:09:56.113 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:56.113 DEBUG:boto:Data: 2024-04-13T22:09:56.113 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.113 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.113 DEBUG:boto:Params: {} 2024-04-13T22:09:56.113 DEBUG:boto:Token: None 2024-04-13T22:09:56.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c2a3958ff9821a-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.115 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.116 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.116 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.116 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.116 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:56.116 DEBUG:boto:Data: 2024-04-13T22:09:56.116 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.116 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.117 DEBUG:boto:Params: {} 2024-04-13T22:09:56.117 DEBUG:boto:Token: None 2024-04-13T22:09:56.117 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e 2024-04-13T22:09:56.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA= 2024-04-13T22:09:56.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA=', 'Content-Length': '0'} 2024-04-13T22:09:56.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a2f01203f66638-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.119 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.119 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.119 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.119 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:56.119 DEBUG:boto:Data: 2024-04-13T22:09:56.119 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.120 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.120 DEBUG:boto:Params: {} 2024-04-13T22:09:56.120 DEBUG:boto:Token: None 2024-04-13T22:09:56.120 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e 2024-04-13T22:09:56.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA= 2024-04-13T22:09:56.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA=', 'Content-Length': '0'} 2024-04-13T22:09:56.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a2c6b4f6bcba3f-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:09:56.122 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.122 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.122 DEBUG:boto:Method: GET 2024-04-13T22:09:56.122 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:56.122 DEBUG:boto:Data: 2024-04-13T22:09:56.123 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.123 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.123 DEBUG:boto:Params: {} 2024-04-13T22:09:56.123 DEBUG:boto:Token: None 2024-04-13T22:09:56.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e 2024-04-13T22:09:56.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c= 2024-04-13T22:09:56.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c=', 'Content-Length': '0'} 2024-04-13T22:09:56.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a2790fb0ec3f8b-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.166 DEBUG:boto:path=/vwnfns-57/e 2024-04-13T22:09:56.167 DEBUG:boto:auth_path=/vwnfns-57/e 2024-04-13T22:09:56.167 DEBUG:boto:Method: GET 2024-04-13T22:09:56.167 DEBUG:boto:Path: /vwnfns-57/e 2024-04-13T22:09:56.167 DEBUG:boto:Data: 2024-04-13T22:09:56.167 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.167 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.167 DEBUG:boto:Params: {} 2024-04-13T22:09:56.167 DEBUG:boto:Token: None 2024-04-13T22:09:56.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/e 2024-04-13T22:09:56.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c= 2024-04-13T22:09:56.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c=', 'Content-Length': '0'} 2024-04-13T22:09:56.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000364464802f33899b-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.170 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.170 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.170 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.170 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.170 DEBUG:boto:Method: GET 2024-04-13T22:09:56.170 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:56.170 DEBUG:boto:Data: 2024-04-13T22:09:56.171 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.171 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.171 DEBUG:boto:Params: {} 2024-04-13T22:09:56.171 DEBUG:boto:Token: None 2024-04-13T22:09:56.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c393cc4c2a474b11-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.174 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.174 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.174 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.174 DEBUG:boto:path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.174 DEBUG:boto:auth_path=/vwnfns-57/?prefix=e 2024-04-13T22:09:56.175 DEBUG:boto:Method: GET 2024-04-13T22:09:56.175 DEBUG:boto:Path: /vwnfns-57/?prefix=e 2024-04-13T22:09:56.175 DEBUG:boto:Data: 2024-04-13T22:09:56.175 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.175 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.175 DEBUG:boto:Params: {} 2024-04-13T22:09:56.175 DEBUG:boto:Token: None 2024-04-13T22:09:56.175 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000155c816482f42c1c-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.177 DEBUG:boto:b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:09:56.178 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.178 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.178 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.178 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.178 DEBUG:boto:Method: GET 2024-04-13T22:09:56.178 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.179 DEBUG:boto:Data: 2024-04-13T22:09:56.179 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.179 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.179 DEBUG:boto:Params: {} 2024-04-13T22:09:56.179 DEBUG:boto:Token: None 2024-04-13T22:09:56.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T22:09:56.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ee2cdf83d1a9f9-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.181 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.181 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.181 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.181 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.182 DEBUG:boto:Method: GET 2024-04-13T22:09:56.182 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.182 DEBUG:boto:Data: 2024-04-13T22:09:56.182 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.182 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.182 DEBUG:boto:Params: {} 2024-04-13T22:09:56.182 DEBUG:boto:Token: None 2024-04-13T22:09:56.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T22:09:56.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07e70e93856a093-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.184 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.185 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.185 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.185 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.185 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.185 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.185 DEBUG:boto:Data: 2024-04-13T22:09:56.185 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.185 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.186 DEBUG:boto:Params: {} 2024-04-13T22:09:56.186 DEBUG:boto:Token: None 2024-04-13T22:09:56.186 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I= 2024-04-13T22:09:56.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I=', 'Content-Length': '0'} 2024-04-13T22:09:56.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9760737820614f-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.188 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.188 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.188 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.188 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.188 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.188 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.188 DEBUG:boto:Data: 2024-04-13T22:09:56.188 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.189 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.189 DEBUG:boto:Params: {} 2024-04-13T22:09:56.189 DEBUG:boto:Token: None 2024-04-13T22:09:56.189 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.189 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I= 2024-04-13T22:09:56.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I=', 'Content-Length': '0'} 2024-04-13T22:09:56.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000721fdde74bc86c8-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:09:56.194 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.194 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.194 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.194 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.194 DEBUG:boto:Method: GET 2024-04-13T22:09:56.194 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.194 DEBUG:boto:Data: 2024-04-13T22:09:56.194 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.194 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.195 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.195 DEBUG:boto:Params: {} 2024-04-13T22:09:56.195 DEBUG:boto:Token: None 2024-04-13T22:09:56.195 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T22:09:56.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab95a1e72b1cbef6-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.197 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.197 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.197 DEBUG:boto:path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.197 DEBUG:boto:auth_path=/vwnfns-57/f?versionId=null 2024-04-13T22:09:56.197 DEBUG:boto:Method: GET 2024-04-13T22:09:56.197 DEBUG:boto:Path: /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.198 DEBUG:boto:Data: 2024-04-13T22:09:56.198 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.198 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.198 DEBUG:boto:Params: {} 2024-04-13T22:09:56.198 DEBUG:boto:Token: None 2024-04-13T22:09:56.198 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f?versionId=null 2024-04-13T22:09:56.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T22:09:56.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed84ff4db71a3aab-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.246 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.247 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.247 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.247 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.247 DEBUG:boto:Method: GET 2024-04-13T22:09:56.247 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:56.247 DEBUG:boto:Data: 2024-04-13T22:09:56.247 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.247 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.247 DEBUG:boto:Params: {} 2024-04-13T22:09:56.248 DEBUG:boto:Token: None 2024-04-13T22:09:56.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f71c45c2e5e2c2f9-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.250 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.251 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.251 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.251 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.251 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.251 DEBUG:boto:Method: GET 2024-04-13T22:09:56.252 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:56.252 DEBUG:boto:Data: 2024-04-13T22:09:56.252 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.252 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.252 DEBUG:boto:Params: {} 2024-04-13T22:09:56.252 DEBUG:boto:Token: None 2024-04-13T22:09:56.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c3f5f8f983abb70-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.254 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.255 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.255 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.255 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.255 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:56.255 DEBUG:boto:Data: 2024-04-13T22:09:56.255 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.255 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.255 DEBUG:boto:Params: {} 2024-04-13T22:09:56.255 DEBUG:boto:Token: None 2024-04-13T22:09:56.256 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f 2024-04-13T22:09:56.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ= 2024-04-13T22:09:56.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c61b83835848898-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.257 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.257 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.258 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.258 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:56.258 DEBUG:boto:Data: 2024-04-13T22:09:56.258 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.258 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.258 DEBUG:boto:Params: {} 2024-04-13T22:09:56.258 DEBUG:boto:Token: None 2024-04-13T22:09:56.258 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f 2024-04-13T22:09:56.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ= 2024-04-13T22:09:56.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c9d2aeff26c988-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:09:56.261 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.261 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.261 DEBUG:boto:Method: GET 2024-04-13T22:09:56.261 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:56.261 DEBUG:boto:Data: 2024-04-13T22:09:56.261 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.261 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.261 DEBUG:boto:Params: {} 2024-04-13T22:09:56.262 DEBUG:boto:Token: None 2024-04-13T22:09:56.262 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f 2024-04-13T22:09:56.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk= 2024-04-13T22:09:56.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk=', 'Content-Length': '0'} 2024-04-13T22:09:56.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf16734ef9bcaef-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.263 DEBUG:boto:path=/vwnfns-57/f 2024-04-13T22:09:56.264 DEBUG:boto:auth_path=/vwnfns-57/f 2024-04-13T22:09:56.264 DEBUG:boto:Method: GET 2024-04-13T22:09:56.264 DEBUG:boto:Path: /vwnfns-57/f 2024-04-13T22:09:56.264 DEBUG:boto:Data: 2024-04-13T22:09:56.264 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.264 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.264 DEBUG:boto:Params: {} 2024-04-13T22:09:56.264 DEBUG:boto:Token: None 2024-04-13T22:09:56.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/f 2024-04-13T22:09:56.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk= 2024-04-13T22:09:56.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk=', 'Content-Length': '0'} 2024-04-13T22:09:56.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33481b32b8edbbf-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.267 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.267 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.267 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.267 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.267 DEBUG:boto:Method: GET 2024-04-13T22:09:56.267 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:56.267 DEBUG:boto:Data: 2024-04-13T22:09:56.267 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.267 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.268 DEBUG:boto:Params: {} 2024-04-13T22:09:56.268 DEBUG:boto:Token: None 2024-04-13T22:09:56.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87bdea1f6beaf5f-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.270 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.271 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.271 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.271 DEBUG:boto:path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.271 DEBUG:boto:auth_path=/vwnfns-57/?prefix=f 2024-04-13T22:09:56.271 DEBUG:boto:Method: GET 2024-04-13T22:09:56.271 DEBUG:boto:Path: /vwnfns-57/?prefix=f 2024-04-13T22:09:56.271 DEBUG:boto:Data: 2024-04-13T22:09:56.271 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.272 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.272 DEBUG:boto:Params: {} 2024-04-13T22:09:56.272 DEBUG:boto:Token: None 2024-04-13T22:09:56.272 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b86a26eca09d51dc-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.274 DEBUG:boto:b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:09:56.274 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.274 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.275 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.275 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.275 DEBUG:boto:Method: GET 2024-04-13T22:09:56.275 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.275 DEBUG:boto:Data: 2024-04-13T22:09:56.275 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.275 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.275 DEBUG:boto:Params: {} 2024-04-13T22:09:56.275 DEBUG:boto:Token: None 2024-04-13T22:09:56.276 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T22:09:56.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T22:09:56.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0056aefa75ff86-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.318 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.319 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.319 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.319 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.319 DEBUG:boto:Method: GET 2024-04-13T22:09:56.319 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.319 DEBUG:boto:Data: 2024-04-13T22:09:56.319 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.319 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.320 DEBUG:boto:Params: {} 2024-04-13T22:09:56.320 DEBUG:boto:Token: None 2024-04-13T22:09:56.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T22:09:56.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T22:09:56.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7298dd675c580a-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.322 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.322 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.322 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.322 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.322 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.322 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.323 DEBUG:boto:Data: 2024-04-13T22:09:56.323 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.323 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.323 DEBUG:boto:Params: {} 2024-04-13T22:09:56.323 DEBUG:boto:Token: None 2024-04-13T22:09:56.323 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk= 2024-04-13T22:09:56.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk=', 'Content-Length': '0'} 2024-04-13T22:09:56.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080032f4e2222e565-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.325 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.325 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.325 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.325 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.325 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.326 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.326 DEBUG:boto:Data: 2024-04-13T22:09:56.326 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.326 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.326 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.326 DEBUG:boto:Params: {} 2024-04-13T22:09:56.326 DEBUG:boto:Token: None 2024-04-13T22:09:56.326 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk= 2024-04-13T22:09:56.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk=', 'Content-Length': '0'} 2024-04-13T22:09:56.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a536242c762bf104-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:09:56.328 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.328 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.328 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.328 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.329 DEBUG:boto:Method: GET 2024-04-13T22:09:56.329 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.329 DEBUG:boto:Data: 2024-04-13T22:09:56.329 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.329 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.329 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.329 DEBUG:boto:Params: {} 2024-04-13T22:09:56.329 DEBUG:boto:Token: None 2024-04-13T22:09:56.329 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T22:09:56.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T22:09:56.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe82b25e7334a23-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.331 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.331 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.331 DEBUG:boto:path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.332 DEBUG:boto:auth_path=/vwnfns-57/g?versionId=null 2024-04-13T22:09:56.332 DEBUG:boto:Method: GET 2024-04-13T22:09:56.332 DEBUG:boto:Path: /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.332 DEBUG:boto:Data: 2024-04-13T22:09:56.332 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.332 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.332 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.332 DEBUG:boto:Params: {} 2024-04-13T22:09:56.332 DEBUG:boto:Token: None 2024-04-13T22:09:56.332 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g?versionId=null 2024-04-13T22:09:56.333 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T22:09:56.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T22:09:56.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009293fe2419d6444c-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.335 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.335 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.335 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.335 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.335 DEBUG:boto:Method: GET 2024-04-13T22:09:56.335 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:56.335 DEBUG:boto:Data: 2024-04-13T22:09:56.335 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.335 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.336 DEBUG:boto:Params: {} 2024-04-13T22:09:56.336 DEBUG:boto:Token: None 2024-04-13T22:09:56.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed6080b0ffc38bc9-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.338 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.339 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.339 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.339 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.339 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.339 DEBUG:boto:Method: GET 2024-04-13T22:09:56.339 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:56.339 DEBUG:boto:Data: 2024-04-13T22:09:56.339 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.339 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.340 DEBUG:boto:Params: {} 2024-04-13T22:09:56.340 DEBUG:boto:Token: None 2024-04-13T22:09:56.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bace98ffb80606ce-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.382 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.383 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.383 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.383 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.383 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:56.383 DEBUG:boto:Data: 2024-04-13T22:09:56.383 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.383 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.384 DEBUG:boto:Params: {} 2024-04-13T22:09:56.384 DEBUG:boto:Token: None 2024-04-13T22:09:56.384 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g 2024-04-13T22:09:56.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs= 2024-04-13T22:09:56.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs=', 'Content-Length': '0'} 2024-04-13T22:09:56.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e3fedb2d5367ee-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.386 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.386 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.386 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.386 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:56.386 DEBUG:boto:Data: 2024-04-13T22:09:56.386 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.387 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.387 DEBUG:boto:Params: {} 2024-04-13T22:09:56.387 DEBUG:boto:Token: None 2024-04-13T22:09:56.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g 2024-04-13T22:09:56.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs= 2024-04-13T22:09:56.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs=', 'Content-Length': '0'} 2024-04-13T22:09:56.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000145fd31fecbcbdcc-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:09:56.389 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.389 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.389 DEBUG:boto:Method: GET 2024-04-13T22:09:56.389 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:56.389 DEBUG:boto:Data: 2024-04-13T22:09:56.390 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.390 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.390 DEBUG:boto:Params: {} 2024-04-13T22:09:56.390 DEBUG:boto:Token: None 2024-04-13T22:09:56.390 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g 2024-04-13T22:09:56.390 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug= 2024-04-13T22:09:56.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug=', 'Content-Length': '0'} 2024-04-13T22:09:56.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000913cf4802cbe8d84-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.392 DEBUG:boto:path=/vwnfns-57/g 2024-04-13T22:09:56.392 DEBUG:boto:auth_path=/vwnfns-57/g 2024-04-13T22:09:56.393 DEBUG:boto:Method: GET 2024-04-13T22:09:56.393 DEBUG:boto:Path: /vwnfns-57/g 2024-04-13T22:09:56.393 DEBUG:boto:Data: 2024-04-13T22:09:56.393 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.393 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.393 DEBUG:boto:Params: {} 2024-04-13T22:09:56.393 DEBUG:boto:Token: None 2024-04-13T22:09:56.393 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/g 2024-04-13T22:09:56.393 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug= 2024-04-13T22:09:56.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug=', 'Content-Length': '0'} 2024-04-13T22:09:56.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea9a79f0dd2283e-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.395 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.395 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.396 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.396 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.396 DEBUG:boto:Method: GET 2024-04-13T22:09:56.396 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:56.396 DEBUG:boto:Data: 2024-04-13T22:09:56.396 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.396 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.396 DEBUG:boto:Params: {} 2024-04-13T22:09:56.396 DEBUG:boto:Token: None 2024-04-13T22:09:56.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000181779977a0206a7-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.399 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.399 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.399 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.400 DEBUG:boto:path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.400 DEBUG:boto:auth_path=/vwnfns-57/?prefix=g 2024-04-13T22:09:56.400 DEBUG:boto:Method: GET 2024-04-13T22:09:56.400 DEBUG:boto:Path: /vwnfns-57/?prefix=g 2024-04-13T22:09:56.400 DEBUG:boto:Data: 2024-04-13T22:09:56.400 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.400 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.400 DEBUG:boto:Params: {} 2024-04-13T22:09:56.400 DEBUG:boto:Token: None 2024-04-13T22:09:56.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6daaa62806914d8-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.403 DEBUG:boto:b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:09:56.403 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.403 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.403 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.403 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.404 DEBUG:boto:Method: GET 2024-04-13T22:09:56.404 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.404 DEBUG:boto:Data: 2024-04-13T22:09:56.404 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.404 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.404 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.404 DEBUG:boto:Params: {} 2024-04-13T22:09:56.404 DEBUG:boto:Token: None 2024-04-13T22:09:56.404 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T22:09:56.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T22:09:56.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022201e3f4cc3a182-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.446 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.447 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.447 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.447 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.447 DEBUG:boto:Method: GET 2024-04-13T22:09:56.447 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.447 DEBUG:boto:Data: 2024-04-13T22:09:56.447 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.447 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.448 DEBUG:boto:Params: {} 2024-04-13T22:09:56.448 DEBUG:boto:Token: None 2024-04-13T22:09:56.448 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T22:09:56.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T22:09:56.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae8e4b5945cb2de-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.450 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.450 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.451 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.451 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.451 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.451 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.451 DEBUG:boto:Data: 2024-04-13T22:09:56.451 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.451 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.451 DEBUG:boto:Params: {} 2024-04-13T22:09:56.451 DEBUG:boto:Token: None 2024-04-13T22:09:56.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ= 2024-04-13T22:09:56.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78321d6263b54ee-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.454 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.454 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.454 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.454 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.454 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.454 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.454 DEBUG:boto:Data: 2024-04-13T22:09:56.454 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.454 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.454 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.455 DEBUG:boto:Params: {} 2024-04-13T22:09:56.455 DEBUG:boto:Token: None 2024-04-13T22:09:56.455 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ= 2024-04-13T22:09:56.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ=', 'Content-Length': '0'} 2024-04-13T22:09:56.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6fca6e80e74b621-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:09:56.457 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.457 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.457 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.457 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.457 DEBUG:boto:Method: GET 2024-04-13T22:09:56.457 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.458 DEBUG:boto:Data: 2024-04-13T22:09:56.458 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.458 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.458 DEBUG:boto:Params: {} 2024-04-13T22:09:56.458 DEBUG:boto:Token: None 2024-04-13T22:09:56.458 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.458 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T22:09:56.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T22:09:56.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a6c89a64e078d3-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.460 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.460 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.460 DEBUG:boto:path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.461 DEBUG:boto:auth_path=/vwnfns-57/h?versionId=null 2024-04-13T22:09:56.461 DEBUG:boto:Method: GET 2024-04-13T22:09:56.461 DEBUG:boto:Path: /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.461 DEBUG:boto:Data: 2024-04-13T22:09:56.461 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.461 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.461 DEBUG:boto:Params: {} 2024-04-13T22:09:56.461 DEBUG:boto:Token: None 2024-04-13T22:09:56.462 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h?versionId=null 2024-04-13T22:09:56.462 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T22:09:56.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T22:09:56.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fcc6e4ded8b45d-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.463 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.463 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.464 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.464 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.464 DEBUG:boto:Method: GET 2024-04-13T22:09:56.464 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:56.464 DEBUG:boto:Data: 2024-04-13T22:09:56.464 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.464 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.464 DEBUG:boto:Params: {} 2024-04-13T22:09:56.464 DEBUG:boto:Token: None 2024-04-13T22:09:56.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d21cc1e373b0909-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.467 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.467 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.468 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.468 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.468 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.468 DEBUG:boto:Method: GET 2024-04-13T22:09:56.468 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:56.468 DEBUG:boto:Data: 2024-04-13T22:09:56.468 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.468 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.468 DEBUG:boto:Params: {} 2024-04-13T22:09:56.468 DEBUG:boto:Token: None 2024-04-13T22:09:56.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b7159e42ff72e9-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.470 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.471 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.471 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.471 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.471 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:56.471 DEBUG:boto:Data: 2024-04-13T22:09:56.471 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.471 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.471 DEBUG:boto:Params: {} 2024-04-13T22:09:56.472 DEBUG:boto:Token: None 2024-04-13T22:09:56.472 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h 2024-04-13T22:09:56.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc= 2024-04-13T22:09:56.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc=', 'Content-Length': '0'} 2024-04-13T22:09:56.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23360d02d682648-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.473 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.474 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.474 DEBUG:boto:Method: HEAD 2024-04-13T22:09:56.474 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:56.474 DEBUG:boto:Data: 2024-04-13T22:09:56.474 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.474 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.474 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.474 DEBUG:boto:Params: {} 2024-04-13T22:09:56.474 DEBUG:boto:Token: None 2024-04-13T22:09:56.475 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h 2024-04-13T22:09:56.475 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc= 2024-04-13T22:09:56.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc=', 'Content-Length': '0'} 2024-04-13T22:09:56.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e817c8fbc2bc8b66-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:09:56.476 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.477 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.477 DEBUG:boto:Method: GET 2024-04-13T22:09:56.477 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:56.477 DEBUG:boto:Data: 2024-04-13T22:09:56.477 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.477 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.477 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.477 DEBUG:boto:Params: {} 2024-04-13T22:09:56.477 DEBUG:boto:Token: None 2024-04-13T22:09:56.477 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h 2024-04-13T22:09:56.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4= 2024-04-13T22:09:56.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4=', 'Content-Length': '0'} 2024-04-13T22:09:56.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf6c49bc5f3260d-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.479 DEBUG:boto:path=/vwnfns-57/h 2024-04-13T22:09:56.480 DEBUG:boto:auth_path=/vwnfns-57/h 2024-04-13T22:09:56.480 DEBUG:boto:Method: GET 2024-04-13T22:09:56.480 DEBUG:boto:Path: /vwnfns-57/h 2024-04-13T22:09:56.480 DEBUG:boto:Data: 2024-04-13T22:09:56.480 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.480 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.480 DEBUG:boto:Params: {} 2024-04-13T22:09:56.480 DEBUG:boto:Token: None 2024-04-13T22:09:56.480 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/h 2024-04-13T22:09:56.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4= 2024-04-13T22:09:56.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4=', 'Content-Length': '0'} 2024-04-13T22:09:56.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c700b7d064483062-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.526 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.526 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.527 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.527 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.527 DEBUG:boto:Method: GET 2024-04-13T22:09:56.527 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:56.527 DEBUG:boto:Data: 2024-04-13T22:09:56.527 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:09:56.527 DEBUG:boto:Port: 8001 2024-04-13T22:09:56.527 DEBUG:boto:Params: {} 2024-04-13T22:09:56.527 DEBUG:boto:Token: None 2024-04-13T22:09:56.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000226f15eccc55027a-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.530 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.530 DEBUG:boto:path=/vwnfns-57/ 2024-04-13T22:09:56.530 DEBUG:boto:auth_path=/vwnfns-57/ 2024-04-13T22:09:56.531 DEBUG:boto:path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.531 DEBUG:boto:auth_path=/vwnfns-57/?prefix=h 2024-04-13T22:09:56.531 DEBUG:boto:Method: GET 2024-04-13T22:09:56.531 DEBUG:boto:Path: /vwnfns-57/?prefix=h 2024-04-13T22:09:56.531 DEBUG:boto:Data: 2024-04-13T22:09:56.531 DEBUG:boto:Headers: {} 2024-04-13T22:09:56.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:56.531 DEBUG:boto:Port: 8000 2024-04-13T22:09:56.531 DEBUG:boto:Params: {} 2024-04-13T22:09:56.531 DEBUG:boto:Token: None 2024-04-13T22:09:56.532 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:09:56 GMT /vwnfns-57/ 2024-04-13T22:09:56.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T22:09:56.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T22:09:56.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a3c046c2bd5420-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:56.534 DEBUG:boto:b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:09:56.534 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-57 zones={a2, a1} 2024-04-13T22:09:56.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:56.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:56.535 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:56.578 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:56.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:56.619 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.618+0000 7f5e0ffe8700 -1 req 13763461066637601712 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-33:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.33,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-33:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.33,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.620 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.618+0000 7f5deffa8700 -1 req 13368004055144513720 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-32:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.32,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-32:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.32,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.620 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.618+0000 7f5dae725700 -1 req 2416174129595301439 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-22:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.22,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-22:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.22,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.624 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.621+0000 7f5d8e6e5700 -1 req 13211626067910606235 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-30:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.30,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-30:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.30,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.624 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.622+0000 7f5d946f1700 -1 req 8741748541085336033 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-25:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.25,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-25:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.25,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.664 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.663+0000 7f5e48859700 -1 req 1938242242939681279 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-40:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.3,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-40:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.3,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.667 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.666+0000 7f5dfe7c5700 -1 req 10447478249211428170 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-39:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.2,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-39:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.2,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.676 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.675+0000 7f5e2a81d700 -1 req 14564861806786498179 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-23:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.23,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-23:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.23,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.706 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.705+0000 7f5e067d5700 -1 req 1710315130551123393 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-24:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.24,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-24:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.24,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.709 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.708+0000 7f5e1e805700 -1 req 11228260614169747268 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-5:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.5,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-5:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.5,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.719 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.718+0000 7f5dfbfc0700 -1 req 13852769238849506418 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-42:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.5,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-42:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.5,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:56.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:56.776 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:56.861 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:56.861 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:56.901 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:09:56.901 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:56.901 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding": "None", 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "current_index": { 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "Normal", 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "normal": { 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:56.902 INFO:teuthology.orchestra.run.smithi132.stdout: "logs": [ 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:56.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.905 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:56.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:56.905 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:09:56.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:56.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:56.906 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:56.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:56.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:56.988 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.987+0000 7f5e1d002700 -1 req 13868092827117849892 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-38:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.1,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-38:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.1,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:56.991 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:56.990+0000 7f5e7d8c3700 -1 req 3477285246539159647 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-29:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.29,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-29:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.29,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:57.079 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:09:57.078+0000 7f5df27ad700 -1 req 3395898003046122895 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-34:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.34,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-34:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.34,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T22:09:57.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:57.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:09:57.151 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:09:57.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:57.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:57.279 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:09:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding": "None", 2024-04-13T22:09:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "current_index": { 2024-04-13T22:09:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "Normal", 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "normal": { 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "logs": [ 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 1, 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "type": "InIndex", 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "in_index": { 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "gen": 2, 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "layout": { 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 3, 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "hash_type": "Mod" 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.284 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:09:57.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-04-13T22:09:57.290 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:09:57.290 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:09:57.290 DEBUG:boto:Method: PUT 2024-04-13T22:09:57.290 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:09:57.290 DEBUG:boto:Data: 2024-04-13T22:09:57.290 DEBUG:boto:Headers: {} 2024-04-13T22:09:57.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:09:57.291 DEBUG:boto:Port: 8000 2024-04-13T22:09:57.291 DEBUG:boto:Params: {} 2024-04-13T22:09:57.291 DEBUG:boto:Token: None 2024-04-13T22:09:57.291 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:09:57 GMT /vwnfns-58/ 2024-04-13T22:09:57.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DOKhdPSp2y1QEJZpS6isYghhCBg= 2024-04-13T22:09:57.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOKhdPSp2y1QEJZpS6isYghhCBg=', 'Content-Length': '0'} 2024-04-13T22:09:57.306 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007d3ca95e3e361985-00661b02b5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:09:57.306 DEBUG:rgw_multi.tests:created bucket=vwnfns-58 2024-04-13T22:09:57.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:57.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:57.306 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:09:57.353 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:57.354 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:57.383 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:09:57.397 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:09:57.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.397 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046197.303904_726.1", 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:09:57.303904Z" 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:09:57.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.399 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:09:57.399 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:09:57.399 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.399 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:09:57.400 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046197.303904_726.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:09:57.400 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:09:57.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:57.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:09:57.401 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:09:57.449 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:09:57.450 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:09:57.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:57.479+0000 7f06b2ec3780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:09:57.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:09:57.480+0000 7f06b2ec3780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:09:57.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046196.624430_721.1", 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:09:56.624430Z", 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:09:57.496 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:09:57.497 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:09:57.498 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:09:57.499 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:09:57.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:09:57.500 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:09:57Z" 2024-04-13T22:09:57.500 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:09:57.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046196.624430_721.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:09:56.624430Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:09:57Z" } 2024-04-13T22:09:57.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046196.624430_721.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:09:56.624430Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:09:57.501 DEBUG:rgw_multi.tests:log_status={0: '1_1713046197.303904_726.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:09:57.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046196.624430_721.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:09:57.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046197.303904_726.1 target=1_1713046196.624430_721.1 2024-04-13T22:10:02.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:10:02.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:10:02.503 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:10:02.559 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:10:02.559 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:10:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:02.590+0000 7f18752ee780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:10:02.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:02.591+0000 7f18752ee780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:10:02.605 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:10:02.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046197.303904_726.1", 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:09:57.303904Z", 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:10:02.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:10:02.608 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:10:02.609 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:10:02.610 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:10:02.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:10:02Z" 2024-04-13T22:10:02.612 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:10:02.614 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046197.303904_726.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:09:57.303904Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:10:02Z" } 2024-04-13T22:10:02.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046197.303904_726.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:09:57.303904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:10:02.615 DEBUG:rgw_multi.tests:log_status={0: '1_1713046197.303904_726.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:10:02.615 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046197.303904_726.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:10:02.615 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:10:02.615 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:02.615 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:02.616 DEBUG:boto:Method: HEAD 2024-04-13T22:10:02.616 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:02.616 DEBUG:boto:Data: 2024-04-13T22:10:02.616 DEBUG:boto:Headers: {} 2024-04-13T22:10:02.616 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.616 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.616 DEBUG:boto:Params: {} 2024-04-13T22:10:02.616 DEBUG:boto:Token: None 2024-04-13T22:10:02.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/ 2024-04-13T22:10:02.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg= 2024-04-13T22:10:02.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg=', 'Content-Length': '0'} 2024-04-13T22:10:02.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f7b865ecf78dcff-00661b02ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.619 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:02.619 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:02.619 DEBUG:boto:Method: PUT 2024-04-13T22:10:02.619 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:10:02.620 DEBUG:boto:Data: 2024-04-13T22:10:02.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:02.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.620 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.620 DEBUG:boto:Params: {} 2024-04-13T22:10:02.620 DEBUG:boto:Token: None 2024-04-13T22:10:02.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/a 2024-04-13T22:10:02.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yQ5X9N7VLPtwUvg6ML6abXwCDUQ= 2024-04-13T22:10:02.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yQ5X9N7VLPtwUvg6ML6abXwCDUQ='} 2024-04-13T22:10:02.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9342b1fbb28a269-00661b02ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.627 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:02.627 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:02.627 DEBUG:boto:Method: HEAD 2024-04-13T22:10:02.628 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:02.628 DEBUG:boto:Data: 2024-04-13T22:10:02.628 DEBUG:boto:Headers: {} 2024-04-13T22:10:02.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.628 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.628 DEBUG:boto:Params: {} 2024-04-13T22:10:02.628 DEBUG:boto:Token: None 2024-04-13T22:10:02.628 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/ 2024-04-13T22:10:02.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg= 2024-04-13T22:10:02.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg=', 'Content-Length': '0'} 2024-04-13T22:10:02.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bcdfdb5d260ccf6-00661b02ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.631 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:02.631 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:02.631 DEBUG:boto:Method: PUT 2024-04-13T22:10:02.631 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:10:02.631 DEBUG:boto:Data: 2024-04-13T22:10:02.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:02.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.632 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.632 DEBUG:boto:Params: {} 2024-04-13T22:10:02.632 DEBUG:boto:Token: None 2024-04-13T22:10:02.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/b 2024-04-13T22:10:02.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QkwwUrBjiRIz0EiAVyQjPlvwrMY= 2024-04-13T22:10:02.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QkwwUrBjiRIz0EiAVyQjPlvwrMY='} 2024-04-13T22:10:02.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d62102a6c9f2e0b-00661b02ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.675 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:02.675 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:02.675 DEBUG:boto:Method: HEAD 2024-04-13T22:10:02.675 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:02.675 DEBUG:boto:Data: 2024-04-13T22:10:02.675 DEBUG:boto:Headers: {} 2024-04-13T22:10:02.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.675 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.675 DEBUG:boto:Params: {} 2024-04-13T22:10:02.676 DEBUG:boto:Token: None 2024-04-13T22:10:02.676 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/ 2024-04-13T22:10:02.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg= 2024-04-13T22:10:02.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg=', 'Content-Length': '0'} 2024-04-13T22:10:02.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007566e21d957435b1-00661b02ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.678 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:02.679 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:02.679 DEBUG:boto:Method: PUT 2024-04-13T22:10:02.679 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:10:02.679 DEBUG:boto:Data: 2024-04-13T22:10:02.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:02.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.679 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.679 DEBUG:boto:Params: {} 2024-04-13T22:10:02.679 DEBUG:boto:Token: None 2024-04-13T22:10:02.679 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/c 2024-04-13T22:10:02.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+ikeURUIFhln0fznbP4z6PWaTyA= 2024-04-13T22:10:02.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+ikeURUIFhln0fznbP4z6PWaTyA='} 2024-04-13T22:10:02.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019fbfda875a4e882-00661b02ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.685 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:02.685 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:02.685 DEBUG:boto:Method: HEAD 2024-04-13T22:10:02.685 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:02.685 DEBUG:boto:Data: 2024-04-13T22:10:02.685 DEBUG:boto:Headers: {} 2024-04-13T22:10:02.685 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.685 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.685 DEBUG:boto:Params: {} 2024-04-13T22:10:02.686 DEBUG:boto:Token: None 2024-04-13T22:10:02.686 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/ 2024-04-13T22:10:02.686 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg= 2024-04-13T22:10:02.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lsKP+lowFYUx/am0XjkRUq1wVPg=', 'Content-Length': '0'} 2024-04-13T22:10:02.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000847ebec728454df8-00661b02ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.688 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:02.688 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:02.689 DEBUG:boto:Method: PUT 2024-04-13T22:10:02.689 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:10:02.689 DEBUG:boto:Data: 2024-04-13T22:10:02.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:02.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:02.689 DEBUG:boto:Port: 8000 2024-04-13T22:10:02.689 DEBUG:boto:Params: {} 2024-04-13T22:10:02.689 DEBUG:boto:Token: None 2024-04-13T22:10:02.689 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:02 GMT /vwnfns-58/d 2024-04-13T22:10:02.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NCtn8xZw0Cdn9fThvE2jVx6D+To= 2024-04-13T22:10:02.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NCtn8xZw0Cdn9fThvE2jVx6D+To='} 2024-04-13T22:10:02.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b37bbe3e3436011-00661b02ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:02.739 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-58 2024-04-13T22:10:02.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:10:02.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:10:02.739 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:10:02.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:10:02.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:10:02.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:02.830+0000 7f25c04e7780 1 waiting to reach incremental sync.. 2024-04-13T22:10:07.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:07.830+0000 7f25c04e7780 1 waiting to reach incremental sync.. 2024-04-13T22:10:12.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:12.831+0000 7f25c04e7780 1 waiting to reach incremental sync.. 2024-04-13T22:10:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:17.832+0000 7f25c04e7780 1 bucket sync caught up with source: 2024-04-13T22:10:17.835 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.19.6, , , , 00000000001.60.6, , , 00000000001.127.6, , 00000000001.31.6] 2024-04-13T22:10:17.835 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.19.6, , , , 00000000001.60.6, , , 00000000001.127.6, , 00000000001.31.6] 2024-04-13T22:10:17.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:17.832+0000 7f25c04e7780 0 bucket checkpoint complete 2024-04-13T22:10:17.852 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-58 2024-04-13T22:10:17.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:10:17.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:10:17.853 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:10:17.913 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:10:17.913 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:10:17.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:17.948+0000 7f81e1a46780 1 waiting to reach incremental sync.. 2024-04-13T22:10:22.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:22.949+0000 7f81e1a46780 1 waiting to reach incremental sync.. 2024-04-13T22:10:27.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:27.952+0000 7f81e1a46780 1 bucket sync caught up with source: 2024-04-13T22:10:27.953 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.1074.6, , , , 00000000001.988.6, , , 00000000001.747.6, , 00000000001.2618.6] 2024-04-13T22:10:27.954 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.1074.6, , , , 00000000001.988.6, , , 00000000001.747.6, , 00000000001.2618.6] 2024-04-13T22:10:27.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:27.952+0000 7f81e1a46780 0 bucket checkpoint complete 2024-04-13T22:10:27.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-58 zones={a2, a1} 2024-04-13T22:10:27.970 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:27.970 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:27.971 DEBUG:boto:Method: HEAD 2024-04-13T22:10:27.971 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:27.971 DEBUG:boto:Data: 2024-04-13T22:10:27.971 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:27.971 DEBUG:boto:Port: 8001 2024-04-13T22:10:27.971 DEBUG:boto:Params: {} 2024-04-13T22:10:27.972 DEBUG:boto:Token: None 2024-04-13T22:10:27.972 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/ 2024-04-13T22:10:27.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6urJxFLgNum/GsHhn4XD4abns6E= 2024-04-13T22:10:27.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6urJxFLgNum/GsHhn4XD4abns6E=', 'Content-Length': '0'} 2024-04-13T22:10:27.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9d5b467b31a7bef-00661b02d3-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:27.975 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:27.975 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:27.975 DEBUG:boto:Method: HEAD 2024-04-13T22:10:27.975 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:27.976 DEBUG:boto:Data: 2024-04-13T22:10:27.976 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:27.976 DEBUG:boto:Port: 8000 2024-04-13T22:10:27.976 DEBUG:boto:Params: {} 2024-04-13T22:10:27.976 DEBUG:boto:Token: None 2024-04-13T22:10:27.976 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/ 2024-04-13T22:10:27.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6urJxFLgNum/GsHhn4XD4abns6E= 2024-04-13T22:10:27.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6urJxFLgNum/GsHhn4XD4abns6E=', 'Content-Length': '0'} 2024-04-13T22:10:27.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023bbb243e07ab9f5-00661b02d3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:27.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:10:27.979 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:27.979 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:27.979 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.979 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.979 DEBUG:boto:Method: GET 2024-04-13T22:10:27.979 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.979 DEBUG:boto:Data: 2024-04-13T22:10:27.979 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.980 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:27.980 DEBUG:boto:Port: 8001 2024-04-13T22:10:27.980 DEBUG:boto:Params: {} 2024-04-13T22:10:27.980 DEBUG:boto:Token: None 2024-04-13T22:10:27.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/?versions 2024-04-13T22:10:27.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc= 2024-04-13T22:10:27.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc=', 'Content-Length': '0'} 2024-04-13T22:10:27.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe5e654855213782-00661b02d3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:27.982 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:27.983 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:10:27.983 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:10:27.984 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:10:27.984 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:10:27.984 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:10:27.984 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:27.984 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:27.984 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.984 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.984 DEBUG:boto:Method: GET 2024-04-13T22:10:27.984 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.984 DEBUG:boto:Data: 2024-04-13T22:10:27.985 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.985 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:27.985 DEBUG:boto:Port: 8000 2024-04-13T22:10:27.985 DEBUG:boto:Params: {} 2024-04-13T22:10:27.985 DEBUG:boto:Token: None 2024-04-13T22:10:27.985 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/?versions 2024-04-13T22:10:27.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc= 2024-04-13T22:10:27.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc=', 'Content-Length': '0'} 2024-04-13T22:10:27.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1cedbffa09bbc6-00661b02d3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:27.988 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:27.988 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:10:27.988 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:10:27.988 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:10:27.988 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:10:27.989 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:27.989 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:27.989 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.989 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.989 DEBUG:boto:Method: GET 2024-04-13T22:10:27.989 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.989 DEBUG:boto:Data: 2024-04-13T22:10:27.989 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.989 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:27.990 DEBUG:boto:Port: 8001 2024-04-13T22:10:27.990 DEBUG:boto:Params: {} 2024-04-13T22:10:27.990 DEBUG:boto:Token: None 2024-04-13T22:10:27.990 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/?versions 2024-04-13T22:10:27.990 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc= 2024-04-13T22:10:27.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc=', 'Content-Length': '0'} 2024-04-13T22:10:27.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000408d3128e8f93803-00661b02d3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:27.992 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:27.993 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:27.993 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:27.993 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.993 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.993 DEBUG:boto:Method: GET 2024-04-13T22:10:27.993 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:10:27.993 DEBUG:boto:Data: 2024-04-13T22:10:27.993 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:27.994 DEBUG:boto:Port: 8000 2024-04-13T22:10:27.994 DEBUG:boto:Params: {} 2024-04-13T22:10:27.994 DEBUG:boto:Token: None 2024-04-13T22:10:27.994 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/?versions 2024-04-13T22:10:27.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc= 2024-04-13T22:10:27.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scylKhLSH1s8mJH005QgQslnXBc=', 'Content-Length': '0'} 2024-04-13T22:10:27.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c16fccd42034ee6-00661b02d3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:27.997 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:27.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:10:27.997 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:27.997 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:27.997 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:27.998 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:27.998 DEBUG:boto:Method: GET 2024-04-13T22:10:27.998 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:10:27.998 DEBUG:boto:Data: 2024-04-13T22:10:27.998 DEBUG:boto:Headers: {} 2024-04-13T22:10:27.998 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:27.998 DEBUG:boto:Port: 8001 2024-04-13T22:10:27.998 DEBUG:boto:Params: {} 2024-04-13T22:10:27.998 DEBUG:boto:Token: None 2024-04-13T22:10:27.998 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:27 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:10:27.999 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2HFzwxg7vWIIryjILqFKWNiK1Fc= 2024-04-13T22:10:27.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2HFzwxg7vWIIryjILqFKWNiK1Fc=', 'Content-Length': '0'} 2024-04-13T22:10:28.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db417b77e338ab9-00661b02d3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.046 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.047 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.047 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.047 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.047 DEBUG:boto:Method: GET 2024-04-13T22:10:28.047 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.047 DEBUG:boto:Data: 2024-04-13T22:10:28.047 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.047 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.047 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.047 DEBUG:boto:Params: {} 2024-04-13T22:10:28.048 DEBUG:boto:Token: None 2024-04-13T22:10:28.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JzikQ4QywJ6q6tiCOFqGruuxLc4= 2024-04-13T22:10:28.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JzikQ4QywJ6q6tiCOFqGruuxLc4=', 'Content-Length': '0'} 2024-04-13T22:10:28.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000901905677501e5c4-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.050 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.050 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.050 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.050 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.051 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.051 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.051 DEBUG:boto:Data: 2024-04-13T22:10:28.051 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.051 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.051 DEBUG:boto:Params: {} 2024-04-13T22:10:28.051 DEBUG:boto:Token: None 2024-04-13T22:10:28.051 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.051 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+9QJwG/Jy90ce2sO5UAz3PM2E8k= 2024-04-13T22:10:28.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+9QJwG/Jy90ce2sO5UAz3PM2E8k=', 'Content-Length': '0'} 2024-04-13T22:10:28.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2736c6465d82ae8-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.054 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.054 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.054 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.054 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.054 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.054 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.054 DEBUG:boto:Data: 2024-04-13T22:10:28.054 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.055 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.055 DEBUG:boto:Params: {} 2024-04-13T22:10:28.055 DEBUG:boto:Token: None 2024-04-13T22:10:28.055 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+9QJwG/Jy90ce2sO5UAz3PM2E8k= 2024-04-13T22:10:28.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+9QJwG/Jy90ce2sO5UAz3PM2E8k=', 'Content-Length': '0'} 2024-04-13T22:10:28.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a85dd356ae5114a-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:10:28.057 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.057 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.058 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.058 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.058 DEBUG:boto:Method: GET 2024-04-13T22:10:28.058 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.058 DEBUG:boto:Data: 2024-04-13T22:10:28.058 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.058 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.058 DEBUG:boto:Params: {} 2024-04-13T22:10:28.058 DEBUG:boto:Token: None 2024-04-13T22:10:28.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JzikQ4QywJ6q6tiCOFqGruuxLc4= 2024-04-13T22:10:28.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JzikQ4QywJ6q6tiCOFqGruuxLc4=', 'Content-Length': '0'} 2024-04-13T22:10:28.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004877680b0cc3051b-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.062 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.062 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.062 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.062 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:10:28.063 DEBUG:boto:Method: GET 2024-04-13T22:10:28.063 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.063 DEBUG:boto:Data: 2024-04-13T22:10:28.063 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.063 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.063 DEBUG:boto:Params: {} 2024-04-13T22:10:28.063 DEBUG:boto:Token: None 2024-04-13T22:10:28.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:10:28.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JzikQ4QywJ6q6tiCOFqGruuxLc4= 2024-04-13T22:10:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JzikQ4QywJ6q6tiCOFqGruuxLc4=', 'Content-Length': '0'} 2024-04-13T22:10:28.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f33079ca1a5ec09-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.066 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.066 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.067 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.067 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.067 DEBUG:boto:Method: GET 2024-04-13T22:10:28.067 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:10:28.067 DEBUG:boto:Data: 2024-04-13T22:10:28.067 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.067 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.068 DEBUG:boto:Params: {} 2024-04-13T22:10:28.068 DEBUG:boto:Token: None 2024-04-13T22:10:28.068 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.068 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e058fad2e215f09a-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.070 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.071 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.071 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.071 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.071 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.071 DEBUG:boto:Method: GET 2024-04-13T22:10:28.071 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:10:28.071 DEBUG:boto:Data: 2024-04-13T22:10:28.071 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.072 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.072 DEBUG:boto:Params: {} 2024-04-13T22:10:28.072 DEBUG:boto:Token: None 2024-04-13T22:10:28.072 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b99687a7add9d2-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.074 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.075 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.075 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.075 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.075 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:10:28.075 DEBUG:boto:Data: 2024-04-13T22:10:28.075 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.075 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.075 DEBUG:boto:Params: {} 2024-04-13T22:10:28.076 DEBUG:boto:Token: None 2024-04-13T22:10:28.076 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a 2024-04-13T22:10:28.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yUJP76XsqZE9w206iOKlt+l6pRM= 2024-04-13T22:10:28.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yUJP76XsqZE9w206iOKlt+l6pRM=', 'Content-Length': '0'} 2024-04-13T22:10:28.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031aec6834c1b62c5-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.078 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.078 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.078 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.078 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:10:28.078 DEBUG:boto:Data: 2024-04-13T22:10:28.078 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.078 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.078 DEBUG:boto:Params: {} 2024-04-13T22:10:28.079 DEBUG:boto:Token: None 2024-04-13T22:10:28.079 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a 2024-04-13T22:10:28.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yUJP76XsqZE9w206iOKlt+l6pRM= 2024-04-13T22:10:28.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yUJP76XsqZE9w206iOKlt+l6pRM=', 'Content-Length': '0'} 2024-04-13T22:10:28.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e10dfff339cc21-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:10:28.081 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.081 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.081 DEBUG:boto:Method: GET 2024-04-13T22:10:28.081 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:10:28.081 DEBUG:boto:Data: 2024-04-13T22:10:28.081 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.081 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.082 DEBUG:boto:Params: {} 2024-04-13T22:10:28.082 DEBUG:boto:Token: None 2024-04-13T22:10:28.082 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a 2024-04-13T22:10:28.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zQqI56qnbXcF0ftPVANezQ6lFfo= 2024-04-13T22:10:28.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zQqI56qnbXcF0ftPVANezQ6lFfo=', 'Content-Length': '0'} 2024-04-13T22:10:28.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0f5a54b5aab19d-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.084 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:10:28.084 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:10:28.085 DEBUG:boto:Method: GET 2024-04-13T22:10:28.085 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:10:28.085 DEBUG:boto:Data: 2024-04-13T22:10:28.085 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.085 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.085 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.085 DEBUG:boto:Params: {} 2024-04-13T22:10:28.085 DEBUG:boto:Token: None 2024-04-13T22:10:28.085 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/a 2024-04-13T22:10:28.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zQqI56qnbXcF0ftPVANezQ6lFfo= 2024-04-13T22:10:28.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zQqI56qnbXcF0ftPVANezQ6lFfo=', 'Content-Length': '0'} 2024-04-13T22:10:28.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f3bd3fe04d2a02-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.130 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.131 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.131 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.131 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.131 DEBUG:boto:Method: GET 2024-04-13T22:10:28.131 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:10:28.131 DEBUG:boto:Data: 2024-04-13T22:10:28.131 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.131 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.131 DEBUG:boto:Params: {} 2024-04-13T22:10:28.132 DEBUG:boto:Token: None 2024-04-13T22:10:28.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f158fb56cc59f5d8-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.134 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.135 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.135 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.135 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.135 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:10:28.135 DEBUG:boto:Method: GET 2024-04-13T22:10:28.135 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:10:28.135 DEBUG:boto:Data: 2024-04-13T22:10:28.135 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.136 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.136 DEBUG:boto:Params: {} 2024-04-13T22:10:28.136 DEBUG:boto:Token: None 2024-04-13T22:10:28.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e50e79689fbd66-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.138 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:10:28.139 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.139 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.139 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.139 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.139 DEBUG:boto:Method: GET 2024-04-13T22:10:28.139 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.139 DEBUG:boto:Data: 2024-04-13T22:10:28.139 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.140 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.140 DEBUG:boto:Params: {} 2024-04-13T22:10:28.140 DEBUG:boto:Token: None 2024-04-13T22:10:28.140 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk= 2024-04-13T22:10:28.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk=', 'Content-Length': '0'} 2024-04-13T22:10:28.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009963e7127622b25-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.182 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.182 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.182 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.183 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.183 DEBUG:boto:Method: GET 2024-04-13T22:10:28.183 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.183 DEBUG:boto:Data: 2024-04-13T22:10:28.183 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.183 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.183 DEBUG:boto:Params: {} 2024-04-13T22:10:28.183 DEBUG:boto:Token: None 2024-04-13T22:10:28.184 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk= 2024-04-13T22:10:28.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk=', 'Content-Length': '0'} 2024-04-13T22:10:28.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bcca8c4a386aaa5-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.186 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.186 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.186 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.186 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.186 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.186 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.186 DEBUG:boto:Data: 2024-04-13T22:10:28.186 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.186 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.187 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.187 DEBUG:boto:Params: {} 2024-04-13T22:10:28.187 DEBUG:boto:Token: None 2024-04-13T22:10:28.187 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.187 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vp7c1ycQFze/bpEHWsZzu0+V5qc= 2024-04-13T22:10:28.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vp7c1ycQFze/bpEHWsZzu0+V5qc=', 'Content-Length': '0'} 2024-04-13T22:10:28.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002888410c70acdffc-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.189 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.189 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.189 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.189 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.190 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.190 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.190 DEBUG:boto:Data: 2024-04-13T22:10:28.190 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.190 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.190 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.190 DEBUG:boto:Params: {} 2024-04-13T22:10:28.190 DEBUG:boto:Token: None 2024-04-13T22:10:28.190 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.191 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vp7c1ycQFze/bpEHWsZzu0+V5qc= 2024-04-13T22:10:28.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vp7c1ycQFze/bpEHWsZzu0+V5qc=', 'Content-Length': '0'} 2024-04-13T22:10:28.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2dabe3e22988329-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:10:28.192 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.193 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.193 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.193 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.193 DEBUG:boto:Method: GET 2024-04-13T22:10:28.193 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.193 DEBUG:boto:Data: 2024-04-13T22:10:28.193 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.193 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.193 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.193 DEBUG:boto:Params: {} 2024-04-13T22:10:28.193 DEBUG:boto:Token: None 2024-04-13T22:10:28.194 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.194 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk= 2024-04-13T22:10:28.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk=', 'Content-Length': '0'} 2024-04-13T22:10:28.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b865155287242bad-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.196 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.196 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.196 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.196 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:10:28.196 DEBUG:boto:Method: GET 2024-04-13T22:10:28.196 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.196 DEBUG:boto:Data: 2024-04-13T22:10:28.196 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.197 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.197 DEBUG:boto:Params: {} 2024-04-13T22:10:28.197 DEBUG:boto:Token: None 2024-04-13T22:10:28.197 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:10:28.197 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk= 2024-04-13T22:10:28.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v8BwGflHCFcCQUuwgawFYrBuYCk=', 'Content-Length': '0'} 2024-04-13T22:10:28.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ef4c743b4bbd43f-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.199 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.199 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.199 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.200 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.200 DEBUG:boto:Method: GET 2024-04-13T22:10:28.200 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:10:28.200 DEBUG:boto:Data: 2024-04-13T22:10:28.200 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.200 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.200 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.200 DEBUG:boto:Params: {} 2024-04-13T22:10:28.200 DEBUG:boto:Token: None 2024-04-13T22:10:28.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.201 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d73b513f91d5cb4-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.203 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.203 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.204 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.204 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.204 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.204 DEBUG:boto:Method: GET 2024-04-13T22:10:28.204 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:10:28.204 DEBUG:boto:Data: 2024-04-13T22:10:28.204 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.204 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.204 DEBUG:boto:Params: {} 2024-04-13T22:10:28.204 DEBUG:boto:Token: None 2024-04-13T22:10:28.205 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.205 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b9e79e0b42dcc6e-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.207 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.207 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.208 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.208 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.208 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:10:28.208 DEBUG:boto:Data: 2024-04-13T22:10:28.208 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.208 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.208 DEBUG:boto:Params: {} 2024-04-13T22:10:28.208 DEBUG:boto:Token: None 2024-04-13T22:10:28.208 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b 2024-04-13T22:10:28.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xK8EVUO+he8R+vELcBYasfL5JqA= 2024-04-13T22:10:28.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xK8EVUO+he8R+vELcBYasfL5JqA=', 'Content-Length': '0'} 2024-04-13T22:10:28.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3011a90aefe88b-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.211 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.211 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.211 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.211 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:10:28.211 DEBUG:boto:Data: 2024-04-13T22:10:28.211 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.211 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.211 DEBUG:boto:Params: {} 2024-04-13T22:10:28.211 DEBUG:boto:Token: None 2024-04-13T22:10:28.212 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b 2024-04-13T22:10:28.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xK8EVUO+he8R+vELcBYasfL5JqA= 2024-04-13T22:10:28.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xK8EVUO+he8R+vELcBYasfL5JqA=', 'Content-Length': '0'} 2024-04-13T22:10:28.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008761b7a7a33c2353-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:10:28.213 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.214 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.214 DEBUG:boto:Method: GET 2024-04-13T22:10:28.214 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:10:28.214 DEBUG:boto:Data: 2024-04-13T22:10:28.214 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.214 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.214 DEBUG:boto:Params: {} 2024-04-13T22:10:28.214 DEBUG:boto:Token: None 2024-04-13T22:10:28.215 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b 2024-04-13T22:10:28.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AxorIoih9bNgu/IpJwl/pABVxR0= 2024-04-13T22:10:28.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AxorIoih9bNgu/IpJwl/pABVxR0=', 'Content-Length': '0'} 2024-04-13T22:10:28.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009017fe076defa403-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.258 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:10:28.259 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:10:28.259 DEBUG:boto:Method: GET 2024-04-13T22:10:28.259 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:10:28.259 DEBUG:boto:Data: 2024-04-13T22:10:28.259 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.259 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.259 DEBUG:boto:Params: {} 2024-04-13T22:10:28.260 DEBUG:boto:Token: None 2024-04-13T22:10:28.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/b 2024-04-13T22:10:28.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AxorIoih9bNgu/IpJwl/pABVxR0= 2024-04-13T22:10:28.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AxorIoih9bNgu/IpJwl/pABVxR0=', 'Content-Length': '0'} 2024-04-13T22:10:28.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d292e15007d7739-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.262 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.262 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.262 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.262 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.262 DEBUG:boto:Method: GET 2024-04-13T22:10:28.262 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:10:28.263 DEBUG:boto:Data: 2024-04-13T22:10:28.263 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.263 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.263 DEBUG:boto:Params: {} 2024-04-13T22:10:28.263 DEBUG:boto:Token: None 2024-04-13T22:10:28.263 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e192e9f76234bfe9-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.265 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.266 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.266 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.266 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.266 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:10:28.266 DEBUG:boto:Method: GET 2024-04-13T22:10:28.266 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:10:28.266 DEBUG:boto:Data: 2024-04-13T22:10:28.266 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.267 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.267 DEBUG:boto:Params: {} 2024-04-13T22:10:28.267 DEBUG:boto:Token: None 2024-04-13T22:10:28.267 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000811fedf82a0301ad-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.269 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:10:28.270 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.270 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.270 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.270 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.270 DEBUG:boto:Method: GET 2024-04-13T22:10:28.270 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.270 DEBUG:boto:Data: 2024-04-13T22:10:28.271 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.271 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.271 DEBUG:boto:Params: {} 2024-04-13T22:10:28.271 DEBUG:boto:Token: None 2024-04-13T22:10:28.271 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM= 2024-04-13T22:10:28.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM=', 'Content-Length': '0'} 2024-04-13T22:10:28.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e3eb92d56f5e21-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.273 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.273 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.273 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.273 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.273 DEBUG:boto:Method: GET 2024-04-13T22:10:28.274 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.274 DEBUG:boto:Data: 2024-04-13T22:10:28.274 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.274 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.274 DEBUG:boto:Params: {} 2024-04-13T22:10:28.274 DEBUG:boto:Token: None 2024-04-13T22:10:28.274 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM= 2024-04-13T22:10:28.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM=', 'Content-Length': '0'} 2024-04-13T22:10:28.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067080007e0558a97-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.277 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.277 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.277 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.277 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.277 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.277 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.277 DEBUG:boto:Data: 2024-04-13T22:10:28.277 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.277 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.277 DEBUG:boto:Params: {} 2024-04-13T22:10:28.277 DEBUG:boto:Token: None 2024-04-13T22:10:28.278 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l71WusqUFeU/QhkSIrvo4Ox8Qwo= 2024-04-13T22:10:28.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l71WusqUFeU/QhkSIrvo4Ox8Qwo=', 'Content-Length': '0'} 2024-04-13T22:10:28.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d02f36985836da-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.280 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.280 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.280 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.280 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.280 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.280 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.280 DEBUG:boto:Data: 2024-04-13T22:10:28.280 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.281 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.281 DEBUG:boto:Params: {} 2024-04-13T22:10:28.281 DEBUG:boto:Token: None 2024-04-13T22:10:28.281 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l71WusqUFeU/QhkSIrvo4Ox8Qwo= 2024-04-13T22:10:28.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l71WusqUFeU/QhkSIrvo4Ox8Qwo=', 'Content-Length': '0'} 2024-04-13T22:10:28.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43046fe4a4c8dad-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:10:28.283 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.283 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.283 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.283 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.283 DEBUG:boto:Method: GET 2024-04-13T22:10:28.283 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.283 DEBUG:boto:Data: 2024-04-13T22:10:28.283 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.284 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.284 DEBUG:boto:Params: {} 2024-04-13T22:10:28.284 DEBUG:boto:Token: None 2024-04-13T22:10:28.284 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM= 2024-04-13T22:10:28.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM=', 'Content-Length': '0'} 2024-04-13T22:10:28.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b54010722d0e96a-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.286 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.286 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.286 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.286 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:10:28.286 DEBUG:boto:Method: GET 2024-04-13T22:10:28.286 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.286 DEBUG:boto:Data: 2024-04-13T22:10:28.287 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.287 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.287 DEBUG:boto:Params: {} 2024-04-13T22:10:28.287 DEBUG:boto:Token: None 2024-04-13T22:10:28.287 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:10:28.287 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM= 2024-04-13T22:10:28.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BJ6DMg5QHrI43IlYDmKMAcOU8DM=', 'Content-Length': '0'} 2024-04-13T22:10:28.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2b4d3d830b29cb-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.289 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.289 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.289 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.289 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.289 DEBUG:boto:Method: GET 2024-04-13T22:10:28.289 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:10:28.289 DEBUG:boto:Data: 2024-04-13T22:10:28.290 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.290 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.290 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.290 DEBUG:boto:Params: {} 2024-04-13T22:10:28.290 DEBUG:boto:Token: None 2024-04-13T22:10:28.290 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.290 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546cd622190aa905-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.292 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.293 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.293 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.293 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.293 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.293 DEBUG:boto:Method: GET 2024-04-13T22:10:28.293 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:10:28.293 DEBUG:boto:Data: 2024-04-13T22:10:28.293 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.294 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.294 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.294 DEBUG:boto:Params: {} 2024-04-13T22:10:28.294 DEBUG:boto:Token: None 2024-04-13T22:10:28.294 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.294 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad01a9c9dc71a12-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.297 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.297 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.297 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.297 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.297 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:10:28.297 DEBUG:boto:Data: 2024-04-13T22:10:28.298 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.298 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.298 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.298 DEBUG:boto:Params: {} 2024-04-13T22:10:28.298 DEBUG:boto:Token: None 2024-04-13T22:10:28.298 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c 2024-04-13T22:10:28.298 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QUVcBAHw9hbv4hx4xvXeO99RZwE= 2024-04-13T22:10:28.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QUVcBAHw9hbv4hx4xvXeO99RZwE=', 'Content-Length': '0'} 2024-04-13T22:10:28.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75606b72450f2da-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.300 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.300 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.300 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.300 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:10:28.300 DEBUG:boto:Data: 2024-04-13T22:10:28.300 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.301 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.301 DEBUG:boto:Params: {} 2024-04-13T22:10:28.301 DEBUG:boto:Token: None 2024-04-13T22:10:28.301 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c 2024-04-13T22:10:28.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QUVcBAHw9hbv4hx4xvXeO99RZwE= 2024-04-13T22:10:28.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QUVcBAHw9hbv4hx4xvXeO99RZwE=', 'Content-Length': '0'} 2024-04-13T22:10:28.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973bb2334374440e-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:10:28.303 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.303 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.303 DEBUG:boto:Method: GET 2024-04-13T22:10:28.304 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:10:28.304 DEBUG:boto:Data: 2024-04-13T22:10:28.304 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.304 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.304 DEBUG:boto:Params: {} 2024-04-13T22:10:28.304 DEBUG:boto:Token: None 2024-04-13T22:10:28.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c 2024-04-13T22:10:28.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L5mFUJZnp09k1rox2rT1o8yQ0Lc= 2024-04-13T22:10:28.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L5mFUJZnp09k1rox2rT1o8yQ0Lc=', 'Content-Length': '0'} 2024-04-13T22:10:28.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000024a71edc1d1bb26-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.306 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:10:28.306 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:10:28.306 DEBUG:boto:Method: GET 2024-04-13T22:10:28.307 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:10:28.307 DEBUG:boto:Data: 2024-04-13T22:10:28.307 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.307 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.307 DEBUG:boto:Params: {} 2024-04-13T22:10:28.307 DEBUG:boto:Token: None 2024-04-13T22:10:28.307 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/c 2024-04-13T22:10:28.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L5mFUJZnp09k1rox2rT1o8yQ0Lc= 2024-04-13T22:10:28.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L5mFUJZnp09k1rox2rT1o8yQ0Lc=', 'Content-Length': '0'} 2024-04-13T22:10:28.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12d2f92566a7ac6-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.309 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.309 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.309 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.310 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.310 DEBUG:boto:Method: GET 2024-04-13T22:10:28.310 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:10:28.310 DEBUG:boto:Data: 2024-04-13T22:10:28.310 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.310 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.310 DEBUG:boto:Params: {} 2024-04-13T22:10:28.310 DEBUG:boto:Token: None 2024-04-13T22:10:28.310 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.311 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ded8ce27dc2d84a2-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.313 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.313 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.313 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.313 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.313 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:10:28.313 DEBUG:boto:Method: GET 2024-04-13T22:10:28.314 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:10:28.314 DEBUG:boto:Data: 2024-04-13T22:10:28.314 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.314 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.314 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.314 DEBUG:boto:Params: {} 2024-04-13T22:10:28.314 DEBUG:boto:Token: None 2024-04-13T22:10:28.314 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38f57b6a1547336-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.317 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:10:28.317 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.317 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.317 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.318 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.318 DEBUG:boto:Method: GET 2024-04-13T22:10:28.318 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.318 DEBUG:boto:Data: 2024-04-13T22:10:28.318 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.318 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.318 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.318 DEBUG:boto:Params: {} 2024-04-13T22:10:28.318 DEBUG:boto:Token: None 2024-04-13T22:10:28.318 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0= 2024-04-13T22:10:28.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0=', 'Content-Length': '0'} 2024-04-13T22:10:28.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf801563e627480-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.320 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.320 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.321 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.321 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.321 DEBUG:boto:Method: GET 2024-04-13T22:10:28.321 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.321 DEBUG:boto:Data: 2024-04-13T22:10:28.321 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.321 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.321 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.321 DEBUG:boto:Params: {} 2024-04-13T22:10:28.321 DEBUG:boto:Token: None 2024-04-13T22:10:28.321 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.322 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0= 2024-04-13T22:10:28.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0=', 'Content-Length': '0'} 2024-04-13T22:10:28.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da8313b83d103f3-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.324 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.324 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.324 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.324 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.324 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.324 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.324 DEBUG:boto:Data: 2024-04-13T22:10:28.324 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.324 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.324 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.324 DEBUG:boto:Params: {} 2024-04-13T22:10:28.325 DEBUG:boto:Token: None 2024-04-13T22:10:28.325 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.325 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lmsd98yuUIq3iq6Y2I357ULbKjM= 2024-04-13T22:10:28.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lmsd98yuUIq3iq6Y2I357ULbKjM=', 'Content-Length': '0'} 2024-04-13T22:10:28.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1217a81956114ef-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.327 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.327 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.327 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.327 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.327 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.327 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.327 DEBUG:boto:Data: 2024-04-13T22:10:28.327 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.328 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.328 DEBUG:boto:Params: {} 2024-04-13T22:10:28.328 DEBUG:boto:Token: None 2024-04-13T22:10:28.328 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lmsd98yuUIq3iq6Y2I357ULbKjM= 2024-04-13T22:10:28.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lmsd98yuUIq3iq6Y2I357ULbKjM=', 'Content-Length': '0'} 2024-04-13T22:10:28.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc9f5bec124e616-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:10:28.331 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.331 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.331 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.331 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.331 DEBUG:boto:Method: GET 2024-04-13T22:10:28.331 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.331 DEBUG:boto:Data: 2024-04-13T22:10:28.331 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.332 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.332 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.332 DEBUG:boto:Params: {} 2024-04-13T22:10:28.332 DEBUG:boto:Token: None 2024-04-13T22:10:28.332 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0= 2024-04-13T22:10:28.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0=', 'Content-Length': '0'} 2024-04-13T22:10:28.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e146842db40936dc-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.335 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.335 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.335 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.335 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:10:28.335 DEBUG:boto:Method: GET 2024-04-13T22:10:28.335 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.336 DEBUG:boto:Data: 2024-04-13T22:10:28.336 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.336 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.336 DEBUG:boto:Params: {} 2024-04-13T22:10:28.336 DEBUG:boto:Token: None 2024-04-13T22:10:28.337 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:10:28.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0= 2024-04-13T22:10:28.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+EPXT8jAcnGuA4pbpFV1iBOlPA0=', 'Content-Length': '0'} 2024-04-13T22:10:28.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000689a72d5456a25c-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.339 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.339 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.339 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.339 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.339 DEBUG:boto:Method: GET 2024-04-13T22:10:28.339 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:10:28.339 DEBUG:boto:Data: 2024-04-13T22:10:28.339 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.339 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.339 DEBUG:boto:Params: {} 2024-04-13T22:10:28.340 DEBUG:boto:Token: None 2024-04-13T22:10:28.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c95376f3e818ab4-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.342 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.343 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.343 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.343 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.343 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.343 DEBUG:boto:Method: GET 2024-04-13T22:10:28.343 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:10:28.343 DEBUG:boto:Data: 2024-04-13T22:10:28.343 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.343 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.343 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.343 DEBUG:boto:Params: {} 2024-04-13T22:10:28.344 DEBUG:boto:Token: None 2024-04-13T22:10:28.344 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3409c5e20b4b659-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.386 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.387 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.387 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.387 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.387 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:10:28.387 DEBUG:boto:Data: 2024-04-13T22:10:28.387 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.388 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.388 DEBUG:boto:Params: {} 2024-04-13T22:10:28.388 DEBUG:boto:Token: None 2024-04-13T22:10:28.388 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d 2024-04-13T22:10:28.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:55pONhyzApk5b4RWPEBbHzhCNJ4= 2024-04-13T22:10:28.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:55pONhyzApk5b4RWPEBbHzhCNJ4=', 'Content-Length': '0'} 2024-04-13T22:10:28.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e973fcd3d91c15-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.390 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.390 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.390 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.390 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:10:28.390 DEBUG:boto:Data: 2024-04-13T22:10:28.390 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.391 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.391 DEBUG:boto:Params: {} 2024-04-13T22:10:28.391 DEBUG:boto:Token: None 2024-04-13T22:10:28.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d 2024-04-13T22:10:28.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:55pONhyzApk5b4RWPEBbHzhCNJ4= 2024-04-13T22:10:28.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:55pONhyzApk5b4RWPEBbHzhCNJ4=', 'Content-Length': '0'} 2024-04-13T22:10:28.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ddf0eeb873b548-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:10:28.393 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.393 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.393 DEBUG:boto:Method: GET 2024-04-13T22:10:28.393 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:10:28.393 DEBUG:boto:Data: 2024-04-13T22:10:28.393 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.394 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.394 DEBUG:boto:Params: {} 2024-04-13T22:10:28.394 DEBUG:boto:Token: None 2024-04-13T22:10:28.394 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d 2024-04-13T22:10:28.394 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7BagF5ESDFX9aBrnN9V15HLrlcU= 2024-04-13T22:10:28.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7BagF5ESDFX9aBrnN9V15HLrlcU=', 'Content-Length': '0'} 2024-04-13T22:10:28.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951fc96bdf44e1b9-00661b02d4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.396 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:10:28.396 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:10:28.396 DEBUG:boto:Method: GET 2024-04-13T22:10:28.396 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:10:28.397 DEBUG:boto:Data: 2024-04-13T22:10:28.397 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.397 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.397 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.397 DEBUG:boto:Params: {} 2024-04-13T22:10:28.397 DEBUG:boto:Token: None 2024-04-13T22:10:28.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/d 2024-04-13T22:10:28.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7BagF5ESDFX9aBrnN9V15HLrlcU= 2024-04-13T22:10:28.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7BagF5ESDFX9aBrnN9V15HLrlcU=', 'Content-Length': '0'} 2024-04-13T22:10:28.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a664bc5fa80557-00661b02d4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.399 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.400 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.400 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.400 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.400 DEBUG:boto:Method: GET 2024-04-13T22:10:28.400 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:10:28.400 DEBUG:boto:Data: 2024-04-13T22:10:28.400 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:10:28.400 DEBUG:boto:Port: 8001 2024-04-13T22:10:28.400 DEBUG:boto:Params: {} 2024-04-13T22:10:28.401 DEBUG:boto:Token: None 2024-04-13T22:10:28.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034bd15028069139d-00661b02d4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.403 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.404 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.404 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.404 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.404 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:10:28.404 DEBUG:boto:Method: GET 2024-04-13T22:10:28.404 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:10:28.404 DEBUG:boto:Data: 2024-04-13T22:10:28.404 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.404 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.404 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.405 DEBUG:boto:Params: {} 2024-04-13T22:10:28.405 DEBUG:boto:Token: None 2024-04-13T22:10:28.405 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM= 2024-04-13T22:10:28.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pjd+VJv9VY4hSIuY4VN2P9NP8LM=', 'Content-Length': '0'} 2024-04-13T22:10:28.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be80fbe312310adc-00661b02d4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.407 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:10:28.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-58 zones={a2, a1} 2024-04-13T22:10:28.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:10:28.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:10:28.408 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:10:28.456 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:10:28.457 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:10:28.495 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:10:28.496 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-58 2024-04-13T22:10:28.499 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 4 2024-04-13T22:10:28.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:10:28.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:10:28.539 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:10:28.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:10:28.588 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:10:28.625 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:10:28.625 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-58 2024-04-13T22:10:28.626 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 4 2024-04-13T22:10:28.653 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.653 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.653 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.653 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:28.653 DEBUG:boto:Data: 2024-04-13T22:10:28.653 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.654 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.654 DEBUG:boto:Params: {} 2024-04-13T22:10:28.654 DEBUG:boto:Token: None 2024-04-13T22:10:28.654 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.654 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4= 2024-04-13T22:10:28.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4=', 'Content-Length': '0'} 2024-04-13T22:10:28.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000963088a0bff44723-00661b02d4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.656 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:10:28.657 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:10:28.657 DEBUG:boto:Method: PUT 2024-04-13T22:10:28.657 DEBUG:boto:Path: /vwnfns-58/e 2024-04-13T22:10:28.657 DEBUG:boto:Data: 2024-04-13T22:10:28.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:28.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.657 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.657 DEBUG:boto:Params: {} 2024-04-13T22:10:28.657 DEBUG:boto:Token: None 2024-04-13T22:10:28.658 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/e 2024-04-13T22:10:28.658 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rJExrFKrQzwngGCjTcLC7svzVvo= 2024-04-13T22:10:28.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rJExrFKrQzwngGCjTcLC7svzVvo='} 2024-04-13T22:10:28.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9c0c71d57a072c0-00661b02d4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.663 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.663 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.663 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.663 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:28.663 DEBUG:boto:Data: 2024-04-13T22:10:28.663 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.663 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.663 DEBUG:boto:Params: {} 2024-04-13T22:10:28.663 DEBUG:boto:Token: None 2024-04-13T22:10:28.664 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4= 2024-04-13T22:10:28.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4=', 'Content-Length': '0'} 2024-04-13T22:10:28.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a582ed13d7526eb-00661b02d4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.666 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:10:28.666 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:10:28.666 DEBUG:boto:Method: PUT 2024-04-13T22:10:28.666 DEBUG:boto:Path: /vwnfns-58/f 2024-04-13T22:10:28.666 DEBUG:boto:Data: 2024-04-13T22:10:28.666 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:28.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.666 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.666 DEBUG:boto:Params: {} 2024-04-13T22:10:28.666 DEBUG:boto:Token: None 2024-04-13T22:10:28.667 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/f 2024-04-13T22:10:28.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R1ZzKHChyA5WUHujF6yF7VLd0Co= 2024-04-13T22:10:28.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R1ZzKHChyA5WUHujF6yF7VLd0Co='} 2024-04-13T22:10:28.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004608dbd4470e55b6-00661b02d4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.711 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.711 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.711 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.711 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:28.711 DEBUG:boto:Data: 2024-04-13T22:10:28.711 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.711 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.711 DEBUG:boto:Params: {} 2024-04-13T22:10:28.712 DEBUG:boto:Token: None 2024-04-13T22:10:28.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4= 2024-04-13T22:10:28.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4=', 'Content-Length': '0'} 2024-04-13T22:10:28.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012b56acd3e185cbe-00661b02d4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.714 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:10:28.714 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:10:28.714 DEBUG:boto:Method: PUT 2024-04-13T22:10:28.714 DEBUG:boto:Path: /vwnfns-58/g 2024-04-13T22:10:28.714 DEBUG:boto:Data: 2024-04-13T22:10:28.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:28.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.715 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.715 DEBUG:boto:Params: {} 2024-04-13T22:10:28.715 DEBUG:boto:Token: None 2024-04-13T22:10:28.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/g 2024-04-13T22:10:28.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aOcX0ATuhbiLo7iQ7DFvBzCf2+4= 2024-04-13T22:10:28.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aOcX0ATuhbiLo7iQ7DFvBzCf2+4='} 2024-04-13T22:10:28.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b78004a58d8a00c-00661b02d4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.720 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:10:28.720 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:10:28.720 DEBUG:boto:Method: HEAD 2024-04-13T22:10:28.720 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:10:28.720 DEBUG:boto:Data: 2024-04-13T22:10:28.720 DEBUG:boto:Headers: {} 2024-04-13T22:10:28.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.720 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.720 DEBUG:boto:Params: {} 2024-04-13T22:10:28.721 DEBUG:boto:Token: None 2024-04-13T22:10:28.721 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/ 2024-04-13T22:10:28.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4= 2024-04-13T22:10:28.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hf3t7FNyppAUaLp/WuCjLgAil4=', 'Content-Length': '0'} 2024-04-13T22:10:28.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89fe71111793811-00661b02d4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.722 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:10:28.723 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:10:28.723 DEBUG:boto:Method: PUT 2024-04-13T22:10:28.723 DEBUG:boto:Path: /vwnfns-58/h 2024-04-13T22:10:28.723 DEBUG:boto:Data: 2024-04-13T22:10:28.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:10:28.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:10:28.723 DEBUG:boto:Port: 8000 2024-04-13T22:10:28.723 DEBUG:boto:Params: {} 2024-04-13T22:10:28.723 DEBUG:boto:Token: None 2024-04-13T22:10:28.723 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:10:28 GMT /vwnfns-58/h 2024-04-13T22:10:28.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kYZLKy8xtBeQXfhlDUFfUTGDxpg= 2024-04-13T22:10:28.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:10:28 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kYZLKy8xtBeQXfhlDUFfUTGDxpg='} 2024-04-13T22:10:28.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a773a8b896c9b35-00661b02d4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:10:28.767 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-58 2024-04-13T22:10:28.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:10:28.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:10:28.767 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:10:28.822 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:10:28.822 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:10:28.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:28.854+0000 7f794d35d780 1 waiting to reach latest gen 2, on 0.. 2024-04-13T22:10:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:33.855+0000 7f794d35d780 1 waiting to reach latest gen 2, on 0.. 2024-04-13T22:10:38.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:38.856+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:10:38.858 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , ] 2024-04-13T22:10:38.858 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:10:43.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:43.860+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:10:43.862 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , ] 2024-04-13T22:10:43.862 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:10:48.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:48.860+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:10:48.864 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , ] 2024-04-13T22:10:48.864 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:10:53.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:53.861+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:10:53.864 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , ] 2024-04-13T22:10:53.864 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:10:58.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:10:58.862+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:10:58.865 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:10:58.865 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:03.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:03.864+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:03.866 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:03.866 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:08.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:08.865+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:08.866 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:08.866 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:13.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:13.866+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:13.867 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:13.867 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:18.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:18.867+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:18.869 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:18.869 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:23.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:23.867+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:23.870 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:23.870 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:28.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:28.868+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:28.870 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:28.870 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:33.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:33.868+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:33.871 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:33.871 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:38.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:38.869+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:38.872 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:38.872 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:43.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:43.871+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:43.873 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:43.873 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:48.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:48.872+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:48.874 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:48.874 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:53.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:53.873+0000 7f794d35d780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:53.874 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, , ] 2024-04-13T22:11:53.875 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:55.127 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:11:55.125+0000 7f5e93db2700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T22:11:58.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:58.872+0000 7f794d35d780 1 bucket sync caught up with source: 2024-04-13T22:11:58.875 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:58.875 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.26.6, 00000000002.38.6, 00000000002.37.6] 2024-04-13T22:11:58.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:58.872+0000 7f794d35d780 0 bucket checkpoint complete 2024-04-13T22:11:58.892 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-58 2024-04-13T22:11:58.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:11:58.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:11:58.893 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:11:58.953 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:11:58.954 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:11:58.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:11:58.989+0000 7f72fdd08780 1 waiting for incremental sync to catch up: 2024-04-13T22:11:58.991 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.1074.6, , , 00000000001.1004.6, 00000000001.988.6, , , 00000000001.747.6, , 00000000002.2627.6] 2024-04-13T22:11:58.991 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.1083.6, , , 00000000001.1004.6, 00000000001.988.6, , , 00000000002.756.6, , 00000000002.2627.6] 2024-04-13T22:12:03.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:03.990+0000 7f72fdd08780 1 bucket sync caught up with source: 2024-04-13T22:12:03.993 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.1083.6, , , 00000000001.1004.6, 00000000001.988.6, , , 00000000002.756.6, , 00000000002.2627.6] 2024-04-13T22:12:03.993 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.1083.6, , , 00000000001.1004.6, 00000000001.988.6, , , 00000000002.756.6, , 00000000002.2627.6] 2024-04-13T22:12:03.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:03.990+0000 7f72fdd08780 0 bucket checkpoint complete 2024-04-13T22:12:04.010 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-58 zones={a2, a1} 2024-04-13T22:12:04.010 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.010 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.010 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.010 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:12:04.011 DEBUG:boto:Data: 2024-04-13T22:12:04.011 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.011 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.011 DEBUG:boto:Params: {} 2024-04-13T22:12:04.011 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:12:04.012 DEBUG:boto:Token: None 2024-04-13T22:12:04.012 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LRInCI+v9Tbod4cJyqlIJNT+fZs= 2024-04-13T22:12:04.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LRInCI+v9Tbod4cJyqlIJNT+fZs=', 'Content-Length': '0'} 2024-04-13T22:12:04.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091728e2129da5945-00661b0334-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.017 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.017 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.018 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.018 DEBUG:boto:Path: /vwnfns-58/ 2024-04-13T22:12:04.018 DEBUG:boto:Data: 2024-04-13T22:12:04.018 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.018 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.018 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.018 DEBUG:boto:Params: {} 2024-04-13T22:12:04.018 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:12:04.018 DEBUG:boto:Token: None 2024-04-13T22:12:04.019 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.019 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LRInCI+v9Tbod4cJyqlIJNT+fZs= 2024-04-13T22:12:04.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LRInCI+v9Tbod4cJyqlIJNT+fZs=', 'Content-Length': '0'} 2024-04-13T22:12:04.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051f2d64d73dcf58e-00661b0334-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.022 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:12:04.023 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.023 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.023 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.023 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.023 DEBUG:boto:Method: GET 2024-04-13T22:12:04.023 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.023 DEBUG:boto:Data: 2024-04-13T22:12:04.023 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.023 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.024 DEBUG:boto:Params: {} 2024-04-13T22:12:04.024 DEBUG:boto:Token: None 2024-04-13T22:12:04.024 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/?versions 2024-04-13T22:12:04.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE= 2024-04-13T22:12:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE=', 'Content-Length': '0'} 2024-04-13T22:12:04.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b0bc69ad832609-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.027 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:12:04.028 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:12:04.029 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:12:04.029 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:12:04.029 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.029 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.029 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.029 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.029 DEBUG:boto:Method: GET 2024-04-13T22:12:04.029 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.029 DEBUG:boto:Data: 2024-04-13T22:12:04.029 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.030 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.030 DEBUG:boto:Params: {} 2024-04-13T22:12:04.030 DEBUG:boto:Token: None 2024-04-13T22:12:04.030 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/?versions 2024-04-13T22:12:04.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE= 2024-04-13T22:12:04.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE=', 'Content-Length': '0'} 2024-04-13T22:12:04.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2c2485f97371f2-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.032 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:12:04.033 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:12:04.034 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.034 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.034 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.034 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.034 DEBUG:boto:Method: GET 2024-04-13T22:12:04.034 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.034 DEBUG:boto:Data: 2024-04-13T22:12:04.034 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.034 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.034 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.035 DEBUG:boto:Params: {} 2024-04-13T22:12:04.035 DEBUG:boto:Token: None 2024-04-13T22:12:04.035 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/?versions 2024-04-13T22:12:04.035 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE= 2024-04-13T22:12:04.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE=', 'Content-Length': '0'} 2024-04-13T22:12:04.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067c641cbf2fc1ba8-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.038 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.038 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.039 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.039 DEBUG:boto:path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.039 DEBUG:boto:auth_path=/vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.039 DEBUG:boto:Method: GET 2024-04-13T22:12:04.039 DEBUG:boto:Path: /vwnfns-58/?versions&max-keys=999 2024-04-13T22:12:04.039 DEBUG:boto:Data: 2024-04-13T22:12:04.039 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.039 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.040 DEBUG:boto:Params: {} 2024-04-13T22:12:04.040 DEBUG:boto:Token: None 2024-04-13T22:12:04.040 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/?versions 2024-04-13T22:12:04.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE= 2024-04-13T22:12:04.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0CNXkdHbGjaJoySo9x2/YoRVjtE=', 'Content-Length': '0'} 2024-04-13T22:12:04.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086af4811a08e1f40-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.042 DEBUG:boto:b'vwnfns-58999falseanulltrue2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:12:04.043 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.043 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.043 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.043 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.043 DEBUG:boto:Method: GET 2024-04-13T22:12:04.043 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.043 DEBUG:boto:Data: 2024-04-13T22:12:04.043 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.043 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.044 DEBUG:boto:Params: {} 2024-04-13T22:12:04.044 DEBUG:boto:Token: None 2024-04-13T22:12:04.044 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw= 2024-04-13T22:12:04.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw=', 'Content-Length': '0'} 2024-04-13T22:12:04.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63ba18bb0eb252d-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.046 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.046 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.046 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.046 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.046 DEBUG:boto:Method: GET 2024-04-13T22:12:04.047 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.047 DEBUG:boto:Data: 2024-04-13T22:12:04.047 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.047 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.047 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.047 DEBUG:boto:Params: {} 2024-04-13T22:12:04.047 DEBUG:boto:Token: None 2024-04-13T22:12:04.047 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.047 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw= 2024-04-13T22:12:04.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw=', 'Content-Length': '0'} 2024-04-13T22:12:04.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6b9f7c3dbaeb78-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.049 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.050 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.050 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.050 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.050 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.050 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.050 DEBUG:boto:Data: 2024-04-13T22:12:04.050 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.050 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.050 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.050 DEBUG:boto:Params: {} 2024-04-13T22:12:04.051 DEBUG:boto:Token: None 2024-04-13T22:12:04.051 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.051 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/qPZ0RDtqqaPjxONzH4SVso0w8Q= 2024-04-13T22:12:04.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/qPZ0RDtqqaPjxONzH4SVso0w8Q=', 'Content-Length': '0'} 2024-04-13T22:12:04.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007589709a4e5b5cf7-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.053 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.053 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.053 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.053 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.053 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.053 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.053 DEBUG:boto:Data: 2024-04-13T22:12:04.053 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.053 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.054 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.054 DEBUG:boto:Params: {} 2024-04-13T22:12:04.054 DEBUG:boto:Token: None 2024-04-13T22:12:04.054 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.054 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/qPZ0RDtqqaPjxONzH4SVso0w8Q= 2024-04-13T22:12:04.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/qPZ0RDtqqaPjxONzH4SVso0w8Q=', 'Content-Length': '0'} 2024-04-13T22:12:04.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe299e51c40f908b-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:12:04.056 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.056 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.057 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.057 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.057 DEBUG:boto:Method: GET 2024-04-13T22:12:04.057 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.057 DEBUG:boto:Data: 2024-04-13T22:12:04.057 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.057 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.057 DEBUG:boto:Params: {} 2024-04-13T22:12:04.057 DEBUG:boto:Token: None 2024-04-13T22:12:04.058 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.058 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw= 2024-04-13T22:12:04.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw=', 'Content-Length': '0'} 2024-04-13T22:12:04.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e776f0a4c60fdc-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.060 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.060 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.060 DEBUG:boto:path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.060 DEBUG:boto:auth_path=/vwnfns-58/a?versionId=null 2024-04-13T22:12:04.060 DEBUG:boto:Method: GET 2024-04-13T22:12:04.060 DEBUG:boto:Path: /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.061 DEBUG:boto:Data: 2024-04-13T22:12:04.061 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.061 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.061 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.061 DEBUG:boto:Params: {} 2024-04-13T22:12:04.061 DEBUG:boto:Token: None 2024-04-13T22:12:04.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a?versionId=null 2024-04-13T22:12:04.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw= 2024-04-13T22:12:04.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HEgGktbprLvjIMJeXSb0gMMkwOw=', 'Content-Length': '0'} 2024-04-13T22:12:04.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044dcd4c5e8b7ded1-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.063 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.064 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.064 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.064 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.064 DEBUG:boto:Method: GET 2024-04-13T22:12:04.064 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:12:04.064 DEBUG:boto:Data: 2024-04-13T22:12:04.064 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.064 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.064 DEBUG:boto:Params: {} 2024-04-13T22:12:04.064 DEBUG:boto:Token: None 2024-04-13T22:12:04.065 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.065 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4ce0e74f52ece7-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.067 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.067 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.067 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.068 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.068 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.068 DEBUG:boto:Method: GET 2024-04-13T22:12:04.068 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:12:04.068 DEBUG:boto:Data: 2024-04-13T22:12:04.068 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.068 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.068 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.068 DEBUG:boto:Params: {} 2024-04-13T22:12:04.068 DEBUG:boto:Token: None 2024-04-13T22:12:04.069 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.069 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000268ca7bd1e7fe51-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.071 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.071 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.071 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.071 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.071 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:12:04.072 DEBUG:boto:Data: 2024-04-13T22:12:04.072 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.072 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.072 DEBUG:boto:Params: {} 2024-04-13T22:12:04.072 DEBUG:boto:Token: None 2024-04-13T22:12:04.072 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a 2024-04-13T22:12:04.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:46ojXVGCVmjbUF9y9p/zCVp9pp4= 2024-04-13T22:12:04.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:46ojXVGCVmjbUF9y9p/zCVp9pp4=', 'Content-Length': '0'} 2024-04-13T22:12:04.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ff6084e8396484-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.074 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.074 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.074 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.075 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:12:04.075 DEBUG:boto:Data: 2024-04-13T22:12:04.075 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.075 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.075 DEBUG:boto:Params: {} 2024-04-13T22:12:04.075 DEBUG:boto:Token: None 2024-04-13T22:12:04.075 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a 2024-04-13T22:12:04.075 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:46ojXVGCVmjbUF9y9p/zCVp9pp4= 2024-04-13T22:12:04.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:46ojXVGCVmjbUF9y9p/zCVp9pp4=', 'Content-Length': '0'} 2024-04-13T22:12:04.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003448c802db6412cb-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:12:04.078 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.078 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.078 DEBUG:boto:Method: GET 2024-04-13T22:12:04.078 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:12:04.078 DEBUG:boto:Data: 2024-04-13T22:12:04.078 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.078 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.078 DEBUG:boto:Params: {} 2024-04-13T22:12:04.079 DEBUG:boto:Token: None 2024-04-13T22:12:04.079 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a 2024-04-13T22:12:04.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:orfpwWOZA/eKbm9fUpez/jzt4ck= 2024-04-13T22:12:04.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:orfpwWOZA/eKbm9fUpez/jzt4ck=', 'Content-Length': '0'} 2024-04-13T22:12:04.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007464680386419b24-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.081 DEBUG:boto:path=/vwnfns-58/a 2024-04-13T22:12:04.081 DEBUG:boto:auth_path=/vwnfns-58/a 2024-04-13T22:12:04.081 DEBUG:boto:Method: GET 2024-04-13T22:12:04.081 DEBUG:boto:Path: /vwnfns-58/a 2024-04-13T22:12:04.081 DEBUG:boto:Data: 2024-04-13T22:12:04.081 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.081 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.081 DEBUG:boto:Params: {} 2024-04-13T22:12:04.082 DEBUG:boto:Token: None 2024-04-13T22:12:04.082 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/a 2024-04-13T22:12:04.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:orfpwWOZA/eKbm9fUpez/jzt4ck= 2024-04-13T22:12:04.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:orfpwWOZA/eKbm9fUpez/jzt4ck=', 'Content-Length': '0'} 2024-04-13T22:12:04.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592d88ea943af55a-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.084 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.084 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.084 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.084 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.084 DEBUG:boto:Method: GET 2024-04-13T22:12:04.084 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:12:04.084 DEBUG:boto:Data: 2024-04-13T22:12:04.084 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.085 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.085 DEBUG:boto:Params: {} 2024-04-13T22:12:04.085 DEBUG:boto:Token: None 2024-04-13T22:12:04.085 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637b90f07ce4a54e-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.087 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.088 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.088 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.088 DEBUG:boto:path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.088 DEBUG:boto:auth_path=/vwnfns-58/?prefix=a 2024-04-13T22:12:04.088 DEBUG:boto:Method: GET 2024-04-13T22:12:04.088 DEBUG:boto:Path: /vwnfns-58/?prefix=a 2024-04-13T22:12:04.088 DEBUG:boto:Data: 2024-04-13T22:12:04.089 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.089 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.089 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.089 DEBUG:boto:Params: {} 2024-04-13T22:12:04.089 DEBUG:boto:Token: None 2024-04-13T22:12:04.089 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc684c3b752bdddb-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.091 DEBUG:boto:b'vwnfns-58a1000falsea2024-04-13T22:10:02.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:12:04.092 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.092 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.092 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.092 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.092 DEBUG:boto:Method: GET 2024-04-13T22:12:04.092 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.092 DEBUG:boto:Data: 2024-04-13T22:12:04.092 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.092 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.092 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.092 DEBUG:boto:Params: {} 2024-04-13T22:12:04.093 DEBUG:boto:Token: None 2024-04-13T22:12:04.093 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.093 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8= 2024-04-13T22:12:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8=', 'Content-Length': '0'} 2024-04-13T22:12:04.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006baa49b142f096d7-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.138 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.139 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.139 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.139 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.139 DEBUG:boto:Method: GET 2024-04-13T22:12:04.139 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.139 DEBUG:boto:Data: 2024-04-13T22:12:04.139 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.139 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.139 DEBUG:boto:Params: {} 2024-04-13T22:12:04.140 DEBUG:boto:Token: None 2024-04-13T22:12:04.140 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8= 2024-04-13T22:12:04.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8=', 'Content-Length': '0'} 2024-04-13T22:12:04.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f304ef71c6196d0-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.143 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.143 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.143 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.143 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.143 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.143 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.143 DEBUG:boto:Data: 2024-04-13T22:12:04.143 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.144 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.144 DEBUG:boto:Params: {} 2024-04-13T22:12:04.144 DEBUG:boto:Token: None 2024-04-13T22:12:04.144 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PGoMDHY9c2BhWrOP0NqqZF1C7jk= 2024-04-13T22:12:04.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PGoMDHY9c2BhWrOP0NqqZF1C7jk=', 'Content-Length': '0'} 2024-04-13T22:12:04.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e4e553218688b4-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.146 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.147 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.147 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.147 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.147 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.147 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.147 DEBUG:boto:Data: 2024-04-13T22:12:04.147 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.147 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.147 DEBUG:boto:Params: {} 2024-04-13T22:12:04.148 DEBUG:boto:Token: None 2024-04-13T22:12:04.148 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PGoMDHY9c2BhWrOP0NqqZF1C7jk= 2024-04-13T22:12:04.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PGoMDHY9c2BhWrOP0NqqZF1C7jk=', 'Content-Length': '0'} 2024-04-13T22:12:04.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d46a9300700b2a04-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:12:04.150 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.150 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.150 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.150 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.150 DEBUG:boto:Method: GET 2024-04-13T22:12:04.150 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.150 DEBUG:boto:Data: 2024-04-13T22:12:04.150 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.150 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.150 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.151 DEBUG:boto:Params: {} 2024-04-13T22:12:04.151 DEBUG:boto:Token: None 2024-04-13T22:12:04.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8= 2024-04-13T22:12:04.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8=', 'Content-Length': '0'} 2024-04-13T22:12:04.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41b5df831026b54-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.153 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.153 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.153 DEBUG:boto:path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.153 DEBUG:boto:auth_path=/vwnfns-58/b?versionId=null 2024-04-13T22:12:04.154 DEBUG:boto:Method: GET 2024-04-13T22:12:04.154 DEBUG:boto:Path: /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.154 DEBUG:boto:Data: 2024-04-13T22:12:04.154 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.154 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.154 DEBUG:boto:Params: {} 2024-04-13T22:12:04.154 DEBUG:boto:Token: None 2024-04-13T22:12:04.154 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b?versionId=null 2024-04-13T22:12:04.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8= 2024-04-13T22:12:04.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bIVgXmaW7yab5M9PSHjgxk/wla8=', 'Content-Length': '0'} 2024-04-13T22:12:04.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056874f5f40bc2f75-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.198 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.198 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.199 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.199 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.199 DEBUG:boto:Method: GET 2024-04-13T22:12:04.199 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:12:04.199 DEBUG:boto:Data: 2024-04-13T22:12:04.199 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.199 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.199 DEBUG:boto:Params: {} 2024-04-13T22:12:04.199 DEBUG:boto:Token: None 2024-04-13T22:12:04.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ddb3903afca693-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.202 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.202 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.203 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.203 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.203 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.203 DEBUG:boto:Method: GET 2024-04-13T22:12:04.203 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:12:04.203 DEBUG:boto:Data: 2024-04-13T22:12:04.203 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.203 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.203 DEBUG:boto:Params: {} 2024-04-13T22:12:04.204 DEBUG:boto:Token: None 2024-04-13T22:12:04.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eced63dbf6bc36c1-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.205 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.206 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.206 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.206 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.206 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:12:04.206 DEBUG:boto:Data: 2024-04-13T22:12:04.207 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.207 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.207 DEBUG:boto:Params: {} 2024-04-13T22:12:04.207 DEBUG:boto:Token: None 2024-04-13T22:12:04.207 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b 2024-04-13T22:12:04.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+I9TWuz589+Ecv4LzkAafx3PzME= 2024-04-13T22:12:04.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+I9TWuz589+Ecv4LzkAafx3PzME=', 'Content-Length': '0'} 2024-04-13T22:12:04.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f06abb2a1cc0548-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.209 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.209 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.209 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.209 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:12:04.209 DEBUG:boto:Data: 2024-04-13T22:12:04.209 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.210 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.210 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.210 DEBUG:boto:Params: {} 2024-04-13T22:12:04.210 DEBUG:boto:Token: None 2024-04-13T22:12:04.210 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b 2024-04-13T22:12:04.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+I9TWuz589+Ecv4LzkAafx3PzME= 2024-04-13T22:12:04.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+I9TWuz589+Ecv4LzkAafx3PzME=', 'Content-Length': '0'} 2024-04-13T22:12:04.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b25766d44700ab6-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:12:04.212 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.212 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.212 DEBUG:boto:Method: GET 2024-04-13T22:12:04.213 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:12:04.213 DEBUG:boto:Data: 2024-04-13T22:12:04.213 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.213 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.213 DEBUG:boto:Params: {} 2024-04-13T22:12:04.213 DEBUG:boto:Token: None 2024-04-13T22:12:04.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b 2024-04-13T22:12:04.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IvikBNj7SRShT9+aW4ex3Oe1seE= 2024-04-13T22:12:04.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IvikBNj7SRShT9+aW4ex3Oe1seE=', 'Content-Length': '0'} 2024-04-13T22:12:04.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621c91297a8bdf60-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.216 DEBUG:boto:path=/vwnfns-58/b 2024-04-13T22:12:04.216 DEBUG:boto:auth_path=/vwnfns-58/b 2024-04-13T22:12:04.216 DEBUG:boto:Method: GET 2024-04-13T22:12:04.216 DEBUG:boto:Path: /vwnfns-58/b 2024-04-13T22:12:04.216 DEBUG:boto:Data: 2024-04-13T22:12:04.216 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.216 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.216 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.216 DEBUG:boto:Params: {} 2024-04-13T22:12:04.217 DEBUG:boto:Token: None 2024-04-13T22:12:04.217 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/b 2024-04-13T22:12:04.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IvikBNj7SRShT9+aW4ex3Oe1seE= 2024-04-13T22:12:04.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IvikBNj7SRShT9+aW4ex3Oe1seE=', 'Content-Length': '0'} 2024-04-13T22:12:04.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b038c39bfc8a489-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.219 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.219 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.219 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.219 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.219 DEBUG:boto:Method: GET 2024-04-13T22:12:04.219 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:12:04.220 DEBUG:boto:Data: 2024-04-13T22:12:04.220 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.220 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.220 DEBUG:boto:Params: {} 2024-04-13T22:12:04.220 DEBUG:boto:Token: None 2024-04-13T22:12:04.220 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8bc03007be5e081-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.223 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.223 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.223 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.223 DEBUG:boto:path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.223 DEBUG:boto:auth_path=/vwnfns-58/?prefix=b 2024-04-13T22:12:04.224 DEBUG:boto:Method: GET 2024-04-13T22:12:04.224 DEBUG:boto:Path: /vwnfns-58/?prefix=b 2024-04-13T22:12:04.224 DEBUG:boto:Data: 2024-04-13T22:12:04.224 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.224 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.224 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.224 DEBUG:boto:Params: {} 2024-04-13T22:12:04.224 DEBUG:boto:Token: None 2024-04-13T22:12:04.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dcbf21c79d9f88b-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.226 DEBUG:boto:b'vwnfns-58b1000falseb2024-04-13T22:10:02.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:12:04.227 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.227 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.227 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.227 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.227 DEBUG:boto:Method: GET 2024-04-13T22:12:04.227 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.227 DEBUG:boto:Data: 2024-04-13T22:12:04.227 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.228 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.228 DEBUG:boto:Params: {} 2024-04-13T22:12:04.228 DEBUG:boto:Token: None 2024-04-13T22:12:04.228 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ= 2024-04-13T22:12:04.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac706f1b6174c13-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.270 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.270 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.271 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.271 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.271 DEBUG:boto:Method: GET 2024-04-13T22:12:04.271 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.271 DEBUG:boto:Data: 2024-04-13T22:12:04.271 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.271 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.271 DEBUG:boto:Params: {} 2024-04-13T22:12:04.271 DEBUG:boto:Token: None 2024-04-13T22:12:04.272 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ= 2024-04-13T22:12:04.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6cf17d55111eceb-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.274 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.274 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.274 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.274 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.275 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.275 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.275 DEBUG:boto:Data: 2024-04-13T22:12:04.275 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.275 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.275 DEBUG:boto:Params: {} 2024-04-13T22:12:04.275 DEBUG:boto:Token: None 2024-04-13T22:12:04.275 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TAqhdt34QPmqI4xqPjXHlo2ewkM= 2024-04-13T22:12:04.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TAqhdt34QPmqI4xqPjXHlo2ewkM=', 'Content-Length': '0'} 2024-04-13T22:12:04.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001437bc8e5d9a466-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.277 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.278 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.278 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.278 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.278 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.278 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.278 DEBUG:boto:Data: 2024-04-13T22:12:04.278 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.278 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.278 DEBUG:boto:Params: {} 2024-04-13T22:12:04.279 DEBUG:boto:Token: None 2024-04-13T22:12:04.279 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TAqhdt34QPmqI4xqPjXHlo2ewkM= 2024-04-13T22:12:04.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TAqhdt34QPmqI4xqPjXHlo2ewkM=', 'Content-Length': '0'} 2024-04-13T22:12:04.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f649ac5e6483fd94-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:12:04.281 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.281 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.281 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.281 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.281 DEBUG:boto:Method: GET 2024-04-13T22:12:04.282 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.282 DEBUG:boto:Data: 2024-04-13T22:12:04.282 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.282 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.282 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.282 DEBUG:boto:Params: {} 2024-04-13T22:12:04.282 DEBUG:boto:Token: None 2024-04-13T22:12:04.282 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.282 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ= 2024-04-13T22:12:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d2768b237fba22-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.285 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.285 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.285 DEBUG:boto:path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.285 DEBUG:boto:auth_path=/vwnfns-58/c?versionId=null 2024-04-13T22:12:04.285 DEBUG:boto:Method: GET 2024-04-13T22:12:04.285 DEBUG:boto:Path: /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.285 DEBUG:boto:Data: 2024-04-13T22:12:04.285 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.285 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.285 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.286 DEBUG:boto:Params: {} 2024-04-13T22:12:04.286 DEBUG:boto:Token: None 2024-04-13T22:12:04.286 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c?versionId=null 2024-04-13T22:12:04.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ= 2024-04-13T22:12:04.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:05rjfV00PRvRQKg0vLmj5jD01JQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e717d4d77ed18bf8-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.288 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.288 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.288 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.288 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.288 DEBUG:boto:Method: GET 2024-04-13T22:12:04.288 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:12:04.289 DEBUG:boto:Data: 2024-04-13T22:12:04.289 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.289 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.289 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.289 DEBUG:boto:Params: {} 2024-04-13T22:12:04.289 DEBUG:boto:Token: None 2024-04-13T22:12:04.289 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.289 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2989c1db82e53f-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.291 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.292 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.292 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.292 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.292 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.292 DEBUG:boto:Method: GET 2024-04-13T22:12:04.292 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:12:04.292 DEBUG:boto:Data: 2024-04-13T22:12:04.292 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.293 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.293 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.293 DEBUG:boto:Params: {} 2024-04-13T22:12:04.293 DEBUG:boto:Token: None 2024-04-13T22:12:04.293 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.293 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d03ada09ce998af8-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.295 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.295 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.295 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.296 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.296 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:12:04.296 DEBUG:boto:Data: 2024-04-13T22:12:04.296 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.296 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.298 DEBUG:boto:Params: {} 2024-04-13T22:12:04.298 DEBUG:boto:Token: None 2024-04-13T22:12:04.298 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c 2024-04-13T22:12:04.298 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/+pSz4nPRFjiiujaGyLbHz8/Sp0= 2024-04-13T22:12:04.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/+pSz4nPRFjiiujaGyLbHz8/Sp0=', 'Content-Length': '0'} 2024-04-13T22:12:04.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88075429126bdc1-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.300 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.300 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.300 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.300 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:12:04.301 DEBUG:boto:Data: 2024-04-13T22:12:04.301 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.301 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.301 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.301 DEBUG:boto:Params: {} 2024-04-13T22:12:04.301 DEBUG:boto:Token: None 2024-04-13T22:12:04.302 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c 2024-04-13T22:12:04.302 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/+pSz4nPRFjiiujaGyLbHz8/Sp0= 2024-04-13T22:12:04.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/+pSz4nPRFjiiujaGyLbHz8/Sp0=', 'Content-Length': '0'} 2024-04-13T22:12:04.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06f1b5bdb20bec2-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:12:04.304 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.305 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.305 DEBUG:boto:Method: GET 2024-04-13T22:12:04.305 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:12:04.305 DEBUG:boto:Data: 2024-04-13T22:12:04.305 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.305 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.305 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.305 DEBUG:boto:Params: {} 2024-04-13T22:12:04.305 DEBUG:boto:Token: None 2024-04-13T22:12:04.305 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c 2024-04-13T22:12:04.306 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B99U/+RrJMKY4FGtGPz5lE4xHow= 2024-04-13T22:12:04.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B99U/+RrJMKY4FGtGPz5lE4xHow=', 'Content-Length': '0'} 2024-04-13T22:12:04.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a146c3d3d4b8126-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.308 DEBUG:boto:path=/vwnfns-58/c 2024-04-13T22:12:04.308 DEBUG:boto:auth_path=/vwnfns-58/c 2024-04-13T22:12:04.308 DEBUG:boto:Method: GET 2024-04-13T22:12:04.308 DEBUG:boto:Path: /vwnfns-58/c 2024-04-13T22:12:04.308 DEBUG:boto:Data: 2024-04-13T22:12:04.308 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.308 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.308 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.308 DEBUG:boto:Params: {} 2024-04-13T22:12:04.308 DEBUG:boto:Token: None 2024-04-13T22:12:04.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/c 2024-04-13T22:12:04.309 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B99U/+RrJMKY4FGtGPz5lE4xHow= 2024-04-13T22:12:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B99U/+RrJMKY4FGtGPz5lE4xHow=', 'Content-Length': '0'} 2024-04-13T22:12:04.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f1caf792c4bdfb-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.355 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.355 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.355 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.355 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.355 DEBUG:boto:Method: GET 2024-04-13T22:12:04.355 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:12:04.355 DEBUG:boto:Data: 2024-04-13T22:12:04.356 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.356 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.356 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.356 DEBUG:boto:Params: {} 2024-04-13T22:12:04.356 DEBUG:boto:Token: None 2024-04-13T22:12:04.356 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed66e56c475ef9f-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.359 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.359 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.359 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.359 DEBUG:boto:path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.359 DEBUG:boto:auth_path=/vwnfns-58/?prefix=c 2024-04-13T22:12:04.360 DEBUG:boto:Method: GET 2024-04-13T22:12:04.360 DEBUG:boto:Path: /vwnfns-58/?prefix=c 2024-04-13T22:12:04.360 DEBUG:boto:Data: 2024-04-13T22:12:04.360 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.360 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.360 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.360 DEBUG:boto:Params: {} 2024-04-13T22:12:04.360 DEBUG:boto:Token: None 2024-04-13T22:12:04.360 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b44dfc0e86890dd3-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.362 DEBUG:boto:b'vwnfns-58c1000falsec2024-04-13T22:10:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:12:04.363 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.363 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.363 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.363 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.363 DEBUG:boto:Method: GET 2024-04-13T22:12:04.363 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.363 DEBUG:boto:Data: 2024-04-13T22:12:04.364 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.364 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.364 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.364 DEBUG:boto:Params: {} 2024-04-13T22:12:04.364 DEBUG:boto:Token: None 2024-04-13T22:12:04.364 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo= 2024-04-13T22:12:04.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo=', 'Content-Length': '0'} 2024-04-13T22:12:04.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7109a2e49cfc0e-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.406 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.406 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.406 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.407 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.407 DEBUG:boto:Method: GET 2024-04-13T22:12:04.407 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.407 DEBUG:boto:Data: 2024-04-13T22:12:04.407 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.407 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.407 DEBUG:boto:Params: {} 2024-04-13T22:12:04.407 DEBUG:boto:Token: None 2024-04-13T22:12:04.407 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo= 2024-04-13T22:12:04.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo=', 'Content-Length': '0'} 2024-04-13T22:12:04.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7d8d8499b8c90e-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.410 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.410 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.410 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.410 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.410 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.410 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.410 DEBUG:boto:Data: 2024-04-13T22:12:04.411 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.411 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.411 DEBUG:boto:Params: {} 2024-04-13T22:12:04.411 DEBUG:boto:Token: None 2024-04-13T22:12:04.411 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QC8kgFXwvR39+M8mAgTuTavAnT8= 2024-04-13T22:12:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QC8kgFXwvR39+M8mAgTuTavAnT8=', 'Content-Length': '0'} 2024-04-13T22:12:04.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093c8339a2a4a63e-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.413 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.413 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.413 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.413 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.413 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.413 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.414 DEBUG:boto:Data: 2024-04-13T22:12:04.414 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.414 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.414 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.414 DEBUG:boto:Params: {} 2024-04-13T22:12:04.414 DEBUG:boto:Token: None 2024-04-13T22:12:04.414 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.414 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QC8kgFXwvR39+M8mAgTuTavAnT8= 2024-04-13T22:12:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QC8kgFXwvR39+M8mAgTuTavAnT8=', 'Content-Length': '0'} 2024-04-13T22:12:04.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba37c5f45e94db4-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:12:04.416 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.416 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.416 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.416 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.417 DEBUG:boto:Method: GET 2024-04-13T22:12:04.417 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.417 DEBUG:boto:Data: 2024-04-13T22:12:04.417 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.417 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.417 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.417 DEBUG:boto:Params: {} 2024-04-13T22:12:04.417 DEBUG:boto:Token: None 2024-04-13T22:12:04.417 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.418 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo= 2024-04-13T22:12:04.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo=', 'Content-Length': '0'} 2024-04-13T22:12:04.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003996d5f25515adf7-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.419 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.420 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.420 DEBUG:boto:path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.420 DEBUG:boto:auth_path=/vwnfns-58/d?versionId=null 2024-04-13T22:12:04.420 DEBUG:boto:Method: GET 2024-04-13T22:12:04.420 DEBUG:boto:Path: /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.420 DEBUG:boto:Data: 2024-04-13T22:12:04.420 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.420 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.420 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.420 DEBUG:boto:Params: {} 2024-04-13T22:12:04.421 DEBUG:boto:Token: None 2024-04-13T22:12:04.421 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d?versionId=null 2024-04-13T22:12:04.421 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo= 2024-04-13T22:12:04.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Pw61SnH/uG2rntuZz38TNjVdgo=', 'Content-Length': '0'} 2024-04-13T22:12:04.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ff57438377715c-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.423 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.423 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.423 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.423 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.423 DEBUG:boto:Method: GET 2024-04-13T22:12:04.423 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:12:04.424 DEBUG:boto:Data: 2024-04-13T22:12:04.424 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.424 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.424 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.424 DEBUG:boto:Params: {} 2024-04-13T22:12:04.424 DEBUG:boto:Token: None 2024-04-13T22:12:04.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005905cfe7b51332f9-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.427 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.427 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.427 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.427 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.427 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.428 DEBUG:boto:Method: GET 2024-04-13T22:12:04.428 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:12:04.428 DEBUG:boto:Data: 2024-04-13T22:12:04.428 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.428 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.428 DEBUG:boto:Params: {} 2024-04-13T22:12:04.428 DEBUG:boto:Token: None 2024-04-13T22:12:04.428 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.428 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac355fce97771b05-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.430 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.431 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.431 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.431 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.431 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:12:04.431 DEBUG:boto:Data: 2024-04-13T22:12:04.431 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.431 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.432 DEBUG:boto:Params: {} 2024-04-13T22:12:04.432 DEBUG:boto:Token: None 2024-04-13T22:12:04.432 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d 2024-04-13T22:12:04.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g8ZFQtRx4Oeh+9ig+WqhdwuY5a4= 2024-04-13T22:12:04.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g8ZFQtRx4Oeh+9ig+WqhdwuY5a4=', 'Content-Length': '0'} 2024-04-13T22:12:04.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013eaa6ccd52f7456-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.434 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.434 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.434 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.434 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:12:04.434 DEBUG:boto:Data: 2024-04-13T22:12:04.434 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.434 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.434 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.434 DEBUG:boto:Params: {} 2024-04-13T22:12:04.435 DEBUG:boto:Token: None 2024-04-13T22:12:04.435 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d 2024-04-13T22:12:04.435 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g8ZFQtRx4Oeh+9ig+WqhdwuY5a4= 2024-04-13T22:12:04.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g8ZFQtRx4Oeh+9ig+WqhdwuY5a4=', 'Content-Length': '0'} 2024-04-13T22:12:04.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5df52e059fbab5-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:12:04.437 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.437 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.437 DEBUG:boto:Method: GET 2024-04-13T22:12:04.437 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:12:04.437 DEBUG:boto:Data: 2024-04-13T22:12:04.437 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.437 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.437 DEBUG:boto:Params: {} 2024-04-13T22:12:04.437 DEBUG:boto:Token: None 2024-04-13T22:12:04.438 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d 2024-04-13T22:12:04.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L7C5h7XqH/oAMhdRwIYV7kdYd3k= 2024-04-13T22:12:04.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L7C5h7XqH/oAMhdRwIYV7kdYd3k=', 'Content-Length': '0'} 2024-04-13T22:12:04.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003007427f99212bea-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.482 DEBUG:boto:path=/vwnfns-58/d 2024-04-13T22:12:04.483 DEBUG:boto:auth_path=/vwnfns-58/d 2024-04-13T22:12:04.483 DEBUG:boto:Method: GET 2024-04-13T22:12:04.483 DEBUG:boto:Path: /vwnfns-58/d 2024-04-13T22:12:04.483 DEBUG:boto:Data: 2024-04-13T22:12:04.483 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.483 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.483 DEBUG:boto:Params: {} 2024-04-13T22:12:04.483 DEBUG:boto:Token: None 2024-04-13T22:12:04.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/d 2024-04-13T22:12:04.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L7C5h7XqH/oAMhdRwIYV7kdYd3k= 2024-04-13T22:12:04.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L7C5h7XqH/oAMhdRwIYV7kdYd3k=', 'Content-Length': '0'} 2024-04-13T22:12:04.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f699e1940aaf23-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.486 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.487 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.487 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.487 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.487 DEBUG:boto:Method: GET 2024-04-13T22:12:04.487 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:12:04.487 DEBUG:boto:Data: 2024-04-13T22:12:04.487 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.487 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.487 DEBUG:boto:Params: {} 2024-04-13T22:12:04.488 DEBUG:boto:Token: None 2024-04-13T22:12:04.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ccfd880700ab08-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.490 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.491 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.491 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.491 DEBUG:boto:path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.491 DEBUG:boto:auth_path=/vwnfns-58/?prefix=d 2024-04-13T22:12:04.491 DEBUG:boto:Method: GET 2024-04-13T22:12:04.491 DEBUG:boto:Path: /vwnfns-58/?prefix=d 2024-04-13T22:12:04.491 DEBUG:boto:Data: 2024-04-13T22:12:04.491 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.492 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.492 DEBUG:boto:Params: {} 2024-04-13T22:12:04.492 DEBUG:boto:Token: None 2024-04-13T22:12:04.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bb0fc5b724d192f-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.494 DEBUG:boto:b'vwnfns-58d1000falsed2024-04-13T22:10:02.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:12:04.494 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.495 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.495 DEBUG:boto:path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.495 DEBUG:boto:auth_path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.495 DEBUG:boto:Method: GET 2024-04-13T22:12:04.495 DEBUG:boto:Path: /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.495 DEBUG:boto:Data: 2024-04-13T22:12:04.495 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.495 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.495 DEBUG:boto:Params: {} 2024-04-13T22:12:04.496 DEBUG:boto:Token: None 2024-04-13T22:12:04.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo= 2024-04-13T22:12:04.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo=', 'Content-Length': '0'} 2024-04-13T22:12:04.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b179e5c0add8a17-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.498 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.498 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.498 DEBUG:boto:path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.498 DEBUG:boto:auth_path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.498 DEBUG:boto:Method: GET 2024-04-13T22:12:04.498 DEBUG:boto:Path: /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.499 DEBUG:boto:Data: 2024-04-13T22:12:04.499 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.499 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.499 DEBUG:boto:Params: {} 2024-04-13T22:12:04.499 DEBUG:boto:Token: None 2024-04-13T22:12:04.499 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.499 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo= 2024-04-13T22:12:04.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo=', 'Content-Length': '0'} 2024-04-13T22:12:04.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c1e4c0001c1332-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.502 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.502 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.502 DEBUG:boto:path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.502 DEBUG:boto:auth_path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.502 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.502 DEBUG:boto:Path: /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.502 DEBUG:boto:Data: 2024-04-13T22:12:04.502 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.502 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.502 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.503 DEBUG:boto:Params: {} 2024-04-13T22:12:04.503 DEBUG:boto:Token: None 2024-04-13T22:12:04.503 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.503 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hmvrqSL3jQej39q0Ne2mrpJRoEY= 2024-04-13T22:12:04.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hmvrqSL3jQej39q0Ne2mrpJRoEY=', 'Content-Length': '0'} 2024-04-13T22:12:04.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d19baf5541d7407-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.505 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.505 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.505 DEBUG:boto:path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.505 DEBUG:boto:auth_path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.505 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.505 DEBUG:boto:Path: /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.505 DEBUG:boto:Data: 2024-04-13T22:12:04.505 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.506 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.506 DEBUG:boto:Params: {} 2024-04-13T22:12:04.506 DEBUG:boto:Token: None 2024-04-13T22:12:04.506 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.506 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hmvrqSL3jQej39q0Ne2mrpJRoEY= 2024-04-13T22:12:04.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hmvrqSL3jQej39q0Ne2mrpJRoEY=', 'Content-Length': '0'} 2024-04-13T22:12:04.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056103079a97dd024-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:12:04.508 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.508 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.508 DEBUG:boto:path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.508 DEBUG:boto:auth_path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.508 DEBUG:boto:Method: GET 2024-04-13T22:12:04.508 DEBUG:boto:Path: /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.508 DEBUG:boto:Data: 2024-04-13T22:12:04.509 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.509 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.509 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.509 DEBUG:boto:Params: {} 2024-04-13T22:12:04.509 DEBUG:boto:Token: None 2024-04-13T22:12:04.509 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo= 2024-04-13T22:12:04.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo=', 'Content-Length': '0'} 2024-04-13T22:12:04.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a88399b20db33d7e-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.511 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.511 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.512 DEBUG:boto:path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.512 DEBUG:boto:auth_path=/vwnfns-58/e?versionId=null 2024-04-13T22:12:04.512 DEBUG:boto:Method: GET 2024-04-13T22:12:04.512 DEBUG:boto:Path: /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.512 DEBUG:boto:Data: 2024-04-13T22:12:04.512 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.512 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.512 DEBUG:boto:Params: {} 2024-04-13T22:12:04.512 DEBUG:boto:Token: None 2024-04-13T22:12:04.513 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e?versionId=null 2024-04-13T22:12:04.513 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo= 2024-04-13T22:12:04.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NaIoNbr9BitEAoQi+J/VR6cigpo=', 'Content-Length': '0'} 2024-04-13T22:12:04.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a89fc70e8117a98-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.515 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.515 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.515 DEBUG:boto:path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.515 DEBUG:boto:auth_path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.515 DEBUG:boto:Method: GET 2024-04-13T22:12:04.515 DEBUG:boto:Path: /vwnfns-58/?prefix=e 2024-04-13T22:12:04.515 DEBUG:boto:Data: 2024-04-13T22:12:04.515 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.515 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.516 DEBUG:boto:Params: {} 2024-04-13T22:12:04.516 DEBUG:boto:Token: None 2024-04-13T22:12:04.516 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.517 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004088475863e7f039-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.562 DEBUG:boto:b'vwnfns-58e1000falsee2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.563 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.563 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.563 DEBUG:boto:path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.563 DEBUG:boto:auth_path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.563 DEBUG:boto:Method: GET 2024-04-13T22:12:04.563 DEBUG:boto:Path: /vwnfns-58/?prefix=e 2024-04-13T22:12:04.563 DEBUG:boto:Data: 2024-04-13T22:12:04.564 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.564 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.564 DEBUG:boto:Params: {} 2024-04-13T22:12:04.564 DEBUG:boto:Token: None 2024-04-13T22:12:04.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2d88a39e183791-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.566 DEBUG:boto:b'vwnfns-58e1000falsee2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.567 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.567 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.567 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.567 DEBUG:boto:Path: /vwnfns-58/e 2024-04-13T22:12:04.567 DEBUG:boto:Data: 2024-04-13T22:12:04.567 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.567 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.567 DEBUG:boto:Params: {} 2024-04-13T22:12:04.567 DEBUG:boto:Token: None 2024-04-13T22:12:04.568 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e 2024-04-13T22:12:04.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U92Xl4mcGdi0/2u3LrR//a4cmTc= 2024-04-13T22:12:04.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U92Xl4mcGdi0/2u3LrR//a4cmTc=', 'Content-Length': '0'} 2024-04-13T22:12:04.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8a3b84fa6a701b5-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.569 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.570 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.570 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.570 DEBUG:boto:Path: /vwnfns-58/e 2024-04-13T22:12:04.570 DEBUG:boto:Data: 2024-04-13T22:12:04.570 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.570 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.570 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.571 DEBUG:boto:Params: {} 2024-04-13T22:12:04.571 DEBUG:boto:Token: None 2024-04-13T22:12:04.571 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e 2024-04-13T22:12:04.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U92Xl4mcGdi0/2u3LrR//a4cmTc= 2024-04-13T22:12:04.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U92Xl4mcGdi0/2u3LrR//a4cmTc=', 'Content-Length': '0'} 2024-04-13T22:12:04.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc88654aeb61bcf-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:12:04.573 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.573 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.573 DEBUG:boto:Method: GET 2024-04-13T22:12:04.573 DEBUG:boto:Path: /vwnfns-58/e 2024-04-13T22:12:04.573 DEBUG:boto:Data: 2024-04-13T22:12:04.573 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.573 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.573 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.574 DEBUG:boto:Params: {} 2024-04-13T22:12:04.574 DEBUG:boto:Token: None 2024-04-13T22:12:04.574 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e 2024-04-13T22:12:04.574 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rr14OGzE1xTZSCecally/i6kmbw= 2024-04-13T22:12:04.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rr14OGzE1xTZSCecally/i6kmbw=', 'Content-Length': '0'} 2024-04-13T22:12:04.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e318b14347dc4171-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.576 DEBUG:boto:path=/vwnfns-58/e 2024-04-13T22:12:04.576 DEBUG:boto:auth_path=/vwnfns-58/e 2024-04-13T22:12:04.576 DEBUG:boto:Method: GET 2024-04-13T22:12:04.576 DEBUG:boto:Path: /vwnfns-58/e 2024-04-13T22:12:04.576 DEBUG:boto:Data: 2024-04-13T22:12:04.576 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.577 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.577 DEBUG:boto:Params: {} 2024-04-13T22:12:04.577 DEBUG:boto:Token: None 2024-04-13T22:12:04.577 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/e 2024-04-13T22:12:04.577 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rr14OGzE1xTZSCecally/i6kmbw= 2024-04-13T22:12:04.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rr14OGzE1xTZSCecally/i6kmbw=', 'Content-Length': '0'} 2024-04-13T22:12:04.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6356fc223a5efe0-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.579 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.579 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.579 DEBUG:boto:path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.579 DEBUG:boto:auth_path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.579 DEBUG:boto:Method: GET 2024-04-13T22:12:04.580 DEBUG:boto:Path: /vwnfns-58/?prefix=e 2024-04-13T22:12:04.580 DEBUG:boto:Data: 2024-04-13T22:12:04.580 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.580 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.580 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.580 DEBUG:boto:Params: {} 2024-04-13T22:12:04.580 DEBUG:boto:Token: None 2024-04-13T22:12:04.580 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f90a5f594bce2df5-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.583 DEBUG:boto:b'vwnfns-58e1000falsee2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.584 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.584 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.584 DEBUG:boto:path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.584 DEBUG:boto:auth_path=/vwnfns-58/?prefix=e 2024-04-13T22:12:04.584 DEBUG:boto:Method: GET 2024-04-13T22:12:04.584 DEBUG:boto:Path: /vwnfns-58/?prefix=e 2024-04-13T22:12:04.584 DEBUG:boto:Data: 2024-04-13T22:12:04.584 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.584 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.584 DEBUG:boto:Params: {} 2024-04-13T22:12:04.585 DEBUG:boto:Token: None 2024-04-13T22:12:04.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c6aa7c3cae7dc5-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.587 DEBUG:boto:b'vwnfns-58e1000falsee2024-04-13T22:10:28.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:12:04.587 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.587 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.588 DEBUG:boto:path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.588 DEBUG:boto:auth_path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.588 DEBUG:boto:Method: GET 2024-04-13T22:12:04.588 DEBUG:boto:Path: /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.588 DEBUG:boto:Data: 2024-04-13T22:12:04.588 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.588 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.588 DEBUG:boto:Params: {} 2024-04-13T22:12:04.588 DEBUG:boto:Token: None 2024-04-13T22:12:04.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ= 2024-04-13T22:12:04.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c121025d37c39c78-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.591 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.591 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.591 DEBUG:boto:path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.591 DEBUG:boto:auth_path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.591 DEBUG:boto:Method: GET 2024-04-13T22:12:04.591 DEBUG:boto:Path: /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.591 DEBUG:boto:Data: 2024-04-13T22:12:04.591 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.591 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.592 DEBUG:boto:Params: {} 2024-04-13T22:12:04.592 DEBUG:boto:Token: None 2024-04-13T22:12:04.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ= 2024-04-13T22:12:04.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f9d1a83c81415a-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.594 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.594 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.594 DEBUG:boto:path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.594 DEBUG:boto:auth_path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.594 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.594 DEBUG:boto:Path: /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.595 DEBUG:boto:Data: 2024-04-13T22:12:04.595 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.595 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.595 DEBUG:boto:Params: {} 2024-04-13T22:12:04.595 DEBUG:boto:Token: None 2024-04-13T22:12:04.595 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.595 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0hd5Gn9BLc1CA5uvknoIbymNKzE= 2024-04-13T22:12:04.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0hd5Gn9BLc1CA5uvknoIbymNKzE=', 'Content-Length': '0'} 2024-04-13T22:12:04.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4e6b672fa10f0b3-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.597 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.597 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.597 DEBUG:boto:path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.597 DEBUG:boto:auth_path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.597 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.597 DEBUG:boto:Path: /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.598 DEBUG:boto:Data: 2024-04-13T22:12:04.598 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.598 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.598 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.598 DEBUG:boto:Params: {} 2024-04-13T22:12:04.598 DEBUG:boto:Token: None 2024-04-13T22:12:04.598 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.598 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0hd5Gn9BLc1CA5uvknoIbymNKzE= 2024-04-13T22:12:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0hd5Gn9BLc1CA5uvknoIbymNKzE=', 'Content-Length': '0'} 2024-04-13T22:12:04.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9826fdedef74587-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:12:04.600 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.601 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.601 DEBUG:boto:path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.601 DEBUG:boto:auth_path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.601 DEBUG:boto:Method: GET 2024-04-13T22:12:04.601 DEBUG:boto:Path: /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.601 DEBUG:boto:Data: 2024-04-13T22:12:04.601 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.601 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.601 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.601 DEBUG:boto:Params: {} 2024-04-13T22:12:04.602 DEBUG:boto:Token: None 2024-04-13T22:12:04.602 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.602 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ= 2024-04-13T22:12:04.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008659cd40c9ffab7e-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.646 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.646 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.647 DEBUG:boto:path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.647 DEBUG:boto:auth_path=/vwnfns-58/f?versionId=null 2024-04-13T22:12:04.647 DEBUG:boto:Method: GET 2024-04-13T22:12:04.647 DEBUG:boto:Path: /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.647 DEBUG:boto:Data: 2024-04-13T22:12:04.647 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.647 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.647 DEBUG:boto:Params: {} 2024-04-13T22:12:04.647 DEBUG:boto:Token: None 2024-04-13T22:12:04.648 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f?versionId=null 2024-04-13T22:12:04.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ= 2024-04-13T22:12:04.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:og9khhXaniI95XZ2sBIN7GZCFWQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb43bcf94b4c207d-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.650 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.650 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.650 DEBUG:boto:path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.650 DEBUG:boto:auth_path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.650 DEBUG:boto:Method: GET 2024-04-13T22:12:04.650 DEBUG:boto:Path: /vwnfns-58/?prefix=f 2024-04-13T22:12:04.650 DEBUG:boto:Data: 2024-04-13T22:12:04.651 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.651 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.651 DEBUG:boto:Params: {} 2024-04-13T22:12:04.651 DEBUG:boto:Token: None 2024-04-13T22:12:04.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e3205c0f37b9cc-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.653 DEBUG:boto:b'vwnfns-58f1000falsef2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.654 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.654 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.654 DEBUG:boto:path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.654 DEBUG:boto:auth_path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.654 DEBUG:boto:Method: GET 2024-04-13T22:12:04.654 DEBUG:boto:Path: /vwnfns-58/?prefix=f 2024-04-13T22:12:04.655 DEBUG:boto:Data: 2024-04-13T22:12:04.655 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.655 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.655 DEBUG:boto:Params: {} 2024-04-13T22:12:04.655 DEBUG:boto:Token: None 2024-04-13T22:12:04.655 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a3ee48ae52f772-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.657 DEBUG:boto:b'vwnfns-58f1000falsef2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.658 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.658 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.658 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.658 DEBUG:boto:Path: /vwnfns-58/f 2024-04-13T22:12:04.658 DEBUG:boto:Data: 2024-04-13T22:12:04.658 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.658 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.658 DEBUG:boto:Params: {} 2024-04-13T22:12:04.659 DEBUG:boto:Token: None 2024-04-13T22:12:04.659 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f 2024-04-13T22:12:04.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaN3wVjX11Vo/wDxhc2Y/48lwk4= 2024-04-13T22:12:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaN3wVjX11Vo/wDxhc2Y/48lwk4=', 'Content-Length': '0'} 2024-04-13T22:12:04.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba37ec47f215f103-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.661 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.661 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.661 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.661 DEBUG:boto:Path: /vwnfns-58/f 2024-04-13T22:12:04.661 DEBUG:boto:Data: 2024-04-13T22:12:04.661 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.661 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.661 DEBUG:boto:Params: {} 2024-04-13T22:12:04.661 DEBUG:boto:Token: None 2024-04-13T22:12:04.662 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f 2024-04-13T22:12:04.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaN3wVjX11Vo/wDxhc2Y/48lwk4= 2024-04-13T22:12:04.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaN3wVjX11Vo/wDxhc2Y/48lwk4=', 'Content-Length': '0'} 2024-04-13T22:12:04.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004750f8738c8c3ba2-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:12:04.664 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.664 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.664 DEBUG:boto:Method: GET 2024-04-13T22:12:04.664 DEBUG:boto:Path: /vwnfns-58/f 2024-04-13T22:12:04.664 DEBUG:boto:Data: 2024-04-13T22:12:04.664 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.664 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.664 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.664 DEBUG:boto:Params: {} 2024-04-13T22:12:04.665 DEBUG:boto:Token: None 2024-04-13T22:12:04.665 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f 2024-04-13T22:12:04.665 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y9RSwCjPKKNoi6M5AgHzjCxy3qo= 2024-04-13T22:12:04.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y9RSwCjPKKNoi6M5AgHzjCxy3qo=', 'Content-Length': '0'} 2024-04-13T22:12:04.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ad320504aab4fb-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.667 DEBUG:boto:path=/vwnfns-58/f 2024-04-13T22:12:04.667 DEBUG:boto:auth_path=/vwnfns-58/f 2024-04-13T22:12:04.667 DEBUG:boto:Method: GET 2024-04-13T22:12:04.667 DEBUG:boto:Path: /vwnfns-58/f 2024-04-13T22:12:04.667 DEBUG:boto:Data: 2024-04-13T22:12:04.667 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.667 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.667 DEBUG:boto:Params: {} 2024-04-13T22:12:04.667 DEBUG:boto:Token: None 2024-04-13T22:12:04.668 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/f 2024-04-13T22:12:04.668 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y9RSwCjPKKNoi6M5AgHzjCxy3qo= 2024-04-13T22:12:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y9RSwCjPKKNoi6M5AgHzjCxy3qo=', 'Content-Length': '0'} 2024-04-13T22:12:04.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad46cf2c527e674a-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.669 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.670 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.670 DEBUG:boto:path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.670 DEBUG:boto:auth_path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.670 DEBUG:boto:Method: GET 2024-04-13T22:12:04.670 DEBUG:boto:Path: /vwnfns-58/?prefix=f 2024-04-13T22:12:04.670 DEBUG:boto:Data: 2024-04-13T22:12:04.670 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.670 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.670 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.670 DEBUG:boto:Params: {} 2024-04-13T22:12:04.671 DEBUG:boto:Token: None 2024-04-13T22:12:04.671 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.671 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a0c63b25e9a8c0f-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.673 DEBUG:boto:b'vwnfns-58f1000falsef2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.673 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.673 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.674 DEBUG:boto:path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.674 DEBUG:boto:auth_path=/vwnfns-58/?prefix=f 2024-04-13T22:12:04.674 DEBUG:boto:Method: GET 2024-04-13T22:12:04.674 DEBUG:boto:Path: /vwnfns-58/?prefix=f 2024-04-13T22:12:04.674 DEBUG:boto:Data: 2024-04-13T22:12:04.674 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.674 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.674 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.674 DEBUG:boto:Params: {} 2024-04-13T22:12:04.674 DEBUG:boto:Token: None 2024-04-13T22:12:04.675 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.675 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b37ca290a952112-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.677 DEBUG:boto:b'vwnfns-58f1000falsef2024-04-13T22:10:28.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:12:04.677 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.677 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.677 DEBUG:boto:path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.677 DEBUG:boto:auth_path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.678 DEBUG:boto:Method: GET 2024-04-13T22:12:04.678 DEBUG:boto:Path: /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.678 DEBUG:boto:Data: 2024-04-13T22:12:04.678 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.678 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.678 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.678 DEBUG:boto:Params: {} 2024-04-13T22:12:04.678 DEBUG:boto:Token: None 2024-04-13T22:12:04.678 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.678 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI= 2024-04-13T22:12:04.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI=', 'Content-Length': '0'} 2024-04-13T22:12:04.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028cd33c74cd079ab-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.681 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.681 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.681 DEBUG:boto:path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.681 DEBUG:boto:auth_path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.681 DEBUG:boto:Method: GET 2024-04-13T22:12:04.681 DEBUG:boto:Path: /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.681 DEBUG:boto:Data: 2024-04-13T22:12:04.681 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.681 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.682 DEBUG:boto:Params: {} 2024-04-13T22:12:04.682 DEBUG:boto:Token: None 2024-04-13T22:12:04.682 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI= 2024-04-13T22:12:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI=', 'Content-Length': '0'} 2024-04-13T22:12:04.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f9036828bcff6c-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.684 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.684 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.684 DEBUG:boto:path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.684 DEBUG:boto:auth_path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.684 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.684 DEBUG:boto:Path: /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.684 DEBUG:boto:Data: 2024-04-13T22:12:04.684 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.685 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.685 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.685 DEBUG:boto:Params: {} 2024-04-13T22:12:04.685 DEBUG:boto:Token: None 2024-04-13T22:12:04.685 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1U4MFIN50E0WKioEPy4EDEHURvc= 2024-04-13T22:12:04.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1U4MFIN50E0WKioEPy4EDEHURvc=', 'Content-Length': '0'} 2024-04-13T22:12:04.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744ba0f6d13cabee-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.687 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.687 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.687 DEBUG:boto:path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.687 DEBUG:boto:auth_path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.687 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.687 DEBUG:boto:Path: /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.687 DEBUG:boto:Data: 2024-04-13T22:12:04.687 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.688 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.688 DEBUG:boto:Params: {} 2024-04-13T22:12:04.688 DEBUG:boto:Token: None 2024-04-13T22:12:04.688 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1U4MFIN50E0WKioEPy4EDEHURvc= 2024-04-13T22:12:04.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1U4MFIN50E0WKioEPy4EDEHURvc=', 'Content-Length': '0'} 2024-04-13T22:12:04.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d3972add84b9684-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:12:04.690 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.690 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.690 DEBUG:boto:path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.691 DEBUG:boto:auth_path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.691 DEBUG:boto:Method: GET 2024-04-13T22:12:04.691 DEBUG:boto:Path: /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.691 DEBUG:boto:Data: 2024-04-13T22:12:04.691 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.691 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.691 DEBUG:boto:Params: {} 2024-04-13T22:12:04.691 DEBUG:boto:Token: None 2024-04-13T22:12:04.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI= 2024-04-13T22:12:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI=', 'Content-Length': '0'} 2024-04-13T22:12:04.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003234aec0f8a7fd95-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.738 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.738 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.739 DEBUG:boto:path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.739 DEBUG:boto:auth_path=/vwnfns-58/g?versionId=null 2024-04-13T22:12:04.739 DEBUG:boto:Method: GET 2024-04-13T22:12:04.739 DEBUG:boto:Path: /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.739 DEBUG:boto:Data: 2024-04-13T22:12:04.739 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.739 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.739 DEBUG:boto:Params: {} 2024-04-13T22:12:04.739 DEBUG:boto:Token: None 2024-04-13T22:12:04.740 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g?versionId=null 2024-04-13T22:12:04.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI= 2024-04-13T22:12:04.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MsXljnxh00wsjrCx2+HNxZGMybI=', 'Content-Length': '0'} 2024-04-13T22:12:04.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7aa6dfccd8f0bb7-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.742 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.742 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.742 DEBUG:boto:path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.742 DEBUG:boto:auth_path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.742 DEBUG:boto:Method: GET 2024-04-13T22:12:04.743 DEBUG:boto:Path: /vwnfns-58/?prefix=g 2024-04-13T22:12:04.743 DEBUG:boto:Data: 2024-04-13T22:12:04.743 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.743 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.743 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.743 DEBUG:boto:Params: {} 2024-04-13T22:12:04.743 DEBUG:boto:Token: None 2024-04-13T22:12:04.743 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f04fbb88c407193c-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.745 DEBUG:boto:b'vwnfns-58g1000falseg2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.746 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.746 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.746 DEBUG:boto:path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.746 DEBUG:boto:auth_path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.746 DEBUG:boto:Method: GET 2024-04-13T22:12:04.746 DEBUG:boto:Path: /vwnfns-58/?prefix=g 2024-04-13T22:12:04.747 DEBUG:boto:Data: 2024-04-13T22:12:04.747 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.747 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.747 DEBUG:boto:Params: {} 2024-04-13T22:12:04.747 DEBUG:boto:Token: None 2024-04-13T22:12:04.747 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a0f0692f8cdb7a-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.749 DEBUG:boto:b'vwnfns-58g1000falseg2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.750 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.750 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.750 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.750 DEBUG:boto:Path: /vwnfns-58/g 2024-04-13T22:12:04.750 DEBUG:boto:Data: 2024-04-13T22:12:04.750 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.750 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.751 DEBUG:boto:Params: {} 2024-04-13T22:12:04.751 DEBUG:boto:Token: None 2024-04-13T22:12:04.751 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g 2024-04-13T22:12:04.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j8qXOB/qwn5eTZjz+sJMzQXLNXg= 2024-04-13T22:12:04.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j8qXOB/qwn5eTZjz+sJMzQXLNXg=', 'Content-Length': '0'} 2024-04-13T22:12:04.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e319cf43da8ed35-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.753 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.753 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.753 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.753 DEBUG:boto:Path: /vwnfns-58/g 2024-04-13T22:12:04.753 DEBUG:boto:Data: 2024-04-13T22:12:04.753 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.753 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.753 DEBUG:boto:Params: {} 2024-04-13T22:12:04.754 DEBUG:boto:Token: None 2024-04-13T22:12:04.754 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g 2024-04-13T22:12:04.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j8qXOB/qwn5eTZjz+sJMzQXLNXg= 2024-04-13T22:12:04.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j8qXOB/qwn5eTZjz+sJMzQXLNXg=', 'Content-Length': '0'} 2024-04-13T22:12:04.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e546679d4e09f8-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:12:04.756 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.756 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.756 DEBUG:boto:Method: GET 2024-04-13T22:12:04.757 DEBUG:boto:Path: /vwnfns-58/g 2024-04-13T22:12:04.757 DEBUG:boto:Data: 2024-04-13T22:12:04.757 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.757 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.757 DEBUG:boto:Params: {} 2024-04-13T22:12:04.757 DEBUG:boto:Token: None 2024-04-13T22:12:04.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g 2024-04-13T22:12:04.757 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kECs89cd5alM4kiKzV7hRz3fCAA= 2024-04-13T22:12:04.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kECs89cd5alM4kiKzV7hRz3fCAA=', 'Content-Length': '0'} 2024-04-13T22:12:04.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcc98d60f5dc7f9-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.759 DEBUG:boto:path=/vwnfns-58/g 2024-04-13T22:12:04.759 DEBUG:boto:auth_path=/vwnfns-58/g 2024-04-13T22:12:04.759 DEBUG:boto:Method: GET 2024-04-13T22:12:04.760 DEBUG:boto:Path: /vwnfns-58/g 2024-04-13T22:12:04.760 DEBUG:boto:Data: 2024-04-13T22:12:04.760 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.760 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.760 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.760 DEBUG:boto:Params: {} 2024-04-13T22:12:04.760 DEBUG:boto:Token: None 2024-04-13T22:12:04.760 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/g 2024-04-13T22:12:04.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kECs89cd5alM4kiKzV7hRz3fCAA= 2024-04-13T22:12:04.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kECs89cd5alM4kiKzV7hRz3fCAA=', 'Content-Length': '0'} 2024-04-13T22:12:04.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162e8c723b538900-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.763 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.763 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.763 DEBUG:boto:path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.763 DEBUG:boto:auth_path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.763 DEBUG:boto:Method: GET 2024-04-13T22:12:04.763 DEBUG:boto:Path: /vwnfns-58/?prefix=g 2024-04-13T22:12:04.763 DEBUG:boto:Data: 2024-04-13T22:12:04.763 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.763 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.763 DEBUG:boto:Params: {} 2024-04-13T22:12:04.764 DEBUG:boto:Token: None 2024-04-13T22:12:04.764 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3376859a86dc1fb-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.766 DEBUG:boto:b'vwnfns-58g1000falseg2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.767 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.767 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.767 DEBUG:boto:path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.767 DEBUG:boto:auth_path=/vwnfns-58/?prefix=g 2024-04-13T22:12:04.767 DEBUG:boto:Method: GET 2024-04-13T22:12:04.767 DEBUG:boto:Path: /vwnfns-58/?prefix=g 2024-04-13T22:12:04.767 DEBUG:boto:Data: 2024-04-13T22:12:04.767 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.767 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.767 DEBUG:boto:Params: {} 2024-04-13T22:12:04.767 DEBUG:boto:Token: None 2024-04-13T22:12:04.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8db263cd3f7f57-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.810 DEBUG:boto:b'vwnfns-58g1000falseg2024-04-13T22:10:28.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:12:04.811 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.811 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.811 DEBUG:boto:path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.811 DEBUG:boto:auth_path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.811 DEBUG:boto:Method: GET 2024-04-13T22:12:04.812 DEBUG:boto:Path: /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.812 DEBUG:boto:Data: 2024-04-13T22:12:04.812 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.812 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.812 DEBUG:boto:Params: {} 2024-04-13T22:12:04.812 DEBUG:boto:Token: None 2024-04-13T22:12:04.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4= 2024-04-13T22:12:04.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4=', 'Content-Length': '0'} 2024-04-13T22:12:04.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062018faea0fb80b6-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.815 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.815 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.815 DEBUG:boto:path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.815 DEBUG:boto:auth_path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.815 DEBUG:boto:Method: GET 2024-04-13T22:12:04.815 DEBUG:boto:Path: /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.815 DEBUG:boto:Data: 2024-04-13T22:12:04.815 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.815 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.815 DEBUG:boto:Params: {} 2024-04-13T22:12:04.816 DEBUG:boto:Token: None 2024-04-13T22:12:04.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4= 2024-04-13T22:12:04.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4=', 'Content-Length': '0'} 2024-04-13T22:12:04.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dbbf0ae847da08f-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.818 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.818 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.819 DEBUG:boto:path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.819 DEBUG:boto:auth_path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.819 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.819 DEBUG:boto:Path: /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.819 DEBUG:boto:Data: 2024-04-13T22:12:04.819 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.819 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.819 DEBUG:boto:Params: {} 2024-04-13T22:12:04.819 DEBUG:boto:Token: None 2024-04-13T22:12:04.819 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kt75/W/LcG3HbRVNDduAs4Kuf1Y= 2024-04-13T22:12:04.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kt75/W/LcG3HbRVNDduAs4Kuf1Y=', 'Content-Length': '0'} 2024-04-13T22:12:04.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dacf86f8f211dfe8-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.821 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.821 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.822 DEBUG:boto:path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.822 DEBUG:boto:auth_path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.822 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.822 DEBUG:boto:Path: /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.822 DEBUG:boto:Data: 2024-04-13T22:12:04.822 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.822 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.822 DEBUG:boto:Params: {} 2024-04-13T22:12:04.822 DEBUG:boto:Token: None 2024-04-13T22:12:04.823 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kt75/W/LcG3HbRVNDduAs4Kuf1Y= 2024-04-13T22:12:04.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kt75/W/LcG3HbRVNDduAs4Kuf1Y=', 'Content-Length': '0'} 2024-04-13T22:12:04.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2181ca151a2b5a5-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:12:04.825 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.825 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.825 DEBUG:boto:path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.825 DEBUG:boto:auth_path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.825 DEBUG:boto:Method: GET 2024-04-13T22:12:04.825 DEBUG:boto:Path: /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.825 DEBUG:boto:Data: 2024-04-13T22:12:04.825 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.826 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.826 DEBUG:boto:Params: {} 2024-04-13T22:12:04.826 DEBUG:boto:Token: None 2024-04-13T22:12:04.826 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.826 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4= 2024-04-13T22:12:04.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4=', 'Content-Length': '0'} 2024-04-13T22:12:04.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000631128696aa51735-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.871 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.871 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.871 DEBUG:boto:path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.871 DEBUG:boto:auth_path=/vwnfns-58/h?versionId=null 2024-04-13T22:12:04.871 DEBUG:boto:Method: GET 2024-04-13T22:12:04.871 DEBUG:boto:Path: /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.871 DEBUG:boto:Data: 2024-04-13T22:12:04.871 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.872 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.872 DEBUG:boto:Params: {} 2024-04-13T22:12:04.872 DEBUG:boto:Token: None 2024-04-13T22:12:04.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h?versionId=null 2024-04-13T22:12:04.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4= 2024-04-13T22:12:04.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vxjmrgm/B75M0XxxLEkS7MqCQd4=', 'Content-Length': '0'} 2024-04-13T22:12:04.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008264a639b229ca69-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.874 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.874 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.874 DEBUG:boto:path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.874 DEBUG:boto:auth_path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.875 DEBUG:boto:Method: GET 2024-04-13T22:12:04.875 DEBUG:boto:Path: /vwnfns-58/?prefix=h 2024-04-13T22:12:04.875 DEBUG:boto:Data: 2024-04-13T22:12:04.875 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.875 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.875 DEBUG:boto:Params: {} 2024-04-13T22:12:04.875 DEBUG:boto:Token: None 2024-04-13T22:12:04.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c60ae6acd62471-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.878 DEBUG:boto:b'vwnfns-58h1000falseh2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.879 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.879 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.879 DEBUG:boto:path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.879 DEBUG:boto:auth_path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.879 DEBUG:boto:Method: GET 2024-04-13T22:12:04.879 DEBUG:boto:Path: /vwnfns-58/?prefix=h 2024-04-13T22:12:04.879 DEBUG:boto:Data: 2024-04-13T22:12:04.879 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.879 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.880 DEBUG:boto:Params: {} 2024-04-13T22:12:04.880 DEBUG:boto:Token: None 2024-04-13T22:12:04.880 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab936da809e14e8-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.882 DEBUG:boto:b'vwnfns-58h1000falseh2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.882 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.882 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.882 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.882 DEBUG:boto:Path: /vwnfns-58/h 2024-04-13T22:12:04.882 DEBUG:boto:Data: 2024-04-13T22:12:04.883 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.883 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.883 DEBUG:boto:Params: {} 2024-04-13T22:12:04.883 DEBUG:boto:Token: None 2024-04-13T22:12:04.883 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h 2024-04-13T22:12:04.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+HYMWFk8CYAZuQIlwRCycZXZ/6s= 2024-04-13T22:12:04.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+HYMWFk8CYAZuQIlwRCycZXZ/6s=', 'Content-Length': '0'} 2024-04-13T22:12:04.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887da8f741cb93f1-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.885 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.885 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.885 DEBUG:boto:Method: HEAD 2024-04-13T22:12:04.885 DEBUG:boto:Path: /vwnfns-58/h 2024-04-13T22:12:04.885 DEBUG:boto:Data: 2024-04-13T22:12:04.886 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.886 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.886 DEBUG:boto:Params: {} 2024-04-13T22:12:04.886 DEBUG:boto:Token: None 2024-04-13T22:12:04.886 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h 2024-04-13T22:12:04.886 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+HYMWFk8CYAZuQIlwRCycZXZ/6s= 2024-04-13T22:12:04.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+HYMWFk8CYAZuQIlwRCycZXZ/6s=', 'Content-Length': '0'} 2024-04-13T22:12:04.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646f20d1e5ec6051-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:12:04.888 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.888 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.888 DEBUG:boto:Method: GET 2024-04-13T22:12:04.888 DEBUG:boto:Path: /vwnfns-58/h 2024-04-13T22:12:04.888 DEBUG:boto:Data: 2024-04-13T22:12:04.889 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.889 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.889 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.889 DEBUG:boto:Params: {} 2024-04-13T22:12:04.889 DEBUG:boto:Token: None 2024-04-13T22:12:04.889 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h 2024-04-13T22:12:04.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n8O5JCEZPq9ArQBrDZGpvERENYs= 2024-04-13T22:12:04.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n8O5JCEZPq9ArQBrDZGpvERENYs=', 'Content-Length': '0'} 2024-04-13T22:12:04.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-58:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28a87a85f79e911-00661b0334-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.891 DEBUG:boto:path=/vwnfns-58/h 2024-04-13T22:12:04.891 DEBUG:boto:auth_path=/vwnfns-58/h 2024-04-13T22:12:04.891 DEBUG:boto:Method: GET 2024-04-13T22:12:04.892 DEBUG:boto:Path: /vwnfns-58/h 2024-04-13T22:12:04.892 DEBUG:boto:Data: 2024-04-13T22:12:04.892 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.892 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.892 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.892 DEBUG:boto:Params: {} 2024-04-13T22:12:04.892 DEBUG:boto:Token: None 2024-04-13T22:12:04.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/h 2024-04-13T22:12:04.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n8O5JCEZPq9ArQBrDZGpvERENYs= 2024-04-13T22:12:04.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n8O5JCEZPq9ArQBrDZGpvERENYs=', 'Content-Length': '0'} 2024-04-13T22:12:04.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:10:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32687f5c0b9982f-00661b0334-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.894 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.894 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.894 DEBUG:boto:path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.895 DEBUG:boto:auth_path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.895 DEBUG:boto:Method: GET 2024-04-13T22:12:04.895 DEBUG:boto:Path: /vwnfns-58/?prefix=h 2024-04-13T22:12:04.895 DEBUG:boto:Data: 2024-04-13T22:12:04.895 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:04.895 DEBUG:boto:Port: 8001 2024-04-13T22:12:04.895 DEBUG:boto:Params: {} 2024-04-13T22:12:04.895 DEBUG:boto:Token: None 2024-04-13T22:12:04.895 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.895 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a74a107341d65306-00661b0334-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.898 DEBUG:boto:b'vwnfns-58h1000falseh2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.899 DEBUG:boto:path=/vwnfns-58/ 2024-04-13T22:12:04.899 DEBUG:boto:auth_path=/vwnfns-58/ 2024-04-13T22:12:04.899 DEBUG:boto:path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.899 DEBUG:boto:auth_path=/vwnfns-58/?prefix=h 2024-04-13T22:12:04.899 DEBUG:boto:Method: GET 2024-04-13T22:12:04.899 DEBUG:boto:Path: /vwnfns-58/?prefix=h 2024-04-13T22:12:04.899 DEBUG:boto:Data: 2024-04-13T22:12:04.899 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.899 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.899 DEBUG:boto:Params: {} 2024-04-13T22:12:04.899 DEBUG:boto:Token: None 2024-04-13T22:12:04.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-58/ 2024-04-13T22:12:04.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ= 2024-04-13T22:12:04.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XPGgJ9rMrGi1hCYA2jBHDC2rZoQ=', 'Content-Length': '0'} 2024-04-13T22:12:04.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b05ce9b288036a-00661b0334-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.901 DEBUG:boto:b'vwnfns-58h1000falseh2024-04-13T22:10:28.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:12:04.902 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-58 zones={a2, a1} 2024-04-13T22:12:04.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2024-04-13T22:12:04.904 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:04.904 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:04.904 DEBUG:boto:Method: PUT 2024-04-13T22:12:04.904 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:04.904 DEBUG:boto:Data: 2024-04-13T22:12:04.904 DEBUG:boto:Headers: {} 2024-04-13T22:12:04.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:04.904 DEBUG:boto:Port: 8000 2024-04-13T22:12:04.904 DEBUG:boto:Params: {} 2024-04-13T22:12:04.904 DEBUG:boto:Token: None 2024-04-13T22:12:04.905 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:12:04 GMT /vwnfns-59/ 2024-04-13T22:12:04.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A4j6BtdVaH07YvliYTS+SzEJ/ks= 2024-04-13T22:12:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A4j6BtdVaH07YvliYTS+SzEJ/ks=', 'Content-Length': '0'} 2024-04-13T22:12:04.936 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e977c9d7c8f511d-00661b0334-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:04.936 DEBUG:rgw_multi.tests:created bucket=vwnfns-59 2024-04-13T22:12:04.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:12:04.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:12:04.936 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:12:04.984 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:12:04.984 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:12:05.020 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:12:05.032 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:12:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046324.932995_747.1", 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:12:04.932995Z" 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:12:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.034 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:12:05.034 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:12:05.034 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:05.034 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:12:05.035 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046324.932995_747.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:12:05.035 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:12:05.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:12:05.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:12:05.036 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:12:05.086 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:12:05.086 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:12:05.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:05.119+0000 7f4fe1450780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:12:05.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:05.120+0000 7f4fe1450780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:12:05.136 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:12:05.136 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:12:05.137 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046228.627158_735.1", 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:10:28.627158Z", 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.138 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:05.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:12:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:05.141 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:12:05Z" 2024-04-13T22:12:05.142 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:12:05.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046228.627158_735.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:10:28.627158Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:12:05Z" } 2024-04-13T22:12:05.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046228.627158_735.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:10:28.627158Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:12:05.144 DEBUG:rgw_multi.tests:log_status={0: '1_1713046324.932995_747.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:12:05.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046228.627158_735.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:12:05.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046324.932995_747.1 target=1_1713046228.627158_735.1 2024-04-13T22:12:10.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:12:10.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:12:10.146 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:12:10.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:12:10.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:12:10.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:10.232+0000 7fdb3317d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:12:10.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:10.233+0000 7fdb3317d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:12:10.249 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:12:10.249 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:10.250 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046324.932995_747.1", 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:12:04.932995Z", 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:10.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:10.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:12:10.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:10.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:12:10.255 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:12:10.256 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:10.257 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:12:10.258 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:12:10.258 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:12:10.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:12:10.258 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:12:10Z" 2024-04-13T22:12:10.258 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:12:10.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046324.932995_747.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:12:04.932995Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:12:10Z" } 2024-04-13T22:12:10.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046324.932995_747.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:12:04.932995Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:12:10.259 DEBUG:rgw_multi.tests:log_status={0: '1_1713046324.932995_747.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:12:10.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046324.932995_747.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:12:10.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:12:10.260 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:12:10.260 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:12:10.261 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:12:10.258+0000 7f5eb15ed700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 49416) UID: 0 2024-04-13T22:12:10.261 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:12:10.258+0000 7f5eb67c27c0 -1 shutting down 2024-04-13T22:12:15.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:12:16.362 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:12:16.362 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.362 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.362 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.362 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.362 DEBUG:boto:Data: 2024-04-13T22:12:16.362 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.363 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.363 DEBUG:boto:Params: {} 2024-04-13T22:12:16.363 DEBUG:boto:Token: None 2024-04-13T22:12:16.363 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd5b3f10eda8bd55-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.367 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:12:16.367 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:12:16.367 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.367 DEBUG:boto:Path: /vwnfns-59/a 2024-04-13T22:12:16.367 DEBUG:boto:Data: 2024-04-13T22:12:16.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.367 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.367 DEBUG:boto:Params: {} 2024-04-13T22:12:16.367 DEBUG:boto:Token: None 2024-04-13T22:12:16.368 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/a 2024-04-13T22:12:16.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VsQc08Qs93QGvwQHxm8ehqHH0hk= 2024-04-13T22:12:16.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VsQc08Qs93QGvwQHxm8ehqHH0hk='} 2024-04-13T22:12:16.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040fdd8e6af5fef63-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.376 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.376 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.376 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.376 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.376 DEBUG:boto:Data: 2024-04-13T22:12:16.376 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.376 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.376 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.376 DEBUG:boto:Params: {} 2024-04-13T22:12:16.376 DEBUG:boto:Token: None 2024-04-13T22:12:16.377 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.377 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029a2aad846729870-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.379 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:12:16.380 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:12:16.380 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.380 DEBUG:boto:Path: /vwnfns-59/b 2024-04-13T22:12:16.380 DEBUG:boto:Data: 2024-04-13T22:12:16.380 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.380 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.380 DEBUG:boto:Params: {} 2024-04-13T22:12:16.380 DEBUG:boto:Token: None 2024-04-13T22:12:16.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/b 2024-04-13T22:12:16.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:301fEiAYGMt9jIb9NMxkOLchvp8= 2024-04-13T22:12:16.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:301fEiAYGMt9jIb9NMxkOLchvp8='} 2024-04-13T22:12:16.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5bb1d99c83f241d-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.427 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.427 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.427 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.427 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.427 DEBUG:boto:Data: 2024-04-13T22:12:16.427 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.428 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.428 DEBUG:boto:Params: {} 2024-04-13T22:12:16.428 DEBUG:boto:Token: None 2024-04-13T22:12:16.428 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.428 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f11e3a152a795287-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.431 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:12:16.431 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:12:16.431 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.431 DEBUG:boto:Path: /vwnfns-59/c 2024-04-13T22:12:16.431 DEBUG:boto:Data: 2024-04-13T22:12:16.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.431 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.431 DEBUG:boto:Params: {} 2024-04-13T22:12:16.431 DEBUG:boto:Token: None 2024-04-13T22:12:16.432 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/c 2024-04-13T22:12:16.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sjAgU9xGpJYEXqcJYxk19hRAquk= 2024-04-13T22:12:16.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sjAgU9xGpJYEXqcJYxk19hRAquk='} 2024-04-13T22:12:16.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037dc61fcebd986ba-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.438 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.438 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.438 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.438 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.438 DEBUG:boto:Data: 2024-04-13T22:12:16.438 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.438 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.438 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.438 DEBUG:boto:Params: {} 2024-04-13T22:12:16.439 DEBUG:boto:Token: None 2024-04-13T22:12:16.439 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.439 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086dd2dd0aa6345a5-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.441 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:12:16.442 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:12:16.442 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.442 DEBUG:boto:Path: /vwnfns-59/d 2024-04-13T22:12:16.442 DEBUG:boto:Data: 2024-04-13T22:12:16.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.442 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.442 DEBUG:boto:Params: {} 2024-04-13T22:12:16.442 DEBUG:boto:Token: None 2024-04-13T22:12:16.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/d 2024-04-13T22:12:16.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b1yW42kAXj04FwsZNtqh4tcGDvM= 2024-04-13T22:12:16.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b1yW42kAXj04FwsZNtqh4tcGDvM='} 2024-04-13T22:12:16.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfcfa56e53b0c317-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-59', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:12:16.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-59', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:12:16.491 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-59 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:12:16.546 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:12:16.546 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:12:16.588 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:12:16.588 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-59 2024-04-13T22:12:16.592 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 4 2024-04-13T22:12:16.630 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.630 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.630 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.631 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.631 DEBUG:boto:Data: 2024-04-13T22:12:16.631 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.631 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.631 DEBUG:boto:Params: {} 2024-04-13T22:12:16.631 DEBUG:boto:Token: None 2024-04-13T22:12:16.631 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.631 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0b7e401ced80319-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.634 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:12:16.634 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:12:16.634 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.634 DEBUG:boto:Path: /vwnfns-59/e 2024-04-13T22:12:16.634 DEBUG:boto:Data: 2024-04-13T22:12:16.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.634 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.635 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.635 DEBUG:boto:Params: {} 2024-04-13T22:12:16.635 DEBUG:boto:Token: None 2024-04-13T22:12:16.635 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/e 2024-04-13T22:12:16.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m0XPhn5BtlpFNxCYR3RgF27TQtI= 2024-04-13T22:12:16.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m0XPhn5BtlpFNxCYR3RgF27TQtI='} 2024-04-13T22:12:16.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2299bcd4ac3f5d7-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.640 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.640 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.640 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.640 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.641 DEBUG:boto:Data: 2024-04-13T22:12:16.641 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.641 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.641 DEBUG:boto:Params: {} 2024-04-13T22:12:16.641 DEBUG:boto:Token: None 2024-04-13T22:12:16.641 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.641 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4541c280abc57e7-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.643 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:12:16.643 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:12:16.643 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.644 DEBUG:boto:Path: /vwnfns-59/f 2024-04-13T22:12:16.644 DEBUG:boto:Data: 2024-04-13T22:12:16.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.644 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.644 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.644 DEBUG:boto:Params: {} 2024-04-13T22:12:16.644 DEBUG:boto:Token: None 2024-04-13T22:12:16.644 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/f 2024-04-13T22:12:16.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sLIWbJMz4p94rMHccKBkEV3wUvk= 2024-04-13T22:12:16.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sLIWbJMz4p94rMHccKBkEV3wUvk='} 2024-04-13T22:12:16.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7568124231a7fd5-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.649 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.650 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.650 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.650 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.650 DEBUG:boto:Data: 2024-04-13T22:12:16.650 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.650 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.650 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.650 DEBUG:boto:Params: {} 2024-04-13T22:12:16.650 DEBUG:boto:Token: None 2024-04-13T22:12:16.650 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea54f08bb3a8ad52-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.652 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:12:16.652 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:12:16.653 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.653 DEBUG:boto:Path: /vwnfns-59/g 2024-04-13T22:12:16.653 DEBUG:boto:Data: 2024-04-13T22:12:16.653 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.653 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.653 DEBUG:boto:Params: {} 2024-04-13T22:12:16.653 DEBUG:boto:Token: None 2024-04-13T22:12:16.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/g 2024-04-13T22:12:16.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kUCwp6lpzSqxHzRgUIFuMAa9bSw= 2024-04-13T22:12:16.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kUCwp6lpzSqxHzRgUIFuMAa9bSw='} 2024-04-13T22:12:16.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072fe57b36467f295-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.703 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:12:16.703 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:12:16.703 DEBUG:boto:Method: HEAD 2024-04-13T22:12:16.703 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:12:16.703 DEBUG:boto:Data: 2024-04-13T22:12:16.703 DEBUG:boto:Headers: {} 2024-04-13T22:12:16.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.704 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.704 DEBUG:boto:Params: {} 2024-04-13T22:12:16.704 DEBUG:boto:Token: None 2024-04-13T22:12:16.704 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/ 2024-04-13T22:12:16.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE= 2024-04-13T22:12:16.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fbRbYgYfLnrhEGKL5dWlEVIk+nE=', 'Content-Length': '0'} 2024-04-13T22:12:16.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f80950b8770abd8-00661b0340-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.706 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:12:16.706 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:12:16.706 DEBUG:boto:Method: PUT 2024-04-13T22:12:16.706 DEBUG:boto:Path: /vwnfns-59/h 2024-04-13T22:12:16.706 DEBUG:boto:Data: 2024-04-13T22:12:16.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:12:16.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:12:16.707 DEBUG:boto:Port: 8000 2024-04-13T22:12:16.707 DEBUG:boto:Params: {} 2024-04-13T22:12:16.707 DEBUG:boto:Token: None 2024-04-13T22:12:16.707 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:12:16 GMT /vwnfns-59/h 2024-04-13T22:12:16.707 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GxA5siShRFcOEQ1pFhlASFGna/o= 2024-04-13T22:12:16.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:12:16 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GxA5siShRFcOEQ1pFhlASFGna/o='} 2024-04-13T22:12:16.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049b55a089b7fb228-00661b0340-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:12:16.711 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:12:16.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:12:16.714 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:12:16.714 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:12:16.734 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:12:16.735 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:12:16.735 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:12:17.736 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:12:17.756 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:12:17.757 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:12:17.758 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:12:17.758 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:12:17.759 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-59 2024-04-13T22:12:17.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:12:17.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:12:17.759 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:12:17.853 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:12:17.853 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:12:17.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:17.886+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:12:22.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:22.887+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:27.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:27.888+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:32.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:32.889+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:37.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:37.888+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:42.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:42.889+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:47.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:47.890+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:52.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:52.890+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:12:57.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:12:57.892+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:02.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:02.892+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:07.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:07.893+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:12.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:12.893+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:17.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:17.893+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:22.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:22.894+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:27.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:27.894+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:32.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:32.895+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:37.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:37.896+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:42.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:42.897+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:47.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:47.898+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:52.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:52.898+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:13:57.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:13:57.898+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:02.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:02.898+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:07.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:07.899+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:12.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:12.900+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:17.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:17.901+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:22.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:22.902+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:27.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:27.903+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:32.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:32.902+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:37.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:37.903+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:42.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:42.904+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:47.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:47.904+0000 7f8405eca780 1 waiting to reach incremental sync.. 2024-04-13T22:14:52.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:52.905+0000 7f8405eca780 1 bucket sync caught up with source: 2024-04-13T22:14:52.909 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000003.47.6, 00000000002.26.6, 00000000002.35.6] 2024-04-13T22:14:52.909 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000003.47.6, 00000000002.26.6, 00000000002.35.6] 2024-04-13T22:14:52.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:52.905+0000 7f8405eca780 0 bucket checkpoint complete 2024-04-13T22:14:52.927 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-59 2024-04-13T22:14:52.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:14:52.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:14:52.928 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:14:52.990 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:52.990 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:53.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:53.026+0000 7ff4e523a780 1 bucket sync caught up with source: 2024-04-13T22:14:53.028 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2829.6, , , 00000000001.825.6, 00000000001.1119.6, , , 00000000002.624.6, , 00000000002.633.6] 2024-04-13T22:14:53.029 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2829.6, , , 00000000001.825.6, 00000000001.1119.6, , , 00000000002.624.6, , 00000000002.633.6] 2024-04-13T22:14:53.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:53.026+0000 7ff4e523a780 0 bucket checkpoint complete 2024-04-13T22:14:53.042 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-59 zones={a2, a1} 2024-04-13T22:14:53.043 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.043 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.044 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.044 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:14:53.044 DEBUG:boto:Data: 2024-04-13T22:14:53.044 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.044 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.044 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.044 DEBUG:boto:Params: {} 2024-04-13T22:14:53.045 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:14:53.045 DEBUG:boto:Token: None 2024-04-13T22:14:53.045 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0L+3c0i6NefPepaaOPTZ2BOyYr0= 2024-04-13T22:14:53.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0L+3c0i6NefPepaaOPTZ2BOyYr0=', 'Content-Length': '0'} 2024-04-13T22:14:53.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ae866bcb3c0c6e2-00661b03dd-1eaf-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.051 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.051 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.052 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.052 DEBUG:boto:Path: /vwnfns-59/ 2024-04-13T22:14:53.052 DEBUG:boto:Data: 2024-04-13T22:14:53.052 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.052 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.052 DEBUG:boto:Params: {} 2024-04-13T22:14:53.052 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:14:53.052 DEBUG:boto:Token: None 2024-04-13T22:14:53.053 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.053 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0L+3c0i6NefPepaaOPTZ2BOyYr0= 2024-04-13T22:14:53.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0L+3c0i6NefPepaaOPTZ2BOyYr0=', 'Content-Length': '0'} 2024-04-13T22:14:53.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076307bf2f972eeb7-00661b03dd-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.057 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:14:53.058 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.058 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.058 DEBUG:boto:path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.058 DEBUG:boto:auth_path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.058 DEBUG:boto:Method: GET 2024-04-13T22:14:53.058 DEBUG:boto:Path: /vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.058 DEBUG:boto:Data: 2024-04-13T22:14:53.058 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.058 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.059 DEBUG:boto:Params: {} 2024-04-13T22:14:53.059 DEBUG:boto:Token: None 2024-04-13T22:14:53.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/?versions 2024-04-13T22:14:53.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg= 2024-04-13T22:14:53.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg=', 'Content-Length': '0'} 2024-04-13T22:14:53.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a55c5d7690895e8f-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.061 DEBUG:boto:b'vwnfns-59999falseanulltrue2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:14:53.063 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:14:53.063 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.064 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.064 DEBUG:boto:path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.064 DEBUG:boto:auth_path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.064 DEBUG:boto:Method: GET 2024-04-13T22:14:53.064 DEBUG:boto:Path: /vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.064 DEBUG:boto:Data: 2024-04-13T22:14:53.064 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.064 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.064 DEBUG:boto:Params: {} 2024-04-13T22:14:53.064 DEBUG:boto:Token: None 2024-04-13T22:14:53.065 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/?versions 2024-04-13T22:14:53.065 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg= 2024-04-13T22:14:53.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg=', 'Content-Length': '0'} 2024-04-13T22:14:53.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045afc68017642f13-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.067 DEBUG:boto:b'vwnfns-59999falseanulltrue2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-13T22:14:53.068 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-13T22:14:53.069 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.069 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.069 DEBUG:boto:path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.069 DEBUG:boto:auth_path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.069 DEBUG:boto:Method: GET 2024-04-13T22:14:53.069 DEBUG:boto:Path: /vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.069 DEBUG:boto:Data: 2024-04-13T22:14:53.069 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.069 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.069 DEBUG:boto:Params: {} 2024-04-13T22:14:53.070 DEBUG:boto:Token: None 2024-04-13T22:14:53.070 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/?versions 2024-04-13T22:14:53.070 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg= 2024-04-13T22:14:53.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg=', 'Content-Length': '0'} 2024-04-13T22:14:53.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c88c6506504aa3ee-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.072 DEBUG:boto:b'vwnfns-59999falseanulltrue2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.073 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.073 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.073 DEBUG:boto:path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.073 DEBUG:boto:auth_path=/vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.073 DEBUG:boto:Method: GET 2024-04-13T22:14:53.073 DEBUG:boto:Path: /vwnfns-59/?versions&max-keys=999 2024-04-13T22:14:53.073 DEBUG:boto:Data: 2024-04-13T22:14:53.074 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.074 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.074 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.074 DEBUG:boto:Params: {} 2024-04-13T22:14:53.074 DEBUG:boto:Token: None 2024-04-13T22:14:53.074 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/?versions 2024-04-13T22:14:53.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg= 2024-04-13T22:14:53.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1mOhrW8tnK31R7QEWhmqj+RmSkg=', 'Content-Length': '0'} 2024-04-13T22:14:53.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050165a8116c78508-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.076 DEBUG:boto:b'vwnfns-59999falseanulltrue2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:14:53.077 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.077 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.077 DEBUG:boto:path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.077 DEBUG:boto:auth_path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.078 DEBUG:boto:Method: GET 2024-04-13T22:14:53.078 DEBUG:boto:Path: /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.078 DEBUG:boto:Data: 2024-04-13T22:14:53.078 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.078 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.078 DEBUG:boto:Params: {} 2024-04-13T22:14:53.078 DEBUG:boto:Token: None 2024-04-13T22:14:53.078 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8= 2024-04-13T22:14:53.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8=', 'Content-Length': '0'} 2024-04-13T22:14:53.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4757e4429970821-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.081 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.081 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.081 DEBUG:boto:path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.081 DEBUG:boto:auth_path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.081 DEBUG:boto:Method: GET 2024-04-13T22:14:53.081 DEBUG:boto:Path: /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.081 DEBUG:boto:Data: 2024-04-13T22:14:53.081 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.081 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.081 DEBUG:boto:Params: {} 2024-04-13T22:14:53.081 DEBUG:boto:Token: None 2024-04-13T22:14:53.082 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8= 2024-04-13T22:14:53.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8=', 'Content-Length': '0'} 2024-04-13T22:14:53.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e30f256262a6c40-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.084 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.084 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.084 DEBUG:boto:path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.084 DEBUG:boto:auth_path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.084 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.084 DEBUG:boto:Path: /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.084 DEBUG:boto:Data: 2024-04-13T22:14:53.085 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.085 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.085 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.085 DEBUG:boto:Params: {} 2024-04-13T22:14:53.085 DEBUG:boto:Token: None 2024-04-13T22:14:53.085 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dMlmE88pe6CPkcGnr/EYv+R088E= 2024-04-13T22:14:53.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dMlmE88pe6CPkcGnr/EYv+R088E=', 'Content-Length': '0'} 2024-04-13T22:14:53.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c6ecac84a00597-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.087 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.088 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.088 DEBUG:boto:path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.088 DEBUG:boto:auth_path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.088 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.088 DEBUG:boto:Path: /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.088 DEBUG:boto:Data: 2024-04-13T22:14:53.088 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.088 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.088 DEBUG:boto:Params: {} 2024-04-13T22:14:53.088 DEBUG:boto:Token: None 2024-04-13T22:14:53.089 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dMlmE88pe6CPkcGnr/EYv+R088E= 2024-04-13T22:14:53.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dMlmE88pe6CPkcGnr/EYv+R088E=', 'Content-Length': '0'} 2024-04-13T22:14:53.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be6c9bc69f286ba-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:14:53.091 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.091 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.091 DEBUG:boto:path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.091 DEBUG:boto:auth_path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.091 DEBUG:boto:Method: GET 2024-04-13T22:14:53.091 DEBUG:boto:Path: /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.091 DEBUG:boto:Data: 2024-04-13T22:14:53.091 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.092 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.092 DEBUG:boto:Params: {} 2024-04-13T22:14:53.092 DEBUG:boto:Token: None 2024-04-13T22:14:53.092 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8= 2024-04-13T22:14:53.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8=', 'Content-Length': '0'} 2024-04-13T22:14:53.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058bc6fce93b05ece-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.094 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.094 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.094 DEBUG:boto:path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.094 DEBUG:boto:auth_path=/vwnfns-59/a?versionId=null 2024-04-13T22:14:53.094 DEBUG:boto:Method: GET 2024-04-13T22:14:53.094 DEBUG:boto:Path: /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.095 DEBUG:boto:Data: 2024-04-13T22:14:53.095 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.095 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.095 DEBUG:boto:Params: {} 2024-04-13T22:14:53.095 DEBUG:boto:Token: None 2024-04-13T22:14:53.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a?versionId=null 2024-04-13T22:14:53.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8= 2024-04-13T22:14:53.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q8Bj42Y2ZrCQgVm3UR9RSTgWLD8=', 'Content-Length': '0'} 2024-04-13T22:14:53.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee506174851e8fc2-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.097 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.098 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.098 DEBUG:boto:path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.098 DEBUG:boto:auth_path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.098 DEBUG:boto:Method: GET 2024-04-13T22:14:53.098 DEBUG:boto:Path: /vwnfns-59/?prefix=a 2024-04-13T22:14:53.098 DEBUG:boto:Data: 2024-04-13T22:14:53.098 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.098 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.098 DEBUG:boto:Params: {} 2024-04-13T22:14:53.099 DEBUG:boto:Token: None 2024-04-13T22:14:53.099 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000212bd16302d0d4f0-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.101 DEBUG:boto:b'vwnfns-59a1000falsea2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.102 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.102 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.102 DEBUG:boto:path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.102 DEBUG:boto:auth_path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.102 DEBUG:boto:Method: GET 2024-04-13T22:14:53.102 DEBUG:boto:Path: /vwnfns-59/?prefix=a 2024-04-13T22:14:53.102 DEBUG:boto:Data: 2024-04-13T22:14:53.102 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.103 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.103 DEBUG:boto:Params: {} 2024-04-13T22:14:53.103 DEBUG:boto:Token: None 2024-04-13T22:14:53.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee16eb548d34f7a-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.105 DEBUG:boto:b'vwnfns-59a1000falsea2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.105 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.105 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.106 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.106 DEBUG:boto:Path: /vwnfns-59/a 2024-04-13T22:14:53.106 DEBUG:boto:Data: 2024-04-13T22:14:53.106 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.106 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.106 DEBUG:boto:Params: {} 2024-04-13T22:14:53.106 DEBUG:boto:Token: None 2024-04-13T22:14:53.106 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a 2024-04-13T22:14:53.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r7nVEihFMLKIEsZEZC3oSNsFWYk= 2024-04-13T22:14:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r7nVEihFMLKIEsZEZC3oSNsFWYk=', 'Content-Length': '0'} 2024-04-13T22:14:53.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076190da07538d346-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.108 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.108 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.109 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.109 DEBUG:boto:Path: /vwnfns-59/a 2024-04-13T22:14:53.109 DEBUG:boto:Data: 2024-04-13T22:14:53.109 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.109 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.109 DEBUG:boto:Params: {} 2024-04-13T22:14:53.109 DEBUG:boto:Token: None 2024-04-13T22:14:53.109 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a 2024-04-13T22:14:53.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r7nVEihFMLKIEsZEZC3oSNsFWYk= 2024-04-13T22:14:53.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r7nVEihFMLKIEsZEZC3oSNsFWYk=', 'Content-Length': '0'} 2024-04-13T22:14:53.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b57f5d10a726aa-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-13T22:14:53.112 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.112 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.112 DEBUG:boto:Method: GET 2024-04-13T22:14:53.112 DEBUG:boto:Path: /vwnfns-59/a 2024-04-13T22:14:53.112 DEBUG:boto:Data: 2024-04-13T22:14:53.112 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.112 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.113 DEBUG:boto:Params: {} 2024-04-13T22:14:53.113 DEBUG:boto:Token: None 2024-04-13T22:14:53.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a 2024-04-13T22:14:53.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t/F0ju9l3sAcZZUbsxkhSLKc4oI= 2024-04-13T22:14:53.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t/F0ju9l3sAcZZUbsxkhSLKc4oI=', 'Content-Length': '0'} 2024-04-13T22:14:53.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0835cbb7e760cf-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.115 DEBUG:boto:path=/vwnfns-59/a 2024-04-13T22:14:53.115 DEBUG:boto:auth_path=/vwnfns-59/a 2024-04-13T22:14:53.115 DEBUG:boto:Method: GET 2024-04-13T22:14:53.115 DEBUG:boto:Path: /vwnfns-59/a 2024-04-13T22:14:53.115 DEBUG:boto:Data: 2024-04-13T22:14:53.115 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.115 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.115 DEBUG:boto:Params: {} 2024-04-13T22:14:53.116 DEBUG:boto:Token: None 2024-04-13T22:14:53.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/a 2024-04-13T22:14:53.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t/F0ju9l3sAcZZUbsxkhSLKc4oI= 2024-04-13T22:14:53.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t/F0ju9l3sAcZZUbsxkhSLKc4oI=', 'Content-Length': '0'} 2024-04-13T22:14:53.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003095ad5922f95bdb-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.118 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.118 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.118 DEBUG:boto:path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.118 DEBUG:boto:auth_path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.118 DEBUG:boto:Method: GET 2024-04-13T22:14:53.118 DEBUG:boto:Path: /vwnfns-59/?prefix=a 2024-04-13T22:14:53.118 DEBUG:boto:Data: 2024-04-13T22:14:53.118 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.118 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.119 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.119 DEBUG:boto:Params: {} 2024-04-13T22:14:53.119 DEBUG:boto:Token: None 2024-04-13T22:14:53.119 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.119 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d9ff6a76ba17e7-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.162 DEBUG:boto:b'vwnfns-59a1000falsea2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.163 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.163 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.163 DEBUG:boto:path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.163 DEBUG:boto:auth_path=/vwnfns-59/?prefix=a 2024-04-13T22:14:53.163 DEBUG:boto:Method: GET 2024-04-13T22:14:53.163 DEBUG:boto:Path: /vwnfns-59/?prefix=a 2024-04-13T22:14:53.163 DEBUG:boto:Data: 2024-04-13T22:14:53.163 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.164 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.164 DEBUG:boto:Params: {} 2024-04-13T22:14:53.164 DEBUG:boto:Token: None 2024-04-13T22:14:53.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002452da98e5d3379-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.166 DEBUG:boto:b'vwnfns-59a1000falsea2024-04-13T22:12:16.369Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:14:53.167 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.167 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.167 DEBUG:boto:path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.167 DEBUG:boto:auth_path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.167 DEBUG:boto:Method: GET 2024-04-13T22:14:53.167 DEBUG:boto:Path: /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.167 DEBUG:boto:Data: 2024-04-13T22:14:53.167 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.167 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.168 DEBUG:boto:Params: {} 2024-04-13T22:14:53.168 DEBUG:boto:Token: None 2024-04-13T22:14:53.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y= 2024-04-13T22:14:53.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y=', 'Content-Length': '0'} 2024-04-13T22:14:53.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26af9e1b0cd5378-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.170 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.170 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.170 DEBUG:boto:path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.170 DEBUG:boto:auth_path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.170 DEBUG:boto:Method: GET 2024-04-13T22:14:53.170 DEBUG:boto:Path: /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.171 DEBUG:boto:Data: 2024-04-13T22:14:53.171 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.171 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.171 DEBUG:boto:Params: {} 2024-04-13T22:14:53.171 DEBUG:boto:Token: None 2024-04-13T22:14:53.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y= 2024-04-13T22:14:53.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y=', 'Content-Length': '0'} 2024-04-13T22:14:53.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076c51f3233f39e2b-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.174 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.174 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.174 DEBUG:boto:path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.174 DEBUG:boto:auth_path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.174 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.174 DEBUG:boto:Path: /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.174 DEBUG:boto:Data: 2024-04-13T22:14:53.174 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.175 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.175 DEBUG:boto:Params: {} 2024-04-13T22:14:53.175 DEBUG:boto:Token: None 2024-04-13T22:14:53.175 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gePpmIaHeAdiuJPD5D82trV1PAI= 2024-04-13T22:14:53.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gePpmIaHeAdiuJPD5D82trV1PAI=', 'Content-Length': '0'} 2024-04-13T22:14:53.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a1789b8b67cbae-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.177 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.177 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.177 DEBUG:boto:path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.177 DEBUG:boto:auth_path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.177 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.177 DEBUG:boto:Path: /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.177 DEBUG:boto:Data: 2024-04-13T22:14:53.178 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.178 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.178 DEBUG:boto:Params: {} 2024-04-13T22:14:53.178 DEBUG:boto:Token: None 2024-04-13T22:14:53.178 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gePpmIaHeAdiuJPD5D82trV1PAI= 2024-04-13T22:14:53.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gePpmIaHeAdiuJPD5D82trV1PAI=', 'Content-Length': '0'} 2024-04-13T22:14:53.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001581f61eb88359ac-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:14:53.180 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.180 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.180 DEBUG:boto:path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.180 DEBUG:boto:auth_path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.181 DEBUG:boto:Method: GET 2024-04-13T22:14:53.181 DEBUG:boto:Path: /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.181 DEBUG:boto:Data: 2024-04-13T22:14:53.181 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.181 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.181 DEBUG:boto:Params: {} 2024-04-13T22:14:53.181 DEBUG:boto:Token: None 2024-04-13T22:14:53.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y= 2024-04-13T22:14:53.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y=', 'Content-Length': '0'} 2024-04-13T22:14:53.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069fa6ac1470553bc-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.183 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.184 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.184 DEBUG:boto:path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.184 DEBUG:boto:auth_path=/vwnfns-59/b?versionId=null 2024-04-13T22:14:53.184 DEBUG:boto:Method: GET 2024-04-13T22:14:53.184 DEBUG:boto:Path: /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.184 DEBUG:boto:Data: 2024-04-13T22:14:53.184 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.184 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.185 DEBUG:boto:Params: {} 2024-04-13T22:14:53.185 DEBUG:boto:Token: None 2024-04-13T22:14:53.185 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b?versionId=null 2024-04-13T22:14:53.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y= 2024-04-13T22:14:53.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UnXN493NCAxawQTHlGB7xGhYV5Y=', 'Content-Length': '0'} 2024-04-13T22:14:53.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999ca05a0d56d11f-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.187 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.187 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.187 DEBUG:boto:path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.187 DEBUG:boto:auth_path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.187 DEBUG:boto:Method: GET 2024-04-13T22:14:53.188 DEBUG:boto:Path: /vwnfns-59/?prefix=b 2024-04-13T22:14:53.188 DEBUG:boto:Data: 2024-04-13T22:14:53.188 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.188 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.188 DEBUG:boto:Params: {} 2024-04-13T22:14:53.188 DEBUG:boto:Token: None 2024-04-13T22:14:53.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dbd1f6f5fc39212-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.191 DEBUG:boto:b'vwnfns-59b1000falseb2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.191 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.191 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.191 DEBUG:boto:path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.191 DEBUG:boto:auth_path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.191 DEBUG:boto:Method: GET 2024-04-13T22:14:53.192 DEBUG:boto:Path: /vwnfns-59/?prefix=b 2024-04-13T22:14:53.192 DEBUG:boto:Data: 2024-04-13T22:14:53.192 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.192 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.192 DEBUG:boto:Params: {} 2024-04-13T22:14:53.192 DEBUG:boto:Token: None 2024-04-13T22:14:53.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aab4088b787d424-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.238 DEBUG:boto:b'vwnfns-59b1000falseb2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.239 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.239 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.239 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.239 DEBUG:boto:Path: /vwnfns-59/b 2024-04-13T22:14:53.239 DEBUG:boto:Data: 2024-04-13T22:14:53.239 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.239 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.239 DEBUG:boto:Params: {} 2024-04-13T22:14:53.239 DEBUG:boto:Token: None 2024-04-13T22:14:53.240 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b 2024-04-13T22:14:53.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nWUHlJOsBbIsz6aE8wCdE5sfVhA= 2024-04-13T22:14:53.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nWUHlJOsBbIsz6aE8wCdE5sfVhA=', 'Content-Length': '0'} 2024-04-13T22:14:53.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a61726a85b7bdc1d-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.241 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.242 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.242 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.242 DEBUG:boto:Path: /vwnfns-59/b 2024-04-13T22:14:53.242 DEBUG:boto:Data: 2024-04-13T22:14:53.242 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.242 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.242 DEBUG:boto:Params: {} 2024-04-13T22:14:53.242 DEBUG:boto:Token: None 2024-04-13T22:14:53.243 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b 2024-04-13T22:14:53.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nWUHlJOsBbIsz6aE8wCdE5sfVhA= 2024-04-13T22:14:53.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nWUHlJOsBbIsz6aE8wCdE5sfVhA=', 'Content-Length': '0'} 2024-04-13T22:14:53.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f00dba996a78453d-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-13T22:14:53.245 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.245 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.245 DEBUG:boto:Method: GET 2024-04-13T22:14:53.245 DEBUG:boto:Path: /vwnfns-59/b 2024-04-13T22:14:53.245 DEBUG:boto:Data: 2024-04-13T22:14:53.245 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.245 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.245 DEBUG:boto:Params: {} 2024-04-13T22:14:53.246 DEBUG:boto:Token: None 2024-04-13T22:14:53.246 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b 2024-04-13T22:14:53.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yXszNqRaG1BuDCPz8YKLE8t2wtQ= 2024-04-13T22:14:53.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yXszNqRaG1BuDCPz8YKLE8t2wtQ=', 'Content-Length': '0'} 2024-04-13T22:14:53.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc9714ee9365ccd8-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.250 DEBUG:boto:path=/vwnfns-59/b 2024-04-13T22:14:53.250 DEBUG:boto:auth_path=/vwnfns-59/b 2024-04-13T22:14:53.250 DEBUG:boto:Method: GET 2024-04-13T22:14:53.250 DEBUG:boto:Path: /vwnfns-59/b 2024-04-13T22:14:53.250 DEBUG:boto:Data: 2024-04-13T22:14:53.250 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.250 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.250 DEBUG:boto:Params: {} 2024-04-13T22:14:53.251 DEBUG:boto:Token: None 2024-04-13T22:14:53.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/b 2024-04-13T22:14:53.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yXszNqRaG1BuDCPz8YKLE8t2wtQ= 2024-04-13T22:14:53.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yXszNqRaG1BuDCPz8YKLE8t2wtQ=', 'Content-Length': '0'} 2024-04-13T22:14:53.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2941bf8c94591d-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.253 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.253 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.253 DEBUG:boto:path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.253 DEBUG:boto:auth_path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.253 DEBUG:boto:Method: GET 2024-04-13T22:14:53.253 DEBUG:boto:Path: /vwnfns-59/?prefix=b 2024-04-13T22:14:53.253 DEBUG:boto:Data: 2024-04-13T22:14:53.254 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.254 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.254 DEBUG:boto:Params: {} 2024-04-13T22:14:53.254 DEBUG:boto:Token: None 2024-04-13T22:14:53.254 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2e54be44c21465d-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.257 DEBUG:boto:b'vwnfns-59b1000falseb2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.257 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.257 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.257 DEBUG:boto:path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.257 DEBUG:boto:auth_path=/vwnfns-59/?prefix=b 2024-04-13T22:14:53.257 DEBUG:boto:Method: GET 2024-04-13T22:14:53.258 DEBUG:boto:Path: /vwnfns-59/?prefix=b 2024-04-13T22:14:53.258 DEBUG:boto:Data: 2024-04-13T22:14:53.258 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.258 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.258 DEBUG:boto:Params: {} 2024-04-13T22:14:53.258 DEBUG:boto:Token: None 2024-04-13T22:14:53.258 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002599a64f7e856b58-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.260 DEBUG:boto:b'vwnfns-59b1000falseb2024-04-13T22:12:16.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:14:53.261 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.261 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.261 DEBUG:boto:path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.261 DEBUG:boto:auth_path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.261 DEBUG:boto:Method: GET 2024-04-13T22:14:53.261 DEBUG:boto:Path: /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.261 DEBUG:boto:Data: 2024-04-13T22:14:53.261 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.262 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.262 DEBUG:boto:Params: {} 2024-04-13T22:14:53.262 DEBUG:boto:Token: None 2024-04-13T22:14:53.262 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E= 2024-04-13T22:14:53.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E=', 'Content-Length': '0'} 2024-04-13T22:14:53.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd127875bb23010-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.264 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.264 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.264 DEBUG:boto:path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.264 DEBUG:boto:auth_path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.264 DEBUG:boto:Method: GET 2024-04-13T22:14:53.264 DEBUG:boto:Path: /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.265 DEBUG:boto:Data: 2024-04-13T22:14:53.265 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.265 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.265 DEBUG:boto:Params: {} 2024-04-13T22:14:53.265 DEBUG:boto:Token: None 2024-04-13T22:14:53.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E= 2024-04-13T22:14:53.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E=', 'Content-Length': '0'} 2024-04-13T22:14:53.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543cde91ff7e6934-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.267 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.267 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.267 DEBUG:boto:path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.267 DEBUG:boto:auth_path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.268 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.268 DEBUG:boto:Path: /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.268 DEBUG:boto:Data: 2024-04-13T22:14:53.268 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.268 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.268 DEBUG:boto:Params: {} 2024-04-13T22:14:53.268 DEBUG:boto:Token: None 2024-04-13T22:14:53.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H1/vpmzk4w/MBqOvCjF6RQDCWIY= 2024-04-13T22:14:53.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H1/vpmzk4w/MBqOvCjF6RQDCWIY=', 'Content-Length': '0'} 2024-04-13T22:14:53.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfacb8760a556bad-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.270 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.270 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.270 DEBUG:boto:path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.271 DEBUG:boto:auth_path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.271 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.271 DEBUG:boto:Path: /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.271 DEBUG:boto:Data: 2024-04-13T22:14:53.271 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.271 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.271 DEBUG:boto:Params: {} 2024-04-13T22:14:53.271 DEBUG:boto:Token: None 2024-04-13T22:14:53.272 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H1/vpmzk4w/MBqOvCjF6RQDCWIY= 2024-04-13T22:14:53.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H1/vpmzk4w/MBqOvCjF6RQDCWIY=', 'Content-Length': '0'} 2024-04-13T22:14:53.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009bfc85e74f89277-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:14:53.274 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.274 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.274 DEBUG:boto:path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.274 DEBUG:boto:auth_path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.274 DEBUG:boto:Method: GET 2024-04-13T22:14:53.274 DEBUG:boto:Path: /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.274 DEBUG:boto:Data: 2024-04-13T22:14:53.274 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.274 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.275 DEBUG:boto:Params: {} 2024-04-13T22:14:53.275 DEBUG:boto:Token: None 2024-04-13T22:14:53.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E= 2024-04-13T22:14:53.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E=', 'Content-Length': '0'} 2024-04-13T22:14:53.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008720438991e53864-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.277 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.277 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.277 DEBUG:boto:path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.277 DEBUG:boto:auth_path=/vwnfns-59/c?versionId=null 2024-04-13T22:14:53.277 DEBUG:boto:Method: GET 2024-04-13T22:14:53.277 DEBUG:boto:Path: /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.277 DEBUG:boto:Data: 2024-04-13T22:14:53.278 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.278 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.278 DEBUG:boto:Params: {} 2024-04-13T22:14:53.278 DEBUG:boto:Token: None 2024-04-13T22:14:53.278 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c?versionId=null 2024-04-13T22:14:53.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E= 2024-04-13T22:14:53.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CcdVt2oezeFjuuLSqkX+zkwVK0E=', 'Content-Length': '0'} 2024-04-13T22:14:53.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc31ec14f240d1e4-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.280 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.280 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.280 DEBUG:boto:path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.280 DEBUG:boto:auth_path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.281 DEBUG:boto:Method: GET 2024-04-13T22:14:53.281 DEBUG:boto:Path: /vwnfns-59/?prefix=c 2024-04-13T22:14:53.281 DEBUG:boto:Data: 2024-04-13T22:14:53.281 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.281 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.281 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.281 DEBUG:boto:Params: {} 2024-04-13T22:14:53.281 DEBUG:boto:Token: None 2024-04-13T22:14:53.281 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029913759d34d7a4d-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.284 DEBUG:boto:b'vwnfns-59c1000falsec2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.285 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.285 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.285 DEBUG:boto:path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.285 DEBUG:boto:auth_path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.285 DEBUG:boto:Method: GET 2024-04-13T22:14:53.285 DEBUG:boto:Path: /vwnfns-59/?prefix=c 2024-04-13T22:14:53.285 DEBUG:boto:Data: 2024-04-13T22:14:53.285 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.285 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.285 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.286 DEBUG:boto:Params: {} 2024-04-13T22:14:53.286 DEBUG:boto:Token: None 2024-04-13T22:14:53.286 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a84270e2a2fc07b-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.288 DEBUG:boto:b'vwnfns-59c1000falsec2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.288 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.289 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.289 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.289 DEBUG:boto:Path: /vwnfns-59/c 2024-04-13T22:14:53.289 DEBUG:boto:Data: 2024-04-13T22:14:53.289 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.289 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.289 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.289 DEBUG:boto:Params: {} 2024-04-13T22:14:53.289 DEBUG:boto:Token: None 2024-04-13T22:14:53.289 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c 2024-04-13T22:14:53.290 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXsvv7qBYMB3T8YJbI1kIBtg0no= 2024-04-13T22:14:53.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXsvv7qBYMB3T8YJbI1kIBtg0no=', 'Content-Length': '0'} 2024-04-13T22:14:53.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a503b19f16f86b69-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.291 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.291 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.291 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.291 DEBUG:boto:Path: /vwnfns-59/c 2024-04-13T22:14:53.292 DEBUG:boto:Data: 2024-04-13T22:14:53.292 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.292 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.292 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.292 DEBUG:boto:Params: {} 2024-04-13T22:14:53.292 DEBUG:boto:Token: None 2024-04-13T22:14:53.292 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c 2024-04-13T22:14:53.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXsvv7qBYMB3T8YJbI1kIBtg0no= 2024-04-13T22:14:53.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXsvv7qBYMB3T8YJbI1kIBtg0no=', 'Content-Length': '0'} 2024-04-13T22:14:53.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2c238ca6f3ca41-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-13T22:14:53.295 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.295 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.295 DEBUG:boto:Method: GET 2024-04-13T22:14:53.295 DEBUG:boto:Path: /vwnfns-59/c 2024-04-13T22:14:53.295 DEBUG:boto:Data: 2024-04-13T22:14:53.295 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.295 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.295 DEBUG:boto:Params: {} 2024-04-13T22:14:53.295 DEBUG:boto:Token: None 2024-04-13T22:14:53.296 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c 2024-04-13T22:14:53.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LUuMResXj/nrnV9mi9PLwT0csqw= 2024-04-13T22:14:53.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LUuMResXj/nrnV9mi9PLwT0csqw=', 'Content-Length': '0'} 2024-04-13T22:14:53.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc10ec0605ae19d4-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.338 DEBUG:boto:path=/vwnfns-59/c 2024-04-13T22:14:53.338 DEBUG:boto:auth_path=/vwnfns-59/c 2024-04-13T22:14:53.339 DEBUG:boto:Method: GET 2024-04-13T22:14:53.339 DEBUG:boto:Path: /vwnfns-59/c 2024-04-13T22:14:53.339 DEBUG:boto:Data: 2024-04-13T22:14:53.339 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.339 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.339 DEBUG:boto:Params: {} 2024-04-13T22:14:53.339 DEBUG:boto:Token: None 2024-04-13T22:14:53.339 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/c 2024-04-13T22:14:53.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LUuMResXj/nrnV9mi9PLwT0csqw= 2024-04-13T22:14:53.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LUuMResXj/nrnV9mi9PLwT0csqw=', 'Content-Length': '0'} 2024-04-13T22:14:53.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813ab8996771f752-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.341 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.341 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.342 DEBUG:boto:path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.342 DEBUG:boto:auth_path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.342 DEBUG:boto:Method: GET 2024-04-13T22:14:53.342 DEBUG:boto:Path: /vwnfns-59/?prefix=c 2024-04-13T22:14:53.342 DEBUG:boto:Data: 2024-04-13T22:14:53.342 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.342 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.342 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.342 DEBUG:boto:Params: {} 2024-04-13T22:14:53.343 DEBUG:boto:Token: None 2024-04-13T22:14:53.343 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.343 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a3b50310f39735-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.345 DEBUG:boto:b'vwnfns-59c1000falsec2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.346 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.346 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.346 DEBUG:boto:path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.346 DEBUG:boto:auth_path=/vwnfns-59/?prefix=c 2024-04-13T22:14:53.346 DEBUG:boto:Method: GET 2024-04-13T22:14:53.346 DEBUG:boto:Path: /vwnfns-59/?prefix=c 2024-04-13T22:14:53.346 DEBUG:boto:Data: 2024-04-13T22:14:53.346 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.346 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.347 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.347 DEBUG:boto:Params: {} 2024-04-13T22:14:53.347 DEBUG:boto:Token: None 2024-04-13T22:14:53.347 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.347 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c383177318e7f946-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.349 DEBUG:boto:b'vwnfns-59c1000falsec2024-04-13T22:12:16.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:14:53.349 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.349 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.349 DEBUG:boto:path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.350 DEBUG:boto:auth_path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.350 DEBUG:boto:Method: GET 2024-04-13T22:14:53.350 DEBUG:boto:Path: /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.350 DEBUG:boto:Data: 2024-04-13T22:14:53.350 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.350 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.350 DEBUG:boto:Params: {} 2024-04-13T22:14:53.350 DEBUG:boto:Token: None 2024-04-13T22:14:53.350 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o= 2024-04-13T22:14:53.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o=', 'Content-Length': '0'} 2024-04-13T22:14:53.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e50b65420a1fc827-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.352 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.353 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.353 DEBUG:boto:path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.353 DEBUG:boto:auth_path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.353 DEBUG:boto:Method: GET 2024-04-13T22:14:53.353 DEBUG:boto:Path: /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.353 DEBUG:boto:Data: 2024-04-13T22:14:53.353 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.353 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.353 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.353 DEBUG:boto:Params: {} 2024-04-13T22:14:53.353 DEBUG:boto:Token: None 2024-04-13T22:14:53.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o= 2024-04-13T22:14:53.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o=', 'Content-Length': '0'} 2024-04-13T22:14:53.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2632feb3234d547-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.356 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.356 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.356 DEBUG:boto:path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.356 DEBUG:boto:auth_path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.356 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.356 DEBUG:boto:Path: /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.356 DEBUG:boto:Data: 2024-04-13T22:14:53.356 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.357 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.357 DEBUG:boto:Params: {} 2024-04-13T22:14:53.357 DEBUG:boto:Token: None 2024-04-13T22:14:53.357 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.357 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2L0+xPwEl95bepJyVqFGHPXg6zs= 2024-04-13T22:14:53.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2L0+xPwEl95bepJyVqFGHPXg6zs=', 'Content-Length': '0'} 2024-04-13T22:14:53.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000402ab2d565dbf8fb-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.359 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.359 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.359 DEBUG:boto:path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.359 DEBUG:boto:auth_path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.359 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.359 DEBUG:boto:Path: /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.359 DEBUG:boto:Data: 2024-04-13T22:14:53.360 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.360 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.360 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.360 DEBUG:boto:Params: {} 2024-04-13T22:14:53.360 DEBUG:boto:Token: None 2024-04-13T22:14:53.360 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2L0+xPwEl95bepJyVqFGHPXg6zs= 2024-04-13T22:14:53.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2L0+xPwEl95bepJyVqFGHPXg6zs=', 'Content-Length': '0'} 2024-04-13T22:14:53.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948a9c0dddeea9f4-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:14:53.362 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.362 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.362 DEBUG:boto:path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.362 DEBUG:boto:auth_path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.363 DEBUG:boto:Method: GET 2024-04-13T22:14:53.363 DEBUG:boto:Path: /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.363 DEBUG:boto:Data: 2024-04-13T22:14:53.363 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.363 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.363 DEBUG:boto:Params: {} 2024-04-13T22:14:53.363 DEBUG:boto:Token: None 2024-04-13T22:14:53.363 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o= 2024-04-13T22:14:53.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o=', 'Content-Length': '0'} 2024-04-13T22:14:53.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f342241d29a60cb-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.366 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.366 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.366 DEBUG:boto:path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.366 DEBUG:boto:auth_path=/vwnfns-59/d?versionId=null 2024-04-13T22:14:53.366 DEBUG:boto:Method: GET 2024-04-13T22:14:53.366 DEBUG:boto:Path: /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.366 DEBUG:boto:Data: 2024-04-13T22:14:53.366 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.366 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.367 DEBUG:boto:Params: {} 2024-04-13T22:14:53.367 DEBUG:boto:Token: None 2024-04-13T22:14:53.367 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d?versionId=null 2024-04-13T22:14:53.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o= 2024-04-13T22:14:53.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QATiBLdLQYgxh7pbN/kS+Pkc56o=', 'Content-Length': '0'} 2024-04-13T22:14:53.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832790dd289348f4-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.369 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.369 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.369 DEBUG:boto:path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.369 DEBUG:boto:auth_path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.369 DEBUG:boto:Method: GET 2024-04-13T22:14:53.369 DEBUG:boto:Path: /vwnfns-59/?prefix=d 2024-04-13T22:14:53.369 DEBUG:boto:Data: 2024-04-13T22:14:53.369 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.370 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.370 DEBUG:boto:Params: {} 2024-04-13T22:14:53.370 DEBUG:boto:Token: None 2024-04-13T22:14:53.370 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.370 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016dd999901a80a54-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.372 DEBUG:boto:b'vwnfns-59d1000falsed2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.373 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.373 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.373 DEBUG:boto:path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.373 DEBUG:boto:auth_path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.373 DEBUG:boto:Method: GET 2024-04-13T22:14:53.373 DEBUG:boto:Path: /vwnfns-59/?prefix=d 2024-04-13T22:14:53.373 DEBUG:boto:Data: 2024-04-13T22:14:53.373 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.373 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.374 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.374 DEBUG:boto:Params: {} 2024-04-13T22:14:53.374 DEBUG:boto:Token: None 2024-04-13T22:14:53.374 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.374 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004cc7ce507058fc4-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.376 DEBUG:boto:b'vwnfns-59d1000falsed2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.376 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.377 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.377 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.377 DEBUG:boto:Path: /vwnfns-59/d 2024-04-13T22:14:53.377 DEBUG:boto:Data: 2024-04-13T22:14:53.377 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.377 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.377 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.377 DEBUG:boto:Params: {} 2024-04-13T22:14:53.377 DEBUG:boto:Token: None 2024-04-13T22:14:53.377 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d 2024-04-13T22:14:53.378 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c8NbkChmttHV671UuEzm3Ios/0Y= 2024-04-13T22:14:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c8NbkChmttHV671UuEzm3Ios/0Y=', 'Content-Length': '0'} 2024-04-13T22:14:53.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fe847e47607227-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.379 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.380 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.380 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.380 DEBUG:boto:Path: /vwnfns-59/d 2024-04-13T22:14:53.380 DEBUG:boto:Data: 2024-04-13T22:14:53.380 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.380 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.380 DEBUG:boto:Params: {} 2024-04-13T22:14:53.380 DEBUG:boto:Token: None 2024-04-13T22:14:53.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d 2024-04-13T22:14:53.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c8NbkChmttHV671UuEzm3Ios/0Y= 2024-04-13T22:14:53.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c8NbkChmttHV671UuEzm3Ios/0Y=', 'Content-Length': '0'} 2024-04-13T22:14:53.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a65e15f23c8155e-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-13T22:14:53.383 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.383 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.383 DEBUG:boto:Method: GET 2024-04-13T22:14:53.383 DEBUG:boto:Path: /vwnfns-59/d 2024-04-13T22:14:53.383 DEBUG:boto:Data: 2024-04-13T22:14:53.383 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.383 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.383 DEBUG:boto:Params: {} 2024-04-13T22:14:53.384 DEBUG:boto:Token: None 2024-04-13T22:14:53.384 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d 2024-04-13T22:14:53.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b93Pbi/lLOAVNFlvqLeW8JXFrVw= 2024-04-13T22:14:53.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b93Pbi/lLOAVNFlvqLeW8JXFrVw=', 'Content-Length': '0'} 2024-04-13T22:14:53.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e483bd977836325-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.386 DEBUG:boto:path=/vwnfns-59/d 2024-04-13T22:14:53.386 DEBUG:boto:auth_path=/vwnfns-59/d 2024-04-13T22:14:53.386 DEBUG:boto:Method: GET 2024-04-13T22:14:53.386 DEBUG:boto:Path: /vwnfns-59/d 2024-04-13T22:14:53.386 DEBUG:boto:Data: 2024-04-13T22:14:53.386 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.386 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.386 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.386 DEBUG:boto:Params: {} 2024-04-13T22:14:53.387 DEBUG:boto:Token: None 2024-04-13T22:14:53.387 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/d 2024-04-13T22:14:53.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b93Pbi/lLOAVNFlvqLeW8JXFrVw= 2024-04-13T22:14:53.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b93Pbi/lLOAVNFlvqLeW8JXFrVw=', 'Content-Length': '0'} 2024-04-13T22:14:53.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424cf6538c4d393d-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.434 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.434 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.435 DEBUG:boto:path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.435 DEBUG:boto:auth_path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.435 DEBUG:boto:Method: GET 2024-04-13T22:14:53.435 DEBUG:boto:Path: /vwnfns-59/?prefix=d 2024-04-13T22:14:53.435 DEBUG:boto:Data: 2024-04-13T22:14:53.435 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.435 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.435 DEBUG:boto:Params: {} 2024-04-13T22:14:53.435 DEBUG:boto:Token: None 2024-04-13T22:14:53.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0fd5565abf158c1-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.438 DEBUG:boto:b'vwnfns-59d1000falsed2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.439 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.439 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.439 DEBUG:boto:path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.439 DEBUG:boto:auth_path=/vwnfns-59/?prefix=d 2024-04-13T22:14:53.439 DEBUG:boto:Method: GET 2024-04-13T22:14:53.439 DEBUG:boto:Path: /vwnfns-59/?prefix=d 2024-04-13T22:14:53.439 DEBUG:boto:Data: 2024-04-13T22:14:53.439 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.440 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.440 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.440 DEBUG:boto:Params: {} 2024-04-13T22:14:53.440 DEBUG:boto:Token: None 2024-04-13T22:14:53.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be39d7e09646545-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.442 DEBUG:boto:b'vwnfns-59d1000falsed2024-04-13T22:12:16.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:14:53.442 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.442 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.443 DEBUG:boto:path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.443 DEBUG:boto:auth_path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.443 DEBUG:boto:Method: GET 2024-04-13T22:14:53.443 DEBUG:boto:Path: /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.443 DEBUG:boto:Data: 2024-04-13T22:14:53.443 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.443 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.443 DEBUG:boto:Params: {} 2024-04-13T22:14:53.443 DEBUG:boto:Token: None 2024-04-13T22:14:53.443 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q= 2024-04-13T22:14:53.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q=', 'Content-Length': '0'} 2024-04-13T22:14:53.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008540e82f3b957ecf-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.445 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.445 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.446 DEBUG:boto:path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.446 DEBUG:boto:auth_path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.446 DEBUG:boto:Method: GET 2024-04-13T22:14:53.446 DEBUG:boto:Path: /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.446 DEBUG:boto:Data: 2024-04-13T22:14:53.446 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.446 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.446 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.446 DEBUG:boto:Params: {} 2024-04-13T22:14:53.446 DEBUG:boto:Token: None 2024-04-13T22:14:53.447 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q= 2024-04-13T22:14:53.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q=', 'Content-Length': '0'} 2024-04-13T22:14:53.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103f174bfb28f248-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.449 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.449 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.449 DEBUG:boto:path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.449 DEBUG:boto:auth_path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.449 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.449 DEBUG:boto:Path: /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.449 DEBUG:boto:Data: 2024-04-13T22:14:53.450 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.450 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.450 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.450 DEBUG:boto:Params: {} 2024-04-13T22:14:53.450 DEBUG:boto:Token: None 2024-04-13T22:14:53.450 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.450 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r3381wADP1nbYNg1OivuR4eVHqA= 2024-04-13T22:14:53.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r3381wADP1nbYNg1OivuR4eVHqA=', 'Content-Length': '0'} 2024-04-13T22:14:53.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441f0f4d15d01fba-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.452 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.452 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.453 DEBUG:boto:path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.453 DEBUG:boto:auth_path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.453 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.453 DEBUG:boto:Path: /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.453 DEBUG:boto:Data: 2024-04-13T22:14:53.453 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.453 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.453 DEBUG:boto:Params: {} 2024-04-13T22:14:53.453 DEBUG:boto:Token: None 2024-04-13T22:14:53.454 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.454 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r3381wADP1nbYNg1OivuR4eVHqA= 2024-04-13T22:14:53.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r3381wADP1nbYNg1OivuR4eVHqA=', 'Content-Length': '0'} 2024-04-13T22:14:53.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cedf6a64a009933c-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:14:53.458 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.458 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.458 DEBUG:boto:path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.458 DEBUG:boto:auth_path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.458 DEBUG:boto:Method: GET 2024-04-13T22:14:53.458 DEBUG:boto:Path: /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.458 DEBUG:boto:Data: 2024-04-13T22:14:53.458 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.459 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.459 DEBUG:boto:Params: {} 2024-04-13T22:14:53.459 DEBUG:boto:Token: None 2024-04-13T22:14:53.459 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q= 2024-04-13T22:14:53.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q=', 'Content-Length': '0'} 2024-04-13T22:14:53.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464c35605158dc73-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.461 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.461 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.461 DEBUG:boto:path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.461 DEBUG:boto:auth_path=/vwnfns-59/e?versionId=null 2024-04-13T22:14:53.461 DEBUG:boto:Method: GET 2024-04-13T22:14:53.461 DEBUG:boto:Path: /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.461 DEBUG:boto:Data: 2024-04-13T22:14:53.462 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.462 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.462 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.462 DEBUG:boto:Params: {} 2024-04-13T22:14:53.462 DEBUG:boto:Token: None 2024-04-13T22:14:53.462 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e?versionId=null 2024-04-13T22:14:53.462 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q= 2024-04-13T22:14:53.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tnTp5/e6ZW15NIN2DA2EiC8OJ/Q=', 'Content-Length': '0'} 2024-04-13T22:14:53.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771fe8ee436d22e0-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.464 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.464 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.464 DEBUG:boto:path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.464 DEBUG:boto:auth_path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.464 DEBUG:boto:Method: GET 2024-04-13T22:14:53.464 DEBUG:boto:Path: /vwnfns-59/?prefix=e 2024-04-13T22:14:53.465 DEBUG:boto:Data: 2024-04-13T22:14:53.465 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.465 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.465 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.465 DEBUG:boto:Params: {} 2024-04-13T22:14:53.465 DEBUG:boto:Token: None 2024-04-13T22:14:53.465 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055517a5c5b2c1259-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.468 DEBUG:boto:b'vwnfns-59e1000falsee2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.468 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.468 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.468 DEBUG:boto:path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.468 DEBUG:boto:auth_path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.468 DEBUG:boto:Method: GET 2024-04-13T22:14:53.469 DEBUG:boto:Path: /vwnfns-59/?prefix=e 2024-04-13T22:14:53.469 DEBUG:boto:Data: 2024-04-13T22:14:53.469 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.469 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.469 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.469 DEBUG:boto:Params: {} 2024-04-13T22:14:53.469 DEBUG:boto:Token: None 2024-04-13T22:14:53.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008264a1d62fe99b26-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.471 DEBUG:boto:b'vwnfns-59e1000falsee2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.471 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.472 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.472 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.472 DEBUG:boto:Path: /vwnfns-59/e 2024-04-13T22:14:53.472 DEBUG:boto:Data: 2024-04-13T22:14:53.472 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.472 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.472 DEBUG:boto:Params: {} 2024-04-13T22:14:53.472 DEBUG:boto:Token: None 2024-04-13T22:14:53.472 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e 2024-04-13T22:14:53.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OtY0BZ0a7X070fp5qhWbt1kB+wk= 2024-04-13T22:14:53.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OtY0BZ0a7X070fp5qhWbt1kB+wk=', 'Content-Length': '0'} 2024-04-13T22:14:53.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b00820098ac0e0ff-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.474 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.475 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.475 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.475 DEBUG:boto:Path: /vwnfns-59/e 2024-04-13T22:14:53.475 DEBUG:boto:Data: 2024-04-13T22:14:53.475 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.475 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.475 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.475 DEBUG:boto:Params: {} 2024-04-13T22:14:53.475 DEBUG:boto:Token: None 2024-04-13T22:14:53.475 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e 2024-04-13T22:14:53.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OtY0BZ0a7X070fp5qhWbt1kB+wk= 2024-04-13T22:14:53.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OtY0BZ0a7X070fp5qhWbt1kB+wk=', 'Content-Length': '0'} 2024-04-13T22:14:53.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdaeb7785c6715cc-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-13T22:14:53.477 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.477 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.478 DEBUG:boto:Method: GET 2024-04-13T22:14:53.478 DEBUG:boto:Path: /vwnfns-59/e 2024-04-13T22:14:53.478 DEBUG:boto:Data: 2024-04-13T22:14:53.478 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.478 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.478 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.478 DEBUG:boto:Params: {} 2024-04-13T22:14:53.478 DEBUG:boto:Token: None 2024-04-13T22:14:53.478 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e 2024-04-13T22:14:53.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Cy5ZyZkbES+VByWWQ1WbZS1lgQ= 2024-04-13T22:14:53.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Cy5ZyZkbES+VByWWQ1WbZS1lgQ=', 'Content-Length': '0'} 2024-04-13T22:14:53.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce2116f88467e99-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.526 DEBUG:boto:path=/vwnfns-59/e 2024-04-13T22:14:53.527 DEBUG:boto:auth_path=/vwnfns-59/e 2024-04-13T22:14:53.527 DEBUG:boto:Method: GET 2024-04-13T22:14:53.527 DEBUG:boto:Path: /vwnfns-59/e 2024-04-13T22:14:53.527 DEBUG:boto:Data: 2024-04-13T22:14:53.527 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.527 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.527 DEBUG:boto:Params: {} 2024-04-13T22:14:53.527 DEBUG:boto:Token: None 2024-04-13T22:14:53.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/e 2024-04-13T22:14:53.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Cy5ZyZkbES+VByWWQ1WbZS1lgQ= 2024-04-13T22:14:53.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Cy5ZyZkbES+VByWWQ1WbZS1lgQ=', 'Content-Length': '0'} 2024-04-13T22:14:53.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b4ca148bac94ea-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.530 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.530 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.530 DEBUG:boto:path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.530 DEBUG:boto:auth_path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.530 DEBUG:boto:Method: GET 2024-04-13T22:14:53.530 DEBUG:boto:Path: /vwnfns-59/?prefix=e 2024-04-13T22:14:53.531 DEBUG:boto:Data: 2024-04-13T22:14:53.531 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.531 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.531 DEBUG:boto:Params: {} 2024-04-13T22:14:53.531 DEBUG:boto:Token: None 2024-04-13T22:14:53.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd1adaefe7d23df-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.534 DEBUG:boto:b'vwnfns-59e1000falsee2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.534 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.535 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.535 DEBUG:boto:path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.535 DEBUG:boto:auth_path=/vwnfns-59/?prefix=e 2024-04-13T22:14:53.535 DEBUG:boto:Method: GET 2024-04-13T22:14:53.535 DEBUG:boto:Path: /vwnfns-59/?prefix=e 2024-04-13T22:14:53.535 DEBUG:boto:Data: 2024-04-13T22:14:53.535 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.535 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.535 DEBUG:boto:Params: {} 2024-04-13T22:14:53.535 DEBUG:boto:Token: None 2024-04-13T22:14:53.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda3b430968aff78-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.537 DEBUG:boto:b'vwnfns-59e1000falsee2024-04-13T22:12:16.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:14:53.538 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.538 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.538 DEBUG:boto:path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.538 DEBUG:boto:auth_path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.538 DEBUG:boto:Method: GET 2024-04-13T22:14:53.538 DEBUG:boto:Path: /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.538 DEBUG:boto:Data: 2024-04-13T22:14:53.539 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.539 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.539 DEBUG:boto:Params: {} 2024-04-13T22:14:53.539 DEBUG:boto:Token: None 2024-04-13T22:14:53.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI= 2024-04-13T22:14:53.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI=', 'Content-Length': '0'} 2024-04-13T22:14:53.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff60995ac069bd64-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.541 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.541 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.541 DEBUG:boto:path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.541 DEBUG:boto:auth_path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.541 DEBUG:boto:Method: GET 2024-04-13T22:14:53.541 DEBUG:boto:Path: /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.541 DEBUG:boto:Data: 2024-04-13T22:14:53.542 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.542 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.542 DEBUG:boto:Params: {} 2024-04-13T22:14:53.542 DEBUG:boto:Token: None 2024-04-13T22:14:53.542 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI= 2024-04-13T22:14:53.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI=', 'Content-Length': '0'} 2024-04-13T22:14:53.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daae4b4be6ed18c4-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.544 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.544 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.544 DEBUG:boto:path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.544 DEBUG:boto:auth_path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.545 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.545 DEBUG:boto:Path: /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.545 DEBUG:boto:Data: 2024-04-13T22:14:53.545 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.545 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.545 DEBUG:boto:Params: {} 2024-04-13T22:14:53.545 DEBUG:boto:Token: None 2024-04-13T22:14:53.546 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4EQLoJiGNCxOa7xDfiHabH/jxwU= 2024-04-13T22:14:53.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4EQLoJiGNCxOa7xDfiHabH/jxwU=', 'Content-Length': '0'} 2024-04-13T22:14:53.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7c8a2e8cb5f1c0e-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.547 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.548 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.548 DEBUG:boto:path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.548 DEBUG:boto:auth_path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.548 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.548 DEBUG:boto:Path: /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.548 DEBUG:boto:Data: 2024-04-13T22:14:53.548 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.548 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.548 DEBUG:boto:Params: {} 2024-04-13T22:14:53.549 DEBUG:boto:Token: None 2024-04-13T22:14:53.549 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4EQLoJiGNCxOa7xDfiHabH/jxwU= 2024-04-13T22:14:53.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4EQLoJiGNCxOa7xDfiHabH/jxwU=', 'Content-Length': '0'} 2024-04-13T22:14:53.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100a2cd4d7e246ac-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:14:53.550 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.551 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.551 DEBUG:boto:path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.551 DEBUG:boto:auth_path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.551 DEBUG:boto:Method: GET 2024-04-13T22:14:53.551 DEBUG:boto:Path: /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.551 DEBUG:boto:Data: 2024-04-13T22:14:53.551 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.551 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.551 DEBUG:boto:Params: {} 2024-04-13T22:14:53.551 DEBUG:boto:Token: None 2024-04-13T22:14:53.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI= 2024-04-13T22:14:53.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI=', 'Content-Length': '0'} 2024-04-13T22:14:53.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d162be7610584b-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.553 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.553 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.554 DEBUG:boto:path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.554 DEBUG:boto:auth_path=/vwnfns-59/f?versionId=null 2024-04-13T22:14:53.554 DEBUG:boto:Method: GET 2024-04-13T22:14:53.554 DEBUG:boto:Path: /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.554 DEBUG:boto:Data: 2024-04-13T22:14:53.554 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.554 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.554 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.554 DEBUG:boto:Params: {} 2024-04-13T22:14:53.554 DEBUG:boto:Token: None 2024-04-13T22:14:53.555 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f?versionId=null 2024-04-13T22:14:53.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI= 2024-04-13T22:14:53.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RCnQgEHRSqxWbAZ7VAATEMpUjwI=', 'Content-Length': '0'} 2024-04-13T22:14:53.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376e22486161be9c-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.556 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.557 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.557 DEBUG:boto:path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.557 DEBUG:boto:auth_path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.557 DEBUG:boto:Method: GET 2024-04-13T22:14:53.557 DEBUG:boto:Path: /vwnfns-59/?prefix=f 2024-04-13T22:14:53.557 DEBUG:boto:Data: 2024-04-13T22:14:53.557 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.557 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.557 DEBUG:boto:Params: {} 2024-04-13T22:14:53.558 DEBUG:boto:Token: None 2024-04-13T22:14:53.558 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.558 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5dc582c92d3ae37-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.560 DEBUG:boto:b'vwnfns-59f1000falsef2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.560 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.560 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.561 DEBUG:boto:path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.561 DEBUG:boto:auth_path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.561 DEBUG:boto:Method: GET 2024-04-13T22:14:53.561 DEBUG:boto:Path: /vwnfns-59/?prefix=f 2024-04-13T22:14:53.561 DEBUG:boto:Data: 2024-04-13T22:14:53.561 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.561 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.561 DEBUG:boto:Params: {} 2024-04-13T22:14:53.561 DEBUG:boto:Token: None 2024-04-13T22:14:53.561 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33c83eac70227f8-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.564 DEBUG:boto:b'vwnfns-59f1000falsef2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.564 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.564 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.564 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.564 DEBUG:boto:Path: /vwnfns-59/f 2024-04-13T22:14:53.564 DEBUG:boto:Data: 2024-04-13T22:14:53.564 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.564 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.565 DEBUG:boto:Params: {} 2024-04-13T22:14:53.565 DEBUG:boto:Token: None 2024-04-13T22:14:53.565 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f 2024-04-13T22:14:53.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5cy9f0vd4AWt1MwzkzFpHkeQDlI= 2024-04-13T22:14:53.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5cy9f0vd4AWt1MwzkzFpHkeQDlI=', 'Content-Length': '0'} 2024-04-13T22:14:53.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a52300cc2589558b-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.567 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.567 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.567 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.567 DEBUG:boto:Path: /vwnfns-59/f 2024-04-13T22:14:53.567 DEBUG:boto:Data: 2024-04-13T22:14:53.567 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.567 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.568 DEBUG:boto:Params: {} 2024-04-13T22:14:53.568 DEBUG:boto:Token: None 2024-04-13T22:14:53.568 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f 2024-04-13T22:14:53.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5cy9f0vd4AWt1MwzkzFpHkeQDlI= 2024-04-13T22:14:53.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5cy9f0vd4AWt1MwzkzFpHkeQDlI=', 'Content-Length': '0'} 2024-04-13T22:14:53.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c686528c1f14dc97-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-13T22:14:53.570 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.570 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.570 DEBUG:boto:Method: GET 2024-04-13T22:14:53.571 DEBUG:boto:Path: /vwnfns-59/f 2024-04-13T22:14:53.571 DEBUG:boto:Data: 2024-04-13T22:14:53.571 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.571 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.571 DEBUG:boto:Params: {} 2024-04-13T22:14:53.571 DEBUG:boto:Token: None 2024-04-13T22:14:53.571 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f 2024-04-13T22:14:53.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c84+kGH229VsmoWU0mhKG4ffTkY= 2024-04-13T22:14:53.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c84+kGH229VsmoWU0mhKG4ffTkY=', 'Content-Length': '0'} 2024-04-13T22:14:53.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077fbe6f296009ce9-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.573 DEBUG:boto:path=/vwnfns-59/f 2024-04-13T22:14:53.573 DEBUG:boto:auth_path=/vwnfns-59/f 2024-04-13T22:14:53.574 DEBUG:boto:Method: GET 2024-04-13T22:14:53.574 DEBUG:boto:Path: /vwnfns-59/f 2024-04-13T22:14:53.574 DEBUG:boto:Data: 2024-04-13T22:14:53.574 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.574 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.574 DEBUG:boto:Params: {} 2024-04-13T22:14:53.574 DEBUG:boto:Token: None 2024-04-13T22:14:53.574 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/f 2024-04-13T22:14:53.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c84+kGH229VsmoWU0mhKG4ffTkY= 2024-04-13T22:14:53.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c84+kGH229VsmoWU0mhKG4ffTkY=', 'Content-Length': '0'} 2024-04-13T22:14:53.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b6c7d2ed177976-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.576 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.576 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.577 DEBUG:boto:path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.577 DEBUG:boto:auth_path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.577 DEBUG:boto:Method: GET 2024-04-13T22:14:53.577 DEBUG:boto:Path: /vwnfns-59/?prefix=f 2024-04-13T22:14:53.577 DEBUG:boto:Data: 2024-04-13T22:14:53.577 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.577 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.577 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.577 DEBUG:boto:Params: {} 2024-04-13T22:14:53.577 DEBUG:boto:Token: None 2024-04-13T22:14:53.578 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.578 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009423ed5ab4da90cd-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.580 DEBUG:boto:b'vwnfns-59f1000falsef2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.580 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.581 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.581 DEBUG:boto:path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.581 DEBUG:boto:auth_path=/vwnfns-59/?prefix=f 2024-04-13T22:14:53.581 DEBUG:boto:Method: GET 2024-04-13T22:14:53.581 DEBUG:boto:Path: /vwnfns-59/?prefix=f 2024-04-13T22:14:53.581 DEBUG:boto:Data: 2024-04-13T22:14:53.581 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.581 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.581 DEBUG:boto:Params: {} 2024-04-13T22:14:53.581 DEBUG:boto:Token: None 2024-04-13T22:14:53.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf80ea66536d156-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.583 DEBUG:boto:b'vwnfns-59f1000falsef2024-04-13T22:12:16.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:14:53.584 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.584 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.584 DEBUG:boto:path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.584 DEBUG:boto:auth_path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.584 DEBUG:boto:Method: GET 2024-04-13T22:14:53.584 DEBUG:boto:Path: /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.584 DEBUG:boto:Data: 2024-04-13T22:14:53.584 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.584 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.585 DEBUG:boto:Params: {} 2024-04-13T22:14:53.585 DEBUG:boto:Token: None 2024-04-13T22:14:53.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c= 2024-04-13T22:14:53.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c=', 'Content-Length': '0'} 2024-04-13T22:14:53.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240dc79df92235f3-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.587 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.587 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.587 DEBUG:boto:path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.587 DEBUG:boto:auth_path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.587 DEBUG:boto:Method: GET 2024-04-13T22:14:53.587 DEBUG:boto:Path: /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.587 DEBUG:boto:Data: 2024-04-13T22:14:53.588 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.588 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.588 DEBUG:boto:Params: {} 2024-04-13T22:14:53.588 DEBUG:boto:Token: None 2024-04-13T22:14:53.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c= 2024-04-13T22:14:53.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c=', 'Content-Length': '0'} 2024-04-13T22:14:53.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcdc0b0d315aaed-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.630 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.630 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.631 DEBUG:boto:path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.631 DEBUG:boto:auth_path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.631 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.631 DEBUG:boto:Path: /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.631 DEBUG:boto:Data: 2024-04-13T22:14:53.631 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.631 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.631 DEBUG:boto:Params: {} 2024-04-13T22:14:53.631 DEBUG:boto:Token: None 2024-04-13T22:14:53.632 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f59tKuT8wu2Q4YJ9QDCkk1aKVqE= 2024-04-13T22:14:53.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f59tKuT8wu2Q4YJ9QDCkk1aKVqE=', 'Content-Length': '0'} 2024-04-13T22:14:53.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006135e444ad64b541-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.634 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.634 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.634 DEBUG:boto:path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.634 DEBUG:boto:auth_path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.634 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.634 DEBUG:boto:Path: /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.634 DEBUG:boto:Data: 2024-04-13T22:14:53.634 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.634 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.635 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.635 DEBUG:boto:Params: {} 2024-04-13T22:14:53.635 DEBUG:boto:Token: None 2024-04-13T22:14:53.635 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f59tKuT8wu2Q4YJ9QDCkk1aKVqE= 2024-04-13T22:14:53.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f59tKuT8wu2Q4YJ9QDCkk1aKVqE=', 'Content-Length': '0'} 2024-04-13T22:14:53.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3fa38df91a561e-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:14:53.637 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.637 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.637 DEBUG:boto:path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.637 DEBUG:boto:auth_path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.637 DEBUG:boto:Method: GET 2024-04-13T22:14:53.637 DEBUG:boto:Path: /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.637 DEBUG:boto:Data: 2024-04-13T22:14:53.637 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.637 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.638 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.638 DEBUG:boto:Params: {} 2024-04-13T22:14:53.638 DEBUG:boto:Token: None 2024-04-13T22:14:53.638 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c= 2024-04-13T22:14:53.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c=', 'Content-Length': '0'} 2024-04-13T22:14:53.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185391d74bc9c965-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.640 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.640 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.640 DEBUG:boto:path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.640 DEBUG:boto:auth_path=/vwnfns-59/g?versionId=null 2024-04-13T22:14:53.640 DEBUG:boto:Method: GET 2024-04-13T22:14:53.640 DEBUG:boto:Path: /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.640 DEBUG:boto:Data: 2024-04-13T22:14:53.640 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.640 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.640 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.641 DEBUG:boto:Params: {} 2024-04-13T22:14:53.641 DEBUG:boto:Token: None 2024-04-13T22:14:53.641 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g?versionId=null 2024-04-13T22:14:53.641 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c= 2024-04-13T22:14:53.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CU02HRHexnS9DBqPrIdzaEpMQ1c=', 'Content-Length': '0'} 2024-04-13T22:14:53.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003287c9c9978327df-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.643 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.643 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.643 DEBUG:boto:path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.643 DEBUG:boto:auth_path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.643 DEBUG:boto:Method: GET 2024-04-13T22:14:53.643 DEBUG:boto:Path: /vwnfns-59/?prefix=g 2024-04-13T22:14:53.643 DEBUG:boto:Data: 2024-04-13T22:14:53.643 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.644 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.644 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.644 DEBUG:boto:Params: {} 2024-04-13T22:14:53.644 DEBUG:boto:Token: None 2024-04-13T22:14:53.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a74b0cc3c33c73cc-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.646 DEBUG:boto:b'vwnfns-59g1000falseg2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.647 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.647 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.647 DEBUG:boto:path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.647 DEBUG:boto:auth_path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.647 DEBUG:boto:Method: GET 2024-04-13T22:14:53.647 DEBUG:boto:Path: /vwnfns-59/?prefix=g 2024-04-13T22:14:53.647 DEBUG:boto:Data: 2024-04-13T22:14:53.647 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.647 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.647 DEBUG:boto:Params: {} 2024-04-13T22:14:53.648 DEBUG:boto:Token: None 2024-04-13T22:14:53.648 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e11fcc8bd8e03be0-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.650 DEBUG:boto:b'vwnfns-59g1000falseg2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.650 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.650 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.650 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.650 DEBUG:boto:Path: /vwnfns-59/g 2024-04-13T22:14:53.650 DEBUG:boto:Data: 2024-04-13T22:14:53.650 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.650 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.651 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.651 DEBUG:boto:Params: {} 2024-04-13T22:14:53.651 DEBUG:boto:Token: None 2024-04-13T22:14:53.651 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g 2024-04-13T22:14:53.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AlmTSZLOOtz765UX9xnvtTqR82M= 2024-04-13T22:14:53.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AlmTSZLOOtz765UX9xnvtTqR82M=', 'Content-Length': '0'} 2024-04-13T22:14:53.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331f120b2d378fe9-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.653 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.653 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.653 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.653 DEBUG:boto:Path: /vwnfns-59/g 2024-04-13T22:14:53.653 DEBUG:boto:Data: 2024-04-13T22:14:53.653 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.653 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.653 DEBUG:boto:Params: {} 2024-04-13T22:14:53.653 DEBUG:boto:Token: None 2024-04-13T22:14:53.654 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g 2024-04-13T22:14:53.654 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AlmTSZLOOtz765UX9xnvtTqR82M= 2024-04-13T22:14:53.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AlmTSZLOOtz765UX9xnvtTqR82M=', 'Content-Length': '0'} 2024-04-13T22:14:53.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ac0f8fc7b5a100-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-13T22:14:53.656 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.656 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.656 DEBUG:boto:Method: GET 2024-04-13T22:14:53.656 DEBUG:boto:Path: /vwnfns-59/g 2024-04-13T22:14:53.656 DEBUG:boto:Data: 2024-04-13T22:14:53.656 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.656 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.657 DEBUG:boto:Params: {} 2024-04-13T22:14:53.657 DEBUG:boto:Token: None 2024-04-13T22:14:53.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g 2024-04-13T22:14:53.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xzrq4/k90ckqQ3TUY8yVVyQJXts= 2024-04-13T22:14:53.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xzrq4/k90ckqQ3TUY8yVVyQJXts=', 'Content-Length': '0'} 2024-04-13T22:14:53.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ccf0ecde86f6966-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.706 DEBUG:boto:path=/vwnfns-59/g 2024-04-13T22:14:53.707 DEBUG:boto:auth_path=/vwnfns-59/g 2024-04-13T22:14:53.707 DEBUG:boto:Method: GET 2024-04-13T22:14:53.707 DEBUG:boto:Path: /vwnfns-59/g 2024-04-13T22:14:53.707 DEBUG:boto:Data: 2024-04-13T22:14:53.707 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.707 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.707 DEBUG:boto:Params: {} 2024-04-13T22:14:53.707 DEBUG:boto:Token: None 2024-04-13T22:14:53.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/g 2024-04-13T22:14:53.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xzrq4/k90ckqQ3TUY8yVVyQJXts= 2024-04-13T22:14:53.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xzrq4/k90ckqQ3TUY8yVVyQJXts=', 'Content-Length': '0'} 2024-04-13T22:14:53.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052742c00c81a9676-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.710 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.710 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.710 DEBUG:boto:path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.710 DEBUG:boto:auth_path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.710 DEBUG:boto:Method: GET 2024-04-13T22:14:53.711 DEBUG:boto:Path: /vwnfns-59/?prefix=g 2024-04-13T22:14:53.711 DEBUG:boto:Data: 2024-04-13T22:14:53.711 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.711 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.711 DEBUG:boto:Params: {} 2024-04-13T22:14:53.711 DEBUG:boto:Token: None 2024-04-13T22:14:53.711 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07f5deab1e2e783-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.714 DEBUG:boto:b'vwnfns-59g1000falseg2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.714 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.714 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.714 DEBUG:boto:path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.714 DEBUG:boto:auth_path=/vwnfns-59/?prefix=g 2024-04-13T22:14:53.715 DEBUG:boto:Method: GET 2024-04-13T22:14:53.715 DEBUG:boto:Path: /vwnfns-59/?prefix=g 2024-04-13T22:14:53.715 DEBUG:boto:Data: 2024-04-13T22:14:53.715 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.715 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.715 DEBUG:boto:Params: {} 2024-04-13T22:14:53.715 DEBUG:boto:Token: None 2024-04-13T22:14:53.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7236b427022c15a-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.717 DEBUG:boto:b'vwnfns-59g1000falseg2024-04-13T22:12:16.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:14:53.718 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.718 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.718 DEBUG:boto:path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.718 DEBUG:boto:auth_path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.718 DEBUG:boto:Method: GET 2024-04-13T22:14:53.719 DEBUG:boto:Path: /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.719 DEBUG:boto:Data: 2024-04-13T22:14:53.719 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.719 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.719 DEBUG:boto:Params: {} 2024-04-13T22:14:53.719 DEBUG:boto:Token: None 2024-04-13T22:14:53.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI= 2024-04-13T22:14:53.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI=', 'Content-Length': '0'} 2024-04-13T22:14:53.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8295765a3e4b74-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.722 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.722 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.722 DEBUG:boto:path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.722 DEBUG:boto:auth_path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.722 DEBUG:boto:Method: GET 2024-04-13T22:14:53.722 DEBUG:boto:Path: /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.722 DEBUG:boto:Data: 2024-04-13T22:14:53.722 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.722 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.723 DEBUG:boto:Params: {} 2024-04-13T22:14:53.723 DEBUG:boto:Token: None 2024-04-13T22:14:53.723 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI= 2024-04-13T22:14:53.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI=', 'Content-Length': '0'} 2024-04-13T22:14:53.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a158fde0ada5f715-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.725 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.725 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.725 DEBUG:boto:path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.725 DEBUG:boto:auth_path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.725 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.726 DEBUG:boto:Path: /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.726 DEBUG:boto:Data: 2024-04-13T22:14:53.726 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.726 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.726 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.726 DEBUG:boto:Params: {} 2024-04-13T22:14:53.726 DEBUG:boto:Token: None 2024-04-13T22:14:53.726 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UvKDtHvOtyHhw3u6xZflnecFxvs= 2024-04-13T22:14:53.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UvKDtHvOtyHhw3u6xZflnecFxvs=', 'Content-Length': '0'} 2024-04-13T22:14:53.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e0940c12aeb2ac-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.728 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.728 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.728 DEBUG:boto:path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.728 DEBUG:boto:auth_path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.729 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.729 DEBUG:boto:Path: /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.729 DEBUG:boto:Data: 2024-04-13T22:14:53.729 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.729 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.729 DEBUG:boto:Params: {} 2024-04-13T22:14:53.729 DEBUG:boto:Token: None 2024-04-13T22:14:53.729 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UvKDtHvOtyHhw3u6xZflnecFxvs= 2024-04-13T22:14:53.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UvKDtHvOtyHhw3u6xZflnecFxvs=', 'Content-Length': '0'} 2024-04-13T22:14:53.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f34bc8901ed8e3-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:14:53.731 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.731 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.731 DEBUG:boto:path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.731 DEBUG:boto:auth_path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.732 DEBUG:boto:Method: GET 2024-04-13T22:14:53.732 DEBUG:boto:Path: /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.732 DEBUG:boto:Data: 2024-04-13T22:14:53.732 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.732 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.732 DEBUG:boto:Params: {} 2024-04-13T22:14:53.732 DEBUG:boto:Token: None 2024-04-13T22:14:53.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI= 2024-04-13T22:14:53.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI=', 'Content-Length': '0'} 2024-04-13T22:14:53.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a72013fbbb3458-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.734 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.734 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.734 DEBUG:boto:path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.734 DEBUG:boto:auth_path=/vwnfns-59/h?versionId=null 2024-04-13T22:14:53.734 DEBUG:boto:Method: GET 2024-04-13T22:14:53.735 DEBUG:boto:Path: /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.735 DEBUG:boto:Data: 2024-04-13T22:14:53.735 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.735 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.735 DEBUG:boto:Params: {} 2024-04-13T22:14:53.735 DEBUG:boto:Token: None 2024-04-13T22:14:53.735 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h?versionId=null 2024-04-13T22:14:53.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI= 2024-04-13T22:14:53.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SFs26X/cOEqPFPY1GOcMPuOZurI=', 'Content-Length': '0'} 2024-04-13T22:14:53.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3a98f1b93679cc-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.778 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.778 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.779 DEBUG:boto:path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.779 DEBUG:boto:auth_path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.779 DEBUG:boto:Method: GET 2024-04-13T22:14:53.779 DEBUG:boto:Path: /vwnfns-59/?prefix=h 2024-04-13T22:14:53.779 DEBUG:boto:Data: 2024-04-13T22:14:53.779 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.779 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.779 DEBUG:boto:Params: {} 2024-04-13T22:14:53.779 DEBUG:boto:Token: None 2024-04-13T22:14:53.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c72b5234bd99fd75-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.782 DEBUG:boto:b'vwnfns-59h1000falseh2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.783 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.783 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.783 DEBUG:boto:path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.783 DEBUG:boto:auth_path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.783 DEBUG:boto:Method: GET 2024-04-13T22:14:53.783 DEBUG:boto:Path: /vwnfns-59/?prefix=h 2024-04-13T22:14:53.783 DEBUG:boto:Data: 2024-04-13T22:14:53.783 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.783 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.784 DEBUG:boto:Params: {} 2024-04-13T22:14:53.784 DEBUG:boto:Token: None 2024-04-13T22:14:53.784 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e588f9257119f2eb-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.786 DEBUG:boto:b'vwnfns-59h1000falseh2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.786 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.786 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.786 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.787 DEBUG:boto:Path: /vwnfns-59/h 2024-04-13T22:14:53.787 DEBUG:boto:Data: 2024-04-13T22:14:53.787 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.787 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.787 DEBUG:boto:Params: {} 2024-04-13T22:14:53.787 DEBUG:boto:Token: None 2024-04-13T22:14:53.787 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h 2024-04-13T22:14:53.787 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pMBZ7ooKimZDbp/n9NsMzz0X0Bo= 2024-04-13T22:14:53.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pMBZ7ooKimZDbp/n9NsMzz0X0Bo=', 'Content-Length': '0'} 2024-04-13T22:14:53.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ee7610c95b3732-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.789 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.789 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.789 DEBUG:boto:Method: HEAD 2024-04-13T22:14:53.789 DEBUG:boto:Path: /vwnfns-59/h 2024-04-13T22:14:53.790 DEBUG:boto:Data: 2024-04-13T22:14:53.790 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.790 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.790 DEBUG:boto:Params: {} 2024-04-13T22:14:53.790 DEBUG:boto:Token: None 2024-04-13T22:14:53.790 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h 2024-04-13T22:14:53.790 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pMBZ7ooKimZDbp/n9NsMzz0X0Bo= 2024-04-13T22:14:53.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pMBZ7ooKimZDbp/n9NsMzz0X0Bo=', 'Content-Length': '0'} 2024-04-13T22:14:53.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b3b26eae437cc5-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-13T22:14:53.792 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.792 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.792 DEBUG:boto:Method: GET 2024-04-13T22:14:53.793 DEBUG:boto:Path: /vwnfns-59/h 2024-04-13T22:14:53.793 DEBUG:boto:Data: 2024-04-13T22:14:53.793 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.793 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.793 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.793 DEBUG:boto:Params: {} 2024-04-13T22:14:53.793 DEBUG:boto:Token: None 2024-04-13T22:14:53.793 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h 2024-04-13T22:14:53.793 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J7V+KkvLkI2FhYq+A+7su9riqlU= 2024-04-13T22:14:53.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J7V+KkvLkI2FhYq+A+7su9riqlU=', 'Content-Length': '0'} 2024-04-13T22:14:53.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-59:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005bbfa3659603d23-00661b03dd-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.795 DEBUG:boto:path=/vwnfns-59/h 2024-04-13T22:14:53.795 DEBUG:boto:auth_path=/vwnfns-59/h 2024-04-13T22:14:53.795 DEBUG:boto:Method: GET 2024-04-13T22:14:53.796 DEBUG:boto:Path: /vwnfns-59/h 2024-04-13T22:14:53.796 DEBUG:boto:Data: 2024-04-13T22:14:53.796 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.796 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.796 DEBUG:boto:Params: {} 2024-04-13T22:14:53.796 DEBUG:boto:Token: None 2024-04-13T22:14:53.796 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/h 2024-04-13T22:14:53.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J7V+KkvLkI2FhYq+A+7su9riqlU= 2024-04-13T22:14:53.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J7V+KkvLkI2FhYq+A+7su9riqlU=', 'Content-Length': '0'} 2024-04-13T22:14:53.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:12:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67ecea1a4672790-00661b03dd-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.798 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.798 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.798 DEBUG:boto:path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.799 DEBUG:boto:auth_path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.799 DEBUG:boto:Method: GET 2024-04-13T22:14:53.799 DEBUG:boto:Path: /vwnfns-59/?prefix=h 2024-04-13T22:14:53.799 DEBUG:boto:Data: 2024-04-13T22:14:53.799 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:53.799 DEBUG:boto:Port: 8001 2024-04-13T22:14:53.799 DEBUG:boto:Params: {} 2024-04-13T22:14:53.799 DEBUG:boto:Token: None 2024-04-13T22:14:53.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25c4798a75375e7-00661b03dd-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.802 DEBUG:boto:b'vwnfns-59h1000falseh2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.802 DEBUG:boto:path=/vwnfns-59/ 2024-04-13T22:14:53.802 DEBUG:boto:auth_path=/vwnfns-59/ 2024-04-13T22:14:53.802 DEBUG:boto:path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.803 DEBUG:boto:auth_path=/vwnfns-59/?prefix=h 2024-04-13T22:14:53.803 DEBUG:boto:Method: GET 2024-04-13T22:14:53.803 DEBUG:boto:Path: /vwnfns-59/?prefix=h 2024-04-13T22:14:53.803 DEBUG:boto:Data: 2024-04-13T22:14:53.803 DEBUG:boto:Headers: {} 2024-04-13T22:14:53.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:53.803 DEBUG:boto:Port: 8000 2024-04-13T22:14:53.803 DEBUG:boto:Params: {} 2024-04-13T22:14:53.803 DEBUG:boto:Token: None 2024-04-13T22:14:53.803 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:53 GMT /vwnfns-59/ 2024-04-13T22:14:53.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s= 2024-04-13T22:14:53.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cFdhwdAY+s1lV8zBOEyBW9GiE3s=', 'Content-Length': '0'} 2024-04-13T22:14:53.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d2191c5f96313c-00661b03dd-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:53.805 DEBUG:boto:b'vwnfns-59h1000falseh2024-04-13T22:12:16.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:14:53.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-59 zones={a2, a1} 2024-04-13T22:14:53.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2024-04-13T22:14:53.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:53.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:53.808 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:14:53.854 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:53.855 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:53.884 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046336.595708_751.1", 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:12:16.595708Z" 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:53.898 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:14:53.899 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046336.595708_751.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:14:53.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:14:53.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:53.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:53.900 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:14:53.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:53.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:53.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:53.976+0000 7fae78da1780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:14:53.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:53.977+0000 7fae78da1780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:14:53.992 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046336.595708_751.1", 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:14:53.993 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:12:16.595708Z", 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:53.994 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:53.995 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:14:53.996 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:53.997 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:14:53Z" 2024-04-13T22:14:53.998 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:14:53.999 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046336.595708_751.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:12:16.595708Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:14:53Z" } 2024-04-13T22:14:53.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046336.595708_751.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:12:16.595708Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:14:53.999 DEBUG:rgw_multi.tests:log_status={0: '1_1713046336.595708_751.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:14:54.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046336.595708_751.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:14:54.000 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:14:54.000 DEBUG:boto:path=/ 2024-04-13T22:14:54.000 DEBUG:boto:auth_path=/ 2024-04-13T22:14:54.000 DEBUG:boto:Method: GET 2024-04-13T22:14:54.000 DEBUG:boto:Path: / 2024-04-13T22:14:54.000 DEBUG:boto:Data: 2024-04-13T22:14:54.000 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.000 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.001 DEBUG:boto:Params: {} 2024-04-13T22:14:54.001 DEBUG:boto:Token: None 2024-04-13T22:14:54.001 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:54 GMT / 2024-04-13T22:14:54.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NYBAnWWVuguDeeBe1Ft+mHArutQ= 2024-04-13T22:14:54.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NYBAnWWVuguDeeBe1Ft+mHArutQ=', 'Content-Length': '0'} 2024-04-13T22:14:54.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f4b704f2647873b-00661b03de-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.006 DEBUG:boto:path=/ 2024-04-13T22:14:54.006 DEBUG:boto:auth_path=/ 2024-04-13T22:14:54.006 DEBUG:boto:Method: GET 2024-04-13T22:14:54.006 DEBUG:boto:Path: / 2024-04-13T22:14:54.006 DEBUG:boto:Data: 2024-04-13T22:14:54.006 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:14:54.006 DEBUG:boto:Port: 8001 2024-04-13T22:14:54.007 DEBUG:boto:Params: {} 2024-04-13T22:14:54.007 DEBUG:boto:Token: None 2024-04-13T22:14:54.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:14:54 GMT / 2024-04-13T22:14:54.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NYBAnWWVuguDeeBe1Ft+mHArutQ= 2024-04-13T22:14:54.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NYBAnWWVuguDeeBe1Ft+mHArutQ=', 'Content-Length': '0'} 2024-04-13T22:14:54.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d0735d254a20ed6-00661b03de-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-04-13T22:14:54.011 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.011 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.011 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.011 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.011 DEBUG:boto:Data: 2024-04-13T22:14:54.011 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.011 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.011 DEBUG:boto:Params: {} 2024-04-13T22:14:54.012 DEBUG:boto:Token: None 2024-04-13T22:14:54.012 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+RH0z16dXrCwP5+oQLs2zYF8PFw= 2024-04-13T22:14:54.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+RH0z16dXrCwP5+oQLs2zYF8PFw=', 'Content-Length': '0'} 2024-04-13T22:14:54.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6f0d5c35f0441f9-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.025 DEBUG:rgw_multi.tests:created bucket=vwnfns-60 2024-04-13T22:14:54.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:54.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:54.026 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:14:54.072 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:54.073 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:54.101 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:14:54.114 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:14:54.022684Z" 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:54.116 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:14:54.117 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:14:54.118 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:14:54.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:54.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:54.118 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:14:54.166 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:54.166 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:54.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:54.199+0000 7f085424c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:14:54.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:54.200+0000 7f085424c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:54.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:14:54.216 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:14:54.218 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:14:54.219 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:14:54.220 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:14:54Z" 2024-04-13T22:14:54.221 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:14:54.221 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:14:54Z" } 2024-04-13T22:14:54.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:14:54.222 DEBUG:rgw_multi.tests:log_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:14:54.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:14:54.222 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:14:54.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:54.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:14:54.223 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:14:54.274 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:54.274 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:54.312 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000005", 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000012", 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000004", 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000011", 2024-04-13T22:14:54.313 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000010", 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000003", 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000013", 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000002", 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000006" 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:14:54.314 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:14:54.315 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.316 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.316 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.316 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.316 DEBUG:boto:Data: 2024-04-13T22:14:54.316 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.316 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.316 DEBUG:boto:Params: {} 2024-04-13T22:14:54.316 DEBUG:boto:Token: None 2024-04-13T22:14:54.317 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.317 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000af0c16a5c91b528-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.320 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:14:54.320 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:14:54.320 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.320 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:14:54.320 DEBUG:boto:Data: 2024-04-13T22:14:54.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.320 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.320 DEBUG:boto:Params: {} 2024-04-13T22:14:54.320 DEBUG:boto:Token: None 2024-04-13T22:14:54.321 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:14:54.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7+Uv5vtvOpGv15P4YEmGPDM7dsY= 2024-04-13T22:14:54.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7+Uv5vtvOpGv15P4YEmGPDM7dsY='} 2024-04-13T22:14:54.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000286372a8106bc949-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.367 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.367 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.367 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.367 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.367 DEBUG:boto:Data: 2024-04-13T22:14:54.367 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.367 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.367 DEBUG:boto:Params: {} 2024-04-13T22:14:54.368 DEBUG:boto:Token: None 2024-04-13T22:14:54.368 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f89899b3f2e2483-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.371 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:14:54.371 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:14:54.371 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.371 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:14:54.371 DEBUG:boto:Data: 2024-04-13T22:14:54.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.372 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.372 DEBUG:boto:Params: {} 2024-04-13T22:14:54.372 DEBUG:boto:Token: None 2024-04-13T22:14:54.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:14:54.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RaXAE9y1Jp2Az+Gm0FgfT8RPi14= 2024-04-13T22:14:54.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RaXAE9y1Jp2Az+Gm0FgfT8RPi14='} 2024-04-13T22:14:54.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000198c0efd8e2869a1-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.377 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.377 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.377 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.377 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.377 DEBUG:boto:Data: 2024-04-13T22:14:54.377 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.377 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.378 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.378 DEBUG:boto:Params: {} 2024-04-13T22:14:54.378 DEBUG:boto:Token: None 2024-04-13T22:14:54.378 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.378 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000320d03fb20cd4e30-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.380 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:14:54.380 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:14:54.380 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.380 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:14:54.380 DEBUG:boto:Data: 2024-04-13T22:14:54.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.381 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.381 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.381 DEBUG:boto:Params: {} 2024-04-13T22:14:54.381 DEBUG:boto:Token: None 2024-04-13T22:14:54.381 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:14:54.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bEthus8hV1YLc7Thy3GskB86+YQ= 2024-04-13T22:14:54.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bEthus8hV1YLc7Thy3GskB86+YQ='} 2024-04-13T22:14:54.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005da2da47e6c75774-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.431 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.431 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.431 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.431 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.431 DEBUG:boto:Data: 2024-04-13T22:14:54.431 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.431 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.432 DEBUG:boto:Params: {} 2024-04-13T22:14:54.432 DEBUG:boto:Token: None 2024-04-13T22:14:54.432 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000810a4adc914ac924-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.435 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:14:54.435 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:14:54.435 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.435 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:14:54.435 DEBUG:boto:Data: 2024-04-13T22:14:54.435 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.435 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.435 DEBUG:boto:Params: {} 2024-04-13T22:14:54.436 DEBUG:boto:Token: None 2024-04-13T22:14:54.436 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:14:54.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Wq71kMW4syTfrq4T8s/6tGcVX8= 2024-04-13T22:14:54.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Wq71kMW4syTfrq4T8s/6tGcVX8='} 2024-04-13T22:14:54.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069e84921fa933eb4-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.441 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.441 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.441 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.441 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.441 DEBUG:boto:Data: 2024-04-13T22:14:54.441 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.442 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.442 DEBUG:boto:Params: {} 2024-04-13T22:14:54.442 DEBUG:boto:Token: None 2024-04-13T22:14:54.442 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.442 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014d4a885d7b59fc0-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.445 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:14:54.445 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:14:54.445 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.445 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:14:54.445 DEBUG:boto:Data: 2024-04-13T22:14:54.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.445 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.445 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.445 DEBUG:boto:Params: {} 2024-04-13T22:14:54.446 DEBUG:boto:Token: None 2024-04-13T22:14:54.446 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:14:54.446 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O0nomogJLrBTOqw79khMrXx4YEA= 2024-04-13T22:14:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O0nomogJLrBTOqw79khMrXx4YEA='} 2024-04-13T22:14:54.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a43f8527fb9b497-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.495 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.495 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.495 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.495 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.495 DEBUG:boto:Data: 2024-04-13T22:14:54.495 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.496 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.496 DEBUG:boto:Params: {} 2024-04-13T22:14:54.496 DEBUG:boto:Token: None 2024-04-13T22:14:54.496 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dbdf7bb2eb0a9fc-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.498 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:14:54.498 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:14:54.498 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.499 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:14:54.499 DEBUG:boto:Data: 2024-04-13T22:14:54.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.499 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.499 DEBUG:boto:Params: {} 2024-04-13T22:14:54.499 DEBUG:boto:Token: None 2024-04-13T22:14:54.499 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:14:54.499 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JX6B37PbMCXSTx1Owws+exArsEU= 2024-04-13T22:14:54.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JX6B37PbMCXSTx1Owws+exArsEU='} 2024-04-13T22:14:54.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007de97235daee6f01-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.504 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.504 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.504 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.505 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.505 DEBUG:boto:Data: 2024-04-13T22:14:54.505 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.505 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.505 DEBUG:boto:Params: {} 2024-04-13T22:14:54.505 DEBUG:boto:Token: None 2024-04-13T22:14:54.505 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.505 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090195de6226435d5-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.508 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:14:54.508 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:14:54.508 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.508 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:14:54.508 DEBUG:boto:Data: 2024-04-13T22:14:54.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.508 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.509 DEBUG:boto:Params: {} 2024-04-13T22:14:54.509 DEBUG:boto:Token: None 2024-04-13T22:14:54.509 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:14:54.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RbhMLTy8PSbhMMcFQ7kYY2DM9l4= 2024-04-13T22:14:54.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RbhMLTy8PSbhMMcFQ7kYY2DM9l4='} 2024-04-13T22:14:54.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011753fe08b94abf3-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.555 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.555 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.555 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.555 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.555 DEBUG:boto:Data: 2024-04-13T22:14:54.555 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.555 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.555 DEBUG:boto:Params: {} 2024-04-13T22:14:54.555 DEBUG:boto:Token: None 2024-04-13T22:14:54.556 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048b560c354f3128a-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.558 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:14:54.558 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:14:54.558 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.558 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:14:54.558 DEBUG:boto:Data: 2024-04-13T22:14:54.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.558 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.558 DEBUG:boto:Params: {} 2024-04-13T22:14:54.558 DEBUG:boto:Token: None 2024-04-13T22:14:54.559 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:14:54.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p0TUt+ocHanFKy49U4gWtbfBdqc= 2024-04-13T22:14:54.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p0TUt+ocHanFKy49U4gWtbfBdqc='} 2024-04-13T22:14:54.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a9513e978489ae9-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.564 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.564 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.564 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.564 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.564 DEBUG:boto:Data: 2024-04-13T22:14:54.564 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.565 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.565 DEBUG:boto:Params: {} 2024-04-13T22:14:54.565 DEBUG:boto:Token: None 2024-04-13T22:14:54.565 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000446733511518952c-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.567 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:14:54.567 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:14:54.568 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.568 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:14:54.568 DEBUG:boto:Data: 2024-04-13T22:14:54.568 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.568 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.568 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.568 DEBUG:boto:Params: {} 2024-04-13T22:14:54.568 DEBUG:boto:Token: None 2024-04-13T22:14:54.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:14:54.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:96L3ZoHHHWlZCmWg9fAvX5XmTEU= 2024-04-13T22:14:54.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:96L3ZoHHHWlZCmWg9fAvX5XmTEU='} 2024-04-13T22:14:54.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099e16057cf754ad2-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.610 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:14:54.611 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:14:54.611 DEBUG:boto:Method: HEAD 2024-04-13T22:14:54.611 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:14:54.611 DEBUG:boto:Data: 2024-04-13T22:14:54.611 DEBUG:boto:Headers: {} 2024-04-13T22:14:54.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.611 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.611 DEBUG:boto:Params: {} 2024-04-13T22:14:54.611 DEBUG:boto:Token: None 2024-04-13T22:14:54.611 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/ 2024-04-13T22:14:54.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM= 2024-04-13T22:14:54.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zcGBZz983QYlor36l6Va288yzzM=', 'Content-Length': '0'} 2024-04-13T22:14:54.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004acff8ab0c5eb5c3-00661b03de-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.614 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:14:54.615 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:14:54.615 DEBUG:boto:Method: PUT 2024-04-13T22:14:54.615 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:14:54.615 DEBUG:boto:Data: 2024-04-13T22:14:54.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:14:54.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:14:54.615 DEBUG:boto:Port: 8000 2024-04-13T22:14:54.615 DEBUG:boto:Params: {} 2024-04-13T22:14:54.615 DEBUG:boto:Token: None 2024-04-13T22:14:54.616 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:14:54 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:14:54.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/IopY5T0j5fxUX1tf9Hin6ikz2I= 2024-04-13T22:14:54.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:14:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/IopY5T0j5fxUX1tf9Hin6ikz2I='} 2024-04-13T22:14:54.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdce07e66b9c9fb8-00661b03de-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:14:54.621 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-60 2024-04-13T22:14:54.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:14:54.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:14:54.622 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:14:54.679 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:54.679 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:54.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:54.712+0000 7f7e4a936780 1 waiting to reach incremental sync.. 2024-04-13T22:14:59.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:59.713+0000 7f7e4a936780 1 bucket sync caught up with source: 2024-04-13T22:14:59.715 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.29.6, 00000000001.86.6, 00000000001.40.6, 00000000001.65.6, 00000000001.40.6, 00000000001.30.6, 00000000001.56.6, 00000000001.39.6, 00000000001.46.6, 00000000001.41.6] 2024-04-13T22:14:59.716 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.29.6, 00000000001.86.6, 00000000001.40.6, 00000000001.65.6, 00000000001.40.6, 00000000001.30.6, 00000000001.56.6, 00000000001.39.6, 00000000001.46.6, 00000000001.41.6] 2024-04-13T22:14:59.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:59.713+0000 7f7e4a936780 0 bucket checkpoint complete 2024-04-13T22:14:59.733 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-60 2024-04-13T22:14:59.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:14:59.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:14:59.734 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:14:59.794 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:14:59.797 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:14:59.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:14:59.830+0000 7efd0e7ee780 1 waiting to reach incremental sync.. 2024-04-13T22:15:04.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:04.830+0000 7efd0e7ee780 1 waiting to reach incremental sync.. 2024-04-13T22:15:09.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:09.832+0000 7efd0e7ee780 1 bucket sync caught up with source: 2024-04-13T22:15:09.835 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.471.6, 00000000001.556.6, 00000000001.461.6, 00000000001.1013.6, 00000000001.2215.6, 00000000001.1239.6, 00000000001.950.6, 00000000001.407.6, 00000000001.1754.6, 00000000001.408.6] 2024-04-13T22:15:09.835 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.471.6, 00000000001.556.6, 00000000001.461.6, 00000000001.1013.6, 00000000001.2215.6, 00000000001.1239.6, 00000000001.950.6, 00000000001.407.6, 00000000001.1754.6, 00000000001.408.6] 2024-04-13T22:15:09.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:09.832+0000 7efd0e7ee780 0 bucket checkpoint complete 2024-04-13T22:15:09.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-60 zones={a2, a1} 2024-04-13T22:15:09.852 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.852 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.853 DEBUG:boto:Method: HEAD 2024-04-13T22:15:09.853 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:09.853 DEBUG:boto:Data: 2024-04-13T22:15:09.853 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.853 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.853 DEBUG:boto:Params: {} 2024-04-13T22:15:09.853 DEBUG:boto:Token: None 2024-04-13T22:15:09.854 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/ 2024-04-13T22:15:09.854 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x6CHdi41Yj/Vh+cCBQyexEnyMAM= 2024-04-13T22:15:09.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x6CHdi41Yj/Vh+cCBQyexEnyMAM=', 'Content-Length': '0'} 2024-04-13T22:15:09.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ddbe32286cba5ee-00661b03ed-1eaf-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.857 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.857 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.857 DEBUG:boto:Method: HEAD 2024-04-13T22:15:09.857 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:09.857 DEBUG:boto:Data: 2024-04-13T22:15:09.857 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.857 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.857 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.857 DEBUG:boto:Params: {} 2024-04-13T22:15:09.857 DEBUG:boto:Token: None 2024-04-13T22:15:09.858 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/ 2024-04-13T22:15:09.858 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x6CHdi41Yj/Vh+cCBQyexEnyMAM= 2024-04-13T22:15:09.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x6CHdi41Yj/Vh+cCBQyexEnyMAM=', 'Content-Length': '0'} 2024-04-13T22:15:09.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e8b3e2c75131c1e-00661b03ed-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.860 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:15:09.860 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.860 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.860 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.860 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.861 DEBUG:boto:Method: GET 2024-04-13T22:15:09.861 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.861 DEBUG:boto:Data: 2024-04-13T22:15:09.861 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.861 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.861 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.861 DEBUG:boto:Params: {} 2024-04-13T22:15:09.861 DEBUG:boto:Token: None 2024-04-13T22:15:09.861 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/?versions 2024-04-13T22:15:09.861 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY= 2024-04-13T22:15:09.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY=', 'Content-Length': '0'} 2024-04-13T22:15:09.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59f25c0105584b2-00661b03ed-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.864 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-13T22:15:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-13T22:15:09.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-13T22:15:09.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-13T22:15:09.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-13T22:15:09.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:15:09.866 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.866 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.866 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.866 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.866 DEBUG:boto:Method: GET 2024-04-13T22:15:09.866 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.867 DEBUG:boto:Data: 2024-04-13T22:15:09.867 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.867 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.867 DEBUG:boto:Params: {} 2024-04-13T22:15:09.867 DEBUG:boto:Token: None 2024-04-13T22:15:09.867 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/?versions 2024-04-13T22:15:09.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY= 2024-04-13T22:15:09.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY=', 'Content-Length': '0'} 2024-04-13T22:15:09.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000131f0d81eae3c2d7-00661b03ed-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.870 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:09.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-13T22:15:09.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-13T22:15:09.871 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.871 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.871 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.872 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.872 DEBUG:boto:Method: GET 2024-04-13T22:15:09.872 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.872 DEBUG:boto:Data: 2024-04-13T22:15:09.872 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.872 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.872 DEBUG:boto:Params: {} 2024-04-13T22:15:09.872 DEBUG:boto:Token: None 2024-04-13T22:15:09.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/?versions 2024-04-13T22:15:09.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY= 2024-04-13T22:15:09.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY=', 'Content-Length': '0'} 2024-04-13T22:15:09.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf637cfb3be0e0e-00661b03ed-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.875 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:09.875 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.876 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.876 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.876 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.876 DEBUG:boto:Method: GET 2024-04-13T22:15:09.876 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:09.876 DEBUG:boto:Data: 2024-04-13T22:15:09.876 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.876 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.876 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.876 DEBUG:boto:Params: {} 2024-04-13T22:15:09.876 DEBUG:boto:Token: None 2024-04-13T22:15:09.877 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/?versions 2024-04-13T22:15:09.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY= 2024-04-13T22:15:09.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+aBfcP3D3dUaNOd7kj/W2pSOKkY=', 'Content-Length': '0'} 2024-04-13T22:15:09.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a0a010ae56dad02-00661b03ed-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.879 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:09.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-13T22:15:09.880 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.880 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.880 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.881 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.881 DEBUG:boto:Method: GET 2024-04-13T22:15:09.881 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.881 DEBUG:boto:Data: 2024-04-13T22:15:09.881 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.881 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.881 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.881 DEBUG:boto:Params: {} 2024-04-13T22:15:09.881 DEBUG:boto:Token: None 2024-04-13T22:15:09.881 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.881 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo= 2024-04-13T22:15:09.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo=', 'Content-Length': '0'} 2024-04-13T22:15:09.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030ad8652dde60238-00661b03ed-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.930 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.931 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.931 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.931 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.931 DEBUG:boto:Method: GET 2024-04-13T22:15:09.931 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.931 DEBUG:boto:Data: 2024-04-13T22:15:09.931 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.931 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.931 DEBUG:boto:Params: {} 2024-04-13T22:15:09.932 DEBUG:boto:Token: None 2024-04-13T22:15:09.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo= 2024-04-13T22:15:09.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo=', 'Content-Length': '0'} 2024-04-13T22:15:09.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ecfe8594c39cf7-00661b03ed-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.934 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.934 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.934 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.934 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.934 DEBUG:boto:Method: HEAD 2024-04-13T22:15:09.935 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.935 DEBUG:boto:Data: 2024-04-13T22:15:09.935 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.935 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.935 DEBUG:boto:Params: {} 2024-04-13T22:15:09.935 DEBUG:boto:Token: None 2024-04-13T22:15:09.935 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:744HHZGNlJToKNjxxOL/RQxAZFc= 2024-04-13T22:15:09.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:744HHZGNlJToKNjxxOL/RQxAZFc=', 'Content-Length': '0'} 2024-04-13T22:15:09.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052865f591da375f8-00661b03ed-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.937 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.938 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.938 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.938 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.938 DEBUG:boto:Method: HEAD 2024-04-13T22:15:09.938 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.938 DEBUG:boto:Data: 2024-04-13T22:15:09.938 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.938 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.938 DEBUG:boto:Params: {} 2024-04-13T22:15:09.939 DEBUG:boto:Token: None 2024-04-13T22:15:09.939 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:744HHZGNlJToKNjxxOL/RQxAZFc= 2024-04-13T22:15:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:744HHZGNlJToKNjxxOL/RQxAZFc=', 'Content-Length': '0'} 2024-04-13T22:15:09.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000defc885aaec97085-00661b03ed-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-13T22:15:09.941 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.941 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.941 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.941 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.941 DEBUG:boto:Method: GET 2024-04-13T22:15:09.941 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.941 DEBUG:boto:Data: 2024-04-13T22:15:09.941 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.942 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.942 DEBUG:boto:Params: {} 2024-04-13T22:15:09.942 DEBUG:boto:Token: None 2024-04-13T22:15:09.942 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo= 2024-04-13T22:15:09.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo=', 'Content-Length': '0'} 2024-04-13T22:15:09.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ed8ec02eba3efd-00661b03ed-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.944 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.944 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.945 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.945 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.945 DEBUG:boto:Method: GET 2024-04-13T22:15:09.945 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.945 DEBUG:boto:Data: 2024-04-13T22:15:09.945 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.945 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.945 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.945 DEBUG:boto:Params: {} 2024-04-13T22:15:09.945 DEBUG:boto:Token: None 2024-04-13T22:15:09.945 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:09.946 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo= 2024-04-13T22:15:09.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:++l1k9gx9I7h49PwzLMyWU7Wxgo=', 'Content-Length': '0'} 2024-04-13T22:15:09.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b9c17ba9b624e5-00661b03ed-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.948 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.948 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.948 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:09.948 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:09.948 DEBUG:boto:Method: GET 2024-04-13T22:15:09.948 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:09.948 DEBUG:boto:Data: 2024-04-13T22:15:09.948 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.948 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.949 DEBUG:boto:Params: {} 2024-04-13T22:15:09.949 DEBUG:boto:Token: None 2024-04-13T22:15:09.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/ 2024-04-13T22:15:09.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kz1s/xvZGr9KjdefZU/IV/AP47c= 2024-04-13T22:15:09.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kz1s/xvZGr9KjdefZU/IV/AP47c=', 'Content-Length': '0'} 2024-04-13T22:15:09.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb77a008e782979-00661b03ed-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.951 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:09.952 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:09.952 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:09.952 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:09.952 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:09.952 DEBUG:boto:Method: GET 2024-04-13T22:15:09.952 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:09.952 DEBUG:boto:Data: 2024-04-13T22:15:09.952 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.953 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.953 DEBUG:boto:Params: {} 2024-04-13T22:15:09.953 DEBUG:boto:Token: None 2024-04-13T22:15:09.953 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/ 2024-04-13T22:15:09.953 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kz1s/xvZGr9KjdefZU/IV/AP47c= 2024-04-13T22:15:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kz1s/xvZGr9KjdefZU/IV/AP47c=', 'Content-Length': '0'} 2024-04-13T22:15:09.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1421488ed9f8225-00661b03ed-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.955 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:09.956 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.956 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.956 DEBUG:boto:Method: HEAD 2024-04-13T22:15:09.956 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.956 DEBUG:boto:Data: 2024-04-13T22:15:09.956 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.956 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.957 DEBUG:boto:Params: {} 2024-04-13T22:15:09.957 DEBUG:boto:Token: None 2024-04-13T22:15:09.957 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.957 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pv4a8Cml8XZgoRsPQu6dAJbJ0A8= 2024-04-13T22:15:09.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pv4a8Cml8XZgoRsPQu6dAJbJ0A8=', 'Content-Length': '0'} 2024-04-13T22:15:09.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076aa7fe50cbce928-00661b03ed-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.959 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.959 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.959 DEBUG:boto:Method: HEAD 2024-04-13T22:15:09.959 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.959 DEBUG:boto:Data: 2024-04-13T22:15:09.959 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.960 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:09.960 DEBUG:boto:Port: 8000 2024-04-13T22:15:09.960 DEBUG:boto:Params: {} 2024-04-13T22:15:09.960 DEBUG:boto:Token: None 2024-04-13T22:15:09.960 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pv4a8Cml8XZgoRsPQu6dAJbJ0A8= 2024-04-13T22:15:09.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pv4a8Cml8XZgoRsPQu6dAJbJ0A8=', 'Content-Length': '0'} 2024-04-13T22:15:09.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf6f5a49f188b2f-00661b03ed-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:09.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-13T22:15:09.962 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.962 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.962 DEBUG:boto:Method: GET 2024-04-13T22:15:09.962 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.963 DEBUG:boto:Data: 2024-04-13T22:15:09.963 DEBUG:boto:Headers: {} 2024-04-13T22:15:09.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:09.963 DEBUG:boto:Port: 8001 2024-04-13T22:15:09.963 DEBUG:boto:Params: {} 2024-04-13T22:15:09.963 DEBUG:boto:Token: None 2024-04-13T22:15:09.963 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:09 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:09.963 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MxR3zq6XY297spRQ/M9ju3ehM7Q= 2024-04-13T22:15:09.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:09 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MxR3zq6XY297spRQ/M9ju3ehM7Q=', 'Content-Length': '0'} 2024-04-13T22:15:09.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000806d6a4f1d09771c-00661b03ed-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.006 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:10.006 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:10.006 DEBUG:boto:Method: GET 2024-04-13T22:15:10.007 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:10.007 DEBUG:boto:Data: 2024-04-13T22:15:10.007 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.007 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.007 DEBUG:boto:Params: {} 2024-04-13T22:15:10.007 DEBUG:boto:Token: None 2024-04-13T22:15:10.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:10.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:i49QOUTQJVnJyRGWS0nJFp7xro4= 2024-04-13T22:15:10.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:i49QOUTQJVnJyRGWS0nJFp7xro4=', 'Content-Length': '0'} 2024-04-13T22:15:10.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d4b962745d643a-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.010 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.010 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.010 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:10.010 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:10.010 DEBUG:boto:Method: GET 2024-04-13T22:15:10.010 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:10.010 DEBUG:boto:Data: 2024-04-13T22:15:10.010 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.011 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.011 DEBUG:boto:Params: {} 2024-04-13T22:15:10.011 DEBUG:boto:Token: None 2024-04-13T22:15:10.011 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000949e5fa000bb371-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.014 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.014 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.014 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.015 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:10.015 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:10.015 DEBUG:boto:Method: GET 2024-04-13T22:15:10.015 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:10.015 DEBUG:boto:Data: 2024-04-13T22:15:10.015 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.015 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.015 DEBUG:boto:Params: {} 2024-04-13T22:15:10.015 DEBUG:boto:Token: None 2024-04-13T22:15:10.015 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aaca71b0f6c8268-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.018 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:14:54.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-13T22:15:10.018 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.019 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.019 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.019 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.019 DEBUG:boto:Method: GET 2024-04-13T22:15:10.019 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.019 DEBUG:boto:Data: 2024-04-13T22:15:10.020 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.020 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.020 DEBUG:boto:Params: {} 2024-04-13T22:15:10.020 DEBUG:boto:Token: None 2024-04-13T22:15:10.021 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk= 2024-04-13T22:15:10.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk=', 'Content-Length': '0'} 2024-04-13T22:15:10.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc24cb46f6ab89d2-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.023 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.023 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.023 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.024 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.024 DEBUG:boto:Method: GET 2024-04-13T22:15:10.024 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.024 DEBUG:boto:Data: 2024-04-13T22:15:10.024 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.024 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.025 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.025 DEBUG:boto:Params: {} 2024-04-13T22:15:10.025 DEBUG:boto:Token: None 2024-04-13T22:15:10.025 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.025 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk= 2024-04-13T22:15:10.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk=', 'Content-Length': '0'} 2024-04-13T22:15:10.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f35ee2fde7a2e02-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.027 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.027 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.028 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.028 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.028 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.028 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.028 DEBUG:boto:Data: 2024-04-13T22:15:10.028 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.028 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.028 DEBUG:boto:Params: {} 2024-04-13T22:15:10.028 DEBUG:boto:Token: None 2024-04-13T22:15:10.029 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.029 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PF2BneUeYaN6AFZfRl2y2+GOcXc= 2024-04-13T22:15:10.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PF2BneUeYaN6AFZfRl2y2+GOcXc=', 'Content-Length': '0'} 2024-04-13T22:15:10.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002166ed2a531db1b2-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.031 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.031 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.031 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.031 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.031 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.032 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.032 DEBUG:boto:Data: 2024-04-13T22:15:10.032 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.032 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.032 DEBUG:boto:Params: {} 2024-04-13T22:15:10.032 DEBUG:boto:Token: None 2024-04-13T22:15:10.032 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PF2BneUeYaN6AFZfRl2y2+GOcXc= 2024-04-13T22:15:10.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PF2BneUeYaN6AFZfRl2y2+GOcXc=', 'Content-Length': '0'} 2024-04-13T22:15:10.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26d1c0d48e87d25-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-13T22:15:10.035 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.035 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.035 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.035 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.035 DEBUG:boto:Method: GET 2024-04-13T22:15:10.035 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.035 DEBUG:boto:Data: 2024-04-13T22:15:10.035 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.035 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.036 DEBUG:boto:Params: {} 2024-04-13T22:15:10.036 DEBUG:boto:Token: None 2024-04-13T22:15:10.036 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk= 2024-04-13T22:15:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk=', 'Content-Length': '0'} 2024-04-13T22:15:10.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003554c7edccd14ecc-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.078 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.078 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.078 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.079 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.079 DEBUG:boto:Method: GET 2024-04-13T22:15:10.079 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.079 DEBUG:boto:Data: 2024-04-13T22:15:10.079 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.079 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.079 DEBUG:boto:Params: {} 2024-04-13T22:15:10.079 DEBUG:boto:Token: None 2024-04-13T22:15:10.080 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:10.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk= 2024-04-13T22:15:10.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:whVbgjZkIqLQZ1E2z12N3EP9Etk=', 'Content-Length': '0'} 2024-04-13T22:15:10.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f007506123b9ad-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.082 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.082 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.082 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.082 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.082 DEBUG:boto:Method: GET 2024-04-13T22:15:10.082 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.083 DEBUG:boto:Data: 2024-04-13T22:15:10.083 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.083 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.083 DEBUG:boto:Params: {} 2024-04-13T22:15:10.083 DEBUG:boto:Token: None 2024-04-13T22:15:10.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b089e109984d95-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.086 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.086 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.086 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.087 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.087 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.087 DEBUG:boto:Method: GET 2024-04-13T22:15:10.087 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.087 DEBUG:boto:Data: 2024-04-13T22:15:10.087 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.087 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.087 DEBUG:boto:Params: {} 2024-04-13T22:15:10.087 DEBUG:boto:Token: None 2024-04-13T22:15:10.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007498897ec2757b8d-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.090 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.090 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.091 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.091 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.091 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.091 DEBUG:boto:Data: 2024-04-13T22:15:10.091 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.091 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.091 DEBUG:boto:Params: {} 2024-04-13T22:15:10.091 DEBUG:boto:Token: None 2024-04-13T22:15:10.091 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PTsP9zts9v+PA+3dN9G1oyc1xno= 2024-04-13T22:15:10.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PTsP9zts9v+PA+3dN9G1oyc1xno=', 'Content-Length': '0'} 2024-04-13T22:15:10.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004181c7e158cd33d3-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.093 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.093 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.094 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.094 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.094 DEBUG:boto:Data: 2024-04-13T22:15:10.094 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.094 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.094 DEBUG:boto:Params: {} 2024-04-13T22:15:10.094 DEBUG:boto:Token: None 2024-04-13T22:15:10.094 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PTsP9zts9v+PA+3dN9G1oyc1xno= 2024-04-13T22:15:10.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PTsP9zts9v+PA+3dN9G1oyc1xno=', 'Content-Length': '0'} 2024-04-13T22:15:10.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd103f4477403b5-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-13T22:15:10.097 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.097 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.097 DEBUG:boto:Method: GET 2024-04-13T22:15:10.097 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.097 DEBUG:boto:Data: 2024-04-13T22:15:10.097 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.097 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.097 DEBUG:boto:Params: {} 2024-04-13T22:15:10.098 DEBUG:boto:Token: None 2024-04-13T22:15:10.098 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iClGJ3xLEtDZ9Fv8jD1jY03ohvQ= 2024-04-13T22:15:10.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iClGJ3xLEtDZ9Fv8jD1jY03ohvQ=', 'Content-Length': '0'} 2024-04-13T22:15:10.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be3b20603eab8db6-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.100 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.100 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.100 DEBUG:boto:Method: GET 2024-04-13T22:15:10.100 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.100 DEBUG:boto:Data: 2024-04-13T22:15:10.100 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.100 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.100 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.100 DEBUG:boto:Params: {} 2024-04-13T22:15:10.101 DEBUG:boto:Token: None 2024-04-13T22:15:10.101 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:10.101 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iClGJ3xLEtDZ9Fv8jD1jY03ohvQ= 2024-04-13T22:15:10.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iClGJ3xLEtDZ9Fv8jD1jY03ohvQ=', 'Content-Length': '0'} 2024-04-13T22:15:10.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7449cb8763d77a-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.103 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.103 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.103 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.103 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.103 DEBUG:boto:Method: GET 2024-04-13T22:15:10.103 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.103 DEBUG:boto:Data: 2024-04-13T22:15:10.104 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.104 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.104 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.104 DEBUG:boto:Params: {} 2024-04-13T22:15:10.104 DEBUG:boto:Token: None 2024-04-13T22:15:10.104 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.104 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4244b6a851801d7-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.106 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.107 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.107 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.107 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.107 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.107 DEBUG:boto:Method: GET 2024-04-13T22:15:10.107 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:10.107 DEBUG:boto:Data: 2024-04-13T22:15:10.108 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.108 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.108 DEBUG:boto:Params: {} 2024-04-13T22:15:10.108 DEBUG:boto:Token: None 2024-04-13T22:15:10.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3e6a7fe68da8f0-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.111 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:14:54.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-13T22:15:10.111 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.111 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.111 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.111 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.112 DEBUG:boto:Method: GET 2024-04-13T22:15:10.112 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.112 DEBUG:boto:Data: 2024-04-13T22:15:10.112 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.112 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.112 DEBUG:boto:Params: {} 2024-04-13T22:15:10.112 DEBUG:boto:Token: None 2024-04-13T22:15:10.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg= 2024-04-13T22:15:10.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40d11b89bdc44e3-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.114 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.114 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.115 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.115 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.115 DEBUG:boto:Method: GET 2024-04-13T22:15:10.115 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.115 DEBUG:boto:Data: 2024-04-13T22:15:10.115 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.115 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.115 DEBUG:boto:Params: {} 2024-04-13T22:15:10.115 DEBUG:boto:Token: None 2024-04-13T22:15:10.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg= 2024-04-13T22:15:10.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43a6c293e3feaad-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.118 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.118 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.118 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.118 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.118 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.118 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.118 DEBUG:boto:Data: 2024-04-13T22:15:10.118 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.119 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.119 DEBUG:boto:Params: {} 2024-04-13T22:15:10.119 DEBUG:boto:Token: None 2024-04-13T22:15:10.119 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.119 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L9As3qvOhB4lFYZcYTbpFUnVdrs= 2024-04-13T22:15:10.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L9As3qvOhB4lFYZcYTbpFUnVdrs=', 'Content-Length': '0'} 2024-04-13T22:15:10.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029eb796b9d33c9de-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.121 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.121 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.121 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.121 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.121 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.121 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.121 DEBUG:boto:Data: 2024-04-13T22:15:10.121 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.122 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.122 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.122 DEBUG:boto:Params: {} 2024-04-13T22:15:10.122 DEBUG:boto:Token: None 2024-04-13T22:15:10.122 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.122 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L9As3qvOhB4lFYZcYTbpFUnVdrs= 2024-04-13T22:15:10.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L9As3qvOhB4lFYZcYTbpFUnVdrs=', 'Content-Length': '0'} 2024-04-13T22:15:10.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000217f6d3863b9dfd7-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-13T22:15:10.124 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.124 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.124 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.124 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.124 DEBUG:boto:Method: GET 2024-04-13T22:15:10.125 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.125 DEBUG:boto:Data: 2024-04-13T22:15:10.125 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.125 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.125 DEBUG:boto:Params: {} 2024-04-13T22:15:10.125 DEBUG:boto:Token: None 2024-04-13T22:15:10.125 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.125 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg= 2024-04-13T22:15:10.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8f29feafe06013-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.127 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.127 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.127 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.127 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.128 DEBUG:boto:Method: GET 2024-04-13T22:15:10.128 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.128 DEBUG:boto:Data: 2024-04-13T22:15:10.128 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.128 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.128 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.128 DEBUG:boto:Params: {} 2024-04-13T22:15:10.128 DEBUG:boto:Token: None 2024-04-13T22:15:10.128 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:10.128 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg= 2024-04-13T22:15:10.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ILArCxJs0jdF0hEtAKXYDiYevUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b2f48992dd1f6d1-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.130 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.131 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.131 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.131 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.131 DEBUG:boto:Method: GET 2024-04-13T22:15:10.131 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.131 DEBUG:boto:Data: 2024-04-13T22:15:10.131 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.131 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.131 DEBUG:boto:Params: {} 2024-04-13T22:15:10.132 DEBUG:boto:Token: None 2024-04-13T22:15:10.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16fe2c96b682279-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.178 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.179 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.179 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.179 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.179 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.179 DEBUG:boto:Method: GET 2024-04-13T22:15:10.179 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.179 DEBUG:boto:Data: 2024-04-13T22:15:10.179 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.180 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.180 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.180 DEBUG:boto:Params: {} 2024-04-13T22:15:10.180 DEBUG:boto:Token: None 2024-04-13T22:15:10.180 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.180 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000233474fb85072a9b-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.183 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.183 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.183 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.183 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.183 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.183 DEBUG:boto:Data: 2024-04-13T22:15:10.183 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.184 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.184 DEBUG:boto:Params: {} 2024-04-13T22:15:10.184 DEBUG:boto:Token: None 2024-04-13T22:15:10.184 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wrealf6PxZH8DilTmqHNNvjDTpo= 2024-04-13T22:15:10.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wrealf6PxZH8DilTmqHNNvjDTpo=', 'Content-Length': '0'} 2024-04-13T22:15:10.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9698b031abdd2a-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.186 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.186 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.186 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.186 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.186 DEBUG:boto:Data: 2024-04-13T22:15:10.186 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.186 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.187 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.187 DEBUG:boto:Params: {} 2024-04-13T22:15:10.187 DEBUG:boto:Token: None 2024-04-13T22:15:10.187 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.187 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wrealf6PxZH8DilTmqHNNvjDTpo= 2024-04-13T22:15:10.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wrealf6PxZH8DilTmqHNNvjDTpo=', 'Content-Length': '0'} 2024-04-13T22:15:10.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d065454f8fc11dba-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-13T22:15:10.189 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.189 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.189 DEBUG:boto:Method: GET 2024-04-13T22:15:10.189 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.189 DEBUG:boto:Data: 2024-04-13T22:15:10.189 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.189 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.190 DEBUG:boto:Params: {} 2024-04-13T22:15:10.190 DEBUG:boto:Token: None 2024-04-13T22:15:10.190 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qAMl16/ktujEVURizEoco0r3ESw= 2024-04-13T22:15:10.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qAMl16/ktujEVURizEoco0r3ESw=', 'Content-Length': '0'} 2024-04-13T22:15:10.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5c1e143607494b-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.192 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.192 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.192 DEBUG:boto:Method: GET 2024-04-13T22:15:10.192 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.192 DEBUG:boto:Data: 2024-04-13T22:15:10.193 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.193 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.193 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.193 DEBUG:boto:Params: {} 2024-04-13T22:15:10.193 DEBUG:boto:Token: None 2024-04-13T22:15:10.193 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:10.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qAMl16/ktujEVURizEoco0r3ESw= 2024-04-13T22:15:10.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qAMl16/ktujEVURizEoco0r3ESw=', 'Content-Length': '0'} 2024-04-13T22:15:10.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d310eb18ab34e791-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.238 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.239 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.239 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.239 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.240 DEBUG:boto:Method: GET 2024-04-13T22:15:10.240 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.240 DEBUG:boto:Data: 2024-04-13T22:15:10.240 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.240 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.240 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.241 DEBUG:boto:Params: {} 2024-04-13T22:15:10.241 DEBUG:boto:Token: None 2024-04-13T22:15:10.241 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.241 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfbacd0222eaae24-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.244 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.245 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.245 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.245 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.245 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.246 DEBUG:boto:Method: GET 2024-04-13T22:15:10.246 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:10.246 DEBUG:boto:Data: 2024-04-13T22:15:10.246 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.246 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.247 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.247 DEBUG:boto:Params: {} 2024-04-13T22:15:10.247 DEBUG:boto:Token: None 2024-04-13T22:15:10.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000713147f5f0136b75-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.250 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:14:54.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-13T22:15:10.251 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.251 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.251 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.251 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.251 DEBUG:boto:Method: GET 2024-04-13T22:15:10.252 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.252 DEBUG:boto:Data: 2024-04-13T22:15:10.252 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.252 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.252 DEBUG:boto:Params: {} 2024-04-13T22:15:10.252 DEBUG:boto:Token: None 2024-04-13T22:15:10.253 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.253 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg= 2024-04-13T22:15:10.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg=', 'Content-Length': '0'} 2024-04-13T22:15:10.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef28a013ebac493b-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.255 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.255 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.255 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.255 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.255 DEBUG:boto:Method: GET 2024-04-13T22:15:10.256 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.256 DEBUG:boto:Data: 2024-04-13T22:15:10.256 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.256 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.256 DEBUG:boto:Params: {} 2024-04-13T22:15:10.256 DEBUG:boto:Token: None 2024-04-13T22:15:10.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg= 2024-04-13T22:15:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg=', 'Content-Length': '0'} 2024-04-13T22:15:10.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a3d6a3ac356516-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.258 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.259 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.259 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.259 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.259 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.259 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.259 DEBUG:boto:Data: 2024-04-13T22:15:10.259 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.259 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.259 DEBUG:boto:Params: {} 2024-04-13T22:15:10.260 DEBUG:boto:Token: None 2024-04-13T22:15:10.260 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oIW+dDisDlbInNEOuH5LwE2717M= 2024-04-13T22:15:10.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oIW+dDisDlbInNEOuH5LwE2717M=', 'Content-Length': '0'} 2024-04-13T22:15:10.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f0918d9c05a3581-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.262 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.262 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.262 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.262 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.262 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.262 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.262 DEBUG:boto:Data: 2024-04-13T22:15:10.262 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.263 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.263 DEBUG:boto:Params: {} 2024-04-13T22:15:10.263 DEBUG:boto:Token: None 2024-04-13T22:15:10.263 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oIW+dDisDlbInNEOuH5LwE2717M= 2024-04-13T22:15:10.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oIW+dDisDlbInNEOuH5LwE2717M=', 'Content-Length': '0'} 2024-04-13T22:15:10.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e6964ab38fd8bc-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-13T22:15:10.265 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.265 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.265 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.265 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.266 DEBUG:boto:Method: GET 2024-04-13T22:15:10.266 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.266 DEBUG:boto:Data: 2024-04-13T22:15:10.266 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.266 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.266 DEBUG:boto:Params: {} 2024-04-13T22:15:10.266 DEBUG:boto:Token: None 2024-04-13T22:15:10.266 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.266 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg= 2024-04-13T22:15:10.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg=', 'Content-Length': '0'} 2024-04-13T22:15:10.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646693800322d23a-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.268 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.268 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.268 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.268 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.269 DEBUG:boto:Method: GET 2024-04-13T22:15:10.269 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.269 DEBUG:boto:Data: 2024-04-13T22:15:10.269 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.269 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.269 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.269 DEBUG:boto:Params: {} 2024-04-13T22:15:10.269 DEBUG:boto:Token: None 2024-04-13T22:15:10.269 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:10.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg= 2024-04-13T22:15:10.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HckdKOrpJFYCgeYiNS4a3XXNhgg=', 'Content-Length': '0'} 2024-04-13T22:15:10.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ab98198a71b6e4-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.271 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.271 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.271 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.272 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.272 DEBUG:boto:Method: GET 2024-04-13T22:15:10.272 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.272 DEBUG:boto:Data: 2024-04-13T22:15:10.272 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.272 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.272 DEBUG:boto:Params: {} 2024-04-13T22:15:10.272 DEBUG:boto:Token: None 2024-04-13T22:15:10.272 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe845adf4cd86db-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.275 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.275 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.275 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.275 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.275 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.276 DEBUG:boto:Method: GET 2024-04-13T22:15:10.276 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.276 DEBUG:boto:Data: 2024-04-13T22:15:10.276 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.276 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.276 DEBUG:boto:Params: {} 2024-04-13T22:15:10.276 DEBUG:boto:Token: None 2024-04-13T22:15:10.276 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380de542d0564b29-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.322 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.323 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.323 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.323 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.323 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.323 DEBUG:boto:Data: 2024-04-13T22:15:10.323 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.324 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.324 DEBUG:boto:Params: {} 2024-04-13T22:15:10.324 DEBUG:boto:Token: None 2024-04-13T22:15:10.324 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hn5veJTI9K/2oDsCVo7qKmipyjU= 2024-04-13T22:15:10.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hn5veJTI9K/2oDsCVo7qKmipyjU=', 'Content-Length': '0'} 2024-04-13T22:15:10.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3d034e62547ece-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.326 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.326 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.326 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.326 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.326 DEBUG:boto:Data: 2024-04-13T22:15:10.327 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.327 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.327 DEBUG:boto:Params: {} 2024-04-13T22:15:10.327 DEBUG:boto:Token: None 2024-04-13T22:15:10.327 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hn5veJTI9K/2oDsCVo7qKmipyjU= 2024-04-13T22:15:10.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hn5veJTI9K/2oDsCVo7qKmipyjU=', 'Content-Length': '0'} 2024-04-13T22:15:10.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d6e47207062011-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-13T22:15:10.329 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.329 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.329 DEBUG:boto:Method: GET 2024-04-13T22:15:10.329 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.329 DEBUG:boto:Data: 2024-04-13T22:15:10.329 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.330 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.330 DEBUG:boto:Params: {} 2024-04-13T22:15:10.330 DEBUG:boto:Token: None 2024-04-13T22:15:10.330 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.330 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qdMxHW3i3y9E2aSdEDGhNc+xCgI= 2024-04-13T22:15:10.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qdMxHW3i3y9E2aSdEDGhNc+xCgI=', 'Content-Length': '0'} 2024-04-13T22:15:10.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006172d4c1ee1230a1-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.332 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.332 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.332 DEBUG:boto:Method: GET 2024-04-13T22:15:10.332 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.332 DEBUG:boto:Data: 2024-04-13T22:15:10.332 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.333 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.333 DEBUG:boto:Params: {} 2024-04-13T22:15:10.333 DEBUG:boto:Token: None 2024-04-13T22:15:10.333 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:10.333 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qdMxHW3i3y9E2aSdEDGhNc+xCgI= 2024-04-13T22:15:10.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qdMxHW3i3y9E2aSdEDGhNc+xCgI=', 'Content-Length': '0'} 2024-04-13T22:15:10.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f17927f4cc85b5e-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.335 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.335 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.335 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.335 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.335 DEBUG:boto:Method: GET 2024-04-13T22:15:10.336 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.336 DEBUG:boto:Data: 2024-04-13T22:15:10.336 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.336 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.336 DEBUG:boto:Params: {} 2024-04-13T22:15:10.336 DEBUG:boto:Token: None 2024-04-13T22:15:10.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d6b56650e4381a-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.339 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.339 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.339 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.339 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.339 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.339 DEBUG:boto:Method: GET 2024-04-13T22:15:10.340 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:10.340 DEBUG:boto:Data: 2024-04-13T22:15:10.340 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.340 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.340 DEBUG:boto:Params: {} 2024-04-13T22:15:10.340 DEBUG:boto:Token: None 2024-04-13T22:15:10.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de95a2f7e9a9787d-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.342 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:14:54.499Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-13T22:15:10.343 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.343 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.343 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.343 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.343 DEBUG:boto:Method: GET 2024-04-13T22:15:10.343 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.344 DEBUG:boto:Data: 2024-04-13T22:15:10.344 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.344 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.344 DEBUG:boto:Params: {} 2024-04-13T22:15:10.344 DEBUG:boto:Token: None 2024-04-13T22:15:10.344 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA= 2024-04-13T22:15:10.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA=', 'Content-Length': '0'} 2024-04-13T22:15:10.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0dcfec0564139d0-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.386 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.386 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.387 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.387 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.387 DEBUG:boto:Method: GET 2024-04-13T22:15:10.387 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.387 DEBUG:boto:Data: 2024-04-13T22:15:10.387 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.387 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.387 DEBUG:boto:Params: {} 2024-04-13T22:15:10.387 DEBUG:boto:Token: None 2024-04-13T22:15:10.388 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA= 2024-04-13T22:15:10.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA=', 'Content-Length': '0'} 2024-04-13T22:15:10.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b23cb65d16e111e5-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.389 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.390 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.390 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.390 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.390 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.390 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.390 DEBUG:boto:Data: 2024-04-13T22:15:10.390 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.390 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.390 DEBUG:boto:Params: {} 2024-04-13T22:15:10.390 DEBUG:boto:Token: None 2024-04-13T22:15:10.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:54AMQ47TqPl8h6jnoWqMiT1PQ3k= 2024-04-13T22:15:10.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:54AMQ47TqPl8h6jnoWqMiT1PQ3k=', 'Content-Length': '0'} 2024-04-13T22:15:10.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe1ac4e22680a7c1-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.392 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.392 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.393 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.393 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.393 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.393 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.393 DEBUG:boto:Data: 2024-04-13T22:15:10.393 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.393 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.393 DEBUG:boto:Params: {} 2024-04-13T22:15:10.393 DEBUG:boto:Token: None 2024-04-13T22:15:10.394 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.394 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:54AMQ47TqPl8h6jnoWqMiT1PQ3k= 2024-04-13T22:15:10.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:54AMQ47TqPl8h6jnoWqMiT1PQ3k=', 'Content-Length': '0'} 2024-04-13T22:15:10.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe63bb4391abc4bb-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-13T22:15:10.396 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.396 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.396 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.396 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.396 DEBUG:boto:Method: GET 2024-04-13T22:15:10.396 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.396 DEBUG:boto:Data: 2024-04-13T22:15:10.396 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.397 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.397 DEBUG:boto:Params: {} 2024-04-13T22:15:10.397 DEBUG:boto:Token: None 2024-04-13T22:15:10.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA= 2024-04-13T22:15:10.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA=', 'Content-Length': '0'} 2024-04-13T22:15:10.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920c3aa0f98bb6e3-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.399 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.399 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.399 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.399 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.399 DEBUG:boto:Method: GET 2024-04-13T22:15:10.399 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.399 DEBUG:boto:Data: 2024-04-13T22:15:10.400 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.400 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.400 DEBUG:boto:Params: {} 2024-04-13T22:15:10.400 DEBUG:boto:Token: None 2024-04-13T22:15:10.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:10.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA= 2024-04-13T22:15:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uK2Rn6Pv+zdpUA00XpnkT0vQ/mA=', 'Content-Length': '0'} 2024-04-13T22:15:10.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b040f025adab5d51-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.402 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.403 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.403 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.403 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.403 DEBUG:boto:Method: GET 2024-04-13T22:15:10.403 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.403 DEBUG:boto:Data: 2024-04-13T22:15:10.403 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.403 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.403 DEBUG:boto:Params: {} 2024-04-13T22:15:10.404 DEBUG:boto:Token: None 2024-04-13T22:15:10.404 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba05882f1a4c783b-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.406 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.407 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.407 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.407 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.407 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.407 DEBUG:boto:Method: GET 2024-04-13T22:15:10.407 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.407 DEBUG:boto:Data: 2024-04-13T22:15:10.407 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.407 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.407 DEBUG:boto:Params: {} 2024-04-13T22:15:10.408 DEBUG:boto:Token: None 2024-04-13T22:15:10.408 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64f3c5b1ecdc1ad-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.450 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.451 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.451 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.451 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.451 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.451 DEBUG:boto:Data: 2024-04-13T22:15:10.451 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.451 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.451 DEBUG:boto:Params: {} 2024-04-13T22:15:10.452 DEBUG:boto:Token: None 2024-04-13T22:15:10.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9AJAfY8kROEqJavo9u9w3fA1RcU= 2024-04-13T22:15:10.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9AJAfY8kROEqJavo9u9w3fA1RcU=', 'Content-Length': '0'} 2024-04-13T22:15:10.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e95c7d13c8cba8d-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.454 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.454 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.454 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.454 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.454 DEBUG:boto:Data: 2024-04-13T22:15:10.454 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.454 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.455 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.455 DEBUG:boto:Params: {} 2024-04-13T22:15:10.455 DEBUG:boto:Token: None 2024-04-13T22:15:10.455 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9AJAfY8kROEqJavo9u9w3fA1RcU= 2024-04-13T22:15:10.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9AJAfY8kROEqJavo9u9w3fA1RcU=', 'Content-Length': '0'} 2024-04-13T22:15:10.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009808f47787b23888-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-13T22:15:10.457 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.457 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.457 DEBUG:boto:Method: GET 2024-04-13T22:15:10.457 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.457 DEBUG:boto:Data: 2024-04-13T22:15:10.457 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.457 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.458 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.458 DEBUG:boto:Params: {} 2024-04-13T22:15:10.458 DEBUG:boto:Token: None 2024-04-13T22:15:10.458 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.458 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pab43dfAZTlF8SHqttLIcCZiMh8= 2024-04-13T22:15:10.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pab43dfAZTlF8SHqttLIcCZiMh8=', 'Content-Length': '0'} 2024-04-13T22:15:10.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c5a9e8071728cb-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.460 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.460 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.460 DEBUG:boto:Method: GET 2024-04-13T22:15:10.460 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.461 DEBUG:boto:Data: 2024-04-13T22:15:10.461 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.461 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.461 DEBUG:boto:Params: {} 2024-04-13T22:15:10.461 DEBUG:boto:Token: None 2024-04-13T22:15:10.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:10.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pab43dfAZTlF8SHqttLIcCZiMh8= 2024-04-13T22:15:10.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pab43dfAZTlF8SHqttLIcCZiMh8=', 'Content-Length': '0'} 2024-04-13T22:15:10.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc568e6edbeb5b2c-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.463 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.463 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.463 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.463 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.463 DEBUG:boto:Method: GET 2024-04-13T22:15:10.464 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.464 DEBUG:boto:Data: 2024-04-13T22:15:10.464 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.464 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.464 DEBUG:boto:Params: {} 2024-04-13T22:15:10.464 DEBUG:boto:Token: None 2024-04-13T22:15:10.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a23d3652b8809ef3-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.467 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.467 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.467 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.467 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.468 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.468 DEBUG:boto:Method: GET 2024-04-13T22:15:10.468 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:10.468 DEBUG:boto:Data: 2024-04-13T22:15:10.468 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.468 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.468 DEBUG:boto:Params: {} 2024-04-13T22:15:10.468 DEBUG:boto:Token: None 2024-04-13T22:15:10.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc1cc14d1746d40-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.471 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:14:54.509Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-13T22:15:10.472 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.472 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.472 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.472 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.472 DEBUG:boto:Method: GET 2024-04-13T22:15:10.472 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.472 DEBUG:boto:Data: 2024-04-13T22:15:10.472 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.473 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.473 DEBUG:boto:Params: {} 2024-04-13T22:15:10.473 DEBUG:boto:Token: None 2024-04-13T22:15:10.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0= 2024-04-13T22:15:10.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0=', 'Content-Length': '0'} 2024-04-13T22:15:10.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb37eee85d6f69a5-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.518 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.518 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.518 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.519 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.519 DEBUG:boto:Method: GET 2024-04-13T22:15:10.519 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.519 DEBUG:boto:Data: 2024-04-13T22:15:10.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.519 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.519 DEBUG:boto:Params: {} 2024-04-13T22:15:10.519 DEBUG:boto:Token: None 2024-04-13T22:15:10.519 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0= 2024-04-13T22:15:10.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0=', 'Content-Length': '0'} 2024-04-13T22:15:10.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c69d75c5fcdfba0-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.522 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.522 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.522 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.522 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.522 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.522 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.522 DEBUG:boto:Data: 2024-04-13T22:15:10.522 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.523 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.523 DEBUG:boto:Params: {} 2024-04-13T22:15:10.523 DEBUG:boto:Token: None 2024-04-13T22:15:10.523 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1iDYHqA7w3++3EnNPk2mtOE8xFg= 2024-04-13T22:15:10.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1iDYHqA7w3++3EnNPk2mtOE8xFg=', 'Content-Length': '0'} 2024-04-13T22:15:10.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f0b3d1519af1be-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.525 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.525 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.525 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.525 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.525 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.525 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.525 DEBUG:boto:Data: 2024-04-13T22:15:10.526 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.526 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.526 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.526 DEBUG:boto:Params: {} 2024-04-13T22:15:10.526 DEBUG:boto:Token: None 2024-04-13T22:15:10.526 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.526 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1iDYHqA7w3++3EnNPk2mtOE8xFg= 2024-04-13T22:15:10.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1iDYHqA7w3++3EnNPk2mtOE8xFg=', 'Content-Length': '0'} 2024-04-13T22:15:10.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624da6eca6def2f0-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-13T22:15:10.528 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.528 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.528 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.528 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.529 DEBUG:boto:Method: GET 2024-04-13T22:15:10.529 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.529 DEBUG:boto:Data: 2024-04-13T22:15:10.529 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.529 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.529 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.529 DEBUG:boto:Params: {} 2024-04-13T22:15:10.529 DEBUG:boto:Token: None 2024-04-13T22:15:10.529 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.529 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0= 2024-04-13T22:15:10.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0=', 'Content-Length': '0'} 2024-04-13T22:15:10.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d884ff7ca5572d1-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.531 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.531 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.532 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.532 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.532 DEBUG:boto:Method: GET 2024-04-13T22:15:10.532 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.532 DEBUG:boto:Data: 2024-04-13T22:15:10.532 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.532 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.532 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.532 DEBUG:boto:Params: {} 2024-04-13T22:15:10.532 DEBUG:boto:Token: None 2024-04-13T22:15:10.532 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:10.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0= 2024-04-13T22:15:10.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f+XEA+NdPe5ef/qaPZOVYykDzY0=', 'Content-Length': '0'} 2024-04-13T22:15:10.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e7a77c95ab7694-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.534 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.535 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.535 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.535 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.535 DEBUG:boto:Method: GET 2024-04-13T22:15:10.535 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.535 DEBUG:boto:Data: 2024-04-13T22:15:10.535 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.535 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.536 DEBUG:boto:Params: {} 2024-04-13T22:15:10.536 DEBUG:boto:Token: None 2024-04-13T22:15:10.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064024569922ebbd2-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.538 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.539 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.539 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.539 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.539 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.539 DEBUG:boto:Method: GET 2024-04-13T22:15:10.539 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.539 DEBUG:boto:Data: 2024-04-13T22:15:10.539 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.540 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.540 DEBUG:boto:Params: {} 2024-04-13T22:15:10.540 DEBUG:boto:Token: None 2024-04-13T22:15:10.540 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021f26045d7540ab4-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.582 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.583 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.583 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.583 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.583 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.583 DEBUG:boto:Data: 2024-04-13T22:15:10.583 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.583 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.584 DEBUG:boto:Params: {} 2024-04-13T22:15:10.584 DEBUG:boto:Token: None 2024-04-13T22:15:10.584 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:asRp8PbJjz0FTFEGqKi5e3wHLSM= 2024-04-13T22:15:10.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:asRp8PbJjz0FTFEGqKi5e3wHLSM=', 'Content-Length': '0'} 2024-04-13T22:15:10.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ae82df256b93a4-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.586 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.586 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.586 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.587 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.587 DEBUG:boto:Data: 2024-04-13T22:15:10.587 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.587 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.587 DEBUG:boto:Params: {} 2024-04-13T22:15:10.587 DEBUG:boto:Token: None 2024-04-13T22:15:10.587 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:asRp8PbJjz0FTFEGqKi5e3wHLSM= 2024-04-13T22:15:10.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:asRp8PbJjz0FTFEGqKi5e3wHLSM=', 'Content-Length': '0'} 2024-04-13T22:15:10.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f6499456db093c-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-13T22:15:10.590 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.590 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.590 DEBUG:boto:Method: GET 2024-04-13T22:15:10.590 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.590 DEBUG:boto:Data: 2024-04-13T22:15:10.590 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.590 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.590 DEBUG:boto:Params: {} 2024-04-13T22:15:10.590 DEBUG:boto:Token: None 2024-04-13T22:15:10.591 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.591 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WnvqLfP8cFZ0TdQLaBmdWFkWyKA= 2024-04-13T22:15:10.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WnvqLfP8cFZ0TdQLaBmdWFkWyKA=', 'Content-Length': '0'} 2024-04-13T22:15:10.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c87260ea78c7550-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.593 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.593 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.593 DEBUG:boto:Method: GET 2024-04-13T22:15:10.593 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.593 DEBUG:boto:Data: 2024-04-13T22:15:10.593 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.594 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.594 DEBUG:boto:Params: {} 2024-04-13T22:15:10.594 DEBUG:boto:Token: None 2024-04-13T22:15:10.594 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:10.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WnvqLfP8cFZ0TdQLaBmdWFkWyKA= 2024-04-13T22:15:10.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WnvqLfP8cFZ0TdQLaBmdWFkWyKA=', 'Content-Length': '0'} 2024-04-13T22:15:10.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139335477cd152e5-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.596 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.596 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.596 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.596 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.597 DEBUG:boto:Method: GET 2024-04-13T22:15:10.597 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.597 DEBUG:boto:Data: 2024-04-13T22:15:10.597 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.597 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.597 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.597 DEBUG:boto:Params: {} 2024-04-13T22:15:10.597 DEBUG:boto:Token: None 2024-04-13T22:15:10.597 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd21f02909b7360a-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.600 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.600 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.600 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.601 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.601 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.601 DEBUG:boto:Method: GET 2024-04-13T22:15:10.601 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:10.601 DEBUG:boto:Data: 2024-04-13T22:15:10.601 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.601 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.601 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.601 DEBUG:boto:Params: {} 2024-04-13T22:15:10.601 DEBUG:boto:Token: None 2024-04-13T22:15:10.602 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.602 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfcb98503fce5cc1-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.604 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:14:54.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-13T22:15:10.604 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.604 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.604 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.605 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.605 DEBUG:boto:Method: GET 2024-04-13T22:15:10.605 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.605 DEBUG:boto:Data: 2024-04-13T22:15:10.605 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.605 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.605 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.605 DEBUG:boto:Params: {} 2024-04-13T22:15:10.605 DEBUG:boto:Token: None 2024-04-13T22:15:10.606 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.606 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE= 2024-04-13T22:15:10.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE=', 'Content-Length': '0'} 2024-04-13T22:15:10.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6fe6b1d565e7c0b-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.650 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.651 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.651 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.651 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.651 DEBUG:boto:Method: GET 2024-04-13T22:15:10.651 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.651 DEBUG:boto:Data: 2024-04-13T22:15:10.651 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.651 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.651 DEBUG:boto:Params: {} 2024-04-13T22:15:10.652 DEBUG:boto:Token: None 2024-04-13T22:15:10.652 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE= 2024-04-13T22:15:10.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE=', 'Content-Length': '0'} 2024-04-13T22:15:10.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e9d805d723d7c2-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.654 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.654 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.655 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.655 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.655 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.655 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.655 DEBUG:boto:Data: 2024-04-13T22:15:10.655 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.655 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.655 DEBUG:boto:Params: {} 2024-04-13T22:15:10.655 DEBUG:boto:Token: None 2024-04-13T22:15:10.656 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MYkQU7BwOc5tQUZxUGIdNh6nrwI= 2024-04-13T22:15:10.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MYkQU7BwOc5tQUZxUGIdNh6nrwI=', 'Content-Length': '0'} 2024-04-13T22:15:10.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f749934421a8897-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.657 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.658 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.658 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.658 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.658 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.658 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.658 DEBUG:boto:Data: 2024-04-13T22:15:10.658 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.658 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.659 DEBUG:boto:Params: {} 2024-04-13T22:15:10.659 DEBUG:boto:Token: None 2024-04-13T22:15:10.659 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MYkQU7BwOc5tQUZxUGIdNh6nrwI= 2024-04-13T22:15:10.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MYkQU7BwOc5tQUZxUGIdNh6nrwI=', 'Content-Length': '0'} 2024-04-13T22:15:10.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027f4a0043a98f96f-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-13T22:15:10.661 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.661 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.661 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.661 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.662 DEBUG:boto:Method: GET 2024-04-13T22:15:10.662 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.662 DEBUG:boto:Data: 2024-04-13T22:15:10.662 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.662 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.662 DEBUG:boto:Params: {} 2024-04-13T22:15:10.662 DEBUG:boto:Token: None 2024-04-13T22:15:10.662 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE= 2024-04-13T22:15:10.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE=', 'Content-Length': '0'} 2024-04-13T22:15:10.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c546ef4c54692d-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.664 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.665 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.665 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.665 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.665 DEBUG:boto:Method: GET 2024-04-13T22:15:10.665 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.665 DEBUG:boto:Data: 2024-04-13T22:15:10.665 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.665 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.665 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.665 DEBUG:boto:Params: {} 2024-04-13T22:15:10.666 DEBUG:boto:Token: None 2024-04-13T22:15:10.666 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:10.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE= 2024-04-13T22:15:10.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DHCpIhfCyCpxs17YknT9eMe1AvE=', 'Content-Length': '0'} 2024-04-13T22:15:10.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074efe0c60d5e0b6c-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.668 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.668 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.668 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.668 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.668 DEBUG:boto:Method: GET 2024-04-13T22:15:10.668 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.668 DEBUG:boto:Data: 2024-04-13T22:15:10.668 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.668 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.668 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.668 DEBUG:boto:Params: {} 2024-04-13T22:15:10.669 DEBUG:boto:Token: None 2024-04-13T22:15:10.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecda2e3e9cbf1ea6-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.671 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.671 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.672 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.672 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.672 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.672 DEBUG:boto:Method: GET 2024-04-13T22:15:10.672 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.672 DEBUG:boto:Data: 2024-04-13T22:15:10.672 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.672 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.672 DEBUG:boto:Params: {} 2024-04-13T22:15:10.673 DEBUG:boto:Token: None 2024-04-13T22:15:10.673 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.673 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099eb8c98070ad55c-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.675 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.676 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.676 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.676 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.676 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.676 DEBUG:boto:Data: 2024-04-13T22:15:10.676 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.676 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.676 DEBUG:boto:Params: {} 2024-04-13T22:15:10.677 DEBUG:boto:Token: None 2024-04-13T22:15:10.677 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LDbWF66gfndzFbEJYijvcr+y4g0= 2024-04-13T22:15:10.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LDbWF66gfndzFbEJYijvcr+y4g0=', 'Content-Length': '0'} 2024-04-13T22:15:10.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df1ec0fdbba938c6-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.678 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.679 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.679 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.679 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.679 DEBUG:boto:Data: 2024-04-13T22:15:10.679 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.679 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.679 DEBUG:boto:Params: {} 2024-04-13T22:15:10.679 DEBUG:boto:Token: None 2024-04-13T22:15:10.679 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LDbWF66gfndzFbEJYijvcr+y4g0= 2024-04-13T22:15:10.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LDbWF66gfndzFbEJYijvcr+y4g0=', 'Content-Length': '0'} 2024-04-13T22:15:10.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fced90dd2dd10e1a-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-13T22:15:10.681 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.681 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.681 DEBUG:boto:Method: GET 2024-04-13T22:15:10.682 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.682 DEBUG:boto:Data: 2024-04-13T22:15:10.682 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.682 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.682 DEBUG:boto:Params: {} 2024-04-13T22:15:10.682 DEBUG:boto:Token: None 2024-04-13T22:15:10.682 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BRWH+VMPYpEXtCGmwG0yPl+YwLU= 2024-04-13T22:15:10.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BRWH+VMPYpEXtCGmwG0yPl+YwLU=', 'Content-Length': '0'} 2024-04-13T22:15:10.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005094eb9cff9ecba2-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.726 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.726 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.727 DEBUG:boto:Method: GET 2024-04-13T22:15:10.727 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.727 DEBUG:boto:Data: 2024-04-13T22:15:10.727 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.727 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.727 DEBUG:boto:Params: {} 2024-04-13T22:15:10.727 DEBUG:boto:Token: None 2024-04-13T22:15:10.727 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:10.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BRWH+VMPYpEXtCGmwG0yPl+YwLU= 2024-04-13T22:15:10.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BRWH+VMPYpEXtCGmwG0yPl+YwLU=', 'Content-Length': '0'} 2024-04-13T22:15:10.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3ede6da1fea04f-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.729 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.729 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.730 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.730 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.730 DEBUG:boto:Method: GET 2024-04-13T22:15:10.730 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.730 DEBUG:boto:Data: 2024-04-13T22:15:10.730 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.730 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.730 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.730 DEBUG:boto:Params: {} 2024-04-13T22:15:10.730 DEBUG:boto:Token: None 2024-04-13T22:15:10.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.731 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308f88f4fc93aaed-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.733 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.733 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.733 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.734 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.734 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.734 DEBUG:boto:Method: GET 2024-04-13T22:15:10.734 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:10.734 DEBUG:boto:Data: 2024-04-13T22:15:10.734 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.734 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.734 DEBUG:boto:Params: {} 2024-04-13T22:15:10.734 DEBUG:boto:Token: None 2024-04-13T22:15:10.735 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5730a2031730abd-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.737 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:14:54.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-13T22:15:10.737 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.737 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.737 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.738 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.738 DEBUG:boto:Method: GET 2024-04-13T22:15:10.738 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.738 DEBUG:boto:Data: 2024-04-13T22:15:10.738 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.738 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.738 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.738 DEBUG:boto:Params: {} 2024-04-13T22:15:10.738 DEBUG:boto:Token: None 2024-04-13T22:15:10.738 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg= 2024-04-13T22:15:10.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bbd2d1a72c44952-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.740 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.740 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.740 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.740 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.741 DEBUG:boto:Method: GET 2024-04-13T22:15:10.741 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.741 DEBUG:boto:Data: 2024-04-13T22:15:10.741 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.741 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.741 DEBUG:boto:Params: {} 2024-04-13T22:15:10.741 DEBUG:boto:Token: None 2024-04-13T22:15:10.741 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg= 2024-04-13T22:15:10.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c07392d2c80183-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.743 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.744 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.744 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.744 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.744 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.744 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.744 DEBUG:boto:Data: 2024-04-13T22:15:10.744 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.744 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.744 DEBUG:boto:Params: {} 2024-04-13T22:15:10.745 DEBUG:boto:Token: None 2024-04-13T22:15:10.745 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.745 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EbMQAhtwXlgvBp7GlnaiJmmp0CI= 2024-04-13T22:15:10.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EbMQAhtwXlgvBp7GlnaiJmmp0CI=', 'Content-Length': '0'} 2024-04-13T22:15:10.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a33606f5d3451d9-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.747 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.747 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.747 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.747 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.747 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.747 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.747 DEBUG:boto:Data: 2024-04-13T22:15:10.747 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.748 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.748 DEBUG:boto:Params: {} 2024-04-13T22:15:10.748 DEBUG:boto:Token: None 2024-04-13T22:15:10.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EbMQAhtwXlgvBp7GlnaiJmmp0CI= 2024-04-13T22:15:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EbMQAhtwXlgvBp7GlnaiJmmp0CI=', 'Content-Length': '0'} 2024-04-13T22:15:10.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d825db4fda2ec353-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-13T22:15:10.750 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.750 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.750 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.750 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.750 DEBUG:boto:Method: GET 2024-04-13T22:15:10.750 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.750 DEBUG:boto:Data: 2024-04-13T22:15:10.750 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.751 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.751 DEBUG:boto:Params: {} 2024-04-13T22:15:10.751 DEBUG:boto:Token: None 2024-04-13T22:15:10.751 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg= 2024-04-13T22:15:10.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33e7fff2b933158-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.753 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.753 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.753 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.753 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.753 DEBUG:boto:Method: GET 2024-04-13T22:15:10.754 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.754 DEBUG:boto:Data: 2024-04-13T22:15:10.754 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.754 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.754 DEBUG:boto:Params: {} 2024-04-13T22:15:10.754 DEBUG:boto:Token: None 2024-04-13T22:15:10.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:10.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg= 2024-04-13T22:15:10.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:znHqwcLNNl2V8ZnL3X4RwWnBrUg=', 'Content-Length': '0'} 2024-04-13T22:15:10.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb61bfdf2fae5bf-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.798 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.798 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.798 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.799 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.799 DEBUG:boto:Method: GET 2024-04-13T22:15:10.799 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.799 DEBUG:boto:Data: 2024-04-13T22:15:10.799 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.799 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.799 DEBUG:boto:Params: {} 2024-04-13T22:15:10.799 DEBUG:boto:Token: None 2024-04-13T22:15:10.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4117e7c00cbd054-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.802 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.802 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.802 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.803 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.803 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.803 DEBUG:boto:Method: GET 2024-04-13T22:15:10.803 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.803 DEBUG:boto:Data: 2024-04-13T22:15:10.803 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.803 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.803 DEBUG:boto:Params: {} 2024-04-13T22:15:10.803 DEBUG:boto:Token: None 2024-04-13T22:15:10.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c738e757b4dff5b-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.806 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.806 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.806 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.807 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.807 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.807 DEBUG:boto:Data: 2024-04-13T22:15:10.807 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.807 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.807 DEBUG:boto:Params: {} 2024-04-13T22:15:10.807 DEBUG:boto:Token: None 2024-04-13T22:15:10.807 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gCWP5yGG1jrXdulQ4C97zl7Ms1Q= 2024-04-13T22:15:10.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gCWP5yGG1jrXdulQ4C97zl7Ms1Q=', 'Content-Length': '0'} 2024-04-13T22:15:10.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec231200348641b-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.809 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.809 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.809 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.810 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.810 DEBUG:boto:Data: 2024-04-13T22:15:10.810 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.810 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.810 DEBUG:boto:Params: {} 2024-04-13T22:15:10.810 DEBUG:boto:Token: None 2024-04-13T22:15:10.810 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.810 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gCWP5yGG1jrXdulQ4C97zl7Ms1Q= 2024-04-13T22:15:10.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gCWP5yGG1jrXdulQ4C97zl7Ms1Q=', 'Content-Length': '0'} 2024-04-13T22:15:10.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006878403cbba33a63-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-13T22:15:10.812 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.813 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.813 DEBUG:boto:Method: GET 2024-04-13T22:15:10.813 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.813 DEBUG:boto:Data: 2024-04-13T22:15:10.813 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.813 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.813 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.813 DEBUG:boto:Params: {} 2024-04-13T22:15:10.813 DEBUG:boto:Token: None 2024-04-13T22:15:10.813 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SKIKVsNZ8fZ3VHnEEwcr36AjpqQ= 2024-04-13T22:15:10.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SKIKVsNZ8fZ3VHnEEwcr36AjpqQ=', 'Content-Length': '0'} 2024-04-13T22:15:10.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74c2fad786f6073-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.815 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.816 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.816 DEBUG:boto:Method: GET 2024-04-13T22:15:10.816 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.816 DEBUG:boto:Data: 2024-04-13T22:15:10.816 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.816 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.816 DEBUG:boto:Params: {} 2024-04-13T22:15:10.816 DEBUG:boto:Token: None 2024-04-13T22:15:10.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:10.817 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SKIKVsNZ8fZ3VHnEEwcr36AjpqQ= 2024-04-13T22:15:10.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SKIKVsNZ8fZ3VHnEEwcr36AjpqQ=', 'Content-Length': '0'} 2024-04-13T22:15:10.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1ae606237c9254-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.818 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.818 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.818 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.819 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.819 DEBUG:boto:Method: GET 2024-04-13T22:15:10.819 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.819 DEBUG:boto:Data: 2024-04-13T22:15:10.819 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.819 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.819 DEBUG:boto:Params: {} 2024-04-13T22:15:10.819 DEBUG:boto:Token: None 2024-04-13T22:15:10.820 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb638eba7cd474fe-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.862 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.863 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.863 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.863 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.863 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.863 DEBUG:boto:Method: GET 2024-04-13T22:15:10.863 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:10.863 DEBUG:boto:Data: 2024-04-13T22:15:10.863 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.863 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.863 DEBUG:boto:Params: {} 2024-04-13T22:15:10.864 DEBUG:boto:Token: None 2024-04-13T22:15:10.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6041078119de8c2-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.866 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:14:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-13T22:15:10.866 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.867 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.867 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.867 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.867 DEBUG:boto:Method: GET 2024-04-13T22:15:10.867 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.867 DEBUG:boto:Data: 2024-04-13T22:15:10.867 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.867 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.867 DEBUG:boto:Params: {} 2024-04-13T22:15:10.868 DEBUG:boto:Token: None 2024-04-13T22:15:10.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8= 2024-04-13T22:15:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8=', 'Content-Length': '0'} 2024-04-13T22:15:10.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8af160e3b0320e-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.870 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.870 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.870 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.870 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.870 DEBUG:boto:Method: GET 2024-04-13T22:15:10.870 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.870 DEBUG:boto:Data: 2024-04-13T22:15:10.870 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.870 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.870 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.871 DEBUG:boto:Params: {} 2024-04-13T22:15:10.871 DEBUG:boto:Token: None 2024-04-13T22:15:10.871 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8= 2024-04-13T22:15:10.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8=', 'Content-Length': '0'} 2024-04-13T22:15:10.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d77938d75124224d-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.873 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.873 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.873 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.873 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.873 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.873 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.873 DEBUG:boto:Data: 2024-04-13T22:15:10.873 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.873 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.873 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.873 DEBUG:boto:Params: {} 2024-04-13T22:15:10.874 DEBUG:boto:Token: None 2024-04-13T22:15:10.874 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.874 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IKzoRNe5E+T6H3uJlgijp6PoFjM= 2024-04-13T22:15:10.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IKzoRNe5E+T6H3uJlgijp6PoFjM=', 'Content-Length': '0'} 2024-04-13T22:15:10.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005388fda13f645177-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.876 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.876 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.876 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.876 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.876 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.876 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.876 DEBUG:boto:Data: 2024-04-13T22:15:10.876 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.876 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.876 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.876 DEBUG:boto:Params: {} 2024-04-13T22:15:10.876 DEBUG:boto:Token: None 2024-04-13T22:15:10.877 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IKzoRNe5E+T6H3uJlgijp6PoFjM= 2024-04-13T22:15:10.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IKzoRNe5E+T6H3uJlgijp6PoFjM=', 'Content-Length': '0'} 2024-04-13T22:15:10.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b02d54841f82b00-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-13T22:15:10.879 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.879 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.879 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.879 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.879 DEBUG:boto:Method: GET 2024-04-13T22:15:10.879 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.880 DEBUG:boto:Data: 2024-04-13T22:15:10.880 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.880 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.880 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.880 DEBUG:boto:Params: {} 2024-04-13T22:15:10.880 DEBUG:boto:Token: None 2024-04-13T22:15:10.880 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8= 2024-04-13T22:15:10.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8=', 'Content-Length': '0'} 2024-04-13T22:15:10.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000179f0034b1118833-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.882 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.882 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.883 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.883 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.883 DEBUG:boto:Method: GET 2024-04-13T22:15:10.883 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.883 DEBUG:boto:Data: 2024-04-13T22:15:10.883 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.883 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.883 DEBUG:boto:Params: {} 2024-04-13T22:15:10.883 DEBUG:boto:Token: None 2024-04-13T22:15:10.883 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:10.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8= 2024-04-13T22:15:10.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MrJj6z60MKU8NoXDGcEhfUuJc+8=', 'Content-Length': '0'} 2024-04-13T22:15:10.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d881d25906549c-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.885 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.886 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.886 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.886 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.886 DEBUG:boto:Method: GET 2024-04-13T22:15:10.886 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.886 DEBUG:boto:Data: 2024-04-13T22:15:10.886 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.887 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.887 DEBUG:boto:Params: {} 2024-04-13T22:15:10.887 DEBUG:boto:Token: None 2024-04-13T22:15:10.887 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002ab05294913bd3c-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.889 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.890 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.890 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.890 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.890 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.890 DEBUG:boto:Method: GET 2024-04-13T22:15:10.891 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.891 DEBUG:boto:Data: 2024-04-13T22:15:10.891 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.892 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.892 DEBUG:boto:Params: {} 2024-04-13T22:15:10.892 DEBUG:boto:Token: None 2024-04-13T22:15:10.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000901ef5ca00bbc9c3-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.895 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.895 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.895 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.895 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.895 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.895 DEBUG:boto:Data: 2024-04-13T22:15:10.896 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.896 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.896 DEBUG:boto:Params: {} 2024-04-13T22:15:10.896 DEBUG:boto:Token: None 2024-04-13T22:15:10.896 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o4xYJu2ag1vUkctU6zAmnHlynbw= 2024-04-13T22:15:10.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o4xYJu2ag1vUkctU6zAmnHlynbw=', 'Content-Length': '0'} 2024-04-13T22:15:10.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d9919346660b64-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.898 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.898 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.898 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.898 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.899 DEBUG:boto:Data: 2024-04-13T22:15:10.899 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.899 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.899 DEBUG:boto:Params: {} 2024-04-13T22:15:10.899 DEBUG:boto:Token: None 2024-04-13T22:15:10.899 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o4xYJu2ag1vUkctU6zAmnHlynbw= 2024-04-13T22:15:10.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o4xYJu2ag1vUkctU6zAmnHlynbw=', 'Content-Length': '0'} 2024-04-13T22:15:10.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1bdd13d02f37623-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-13T22:15:10.901 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.901 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.901 DEBUG:boto:Method: GET 2024-04-13T22:15:10.901 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.901 DEBUG:boto:Data: 2024-04-13T22:15:10.902 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.902 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.902 DEBUG:boto:Params: {} 2024-04-13T22:15:10.902 DEBUG:boto:Token: None 2024-04-13T22:15:10.902 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.902 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dBdnXYPJTkQBAwDF9i4dhuCkrFs= 2024-04-13T22:15:10.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dBdnXYPJTkQBAwDF9i4dhuCkrFs=', 'Content-Length': '0'} 2024-04-13T22:15:10.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b2af841378a9bb-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.904 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.904 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.904 DEBUG:boto:Method: GET 2024-04-13T22:15:10.904 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.904 DEBUG:boto:Data: 2024-04-13T22:15:10.904 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.905 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.905 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.905 DEBUG:boto:Params: {} 2024-04-13T22:15:10.905 DEBUG:boto:Token: None 2024-04-13T22:15:10.905 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:10.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dBdnXYPJTkQBAwDF9i4dhuCkrFs= 2024-04-13T22:15:10.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dBdnXYPJTkQBAwDF9i4dhuCkrFs=', 'Content-Length': '0'} 2024-04-13T22:15:10.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd88d557cef5f44-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.950 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.950 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.951 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.951 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.951 DEBUG:boto:Method: GET 2024-04-13T22:15:10.951 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.951 DEBUG:boto:Data: 2024-04-13T22:15:10.951 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.952 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.952 DEBUG:boto:Params: {} 2024-04-13T22:15:10.952 DEBUG:boto:Token: None 2024-04-13T22:15:10.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037110fc125ff4e37-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.954 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.955 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.955 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.955 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.955 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.955 DEBUG:boto:Method: GET 2024-04-13T22:15:10.955 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:10.955 DEBUG:boto:Data: 2024-04-13T22:15:10.955 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.956 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.956 DEBUG:boto:Params: {} 2024-04-13T22:15:10.956 DEBUG:boto:Token: None 2024-04-13T22:15:10.956 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ff32aec6b6fd82-00661b03ee-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.959 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:14:54.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:10.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-13T22:15:10.959 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.959 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.959 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.960 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.960 DEBUG:boto:Method: GET 2024-04-13T22:15:10.960 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.960 DEBUG:boto:Data: 2024-04-13T22:15:10.960 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.960 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.960 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.960 DEBUG:boto:Params: {} 2024-04-13T22:15:10.960 DEBUG:boto:Token: None 2024-04-13T22:15:10.960 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.961 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c= 2024-04-13T22:15:10.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c=', 'Content-Length': '0'} 2024-04-13T22:15:10.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84c6921ffe627c5-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.962 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.963 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.963 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.963 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.963 DEBUG:boto:Method: GET 2024-04-13T22:15:10.963 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.963 DEBUG:boto:Data: 2024-04-13T22:15:10.963 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.963 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.963 DEBUG:boto:Params: {} 2024-04-13T22:15:10.964 DEBUG:boto:Token: None 2024-04-13T22:15:10.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c= 2024-04-13T22:15:10.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c=', 'Content-Length': '0'} 2024-04-13T22:15:10.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c967d1ca59801b-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.966 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.966 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.966 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.966 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.966 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.966 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.967 DEBUG:boto:Data: 2024-04-13T22:15:10.967 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.967 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.967 DEBUG:boto:Params: {} 2024-04-13T22:15:10.967 DEBUG:boto:Token: None 2024-04-13T22:15:10.967 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.967 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bBZGbPWDHH9FXH08dSztCr3mmbw= 2024-04-13T22:15:10.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bBZGbPWDHH9FXH08dSztCr3mmbw=', 'Content-Length': '0'} 2024-04-13T22:15:10.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047c7f81193bff9c8-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.969 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.969 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.969 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.969 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.969 DEBUG:boto:Method: HEAD 2024-04-13T22:15:10.969 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.969 DEBUG:boto:Data: 2024-04-13T22:15:10.970 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.970 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.970 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.970 DEBUG:boto:Params: {} 2024-04-13T22:15:10.970 DEBUG:boto:Token: None 2024-04-13T22:15:10.970 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.970 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bBZGbPWDHH9FXH08dSztCr3mmbw= 2024-04-13T22:15:10.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bBZGbPWDHH9FXH08dSztCr3mmbw=', 'Content-Length': '0'} 2024-04-13T22:15:10.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7318667cc1adb29-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-13T22:15:10.972 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.972 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.972 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.972 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.972 DEBUG:boto:Method: GET 2024-04-13T22:15:10.972 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.973 DEBUG:boto:Data: 2024-04-13T22:15:10.973 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.973 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.973 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.973 DEBUG:boto:Params: {} 2024-04-13T22:15:10.973 DEBUG:boto:Token: None 2024-04-13T22:15:10.973 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.973 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c= 2024-04-13T22:15:10.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c=', 'Content-Length': '0'} 2024-04-13T22:15:10.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015cb7076e75bbf67-00661b03ee-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.975 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.975 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:10.975 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.975 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.975 DEBUG:boto:Method: GET 2024-04-13T22:15:10.975 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.976 DEBUG:boto:Data: 2024-04-13T22:15:10.976 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:10.976 DEBUG:boto:Port: 8000 2024-04-13T22:15:10.976 DEBUG:boto:Params: {} 2024-04-13T22:15:10.976 DEBUG:boto:Token: None 2024-04-13T22:15:10.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:10.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c= 2024-04-13T22:15:10.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W3Nxu9oUrc3tM6C/7olNhxgbJ/c=', 'Content-Length': '0'} 2024-04-13T22:15:10.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a48afda1aff9ab26-00661b03ee-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:10.979 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:10.979 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:10.979 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:10.979 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:10.979 DEBUG:boto:Method: GET 2024-04-13T22:15:10.979 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:10.979 DEBUG:boto:Data: 2024-04-13T22:15:10.979 DEBUG:boto:Headers: {} 2024-04-13T22:15:10.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:10.979 DEBUG:boto:Port: 8001 2024-04-13T22:15:10.980 DEBUG:boto:Params: {} 2024-04-13T22:15:10.980 DEBUG:boto:Token: None 2024-04-13T22:15:10.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:10 GMT /vwnfns-60/ 2024-04-13T22:15:10.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4= 2024-04-13T22:15:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YbWoLBWHU+xeJKzi/J1y7uDE0h4=', 'Content-Length': '0'} 2024-04-13T22:15:10.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020befa58b1504bf7-00661b03ee-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.022 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:11.023 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:11.023 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:11.023 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.023 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.023 DEBUG:boto:Method: GET 2024-04-13T22:15:11.023 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.023 DEBUG:boto:Data: 2024-04-13T22:15:11.023 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:11.023 DEBUG:boto:Port: 8000 2024-04-13T22:15:11.024 DEBUG:boto:Params: {} 2024-04-13T22:15:11.024 DEBUG:boto:Token: None 2024-04-13T22:15:11.024 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/ 2024-04-13T22:15:11.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pDKW5aArVeN20PJJknWVfQW9A2Y= 2024-04-13T22:15:11.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pDKW5aArVeN20PJJknWVfQW9A2Y=', 'Content-Length': '0'} 2024-04-13T22:15:11.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb029c7b64d841ce-00661b03ef-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.026 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:11.026 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.026 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.027 DEBUG:boto:Method: HEAD 2024-04-13T22:15:11.027 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.027 DEBUG:boto:Data: 2024-04-13T22:15:11.027 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:11.027 DEBUG:boto:Port: 8001 2024-04-13T22:15:11.027 DEBUG:boto:Params: {} 2024-04-13T22:15:11.027 DEBUG:boto:Token: None 2024-04-13T22:15:11.027 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0lM4JkPfHrGP2g7jidjaPxQsjo8= 2024-04-13T22:15:11.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0lM4JkPfHrGP2g7jidjaPxQsjo8=', 'Content-Length': '0'} 2024-04-13T22:15:11.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b8cad9373b0906-00661b03ef-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.029 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.029 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.030 DEBUG:boto:Method: HEAD 2024-04-13T22:15:11.030 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.030 DEBUG:boto:Data: 2024-04-13T22:15:11.030 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.030 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:11.030 DEBUG:boto:Port: 8000 2024-04-13T22:15:11.030 DEBUG:boto:Params: {} 2024-04-13T22:15:11.030 DEBUG:boto:Token: None 2024-04-13T22:15:11.030 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0lM4JkPfHrGP2g7jidjaPxQsjo8= 2024-04-13T22:15:11.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0lM4JkPfHrGP2g7jidjaPxQsjo8=', 'Content-Length': '0'} 2024-04-13T22:15:11.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed69a67930bdcfa2-00661b03ef-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-13T22:15:11.033 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.033 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.033 DEBUG:boto:Method: GET 2024-04-13T22:15:11.033 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.033 DEBUG:boto:Data: 2024-04-13T22:15:11.033 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.033 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:11.033 DEBUG:boto:Port: 8001 2024-04-13T22:15:11.033 DEBUG:boto:Params: {} 2024-04-13T22:15:11.033 DEBUG:boto:Token: None 2024-04-13T22:15:11.033 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.034 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ydYoHeImxiN96EenUFROkvLwm1w= 2024-04-13T22:15:11.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ydYoHeImxiN96EenUFROkvLwm1w=', 'Content-Length': '0'} 2024-04-13T22:15:11.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c15f6bd5bb9625-00661b03ef-1eaf-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.035 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.037 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.037 DEBUG:boto:Method: GET 2024-04-13T22:15:11.038 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.038 DEBUG:boto:Data: 2024-04-13T22:15:11.038 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.038 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:11.038 DEBUG:boto:Port: 8000 2024-04-13T22:15:11.038 DEBUG:boto:Params: {} 2024-04-13T22:15:11.038 DEBUG:boto:Token: None 2024-04-13T22:15:11.038 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:11.038 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ydYoHeImxiN96EenUFROkvLwm1w= 2024-04-13T22:15:11.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ydYoHeImxiN96EenUFROkvLwm1w=', 'Content-Length': '0'} 2024-04-13T22:15:11.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:14:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002058472205eb9a4d-00661b03ef-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.040 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:11.041 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:11.041 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.041 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.041 DEBUG:boto:Method: GET 2024-04-13T22:15:11.041 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.041 DEBUG:boto:Data: 2024-04-13T22:15:11.041 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.041 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:11.041 DEBUG:boto:Port: 8001 2024-04-13T22:15:11.041 DEBUG:boto:Params: {} 2024-04-13T22:15:11.041 DEBUG:boto:Token: None 2024-04-13T22:15:11.041 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/ 2024-04-13T22:15:11.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pDKW5aArVeN20PJJknWVfQW9A2Y= 2024-04-13T22:15:11.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pDKW5aArVeN20PJJknWVfQW9A2Y=', 'Content-Length': '0'} 2024-04-13T22:15:11.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004177e370c60b99c4-00661b03ef-1eaf-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.044 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:11.044 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:11.045 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:11.045 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.045 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.045 DEBUG:boto:Method: GET 2024-04-13T22:15:11.045 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:11.045 DEBUG:boto:Data: 2024-04-13T22:15:11.045 DEBUG:boto:Headers: {} 2024-04-13T22:15:11.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:11.045 DEBUG:boto:Port: 8000 2024-04-13T22:15:11.045 DEBUG:boto:Params: {} 2024-04-13T22:15:11.045 DEBUG:boto:Token: None 2024-04-13T22:15:11.046 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:11 GMT /vwnfns-60/ 2024-04-13T22:15:11.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pDKW5aArVeN20PJJknWVfQW9A2Y= 2024-04-13T22:15:11.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pDKW5aArVeN20PJJknWVfQW9A2Y=', 'Content-Length': '0'} 2024-04-13T22:15:11.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ba1eb26783f505-00661b03ef-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:11.048 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:14:54.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:11.048 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-60 zones={a2, a1} 2024-04-13T22:15:11.049 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:15:11.049 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:15:11.050 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:15:11.047+0000 7fa83315e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 97094) UID: 0 2024-04-13T22:15:11.050 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:15:11.047+0000 7fa8383337c0 -1 shutting down 2024-04-13T22:15:12.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:15:17.150 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:15:17.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:17.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:17.150 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:17.208 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:17.208 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:17.254 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:15:17.255 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-60 2024-04-13T22:15:17.259 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 10 2024-04-13T22:15:17.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:17.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:17.297 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:17.346 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:17.347 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:17.387 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:17.387 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:15:17.387 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000007", 2024-04-13T22:15:17.387 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000016", 2024-04-13T22:15:17.387 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000024", 2024-04-13T22:15:17.387 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000020", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000028", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000032", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000010", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000019", 2024-04-13T22:15:17.388 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000013", 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000021", 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000030", 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000026", 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000033", 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000004", 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000076" 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:17.389 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:17.390 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.390 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.390 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.390 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.391 DEBUG:boto:Data: 2024-04-13T22:15:17.391 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.391 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.391 DEBUG:boto:Params: {} 2024-04-13T22:15:17.391 DEBUG:boto:Token: None 2024-04-13T22:15:17.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0de215eab4bb303-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.395 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:17.396 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:17.396 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.396 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:17.396 DEBUG:boto:Data: 2024-04-13T22:15:17.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.396 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.396 DEBUG:boto:Params: {} 2024-04-13T22:15:17.396 DEBUG:boto:Token: None 2024-04-13T22:15:17.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:17.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aeU9rNzzbYkZbe8bXWLxIV2QerA= 2024-04-13T22:15:17.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aeU9rNzzbYkZbe8bXWLxIV2QerA='} 2024-04-13T22:15:17.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000626a556fdc766deb-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.439 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.439 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.439 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.439 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.440 DEBUG:boto:Data: 2024-04-13T22:15:17.440 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.440 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.440 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.440 DEBUG:boto:Params: {} 2024-04-13T22:15:17.441 DEBUG:boto:Token: None 2024-04-13T22:15:17.441 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a822eaf2f854fce5-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.444 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:17.445 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:17.445 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.445 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:17.445 DEBUG:boto:Data: 2024-04-13T22:15:17.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.446 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.446 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.446 DEBUG:boto:Params: {} 2024-04-13T22:15:17.446 DEBUG:boto:Token: None 2024-04-13T22:15:17.447 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:17.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mHKfM+gEZ8pFV+79GWc50L7tVHY= 2024-04-13T22:15:17.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mHKfM+gEZ8pFV+79GWc50L7tVHY='} 2024-04-13T22:15:17.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021d32c46d6b385de-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.453 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.453 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.454 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.454 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.454 DEBUG:boto:Data: 2024-04-13T22:15:17.454 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.454 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.455 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.455 DEBUG:boto:Params: {} 2024-04-13T22:15:17.455 DEBUG:boto:Token: None 2024-04-13T22:15:17.455 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004265a77f3619b269-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.459 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:17.459 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:17.459 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.460 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:17.460 DEBUG:boto:Data: 2024-04-13T22:15:17.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.460 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.461 DEBUG:boto:Params: {} 2024-04-13T22:15:17.461 DEBUG:boto:Token: None 2024-04-13T22:15:17.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:17.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mPuL1gBJmWihYDkkgjr8HKufhH8= 2024-04-13T22:15:17.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mPuL1gBJmWihYDkkgjr8HKufhH8='} 2024-04-13T22:15:17.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000120202f63c1952e0-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.507 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.507 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.507 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.507 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.507 DEBUG:boto:Data: 2024-04-13T22:15:17.507 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.508 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.508 DEBUG:boto:Params: {} 2024-04-13T22:15:17.508 DEBUG:boto:Token: None 2024-04-13T22:15:17.508 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b06209eb2778acd-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.511 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:17.511 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:17.511 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.511 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:17.512 DEBUG:boto:Data: 2024-04-13T22:15:17.512 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.512 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.512 DEBUG:boto:Params: {} 2024-04-13T22:15:17.512 DEBUG:boto:Token: None 2024-04-13T22:15:17.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:17.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+DGT9YeVI3R0wWlYgem3fnfQgnI= 2024-04-13T22:15:17.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+DGT9YeVI3R0wWlYgem3fnfQgnI='} 2024-04-13T22:15:17.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097769c3a7a3d7f50-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.519 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.519 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.519 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.519 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.519 DEBUG:boto:Data: 2024-04-13T22:15:17.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.519 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.519 DEBUG:boto:Params: {} 2024-04-13T22:15:17.519 DEBUG:boto:Token: None 2024-04-13T22:15:17.519 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097187bf6b485a0ed-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.522 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:17.522 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:17.522 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.522 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:17.523 DEBUG:boto:Data: 2024-04-13T22:15:17.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.523 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.523 DEBUG:boto:Params: {} 2024-04-13T22:15:17.523 DEBUG:boto:Token: None 2024-04-13T22:15:17.523 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:17.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kMcL2CAmQmQiacpI8x3seXZ4OyM= 2024-04-13T22:15:17.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kMcL2CAmQmQiacpI8x3seXZ4OyM='} 2024-04-13T22:15:17.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000127ffa80d62ec0ea-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.571 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.571 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.571 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.571 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.571 DEBUG:boto:Data: 2024-04-13T22:15:17.571 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.571 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.571 DEBUG:boto:Params: {} 2024-04-13T22:15:17.571 DEBUG:boto:Token: None 2024-04-13T22:15:17.572 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c722807837dc445-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.576 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:17.576 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:17.576 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.577 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:17.577 DEBUG:boto:Data: 2024-04-13T22:15:17.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.577 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.577 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.577 DEBUG:boto:Params: {} 2024-04-13T22:15:17.577 DEBUG:boto:Token: None 2024-04-13T22:15:17.577 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:17.578 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:12Dz5buQPsWHfJfekwqEJJab2aY= 2024-04-13T22:15:17.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:12Dz5buQPsWHfJfekwqEJJab2aY='} 2024-04-13T22:15:17.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2e90527d5026ecd-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.583 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.583 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.583 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.583 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.583 DEBUG:boto:Data: 2024-04-13T22:15:17.583 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.583 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.583 DEBUG:boto:Params: {} 2024-04-13T22:15:17.583 DEBUG:boto:Token: None 2024-04-13T22:15:17.584 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007181669d86ebfeb2-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.586 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:17.586 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:17.586 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.587 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:17.587 DEBUG:boto:Data: 2024-04-13T22:15:17.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.587 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.587 DEBUG:boto:Params: {} 2024-04-13T22:15:17.587 DEBUG:boto:Token: None 2024-04-13T22:15:17.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:17.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0xnGrjSl61YiKK0PpKF5lukBaao= 2024-04-13T22:15:17.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0xnGrjSl61YiKK0PpKF5lukBaao='} 2024-04-13T22:15:17.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072f6bbef7d7f9fe5-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.631 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.631 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.631 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.631 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.631 DEBUG:boto:Data: 2024-04-13T22:15:17.631 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.631 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.631 DEBUG:boto:Params: {} 2024-04-13T22:15:17.631 DEBUG:boto:Token: None 2024-04-13T22:15:17.631 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047cace3b08291e4a-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.634 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:17.634 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:17.634 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.634 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:17.635 DEBUG:boto:Data: 2024-04-13T22:15:17.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.635 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.635 DEBUG:boto:Params: {} 2024-04-13T22:15:17.635 DEBUG:boto:Token: None 2024-04-13T22:15:17.635 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:17.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qR8v3NKV9VS15a57U0pAITNsqEc= 2024-04-13T22:15:17.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qR8v3NKV9VS15a57U0pAITNsqEc='} 2024-04-13T22:15:17.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006150d9ed80e53337-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.640 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.641 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.641 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.641 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.641 DEBUG:boto:Data: 2024-04-13T22:15:17.641 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.641 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.641 DEBUG:boto:Params: {} 2024-04-13T22:15:17.641 DEBUG:boto:Token: None 2024-04-13T22:15:17.641 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.641 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc5304788b4c5667-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.644 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:17.644 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:17.644 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.644 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:17.644 DEBUG:boto:Data: 2024-04-13T22:15:17.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.645 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.645 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.645 DEBUG:boto:Params: {} 2024-04-13T22:15:17.645 DEBUG:boto:Token: None 2024-04-13T22:15:17.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:17.645 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2S0Q8sFc/KnaXaJNmCag5fF6juc= 2024-04-13T22:15:17.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2S0Q8sFc/KnaXaJNmCag5fF6juc='} 2024-04-13T22:15:17.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c37a40c91080ee19-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.695 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.695 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.695 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.695 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.695 DEBUG:boto:Data: 2024-04-13T22:15:17.695 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.695 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.695 DEBUG:boto:Params: {} 2024-04-13T22:15:17.695 DEBUG:boto:Token: None 2024-04-13T22:15:17.695 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c28f64859de6b725-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.698 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:17.698 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:17.698 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.699 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:17.699 DEBUG:boto:Data: 2024-04-13T22:15:17.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.699 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.699 DEBUG:boto:Params: {} 2024-04-13T22:15:17.699 DEBUG:boto:Token: None 2024-04-13T22:15:17.699 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:17.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dE9103kFg+MdPyjR4INL/gWx37M= 2024-04-13T22:15:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dE9103kFg+MdPyjR4INL/gWx37M='} 2024-04-13T22:15:17.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c47437c4c225064c-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.706 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.706 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.706 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.706 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.706 DEBUG:boto:Data: 2024-04-13T22:15:17.706 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.706 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.706 DEBUG:boto:Params: {} 2024-04-13T22:15:17.707 DEBUG:boto:Token: None 2024-04-13T22:15:17.707 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.707 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bc2d3a1189de3d8-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.709 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:17.710 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:17.710 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.710 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:17.710 DEBUG:boto:Data: 2024-04-13T22:15:17.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.710 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.710 DEBUG:boto:Params: {} 2024-04-13T22:15:17.710 DEBUG:boto:Token: None 2024-04-13T22:15:17.711 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:17.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QM0/7Vcv6GFpAoF04qLO56So9vY= 2024-04-13T22:15:17.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QM0/7Vcv6GFpAoF04qLO56So9vY='} 2024-04-13T22:15:17.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e19b9ee3d96e618-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.717 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.717 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.717 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.717 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.717 DEBUG:boto:Data: 2024-04-13T22:15:17.717 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.717 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.717 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.717 DEBUG:boto:Params: {} 2024-04-13T22:15:17.718 DEBUG:boto:Token: None 2024-04-13T22:15:17.718 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000451b095c4f9c1612-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.721 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:17.721 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:17.721 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.721 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:17.721 DEBUG:boto:Data: 2024-04-13T22:15:17.721 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.721 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.721 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.721 DEBUG:boto:Params: {} 2024-04-13T22:15:17.722 DEBUG:boto:Token: None 2024-04-13T22:15:17.722 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:17.722 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O0fDAs26DC1VNiizCOUmy1DT0a8= 2024-04-13T22:15:17.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O0fDAs26DC1VNiizCOUmy1DT0a8='} 2024-04-13T22:15:17.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a6770ae10920637-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.767 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.767 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.767 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.767 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.767 DEBUG:boto:Data: 2024-04-13T22:15:17.767 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.768 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.768 DEBUG:boto:Params: {} 2024-04-13T22:15:17.768 DEBUG:boto:Token: None 2024-04-13T22:15:17.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007491b049bf549334-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.771 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:17.771 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:17.771 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.771 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:17.771 DEBUG:boto:Data: 2024-04-13T22:15:17.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.771 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.771 DEBUG:boto:Params: {} 2024-04-13T22:15:17.772 DEBUG:boto:Token: None 2024-04-13T22:15:17.772 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:17.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hxo57JqiRmKJgLkkGnA3Ax6Yp/U= 2024-04-13T22:15:17.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hxo57JqiRmKJgLkkGnA3Ax6Yp/U='} 2024-04-13T22:15:17.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e589fa9cf52485f7-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.777 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.777 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.777 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.777 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.777 DEBUG:boto:Data: 2024-04-13T22:15:17.777 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.777 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.777 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.777 DEBUG:boto:Params: {} 2024-04-13T22:15:17.777 DEBUG:boto:Token: None 2024-04-13T22:15:17.778 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.778 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c462ce7d5457ac7-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.781 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:17.781 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:17.781 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.782 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:17.782 DEBUG:boto:Data: 2024-04-13T22:15:17.782 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.782 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.782 DEBUG:boto:Params: {} 2024-04-13T22:15:17.782 DEBUG:boto:Token: None 2024-04-13T22:15:17.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:17.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0s7AXR3msufLvWPWK+5kcbvEF64= 2024-04-13T22:15:17.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0s7AXR3msufLvWPWK+5kcbvEF64='} 2024-04-13T22:15:17.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb94ad034d86a1b4-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.827 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.827 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.827 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.827 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.827 DEBUG:boto:Data: 2024-04-13T22:15:17.828 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.828 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.828 DEBUG:boto:Params: {} 2024-04-13T22:15:17.828 DEBUG:boto:Token: None 2024-04-13T22:15:17.829 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.829 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038a9610c0d9feee2-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.831 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:17.831 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:17.832 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.832 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:17.832 DEBUG:boto:Data: 2024-04-13T22:15:17.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.832 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.832 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.832 DEBUG:boto:Params: {} 2024-04-13T22:15:17.832 DEBUG:boto:Token: None 2024-04-13T22:15:17.832 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:17.832 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nqsYq2Y9/VteEdnCGf96YwkCHvY= 2024-04-13T22:15:17.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nqsYq2Y9/VteEdnCGf96YwkCHvY='} 2024-04-13T22:15:17.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a8fa761302fb0c4-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.837 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:17.838 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:17.838 DEBUG:boto:Method: HEAD 2024-04-13T22:15:17.838 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:17.838 DEBUG:boto:Data: 2024-04-13T22:15:17.838 DEBUG:boto:Headers: {} 2024-04-13T22:15:17.838 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.838 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.838 DEBUG:boto:Params: {} 2024-04-13T22:15:17.838 DEBUG:boto:Token: None 2024-04-13T22:15:17.838 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/ 2024-04-13T22:15:17.839 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY= 2024-04-13T22:15:17.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pOmIS39AXOMIneIBPIS182Ft8YY=', 'Content-Length': '0'} 2024-04-13T22:15:17.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c146326eca1ec50-00661b03f5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.841 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:17.841 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:17.841 DEBUG:boto:Method: PUT 2024-04-13T22:15:17.841 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:17.841 DEBUG:boto:Data: 2024-04-13T22:15:17.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:17.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:17.842 DEBUG:boto:Port: 8000 2024-04-13T22:15:17.842 DEBUG:boto:Params: {} 2024-04-13T22:15:17.842 DEBUG:boto:Token: None 2024-04-13T22:15:17.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:17 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:17.842 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W4+E0o3kt6L27hdskGGnxVMN1XA= 2024-04-13T22:15:17.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:17 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W4+E0o3kt6L27hdskGGnxVMN1XA='} 2024-04-13T22:15:17.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1f6f30d2fc5a2ac-00661b03f5-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:17.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:17.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:17.887 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:17.938 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:17.938 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:17.981 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:15:17.982 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-60 2024-04-13T22:15:17.988 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 22 2024-04-13T22:15:18.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:18.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:18.034 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:18.084 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:18.084 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:18.124 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:18.124 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:15:18.124 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000005", 2024-04-13T22:15:18.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000010", 2024-04-13T22:15:18.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000016", 2024-04-13T22:15:18.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000021", 2024-04-13T22:15:18.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000004", 2024-04-13T22:15:18.125 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000025", 2024-04-13T22:15:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000030", 2024-04-13T22:15:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000015", 2024-04-13T22:15:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000019", 2024-04-13T22:15:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000003", 2024-04-13T22:15:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000009", 2024-04-13T22:15:18.127 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000014", 2024-04-13T22:15:18.127 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000012", 2024-04-13T22:15:18.127 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000002", 2024-04-13T22:15:18.127 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:15:18.127 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000006", 2024-04-13T22:15:18.128 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000011", 2024-04-13T22:15:18.128 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:15:18.128 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000007" 2024-04-13T22:15:18.128 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:18.128 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:18.130 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.130 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.131 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.131 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.131 DEBUG:boto:Data: 2024-04-13T22:15:18.131 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.132 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.132 DEBUG:boto:Params: {} 2024-04-13T22:15:18.132 DEBUG:boto:Token: None 2024-04-13T22:15:18.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001862292fc773d289-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.136 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:18.136 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:18.136 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.136 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:18.136 DEBUG:boto:Data: 2024-04-13T22:15:18.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.136 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.136 DEBUG:boto:Params: {} 2024-04-13T22:15:18.137 DEBUG:boto:Token: None 2024-04-13T22:15:18.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:18.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0pRlWGDCeJgoSfu0Tvr03p8ZhLo= 2024-04-13T22:15:18.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0pRlWGDCeJgoSfu0Tvr03p8ZhLo='} 2024-04-13T22:15:18.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000361b68a92a7bb84b-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.143 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.143 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.143 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.143 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.143 DEBUG:boto:Data: 2024-04-13T22:15:18.143 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.144 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.144 DEBUG:boto:Params: {} 2024-04-13T22:15:18.144 DEBUG:boto:Token: None 2024-04-13T22:15:18.144 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f664c7584b27b7f4-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.147 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:18.147 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:18.147 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.147 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:18.147 DEBUG:boto:Data: 2024-04-13T22:15:18.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.147 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.147 DEBUG:boto:Params: {} 2024-04-13T22:15:18.148 DEBUG:boto:Token: None 2024-04-13T22:15:18.148 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:18.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s03gybK1B1zq0ss/jjgirDx/89Q= 2024-04-13T22:15:18.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s03gybK1B1zq0ss/jjgirDx/89Q='} 2024-04-13T22:15:18.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cf2b92f32ed45fc-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.190 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.191 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.191 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.191 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.191 DEBUG:boto:Data: 2024-04-13T22:15:18.191 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.191 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.191 DEBUG:boto:Params: {} 2024-04-13T22:15:18.191 DEBUG:boto:Token: None 2024-04-13T22:15:18.191 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047f7eb15aaf74521-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.194 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:18.194 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:18.194 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.194 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:18.194 DEBUG:boto:Data: 2024-04-13T22:15:18.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.195 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.195 DEBUG:boto:Params: {} 2024-04-13T22:15:18.195 DEBUG:boto:Token: None 2024-04-13T22:15:18.195 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:18.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aaxbh4cK2di+Ma1sdnx+ANerbRE= 2024-04-13T22:15:18.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aaxbh4cK2di+Ma1sdnx+ANerbRE='} 2024-04-13T22:15:18.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:15:18.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dc6bfab0f89f355-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.200 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.200 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.200 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.200 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.201 DEBUG:boto:Data: 2024-04-13T22:15:18.201 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.201 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.201 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.201 DEBUG:boto:Params: {} 2024-04-13T22:15:18.201 DEBUG:boto:Token: None 2024-04-13T22:15:18.201 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.201 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010554c48deb1ba81-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.204 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:18.204 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:18.204 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.204 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:18.204 DEBUG:boto:Data: 2024-04-13T22:15:18.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.205 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.205 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.205 DEBUG:boto:Params: {} 2024-04-13T22:15:18.205 DEBUG:boto:Token: None 2024-04-13T22:15:18.205 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:18.205 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V5tZ44Z/3orL7iD0CoG6LmiMA/o= 2024-04-13T22:15:18.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V5tZ44Z/3orL7iD0CoG6LmiMA/o='} 2024-04-13T22:15:18.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dfa9c09e84ca67a-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.247 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.247 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.247 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.247 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.247 DEBUG:boto:Data: 2024-04-13T22:15:18.247 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.247 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.247 DEBUG:boto:Params: {} 2024-04-13T22:15:18.247 DEBUG:boto:Token: None 2024-04-13T22:15:18.248 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbfd262221b41f9d-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.250 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:18.250 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:18.251 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.251 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:18.251 DEBUG:boto:Data: 2024-04-13T22:15:18.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.251 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.251 DEBUG:boto:Params: {} 2024-04-13T22:15:18.251 DEBUG:boto:Token: None 2024-04-13T22:15:18.251 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:18.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p8bGdm3WXYzh6HUN2+DVWusiPd0= 2024-04-13T22:15:18.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p8bGdm3WXYzh6HUN2+DVWusiPd0='} 2024-04-13T22:15:18.257 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9e9ccd26168bed4-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.258 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.258 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.258 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.258 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.258 DEBUG:boto:Data: 2024-04-13T22:15:18.258 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.258 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.258 DEBUG:boto:Params: {} 2024-04-13T22:15:18.258 DEBUG:boto:Token: None 2024-04-13T22:15:18.259 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5c162e15f826f3f-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.261 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:18.261 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:18.261 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.261 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:18.261 DEBUG:boto:Data: 2024-04-13T22:15:18.262 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.262 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.262 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.262 DEBUG:boto:Params: {} 2024-04-13T22:15:18.262 DEBUG:boto:Token: None 2024-04-13T22:15:18.262 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:18.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w4U+Q/QbcLNxAhVS26Zz9iMa2b4= 2024-04-13T22:15:18.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w4U+Q/QbcLNxAhVS26Zz9iMa2b4='} 2024-04-13T22:15:18.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f337ea13eaf5b165-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.269 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.269 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.269 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.269 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.269 DEBUG:boto:Data: 2024-04-13T22:15:18.269 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.269 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.269 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.269 DEBUG:boto:Params: {} 2024-04-13T22:15:18.269 DEBUG:boto:Token: None 2024-04-13T22:15:18.270 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.270 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5396756227b9786-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.272 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:18.272 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:18.272 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.272 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:18.273 DEBUG:boto:Data: 2024-04-13T22:15:18.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.273 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.273 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.273 DEBUG:boto:Params: {} 2024-04-13T22:15:18.273 DEBUG:boto:Token: None 2024-04-13T22:15:18.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:18.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HZYyO8rVXRvemex3Cyz12oK55Eg= 2024-04-13T22:15:18.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HZYyO8rVXRvemex3Cyz12oK55Eg='} 2024-04-13T22:15:18.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa7899207d3aa0f1-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.318 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.319 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.319 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.319 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.319 DEBUG:boto:Data: 2024-04-13T22:15:18.319 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.319 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.319 DEBUG:boto:Params: {} 2024-04-13T22:15:18.319 DEBUG:boto:Token: None 2024-04-13T22:15:18.319 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fd100b005c59b57-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.322 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.322 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.322 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.323 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.323 DEBUG:boto:Data: 2024-04-13T22:15:18.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.323 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.323 DEBUG:boto:Params: {} 2024-04-13T22:15:18.323 DEBUG:boto:Token: None 2024-04-13T22:15:18.323 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eou8L8aoe7olzIAQUzwrQfLVTS0= 2024-04-13T22:15:18.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eou8L8aoe7olzIAQUzwrQfLVTS0='} 2024-04-13T22:15:18.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7a0991562c59df8-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.330 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.330 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.330 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.330 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.330 DEBUG:boto:Data: 2024-04-13T22:15:18.330 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.330 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.330 DEBUG:boto:Params: {} 2024-04-13T22:15:18.330 DEBUG:boto:Token: None 2024-04-13T22:15:18.331 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.331 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8254db8f1918043-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.334 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:18.334 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:18.334 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.334 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:18.334 DEBUG:boto:Data: 2024-04-13T22:15:18.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.334 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.334 DEBUG:boto:Params: {} 2024-04-13T22:15:18.335 DEBUG:boto:Token: None 2024-04-13T22:15:18.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:18.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LinY1l0uRZFhFw5oxRqefohbaH0= 2024-04-13T22:15:18.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LinY1l0uRZFhFw5oxRqefohbaH0='} 2024-04-13T22:15:18.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa274fcd6d370ca1-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.341 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.341 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.341 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.341 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.341 DEBUG:boto:Data: 2024-04-13T22:15:18.341 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.342 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.342 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.342 DEBUG:boto:Params: {} 2024-04-13T22:15:18.342 DEBUG:boto:Token: None 2024-04-13T22:15:18.342 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000b662910928f34b-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.345 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:18.345 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:18.345 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.345 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:18.346 DEBUG:boto:Data: 2024-04-13T22:15:18.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.346 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.346 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.346 DEBUG:boto:Params: {} 2024-04-13T22:15:18.346 DEBUG:boto:Token: None 2024-04-13T22:15:18.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:18.346 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tLCadHdv80m/MwnNfopVIB1sNzM= 2024-04-13T22:15:18.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tLCadHdv80m/MwnNfopVIB1sNzM='} 2024-04-13T22:15:18.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071e23b24afaa831b-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.391 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.391 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.391 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.391 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.391 DEBUG:boto:Data: 2024-04-13T22:15:18.391 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.392 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.392 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.392 DEBUG:boto:Params: {} 2024-04-13T22:15:18.392 DEBUG:boto:Token: None 2024-04-13T22:15:18.392 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb3fcc97c626bfca-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.395 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:18.395 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:18.395 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.395 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:18.395 DEBUG:boto:Data: 2024-04-13T22:15:18.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.396 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.396 DEBUG:boto:Params: {} 2024-04-13T22:15:18.396 DEBUG:boto:Token: None 2024-04-13T22:15:18.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:18.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tIoC+kl8ihN6l2VQ4U/rtoDFw/4= 2024-04-13T22:15:18.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tIoC+kl8ihN6l2VQ4U/rtoDFw/4='} 2024-04-13T22:15:18.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052bffe872ac0037f-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.402 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.402 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.402 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.402 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.402 DEBUG:boto:Data: 2024-04-13T22:15:18.402 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.402 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.402 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.403 DEBUG:boto:Params: {} 2024-04-13T22:15:18.403 DEBUG:boto:Token: None 2024-04-13T22:15:18.403 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.403 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c56cc5e5bad3699-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.405 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:18.406 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:18.406 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.406 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:18.406 DEBUG:boto:Data: 2024-04-13T22:15:18.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.406 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.406 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.406 DEBUG:boto:Params: {} 2024-04-13T22:15:18.406 DEBUG:boto:Token: None 2024-04-13T22:15:18.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:18.407 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ztC5K8axSC79C5JvsQy/U46IQQA= 2024-04-13T22:15:18.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ztC5K8axSC79C5JvsQy/U46IQQA='} 2024-04-13T22:15:18.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4439380d006be72-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.451 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.451 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.451 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.451 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.451 DEBUG:boto:Data: 2024-04-13T22:15:18.451 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.452 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.452 DEBUG:boto:Params: {} 2024-04-13T22:15:18.452 DEBUG:boto:Token: None 2024-04-13T22:15:18.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097717d48d9ecf8fc-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.455 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.455 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.455 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.455 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.456 DEBUG:boto:Data: 2024-04-13T22:15:18.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.456 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.456 DEBUG:boto:Params: {} 2024-04-13T22:15:18.457 DEBUG:boto:Token: None 2024-04-13T22:15:18.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.457 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yv0Lx1LI20hEqRgIWsy3l8UzrhQ= 2024-04-13T22:15:18.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yv0Lx1LI20hEqRgIWsy3l8UzrhQ='} 2024-04-13T22:15:18.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e10ccf1d1bb36e5c-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.467 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.467 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.467 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.467 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.467 DEBUG:boto:Data: 2024-04-13T22:15:18.468 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.468 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.468 DEBUG:boto:Params: {} 2024-04-13T22:15:18.468 DEBUG:boto:Token: None 2024-04-13T22:15:18.468 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b45d7dddfa7c044-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.471 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:18.471 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:18.471 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.471 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:18.472 DEBUG:boto:Data: 2024-04-13T22:15:18.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.472 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.472 DEBUG:boto:Params: {} 2024-04-13T22:15:18.472 DEBUG:boto:Token: None 2024-04-13T22:15:18.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:18.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8G7b3nyTTwPEfGmNWRwFB/Fny8U= 2024-04-13T22:15:18.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8G7b3nyTTwPEfGmNWRwFB/Fny8U='} 2024-04-13T22:15:18.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042a2dcab9a00d705-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.515 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.515 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.515 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.515 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.515 DEBUG:boto:Data: 2024-04-13T22:15:18.515 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.515 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.515 DEBUG:boto:Params: {} 2024-04-13T22:15:18.515 DEBUG:boto:Token: None 2024-04-13T22:15:18.516 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006272f9bd4b0994db-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.519 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:18.519 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:18.519 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.519 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:18.519 DEBUG:boto:Data: 2024-04-13T22:15:18.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.519 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.519 DEBUG:boto:Params: {} 2024-04-13T22:15:18.519 DEBUG:boto:Token: None 2024-04-13T22:15:18.519 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:18.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TrQs7M7hyQ42h7/ZJSqXwfsKj+s= 2024-04-13T22:15:18.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TrQs7M7hyQ42h7/ZJSqXwfsKj+s='} 2024-04-13T22:15:18.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d668c30c961b3387-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.526 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.526 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.527 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.527 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.527 DEBUG:boto:Data: 2024-04-13T22:15:18.527 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.527 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.527 DEBUG:boto:Params: {} 2024-04-13T22:15:18.527 DEBUG:boto:Token: None 2024-04-13T22:15:18.527 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c7237a3cb7d70a5-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.530 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:18.530 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:18.530 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.530 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:18.530 DEBUG:boto:Data: 2024-04-13T22:15:18.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.531 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.531 DEBUG:boto:Params: {} 2024-04-13T22:15:18.531 DEBUG:boto:Token: None 2024-04-13T22:15:18.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:18.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M7jmsm7vTMKbYlZL9DwLplxsyWE= 2024-04-13T22:15:18.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M7jmsm7vTMKbYlZL9DwLplxsyWE='} 2024-04-13T22:15:18.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf21d64e3075417f-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.574 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.575 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.575 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.575 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.575 DEBUG:boto:Data: 2024-04-13T22:15:18.575 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.575 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.575 DEBUG:boto:Params: {} 2024-04-13T22:15:18.575 DEBUG:boto:Token: None 2024-04-13T22:15:18.575 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0937cfc439abcec-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.578 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:18.578 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:18.578 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.578 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:18.578 DEBUG:boto:Data: 2024-04-13T22:15:18.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.579 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.579 DEBUG:boto:Params: {} 2024-04-13T22:15:18.579 DEBUG:boto:Token: None 2024-04-13T22:15:18.579 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:18.579 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C+t9znf7a55VW0kONBvppBPxf98= 2024-04-13T22:15:18.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C+t9znf7a55VW0kONBvppBPxf98='} 2024-04-13T22:15:18.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055d7b8f3b401b12e-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.585 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.585 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.585 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.585 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.585 DEBUG:boto:Data: 2024-04-13T22:15:18.585 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.586 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.586 DEBUG:boto:Params: {} 2024-04-13T22:15:18.586 DEBUG:boto:Token: None 2024-04-13T22:15:18.586 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.586 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d908d256c032c142-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.589 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:18.589 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:18.589 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.589 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:18.589 DEBUG:boto:Data: 2024-04-13T22:15:18.589 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.589 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.589 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.590 DEBUG:boto:Params: {} 2024-04-13T22:15:18.590 DEBUG:boto:Token: None 2024-04-13T22:15:18.590 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:18.590 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6/Pbhf4DDqx9BQAMtRUok5YAqAI= 2024-04-13T22:15:18.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6/Pbhf4DDqx9BQAMtRUok5YAqAI='} 2024-04-13T22:15:18.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b6964f3fb20ba5b-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:18.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:18.595 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:18.647 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:18.647 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:18.692 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:15:18.692 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-60 2024-04-13T22:15:18.699 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 26 2024-04-13T22:15:18.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:18.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:18.746 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:18.795 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:18.795 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:18.835 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000058", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000026", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000011", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000046", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000014", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000050", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000032", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000002", 2024-04-13T22:15:18.836 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000039", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000007", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000006", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000040", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000009", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000045", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000013", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000012", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000033", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:15:18.837 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000021", 2024-04-13T22:15:18.838 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000003", 2024-04-13T22:15:18.838 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000055", 2024-04-13T22:15:18.838 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000008" 2024-04-13T22:15:18.838 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:18.838 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:18.839 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.839 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.839 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.839 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.839 DEBUG:boto:Data: 2024-04-13T22:15:18.839 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.840 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.840 DEBUG:boto:Params: {} 2024-04-13T22:15:18.840 DEBUG:boto:Token: None 2024-04-13T22:15:18.840 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007727b7ca44898280-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.843 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.844 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.844 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.844 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.844 DEBUG:boto:Data: 2024-04-13T22:15:18.844 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.844 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.844 DEBUG:boto:Params: {} 2024-04-13T22:15:18.844 DEBUG:boto:Token: None 2024-04-13T22:15:18.844 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:18.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eou8L8aoe7olzIAQUzwrQfLVTS0= 2024-04-13T22:15:18.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eou8L8aoe7olzIAQUzwrQfLVTS0='} 2024-04-13T22:15:18.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3470ae32d1f1a4a-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.887 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.887 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.887 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.887 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.887 DEBUG:boto:Data: 2024-04-13T22:15:18.887 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.887 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.887 DEBUG:boto:Params: {} 2024-04-13T22:15:18.888 DEBUG:boto:Token: None 2024-04-13T22:15:18.888 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000768b12076e6ece81-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.891 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:18.891 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:18.891 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.891 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:18.891 DEBUG:boto:Data: 2024-04-13T22:15:18.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.892 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.892 DEBUG:boto:Params: {} 2024-04-13T22:15:18.892 DEBUG:boto:Token: None 2024-04-13T22:15:18.892 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:18.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xseub/PamLRPkb7oA6YhvkKGQhY= 2024-04-13T22:15:18.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xseub/PamLRPkb7oA6YhvkKGQhY='} 2024-04-13T22:15:18.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047b7211e70e37e79-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.898 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.898 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.898 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.898 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.899 DEBUG:boto:Data: 2024-04-13T22:15:18.899 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.899 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.899 DEBUG:boto:Params: {} 2024-04-13T22:15:18.899 DEBUG:boto:Token: None 2024-04-13T22:15:18.899 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ba2cded4c727a56-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.902 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.902 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.902 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.902 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.902 DEBUG:boto:Data: 2024-04-13T22:15:18.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.903 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.903 DEBUG:boto:Params: {} 2024-04-13T22:15:18.903 DEBUG:boto:Token: None 2024-04-13T22:15:18.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:18.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yv0Lx1LI20hEqRgIWsy3l8UzrhQ= 2024-04-13T22:15:18.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yv0Lx1LI20hEqRgIWsy3l8UzrhQ='} 2024-04-13T22:15:18.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b28a0d7440cd4580-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.946 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.947 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.947 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.947 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.947 DEBUG:boto:Data: 2024-04-13T22:15:18.947 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.947 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.947 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.947 DEBUG:boto:Params: {} 2024-04-13T22:15:18.947 DEBUG:boto:Token: None 2024-04-13T22:15:18.947 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076aa4e73736189d9-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.950 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:18.950 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:18.950 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.950 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:18.950 DEBUG:boto:Data: 2024-04-13T22:15:18.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.951 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.951 DEBUG:boto:Params: {} 2024-04-13T22:15:18.951 DEBUG:boto:Token: None 2024-04-13T22:15:18.951 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:18.951 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cOoA8Pot/BElLyANn8VKBYQpTN0= 2024-04-13T22:15:18.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cOoA8Pot/BElLyANn8VKBYQpTN0='} 2024-04-13T22:15:18.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076aaeac4c38c6de9-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.956 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:18.957 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:18.957 DEBUG:boto:Method: HEAD 2024-04-13T22:15:18.957 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:18.957 DEBUG:boto:Data: 2024-04-13T22:15:18.957 DEBUG:boto:Headers: {} 2024-04-13T22:15:18.957 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.957 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.957 DEBUG:boto:Params: {} 2024-04-13T22:15:18.957 DEBUG:boto:Token: None 2024-04-13T22:15:18.957 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/ 2024-04-13T22:15:18.957 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc= 2024-04-13T22:15:18.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N6xnegAfl5hU7a5kQTYUASFABnc=', 'Content-Length': '0'} 2024-04-13T22:15:18.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbadeb115f9348f8-00661b03f6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:18.960 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:18.960 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:18.960 DEBUG:boto:Method: PUT 2024-04-13T22:15:18.961 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:18.961 DEBUG:boto:Data: 2024-04-13T22:15:18.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:18.961 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:18.961 DEBUG:boto:Port: 8000 2024-04-13T22:15:18.961 DEBUG:boto:Params: {} 2024-04-13T22:15:18.961 DEBUG:boto:Token: None 2024-04-13T22:15:18.961 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:18 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:18.961 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:staCl4TrFOyEUtuCwWWbq03bpcc= 2024-04-13T22:15:18.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:staCl4TrFOyEUtuCwWWbq03bpcc='} 2024-04-13T22:15:19.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c38d5feadef6125b-00661b03f6-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.007 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.007 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.007 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.007 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.007 DEBUG:boto:Data: 2024-04-13T22:15:19.007 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.007 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.007 DEBUG:boto:Params: {} 2024-04-13T22:15:19.008 DEBUG:boto:Token: None 2024-04-13T22:15:19.008 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d67133686a0ac228-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.011 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:19.011 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:19.011 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.011 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:19.011 DEBUG:boto:Data: 2024-04-13T22:15:19.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.011 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.012 DEBUG:boto:Params: {} 2024-04-13T22:15:19.012 DEBUG:boto:Token: None 2024-04-13T22:15:19.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:19.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w0c/m713k8e7iA6JbWw6T+y66aw= 2024-04-13T22:15:19.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w0c/m713k8e7iA6JbWw6T+y66aw='} 2024-04-13T22:15:19.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000745b6669e7d5983c-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.017 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.017 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.017 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.017 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.017 DEBUG:boto:Data: 2024-04-13T22:15:19.017 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.018 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.018 DEBUG:boto:Params: {} 2024-04-13T22:15:19.018 DEBUG:boto:Token: None 2024-04-13T22:15:19.018 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.018 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b93fdacd396f331c-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.021 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:19.021 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:19.021 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.021 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:19.021 DEBUG:boto:Data: 2024-04-13T22:15:19.021 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.021 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.021 DEBUG:boto:Params: {} 2024-04-13T22:15:19.022 DEBUG:boto:Token: None 2024-04-13T22:15:19.022 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:19.022 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XH4yzrcmj4qdawOyx0+kfftlQag= 2024-04-13T22:15:19.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XH4yzrcmj4qdawOyx0+kfftlQag='} 2024-04-13T22:15:19.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9a9787943a57931-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.067 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.067 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.067 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.067 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.067 DEBUG:boto:Data: 2024-04-13T22:15:19.067 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.067 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.067 DEBUG:boto:Params: {} 2024-04-13T22:15:19.067 DEBUG:boto:Token: None 2024-04-13T22:15:19.068 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.068 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.070 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a1c785abdd2cebe-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.071 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:19.071 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:19.071 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.071 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:19.071 DEBUG:boto:Data: 2024-04-13T22:15:19.071 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.071 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.071 DEBUG:boto:Params: {} 2024-04-13T22:15:19.071 DEBUG:boto:Token: None 2024-04-13T22:15:19.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:19.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zdyzrM9tMGW+DHUYbnYnrBfOrKY= 2024-04-13T22:15:19.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zdyzrM9tMGW+DHUYbnYnrBfOrKY='} 2024-04-13T22:15:19.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c0740ca70421116-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.077 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.078 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.078 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.078 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.078 DEBUG:boto:Data: 2024-04-13T22:15:19.078 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.078 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.078 DEBUG:boto:Params: {} 2024-04-13T22:15:19.078 DEBUG:boto:Token: None 2024-04-13T22:15:19.078 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adbe37b18f7ead36-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.081 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:19.081 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:19.082 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.082 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:19.082 DEBUG:boto:Data: 2024-04-13T22:15:19.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.082 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.082 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.082 DEBUG:boto:Params: {} 2024-04-13T22:15:19.082 DEBUG:boto:Token: None 2024-04-13T22:15:19.082 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:19.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FDMocnpFu2mkqNAZoIGA9f/W2NQ= 2024-04-13T22:15:19.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FDMocnpFu2mkqNAZoIGA9f/W2NQ='} 2024-04-13T22:15:19.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000592e56197d7551f1-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.126 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.127 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.127 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.127 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.127 DEBUG:boto:Data: 2024-04-13T22:15:19.127 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.127 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.127 DEBUG:boto:Params: {} 2024-04-13T22:15:19.127 DEBUG:boto:Token: None 2024-04-13T22:15:19.127 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.128 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000586b26e1e3e0449e-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.130 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.130 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.130 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.130 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.131 DEBUG:boto:Data: 2024-04-13T22:15:19.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.131 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.131 DEBUG:boto:Params: {} 2024-04-13T22:15:19.131 DEBUG:boto:Token: None 2024-04-13T22:15:19.131 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.131 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:edS1cFcWbHf/tQpIs25yq7h+ljY= 2024-04-13T22:15:19.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edS1cFcWbHf/tQpIs25yq7h+ljY='} 2024-04-13T22:15:19.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000101bd78ae75b103-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.137 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.138 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.138 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.138 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.138 DEBUG:boto:Data: 2024-04-13T22:15:19.138 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.138 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.138 DEBUG:boto:Params: {} 2024-04-13T22:15:19.138 DEBUG:boto:Token: None 2024-04-13T22:15:19.139 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd4737f5a83ce50b-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.141 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.141 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.142 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.142 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.142 DEBUG:boto:Data: 2024-04-13T22:15:19.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.142 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.142 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.142 DEBUG:boto:Params: {} 2024-04-13T22:15:19.142 DEBUG:boto:Token: None 2024-04-13T22:15:19.142 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LRHaW36uQ/XGj55flEvLMe1AGSQ= 2024-04-13T22:15:19.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LRHaW36uQ/XGj55flEvLMe1AGSQ='} 2024-04-13T22:15:19.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070dc7515efa5d65e-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.148 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.148 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.149 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.149 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.149 DEBUG:boto:Data: 2024-04-13T22:15:19.149 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.149 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.149 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.149 DEBUG:boto:Params: {} 2024-04-13T22:15:19.149 DEBUG:boto:Token: None 2024-04-13T22:15:19.149 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c29401c3fbb326bb-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.152 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:19.152 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:19.152 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.152 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:19.153 DEBUG:boto:Data: 2024-04-13T22:15:19.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.153 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.153 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.153 DEBUG:boto:Params: {} 2024-04-13T22:15:19.153 DEBUG:boto:Token: None 2024-04-13T22:15:19.153 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:19.153 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t8B59Z1XNsBub8ZcTxqOmj9jF8Q= 2024-04-13T22:15:19.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t8B59Z1XNsBub8ZcTxqOmj9jF8Q='} 2024-04-13T22:15:19.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044feda4f5b62d533-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.198 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.199 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.199 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.199 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.199 DEBUG:boto:Data: 2024-04-13T22:15:19.199 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.199 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.199 DEBUG:boto:Params: {} 2024-04-13T22:15:19.199 DEBUG:boto:Token: None 2024-04-13T22:15:19.199 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059765d4d6df70343-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.202 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.202 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.202 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.202 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.203 DEBUG:boto:Data: 2024-04-13T22:15:19.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.203 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.203 DEBUG:boto:Params: {} 2024-04-13T22:15:19.203 DEBUG:boto:Token: None 2024-04-13T22:15:19.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.203 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vy6KuQn3fiBjTbRzB0YJ++E4STQ= 2024-04-13T22:15:19.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vy6KuQn3fiBjTbRzB0YJ++E4STQ='} 2024-04-13T22:15:19.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006acf79750b21b09f-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.208 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.208 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.209 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.209 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.209 DEBUG:boto:Data: 2024-04-13T22:15:19.209 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.209 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.209 DEBUG:boto:Params: {} 2024-04-13T22:15:19.209 DEBUG:boto:Token: None 2024-04-13T22:15:19.209 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059481f9237027a06-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.212 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:19.212 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:19.212 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.212 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:19.212 DEBUG:boto:Data: 2024-04-13T22:15:19.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.212 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.213 DEBUG:boto:Params: {} 2024-04-13T22:15:19.213 DEBUG:boto:Token: None 2024-04-13T22:15:19.213 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:19.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wVwbnTaKnD2SnFLCSA+iqU1CDN4= 2024-04-13T22:15:19.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wVwbnTaKnD2SnFLCSA+iqU1CDN4='} 2024-04-13T22:15:19.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e1630abcd52d8b3-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.255 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.255 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.255 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.255 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.255 DEBUG:boto:Data: 2024-04-13T22:15:19.255 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.255 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.255 DEBUG:boto:Params: {} 2024-04-13T22:15:19.255 DEBUG:boto:Token: None 2024-04-13T22:15:19.255 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045f55abf465d4690-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.258 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:19.258 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:19.258 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.258 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:19.259 DEBUG:boto:Data: 2024-04-13T22:15:19.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.259 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.259 DEBUG:boto:Params: {} 2024-04-13T22:15:19.259 DEBUG:boto:Token: None 2024-04-13T22:15:19.259 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:19.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+IyeC7KMIMBsDVxwwwDrsX7uQQo= 2024-04-13T22:15:19.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+IyeC7KMIMBsDVxwwwDrsX7uQQo='} 2024-04-13T22:15:19.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006208e5b6e420770d-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.265 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.265 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.266 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.266 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.266 DEBUG:boto:Data: 2024-04-13T22:15:19.266 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.266 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.266 DEBUG:boto:Params: {} 2024-04-13T22:15:19.266 DEBUG:boto:Token: None 2024-04-13T22:15:19.266 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.266 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067c7e223c8813b74-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.269 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:19.269 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:19.269 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.270 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:19.270 DEBUG:boto:Data: 2024-04-13T22:15:19.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.270 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.270 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.270 DEBUG:boto:Params: {} 2024-04-13T22:15:19.270 DEBUG:boto:Token: None 2024-04-13T22:15:19.270 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:19.270 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U3lKTy0C0jnVbDVfGagrzM1qJNM= 2024-04-13T22:15:19.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U3lKTy0C0jnVbDVfGagrzM1qJNM='} 2024-04-13T22:15:19.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c5618755f68e9d5-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.275 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.275 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.275 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.276 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.276 DEBUG:boto:Data: 2024-04-13T22:15:19.276 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.276 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.276 DEBUG:boto:Params: {} 2024-04-13T22:15:19.276 DEBUG:boto:Token: None 2024-04-13T22:15:19.276 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044e1eb3004a8d920-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.279 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:19.279 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:19.279 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.279 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:19.279 DEBUG:boto:Data: 2024-04-13T22:15:19.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.279 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.279 DEBUG:boto:Params: {} 2024-04-13T22:15:19.280 DEBUG:boto:Token: None 2024-04-13T22:15:19.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:19.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sxt/QLXkh+rJzpDVRf7/1WntUKM= 2024-04-13T22:15:19.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sxt/QLXkh+rJzpDVRf7/1WntUKM='} 2024-04-13T22:15:19.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035e5c6cca92718d7-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.322 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.323 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.323 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.323 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.323 DEBUG:boto:Data: 2024-04-13T22:15:19.323 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.323 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.323 DEBUG:boto:Params: {} 2024-04-13T22:15:19.323 DEBUG:boto:Token: None 2024-04-13T22:15:19.323 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000806bec58d925ea59-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.326 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:19.326 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:19.327 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.327 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:19.327 DEBUG:boto:Data: 2024-04-13T22:15:19.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.327 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.327 DEBUG:boto:Params: {} 2024-04-13T22:15:19.327 DEBUG:boto:Token: None 2024-04-13T22:15:19.327 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:19.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H9PQcB/nfYbYGuaQuOXE6iNDZ3c= 2024-04-13T22:15:19.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H9PQcB/nfYbYGuaQuOXE6iNDZ3c='} 2024-04-13T22:15:19.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000feb6f91bc2612d80-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.333 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.333 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.333 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.334 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.334 DEBUG:boto:Data: 2024-04-13T22:15:19.334 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.334 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.334 DEBUG:boto:Params: {} 2024-04-13T22:15:19.334 DEBUG:boto:Token: None 2024-04-13T22:15:19.334 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bec6df84e9fbab0-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.337 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:19.337 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:19.337 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.337 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:19.337 DEBUG:boto:Data: 2024-04-13T22:15:19.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.337 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.338 DEBUG:boto:Params: {} 2024-04-13T22:15:19.338 DEBUG:boto:Token: None 2024-04-13T22:15:19.338 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:19.338 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TO0EPFbldp3NjCYUUm0wvF3xPjE= 2024-04-13T22:15:19.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TO0EPFbldp3NjCYUUm0wvF3xPjE='} 2024-04-13T22:15:19.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008163c53174c1636c-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.349 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.349 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.349 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.349 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.349 DEBUG:boto:Data: 2024-04-13T22:15:19.349 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.349 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.349 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.350 DEBUG:boto:Params: {} 2024-04-13T22:15:19.350 DEBUG:boto:Token: None 2024-04-13T22:15:19.350 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.350 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e04736f6c5b9bef0-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.353 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:19.353 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:19.353 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.353 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:19.353 DEBUG:boto:Data: 2024-04-13T22:15:19.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.353 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.353 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.353 DEBUG:boto:Params: {} 2024-04-13T22:15:19.354 DEBUG:boto:Token: None 2024-04-13T22:15:19.354 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:19.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Edk4KSAnGtD38yeJTQsLjJv0otw= 2024-04-13T22:15:19.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Edk4KSAnGtD38yeJTQsLjJv0otw='} 2024-04-13T22:15:19.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009ad3cfa72481b7c-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.403 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.403 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.403 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.403 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.403 DEBUG:boto:Data: 2024-04-13T22:15:19.403 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.403 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.403 DEBUG:boto:Params: {} 2024-04-13T22:15:19.403 DEBUG:boto:Token: None 2024-04-13T22:15:19.403 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bc92845b7c33d73-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.406 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:19.406 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:19.407 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.407 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:19.407 DEBUG:boto:Data: 2024-04-13T22:15:19.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.407 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.407 DEBUG:boto:Params: {} 2024-04-13T22:15:19.407 DEBUG:boto:Token: None 2024-04-13T22:15:19.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:19.407 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r+RFLizBXnu2opsTkQzqGeoFu7c= 2024-04-13T22:15:19.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r+RFLizBXnu2opsTkQzqGeoFu7c='} 2024-04-13T22:15:19.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d75b9a0f2053a2b9-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:19.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:19.414 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:19.467 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:19.467 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:19.520 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:15:19.521 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-60 2024-04-13T22:15:19.529 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 34 2024-04-13T22:15:19.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:19.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:19.581 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:19.630 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:19.630 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:19.670 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:19.670 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000006", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000059", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000027", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000004", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000011", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000002", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000019", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000049", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000017", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000008", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000015", 2024-04-13T22:15:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000031", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000022", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000013", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000020", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000037", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000005", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000012", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000003", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000042", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000010", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000009", 2024-04-13T22:15:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000025", 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000016", 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000014", 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000030", 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000021" 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:19.673 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:19.674 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.674 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.674 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.674 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.674 DEBUG:boto:Data: 2024-04-13T22:15:19.674 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.674 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.675 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.675 DEBUG:boto:Params: {} 2024-04-13T22:15:19.675 DEBUG:boto:Token: None 2024-04-13T22:15:19.675 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.675 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d89d68a72aa6e6-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.678 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:19.678 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:19.679 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.679 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:19.679 DEBUG:boto:Data: 2024-04-13T22:15:19.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.679 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.679 DEBUG:boto:Params: {} 2024-04-13T22:15:19.679 DEBUG:boto:Token: None 2024-04-13T22:15:19.679 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:19.679 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DJJvdhUwTD4tpdfY0sBRcEccHQY= 2024-04-13T22:15:19.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DJJvdhUwTD4tpdfY0sBRcEccHQY='} 2024-04-13T22:15:19.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080f6017629de1dee-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.684 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.685 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.685 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.685 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.685 DEBUG:boto:Data: 2024-04-13T22:15:19.685 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.685 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.685 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.685 DEBUG:boto:Params: {} 2024-04-13T22:15:19.685 DEBUG:boto:Token: None 2024-04-13T22:15:19.685 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9d032403fe95a86-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.689 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:19.689 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:19.689 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.689 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:19.689 DEBUG:boto:Data: 2024-04-13T22:15:19.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.689 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.689 DEBUG:boto:Params: {} 2024-04-13T22:15:19.689 DEBUG:boto:Token: None 2024-04-13T22:15:19.690 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:19.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3ckRAjqWlke7+6nKBeZ2HMHSJPI= 2024-04-13T22:15:19.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3ckRAjqWlke7+6nKBeZ2HMHSJPI='} 2024-04-13T22:15:19.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c04c68822f941150-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.735 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.735 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.735 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.735 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.735 DEBUG:boto:Data: 2024-04-13T22:15:19.735 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.735 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.735 DEBUG:boto:Params: {} 2024-04-13T22:15:19.735 DEBUG:boto:Token: None 2024-04-13T22:15:19.735 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089b126e87420917c-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.739 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:19.739 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:19.739 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.739 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:19.739 DEBUG:boto:Data: 2024-04-13T22:15:19.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.739 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.739 DEBUG:boto:Params: {} 2024-04-13T22:15:19.740 DEBUG:boto:Token: None 2024-04-13T22:15:19.740 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:19.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:usN3lx2tz0RJFlHFh+sfdv/zl3c= 2024-04-13T22:15:19.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:usN3lx2tz0RJFlHFh+sfdv/zl3c='} 2024-04-13T22:15:19.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f38e18159d69c2ac-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.747 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.747 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.747 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.747 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.747 DEBUG:boto:Data: 2024-04-13T22:15:19.747 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.747 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.747 DEBUG:boto:Params: {} 2024-04-13T22:15:19.748 DEBUG:boto:Token: None 2024-04-13T22:15:19.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bdacdd1e4924283-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.751 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.751 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.751 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.751 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.751 DEBUG:boto:Data: 2024-04-13T22:15:19.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.751 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.751 DEBUG:boto:Params: {} 2024-04-13T22:15:19.752 DEBUG:boto:Token: None 2024-04-13T22:15:19.752 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:19.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LRHaW36uQ/XGj55flEvLMe1AGSQ= 2024-04-13T22:15:19.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LRHaW36uQ/XGj55flEvLMe1AGSQ='} 2024-04-13T22:15:19.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f77b0b47805aa927-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.795 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.795 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.795 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.795 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.795 DEBUG:boto:Data: 2024-04-13T22:15:19.795 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.795 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.795 DEBUG:boto:Params: {} 2024-04-13T22:15:19.796 DEBUG:boto:Token: None 2024-04-13T22:15:19.796 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046c9aeb5aab4b02b-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.799 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.799 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.799 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.799 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.799 DEBUG:boto:Data: 2024-04-13T22:15:19.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.799 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.799 DEBUG:boto:Params: {} 2024-04-13T22:15:19.800 DEBUG:boto:Token: None 2024-04-13T22:15:19.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:19.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vy6KuQn3fiBjTbRzB0YJ++E4STQ= 2024-04-13T22:15:19.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vy6KuQn3fiBjTbRzB0YJ++E4STQ='} 2024-04-13T22:15:19.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f488d8903664583c-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.806 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.806 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.806 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.806 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.806 DEBUG:boto:Data: 2024-04-13T22:15:19.806 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.807 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.807 DEBUG:boto:Params: {} 2024-04-13T22:15:19.807 DEBUG:boto:Token: None 2024-04-13T22:15:19.807 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005072e2d40b496665-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.810 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:19.810 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:19.810 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.811 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:19.811 DEBUG:boto:Data: 2024-04-13T22:15:19.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.811 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.811 DEBUG:boto:Params: {} 2024-04-13T22:15:19.811 DEBUG:boto:Token: None 2024-04-13T22:15:19.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:19.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K84To7l8Brps6aKs36+eVsPweR4= 2024-04-13T22:15:19.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K84To7l8Brps6aKs36+eVsPweR4='} 2024-04-13T22:15:19.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfb25540ab2fb99c-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.855 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.855 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.855 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.855 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.855 DEBUG:boto:Data: 2024-04-13T22:15:19.855 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.855 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.855 DEBUG:boto:Params: {} 2024-04-13T22:15:19.855 DEBUG:boto:Token: None 2024-04-13T22:15:19.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff1b6186c0d61dd5-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.859 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:19.859 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:19.859 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.859 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:19.859 DEBUG:boto:Data: 2024-04-13T22:15:19.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.859 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.859 DEBUG:boto:Params: {} 2024-04-13T22:15:19.860 DEBUG:boto:Token: None 2024-04-13T22:15:19.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:19.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f89CEWzdlCZIwPy4bdftc5NZ0P0= 2024-04-13T22:15:19.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f89CEWzdlCZIwPy4bdftc5NZ0P0='} 2024-04-13T22:15:19.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003108018c451c0db4-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.866 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.866 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.866 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.866 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.866 DEBUG:boto:Data: 2024-04-13T22:15:19.867 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.867 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.867 DEBUG:boto:Params: {} 2024-04-13T22:15:19.867 DEBUG:boto:Token: None 2024-04-13T22:15:19.867 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b30570c3884fd10-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.870 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.871 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.871 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.871 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.871 DEBUG:boto:Data: 2024-04-13T22:15:19.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.871 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.871 DEBUG:boto:Params: {} 2024-04-13T22:15:19.871 DEBUG:boto:Token: None 2024-04-13T22:15:19.871 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:19.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:edS1cFcWbHf/tQpIs25yq7h+ljY= 2024-04-13T22:15:19.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edS1cFcWbHf/tQpIs25yq7h+ljY='} 2024-04-13T22:15:19.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7def627be6c695b-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.915 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.915 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.915 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.915 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.915 DEBUG:boto:Data: 2024-04-13T22:15:19.915 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.915 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.915 DEBUG:boto:Params: {} 2024-04-13T22:15:19.915 DEBUG:boto:Token: None 2024-04-13T22:15:19.915 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000438014b48d9fa582-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.919 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:19.919 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:19.919 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.919 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:19.919 DEBUG:boto:Data: 2024-04-13T22:15:19.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.919 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.919 DEBUG:boto:Params: {} 2024-04-13T22:15:19.920 DEBUG:boto:Token: None 2024-04-13T22:15:19.920 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:19.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TXukkE7fAmjfY6ueadOfmSMaswU= 2024-04-13T22:15:19.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TXukkE7fAmjfY6ueadOfmSMaswU='} 2024-04-13T22:15:19.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b6f906a0528679e-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.925 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.925 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.925 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.925 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.925 DEBUG:boto:Data: 2024-04-13T22:15:19.925 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.925 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.926 DEBUG:boto:Params: {} 2024-04-13T22:15:19.926 DEBUG:boto:Token: None 2024-04-13T22:15:19.926 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078723409c1a5ecae-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.929 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:19.929 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:19.929 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.929 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:19.929 DEBUG:boto:Data: 2024-04-13T22:15:19.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.929 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.929 DEBUG:boto:Params: {} 2024-04-13T22:15:19.930 DEBUG:boto:Token: None 2024-04-13T22:15:19.930 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:19.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IWvULelOJIqsjSakMsRRnYOHsMI= 2024-04-13T22:15:19.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IWvULelOJIqsjSakMsRRnYOHsMI='} 2024-04-13T22:15:19.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5e5346c3e18292a-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.975 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.975 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.975 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.975 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.975 DEBUG:boto:Data: 2024-04-13T22:15:19.975 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.975 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.975 DEBUG:boto:Params: {} 2024-04-13T22:15:19.976 DEBUG:boto:Token: None 2024-04-13T22:15:19.976 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aab96ae4bb4e8fdb-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.979 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:19.979 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:19.979 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.979 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:19.979 DEBUG:boto:Data: 2024-04-13T22:15:19.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.980 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.980 DEBUG:boto:Params: {} 2024-04-13T22:15:19.980 DEBUG:boto:Token: None 2024-04-13T22:15:19.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:19.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bj9zC72QEi1u/m3xC5I8loulky4= 2024-04-13T22:15:19.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bj9zC72QEi1u/m3xC5I8loulky4='} 2024-04-13T22:15:19.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db7a818aa1b704cc-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.987 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:19.987 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:19.987 DEBUG:boto:Method: HEAD 2024-04-13T22:15:19.987 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:19.987 DEBUG:boto:Data: 2024-04-13T22:15:19.987 DEBUG:boto:Headers: {} 2024-04-13T22:15:19.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.988 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.988 DEBUG:boto:Params: {} 2024-04-13T22:15:19.988 DEBUG:boto:Token: None 2024-04-13T22:15:19.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/ 2024-04-13T22:15:19.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew= 2024-04-13T22:15:19.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zZO4boM1veUwimZkgMVpeEovtew=', 'Content-Length': '0'} 2024-04-13T22:15:19.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfa823a030d8e3cc-00661b03f7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:19.991 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:19.992 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:19.992 DEBUG:boto:Method: PUT 2024-04-13T22:15:19.992 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:19.992 DEBUG:boto:Data: 2024-04-13T22:15:19.992 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:19.992 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:19.992 DEBUG:boto:Port: 8000 2024-04-13T22:15:19.992 DEBUG:boto:Params: {} 2024-04-13T22:15:19.992 DEBUG:boto:Token: None 2024-04-13T22:15:19.992 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:19 GMT /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:19.993 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o/hOUFKNl/N4g/smD3lUl6Raf+0= 2024-04-13T22:15:19.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o/hOUFKNl/N4g/smD3lUl6Raf+0='} 2024-04-13T22:15:20.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c666f7f00ac9344-00661b03f7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.035 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.035 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.035 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.035 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.035 DEBUG:boto:Data: 2024-04-13T22:15:20.035 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.035 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.035 DEBUG:boto:Params: {} 2024-04-13T22:15:20.035 DEBUG:boto:Token: None 2024-04-13T22:15:20.036 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef58d78aa5fdf52d-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.039 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:20.039 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:20.039 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.040 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:20.040 DEBUG:boto:Data: 2024-04-13T22:15:20.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.040 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.040 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.040 DEBUG:boto:Params: {} 2024-04-13T22:15:20.040 DEBUG:boto:Token: None 2024-04-13T22:15:20.040 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:20.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5zbcOiurhPtEecHRU9EOjxoIBDk= 2024-04-13T22:15:20.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5zbcOiurhPtEecHRU9EOjxoIBDk='} 2024-04-13T22:15:20.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025acc9ba4f2d27b4-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.047 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.047 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.047 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.047 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.047 DEBUG:boto:Data: 2024-04-13T22:15:20.047 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.047 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.048 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.048 DEBUG:boto:Params: {} 2024-04-13T22:15:20.048 DEBUG:boto:Token: None 2024-04-13T22:15:20.048 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d1476e7a0c3d903-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.051 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:20.052 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:20.052 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.052 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:20.052 DEBUG:boto:Data: 2024-04-13T22:15:20.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.052 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.052 DEBUG:boto:Params: {} 2024-04-13T22:15:20.052 DEBUG:boto:Token: None 2024-04-13T22:15:20.052 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:20.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xXCCUcosBDN2p24sNWlSgcQaBcQ= 2024-04-13T22:15:20.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xXCCUcosBDN2p24sNWlSgcQaBcQ='} 2024-04-13T22:15:20.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004acf05d58fd5438b-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.095 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.095 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.095 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.095 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.095 DEBUG:boto:Data: 2024-04-13T22:15:20.095 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.095 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.095 DEBUG:boto:Params: {} 2024-04-13T22:15:20.095 DEBUG:boto:Token: None 2024-04-13T22:15:20.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbd967104c98ae44-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.099 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:20.099 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:20.099 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.099 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:20.099 DEBUG:boto:Data: 2024-04-13T22:15:20.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.099 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.099 DEBUG:boto:Params: {} 2024-04-13T22:15:20.099 DEBUG:boto:Token: None 2024-04-13T22:15:20.100 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:20.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0gshfC2DBwNSFrcoAjLHIgjxOAI= 2024-04-13T22:15:20.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0gshfC2DBwNSFrcoAjLHIgjxOAI='} 2024-04-13T22:15:20.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf9828463e9a8998-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.106 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.106 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.106 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.106 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.107 DEBUG:boto:Data: 2024-04-13T22:15:20.107 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.107 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.107 DEBUG:boto:Params: {} 2024-04-13T22:15:20.107 DEBUG:boto:Token: None 2024-04-13T22:15:20.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dee110ad189b0fc2-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.110 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:20.110 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:20.110 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.111 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:20.111 DEBUG:boto:Data: 2024-04-13T22:15:20.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.111 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.111 DEBUG:boto:Params: {} 2024-04-13T22:15:20.111 DEBUG:boto:Token: None 2024-04-13T22:15:20.111 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:20.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KKYTxHTldVOMa4kkhazojK8nEUs= 2024-04-13T22:15:20.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KKYTxHTldVOMa4kkhazojK8nEUs='} 2024-04-13T22:15:20.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f46a8b60ac22783c-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.155 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.155 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.155 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.155 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.155 DEBUG:boto:Data: 2024-04-13T22:15:20.155 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.155 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.155 DEBUG:boto:Params: {} 2024-04-13T22:15:20.155 DEBUG:boto:Token: None 2024-04-13T22:15:20.155 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9b1a52d14e39843-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.159 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:20.159 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:20.159 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.159 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:20.159 DEBUG:boto:Data: 2024-04-13T22:15:20.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.159 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.159 DEBUG:boto:Params: {} 2024-04-13T22:15:20.160 DEBUG:boto:Token: None 2024-04-13T22:15:20.160 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:20.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y70wiU1aqqe/DQAxJrssse1nGa0= 2024-04-13T22:15:20.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y70wiU1aqqe/DQAxJrssse1nGa0='} 2024-04-13T22:15:20.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063d1dab13d9378cb-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.165 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.165 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.165 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.165 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.165 DEBUG:boto:Data: 2024-04-13T22:15:20.165 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.165 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.165 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.165 DEBUG:boto:Params: {} 2024-04-13T22:15:20.165 DEBUG:boto:Token: None 2024-04-13T22:15:20.165 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.166 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003665729ba2f96e8a-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.169 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:20.169 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:20.169 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.169 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:20.169 DEBUG:boto:Data: 2024-04-13T22:15:20.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.169 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.169 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.169 DEBUG:boto:Params: {} 2024-04-13T22:15:20.170 DEBUG:boto:Token: None 2024-04-13T22:15:20.170 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:20.170 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eMTFUVJjWYnzb6WJr//CLw50htQ= 2024-04-13T22:15:20.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eMTFUVJjWYnzb6WJr//CLw50htQ='} 2024-04-13T22:15:20.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b255b454e81063af-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.215 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.215 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.215 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.215 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.215 DEBUG:boto:Data: 2024-04-13T22:15:20.215 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.215 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.215 DEBUG:boto:Params: {} 2024-04-13T22:15:20.215 DEBUG:boto:Token: None 2024-04-13T22:15:20.215 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a6ff78d700b542b-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.218 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.219 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.219 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.219 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.219 DEBUG:boto:Data: 2024-04-13T22:15:20.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.219 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.219 DEBUG:boto:Params: {} 2024-04-13T22:15:20.219 DEBUG:boto:Token: None 2024-04-13T22:15:20.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLRySQ2pbNhqP6V0WkfZHzHXh6c= 2024-04-13T22:15:20.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLRySQ2pbNhqP6V0WkfZHzHXh6c='} 2024-04-13T22:15:20.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c74b5a702b12d149-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.226 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.226 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.226 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.226 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.226 DEBUG:boto:Data: 2024-04-13T22:15:20.227 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.227 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.227 DEBUG:boto:Params: {} 2024-04-13T22:15:20.229 DEBUG:boto:Token: None 2024-04-13T22:15:20.229 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051a5e3b67d7b9099-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.232 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:20.233 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:20.233 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.233 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:20.233 DEBUG:boto:Data: 2024-04-13T22:15:20.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.233 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.233 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.233 DEBUG:boto:Params: {} 2024-04-13T22:15:20.233 DEBUG:boto:Token: None 2024-04-13T22:15:20.234 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:20.234 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cf8CLGinvVQf2PqohF33/EhCIw4= 2024-04-13T22:15:20.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cf8CLGinvVQf2PqohF33/EhCIw4='} 2024-04-13T22:15:20.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a5dc70f1bfae2dd-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.279 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.279 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.279 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.279 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.279 DEBUG:boto:Data: 2024-04-13T22:15:20.279 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.279 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.279 DEBUG:boto:Params: {} 2024-04-13T22:15:20.279 DEBUG:boto:Token: None 2024-04-13T22:15:20.280 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ef9b1e7ba388d14-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.283 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.283 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.283 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.283 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.283 DEBUG:boto:Data: 2024-04-13T22:15:20.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.284 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.284 DEBUG:boto:Params: {} 2024-04-13T22:15:20.284 DEBUG:boto:Token: None 2024-04-13T22:15:20.284 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lHAGAlHCpEF3FvxWD7FXihrBc2k= 2024-04-13T22:15:20.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lHAGAlHCpEF3FvxWD7FXihrBc2k='} 2024-04-13T22:15:20.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b88952289cbf43ae-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.290 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.290 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.290 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.291 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.291 DEBUG:boto:Data: 2024-04-13T22:15:20.291 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.291 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.291 DEBUG:boto:Params: {} 2024-04-13T22:15:20.291 DEBUG:boto:Token: None 2024-04-13T22:15:20.291 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2595fb269d1174a-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.295 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:20.295 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:20.295 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.295 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:20.295 DEBUG:boto:Data: 2024-04-13T22:15:20.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.296 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.296 DEBUG:boto:Params: {} 2024-04-13T22:15:20.296 DEBUG:boto:Token: None 2024-04-13T22:15:20.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:20.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sw+ccNH+o9tEf8vZiRtBrLPlmMo= 2024-04-13T22:15:20.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sw+ccNH+o9tEf8vZiRtBrLPlmMo='} 2024-04-13T22:15:20.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b185d811ce446e7c-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.339 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.339 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.339 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.339 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.339 DEBUG:boto:Data: 2024-04-13T22:15:20.339 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.339 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.339 DEBUG:boto:Params: {} 2024-04-13T22:15:20.339 DEBUG:boto:Token: None 2024-04-13T22:15:20.340 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9f8d37d76291650-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.343 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:20.343 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:20.344 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.344 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:20.344 DEBUG:boto:Data: 2024-04-13T22:15:20.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.344 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.344 DEBUG:boto:Params: {} 2024-04-13T22:15:20.344 DEBUG:boto:Token: None 2024-04-13T22:15:20.344 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:20.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uK+eDYNXUtWoPRQ7hX1gCIMOF/Y= 2024-04-13T22:15:20.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uK+eDYNXUtWoPRQ7hX1gCIMOF/Y='} 2024-04-13T22:15:20.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f1fb710b9db0cad-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.351 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.351 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.351 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.351 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.351 DEBUG:boto:Data: 2024-04-13T22:15:20.351 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.351 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.352 DEBUG:boto:Params: {} 2024-04-13T22:15:20.352 DEBUG:boto:Token: None 2024-04-13T22:15:20.352 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.352 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007970dd44dbdc09c1-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.355 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:20.355 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:20.355 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.355 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:20.355 DEBUG:boto:Data: 2024-04-13T22:15:20.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.355 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.356 DEBUG:boto:Params: {} 2024-04-13T22:15:20.356 DEBUG:boto:Token: None 2024-04-13T22:15:20.356 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:20.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4C7Nv2SfqTAPKl9wsiKSQiSuMVk= 2024-04-13T22:15:20.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4C7Nv2SfqTAPKl9wsiKSQiSuMVk='} 2024-04-13T22:15:20.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000414ebb952ac0c1a7-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.399 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.399 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.399 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.399 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.399 DEBUG:boto:Data: 2024-04-13T22:15:20.399 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.399 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.399 DEBUG:boto:Params: {} 2024-04-13T22:15:20.399 DEBUG:boto:Token: None 2024-04-13T22:15:20.399 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7d6b666036abbd6-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.403 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.403 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.403 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.403 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.403 DEBUG:boto:Data: 2024-04-13T22:15:20.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.403 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.403 DEBUG:boto:Params: {} 2024-04-13T22:15:20.404 DEBUG:boto:Token: None 2024-04-13T22:15:20.404 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cgkizdv2RZOQ0uLRf/4rRbXLc+8= 2024-04-13T22:15:20.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cgkizdv2RZOQ0uLRf/4rRbXLc+8='} 2024-04-13T22:15:20.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028fd2575e358e373-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.410 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.410 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.410 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.410 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.410 DEBUG:boto:Data: 2024-04-13T22:15:20.410 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.411 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.411 DEBUG:boto:Params: {} 2024-04-13T22:15:20.411 DEBUG:boto:Token: None 2024-04-13T22:15:20.411 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001baf27b1a1f1ed80-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.414 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:20.414 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:20.415 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.415 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:20.415 DEBUG:boto:Data: 2024-04-13T22:15:20.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.415 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.415 DEBUG:boto:Params: {} 2024-04-13T22:15:20.415 DEBUG:boto:Token: None 2024-04-13T22:15:20.415 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:20.415 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lzdk+EugOq/MwKLSQDsdb8KreP4= 2024-04-13T22:15:20.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lzdk+EugOq/MwKLSQDsdb8KreP4='} 2024-04-13T22:15:20.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000794d165d82bc1431-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.458 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.459 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.459 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.459 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.459 DEBUG:boto:Data: 2024-04-13T22:15:20.459 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.459 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.459 DEBUG:boto:Params: {} 2024-04-13T22:15:20.459 DEBUG:boto:Token: None 2024-04-13T22:15:20.459 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092e33a43bc8205de-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.462 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:20.463 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:20.463 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.463 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:20.463 DEBUG:boto:Data: 2024-04-13T22:15:20.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.463 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.463 DEBUG:boto:Params: {} 2024-04-13T22:15:20.463 DEBUG:boto:Token: None 2024-04-13T22:15:20.463 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:20.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2ZqHucwvnMuW2jeDFzUNEu6UdHE= 2024-04-13T22:15:20.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2ZqHucwvnMuW2jeDFzUNEu6UdHE='} 2024-04-13T22:15:20.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028d921c76b0d875d-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:20.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:20.469 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:20.520 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:20.520 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:20.569 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:15:20.569 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-60 2024-04-13T22:15:20.580 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 43 2024-04-13T22:15:20.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:20.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:20.640 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:20.691 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:20.691 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:20.732 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:15:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000019", 2024-04-13T22:15:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000006", 2024-04-13T22:15:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000467", 2024-04-13T22:15:20.732 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000005", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000056", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000004", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000023", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000317", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000022", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000021", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000009", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000271", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000008", 2024-04-13T22:15:20.733 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000018", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000007", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000017", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000162", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000016", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000003", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000423", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000002", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000012", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000011", 2024-04-13T22:15:20.734 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000283", 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000010", 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000061", 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000050", 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000029", 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000133" 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:20.735 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:20.736 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.736 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.736 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.736 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.737 DEBUG:boto:Data: 2024-04-13T22:15:20.737 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.737 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.737 DEBUG:boto:Params: {} 2024-04-13T22:15:20.737 DEBUG:boto:Token: None 2024-04-13T22:15:20.737 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009078f307a75806e-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.741 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:20.741 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:20.741 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.741 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:20.742 DEBUG:boto:Data: 2024-04-13T22:15:20.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.742 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.742 DEBUG:boto:Params: {} 2024-04-13T22:15:20.742 DEBUG:boto:Token: None 2024-04-13T22:15:20.742 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:20.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hVbi+fozBAnuVev1jOh/w8A8OWs= 2024-04-13T22:15:20.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hVbi+fozBAnuVev1jOh/w8A8OWs='} 2024-04-13T22:15:20.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ceac522bebfad7e-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.787 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.787 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.787 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.787 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.787 DEBUG:boto:Data: 2024-04-13T22:15:20.787 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.787 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.787 DEBUG:boto:Params: {} 2024-04-13T22:15:20.787 DEBUG:boto:Token: None 2024-04-13T22:15:20.788 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c71de141d8f1eaa-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.791 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.791 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.791 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.791 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.791 DEBUG:boto:Data: 2024-04-13T22:15:20.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.791 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.792 DEBUG:boto:Params: {} 2024-04-13T22:15:20.792 DEBUG:boto:Token: None 2024-04-13T22:15:20.792 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:20.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lHAGAlHCpEF3FvxWD7FXihrBc2k= 2024-04-13T22:15:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lHAGAlHCpEF3FvxWD7FXihrBc2k='} 2024-04-13T22:15:20.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebc117e6f50fe1bb-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.798 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.798 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.798 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.799 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.799 DEBUG:boto:Data: 2024-04-13T22:15:20.799 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.799 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.799 DEBUG:boto:Params: {} 2024-04-13T22:15:20.799 DEBUG:boto:Token: None 2024-04-13T22:15:20.799 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.799 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d5bca98f5353afd-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.802 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.803 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.803 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.803 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.803 DEBUG:boto:Data: 2024-04-13T22:15:20.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.803 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.803 DEBUG:boto:Params: {} 2024-04-13T22:15:20.803 DEBUG:boto:Token: None 2024-04-13T22:15:20.803 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:20.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cgkizdv2RZOQ0uLRf/4rRbXLc+8= 2024-04-13T22:15:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cgkizdv2RZOQ0uLRf/4rRbXLc+8='} 2024-04-13T22:15:20.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4cfa485c5398b83-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.851 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.851 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.851 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.851 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.851 DEBUG:boto:Data: 2024-04-13T22:15:20.851 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.851 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.851 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.851 DEBUG:boto:Params: {} 2024-04-13T22:15:20.851 DEBUG:boto:Token: None 2024-04-13T22:15:20.851 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000376334abc6a1f9d4-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.855 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:20.855 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:20.855 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.855 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:20.855 DEBUG:boto:Data: 2024-04-13T22:15:20.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.856 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.856 DEBUG:boto:Params: {} 2024-04-13T22:15:20.856 DEBUG:boto:Token: None 2024-04-13T22:15:20.856 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:20.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6iGool9NUwX9RF11CC/mhUhz9WU= 2024-04-13T22:15:20.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6iGool9NUwX9RF11CC/mhUhz9WU='} 2024-04-13T22:15:20.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ca8078d04711ddd-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.861 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.861 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.861 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.861 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.862 DEBUG:boto:Data: 2024-04-13T22:15:20.862 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.862 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.862 DEBUG:boto:Params: {} 2024-04-13T22:15:20.862 DEBUG:boto:Token: None 2024-04-13T22:15:20.862 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.862 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000870604421bed571e-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.866 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:20.866 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:20.866 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.866 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:20.866 DEBUG:boto:Data: 2024-04-13T22:15:20.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.866 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.866 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.866 DEBUG:boto:Params: {} 2024-04-13T22:15:20.866 DEBUG:boto:Token: None 2024-04-13T22:15:20.867 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:20.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OLVHMzKtE6+l5Mu0tuJXefumNCg= 2024-04-13T22:15:20.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OLVHMzKtE6+l5Mu0tuJXefumNCg='} 2024-04-13T22:15:20.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1032a9789b93d2c-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.911 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.911 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.911 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.911 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.911 DEBUG:boto:Data: 2024-04-13T22:15:20.911 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.911 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.911 DEBUG:boto:Params: {} 2024-04-13T22:15:20.911 DEBUG:boto:Token: None 2024-04-13T22:15:20.912 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ce755dc1bcfbb12-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.915 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:20.915 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:20.915 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.916 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:20.916 DEBUG:boto:Data: 2024-04-13T22:15:20.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.916 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.916 DEBUG:boto:Params: {} 2024-04-13T22:15:20.916 DEBUG:boto:Token: None 2024-04-13T22:15:20.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:20.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PCbYtjqgjQjwCOgdd38RneToe3M= 2024-04-13T22:15:20.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PCbYtjqgjQjwCOgdd38RneToe3M='} 2024-04-13T22:15:20.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4cc7485e2be0df8-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.923 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.923 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.923 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.923 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.923 DEBUG:boto:Data: 2024-04-13T22:15:20.923 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.923 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.923 DEBUG:boto:Params: {} 2024-04-13T22:15:20.924 DEBUG:boto:Token: None 2024-04-13T22:15:20.924 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060cca7d5b50c6fbf-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.927 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:20.928 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:20.928 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.928 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:20.928 DEBUG:boto:Data: 2024-04-13T22:15:20.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.928 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.928 DEBUG:boto:Params: {} 2024-04-13T22:15:20.928 DEBUG:boto:Token: None 2024-04-13T22:15:20.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:20.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vep35B8Yo2sSX7B4PjOaYtqbeMI= 2024-04-13T22:15:20.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vep35B8Yo2sSX7B4PjOaYtqbeMI='} 2024-04-13T22:15:20.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b745a7139ec002f-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.971 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.971 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.971 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.971 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.971 DEBUG:boto:Data: 2024-04-13T22:15:20.971 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.971 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.971 DEBUG:boto:Params: {} 2024-04-13T22:15:20.972 DEBUG:boto:Token: None 2024-04-13T22:15:20.972 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000223bc27af9cbcef7-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.975 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:20.975 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:20.976 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.976 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:20.976 DEBUG:boto:Data: 2024-04-13T22:15:20.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.976 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.976 DEBUG:boto:Params: {} 2024-04-13T22:15:20.976 DEBUG:boto:Token: None 2024-04-13T22:15:20.976 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:20.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KYUPm7+fNaQZWvqS/s1DpNkUMTY= 2024-04-13T22:15:20.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KYUPm7+fNaQZWvqS/s1DpNkUMTY='} 2024-04-13T22:15:20.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006dd63f0f6234e1e3-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.982 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:20.982 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:20.982 DEBUG:boto:Method: HEAD 2024-04-13T22:15:20.982 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:20.982 DEBUG:boto:Data: 2024-04-13T22:15:20.982 DEBUG:boto:Headers: {} 2024-04-13T22:15:20.982 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.982 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.982 DEBUG:boto:Params: {} 2024-04-13T22:15:20.983 DEBUG:boto:Token: None 2024-04-13T22:15:20.983 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/ 2024-04-13T22:15:20.983 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc= 2024-04-13T22:15:20.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P56tgg/iEpDPq0gUMAVBhtljMPc=', 'Content-Length': '0'} 2024-04-13T22:15:20.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f173866ce9326c4f-00661b03f8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:20.986 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.986 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.986 DEBUG:boto:Method: PUT 2024-04-13T22:15:20.987 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.987 DEBUG:boto:Data: 2024-04-13T22:15:20.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:20.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:20.987 DEBUG:boto:Port: 8000 2024-04-13T22:15:20.987 DEBUG:boto:Params: {} 2024-04-13T22:15:20.987 DEBUG:boto:Token: None 2024-04-13T22:15:20.987 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:20 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:20.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLRySQ2pbNhqP6V0WkfZHzHXh6c= 2024-04-13T22:15:20.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:20 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLRySQ2pbNhqP6V0WkfZHzHXh6c='} 2024-04-13T22:15:21.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f340e07883a35e9-00661b03f8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.031 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.031 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.031 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.031 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.031 DEBUG:boto:Data: 2024-04-13T22:15:21.031 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.031 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.031 DEBUG:boto:Params: {} 2024-04-13T22:15:21.031 DEBUG:boto:Token: None 2024-04-13T22:15:21.032 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.036 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f9e125ca7175860-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.036 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:21.036 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:21.036 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.036 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:21.037 DEBUG:boto:Data: 2024-04-13T22:15:21.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.037 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.037 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.037 DEBUG:boto:Params: {} 2024-04-13T22:15:21.037 DEBUG:boto:Token: None 2024-04-13T22:15:21.037 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:21.037 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F0/PvzWrmUr6J+WjQjo+hfFhTSc= 2024-04-13T22:15:21.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F0/PvzWrmUr6J+WjQjo+hfFhTSc='} 2024-04-13T22:15:21.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf2e762d09ea42f2-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.044 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.044 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.044 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.044 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.044 DEBUG:boto:Data: 2024-04-13T22:15:21.044 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.044 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.044 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.044 DEBUG:boto:Params: {} 2024-04-13T22:15:21.045 DEBUG:boto:Token: None 2024-04-13T22:15:21.045 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000991a813c51cdb993-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.048 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:21.048 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:21.048 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.048 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:21.049 DEBUG:boto:Data: 2024-04-13T22:15:21.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.049 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.049 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.049 DEBUG:boto:Params: {} 2024-04-13T22:15:21.049 DEBUG:boto:Token: None 2024-04-13T22:15:21.049 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:21.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w4hNk9mnJJ1I0lCAsdG2fDflhWQ= 2024-04-13T22:15:21.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w4hNk9mnJJ1I0lCAsdG2fDflhWQ='} 2024-04-13T22:15:21.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f145f43e0a381d8a-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.095 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.095 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.095 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.095 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.095 DEBUG:boto:Data: 2024-04-13T22:15:21.095 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.095 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.095 DEBUG:boto:Params: {} 2024-04-13T22:15:21.096 DEBUG:boto:Token: None 2024-04-13T22:15:21.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da71d87466af9fbb-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.099 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.099 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.099 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.099 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.100 DEBUG:boto:Data: 2024-04-13T22:15:21.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.100 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.100 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.100 DEBUG:boto:Params: {} 2024-04-13T22:15:21.100 DEBUG:boto:Token: None 2024-04-13T22:15:21.100 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r/UeEFhYTYgOa2MMT8rJTwoU//o= 2024-04-13T22:15:21.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r/UeEFhYTYgOa2MMT8rJTwoU//o='} 2024-04-13T22:15:21.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000918bf85cb28f8112-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.107 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.107 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.107 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.107 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.107 DEBUG:boto:Data: 2024-04-13T22:15:21.107 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.107 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.107 DEBUG:boto:Params: {} 2024-04-13T22:15:21.107 DEBUG:boto:Token: None 2024-04-13T22:15:21.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fb31bfb3117d94e-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.110 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:21.111 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:21.111 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.111 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:21.111 DEBUG:boto:Data: 2024-04-13T22:15:21.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.111 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.111 DEBUG:boto:Params: {} 2024-04-13T22:15:21.111 DEBUG:boto:Token: None 2024-04-13T22:15:21.111 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:21.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hUp0Z9jZOpzYL/bI4WZIi+nMshQ= 2024-04-13T22:15:21.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hUp0Z9jZOpzYL/bI4WZIi+nMshQ='} 2024-04-13T22:15:21.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096836eb6c3a80bd4-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.155 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.155 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.155 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.155 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.155 DEBUG:boto:Data: 2024-04-13T22:15:21.155 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.156 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.156 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.156 DEBUG:boto:Params: {} 2024-04-13T22:15:21.156 DEBUG:boto:Token: None 2024-04-13T22:15:21.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b3746fba74dd669-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.160 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:21.160 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:21.160 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.160 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:21.161 DEBUG:boto:Data: 2024-04-13T22:15:21.161 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.161 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.161 DEBUG:boto:Params: {} 2024-04-13T22:15:21.161 DEBUG:boto:Token: None 2024-04-13T22:15:21.161 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:21.161 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4d7jR3AqBESBpomqPUD7seyYJW8= 2024-04-13T22:15:21.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4d7jR3AqBESBpomqPUD7seyYJW8='} 2024-04-13T22:15:21.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7384db3060c7bfd-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.168 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.168 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.168 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.168 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.168 DEBUG:boto:Data: 2024-04-13T22:15:21.168 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.168 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.168 DEBUG:boto:Params: {} 2024-04-13T22:15:21.168 DEBUG:boto:Token: None 2024-04-13T22:15:21.168 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bd8d0d27f8a7fbf-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.172 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:21.172 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:21.172 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.172 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:21.172 DEBUG:boto:Data: 2024-04-13T22:15:21.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.172 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.172 DEBUG:boto:Params: {} 2024-04-13T22:15:21.173 DEBUG:boto:Token: None 2024-04-13T22:15:21.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:21.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zR2YrlHHlLWVaboMcYFHPFHfe10= 2024-04-13T22:15:21.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zR2YrlHHlLWVaboMcYFHPFHfe10='} 2024-04-13T22:15:21.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc4a10f474196636-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.215 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.215 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.215 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.215 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.215 DEBUG:boto:Data: 2024-04-13T22:15:21.215 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.215 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.216 DEBUG:boto:Params: {} 2024-04-13T22:15:21.216 DEBUG:boto:Token: None 2024-04-13T22:15:21.216 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1c6f468d29776f5-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.258 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:21.258 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:21.258 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.258 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:21.258 DEBUG:boto:Data: 2024-04-13T22:15:21.258 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.258 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.259 DEBUG:boto:Params: {} 2024-04-13T22:15:21.259 DEBUG:boto:Token: None 2024-04-13T22:15:21.259 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:21.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LJoOprewhzKERT2h5q7B6mTnR8w= 2024-04-13T22:15:21.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LJoOprewhzKERT2h5q7B6mTnR8w='} 2024-04-13T22:15:21.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021da3fd9f7c36763-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.266 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.266 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.266 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.266 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.267 DEBUG:boto:Data: 2024-04-13T22:15:21.267 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.267 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.267 DEBUG:boto:Params: {} 2024-04-13T22:15:21.267 DEBUG:boto:Token: None 2024-04-13T22:15:21.267 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031745231d1b00e7a-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.271 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:21.271 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:21.271 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.271 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:21.271 DEBUG:boto:Data: 2024-04-13T22:15:21.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.272 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.272 DEBUG:boto:Params: {} 2024-04-13T22:15:21.272 DEBUG:boto:Token: None 2024-04-13T22:15:21.272 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:21.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WGqSY838ba/QTkDFR2eaNGRuZIU= 2024-04-13T22:15:21.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WGqSY838ba/QTkDFR2eaNGRuZIU='} 2024-04-13T22:15:21.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068457f32342f0da6-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.315 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.315 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.315 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.315 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.315 DEBUG:boto:Data: 2024-04-13T22:15:21.315 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.315 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.315 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.316 DEBUG:boto:Params: {} 2024-04-13T22:15:21.316 DEBUG:boto:Token: None 2024-04-13T22:15:21.316 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026444163fd31bd74-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.319 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:21.320 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:21.320 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.320 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:21.320 DEBUG:boto:Data: 2024-04-13T22:15:21.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.320 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.320 DEBUG:boto:Params: {} 2024-04-13T22:15:21.320 DEBUG:boto:Token: None 2024-04-13T22:15:21.320 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:21.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:guspZELaNUWA0eB9dMLQ5KQ0Q/I= 2024-04-13T22:15:21.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:guspZELaNUWA0eB9dMLQ5KQ0Q/I='} 2024-04-13T22:15:21.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cd6d530a533cd86-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.326 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.326 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.326 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.326 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.326 DEBUG:boto:Data: 2024-04-13T22:15:21.326 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.326 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.326 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.326 DEBUG:boto:Params: {} 2024-04-13T22:15:21.326 DEBUG:boto:Token: None 2024-04-13T22:15:21.327 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0b3f51faa77c2a3-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.330 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:21.330 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:21.330 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.330 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:21.330 DEBUG:boto:Data: 2024-04-13T22:15:21.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.330 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.330 DEBUG:boto:Params: {} 2024-04-13T22:15:21.330 DEBUG:boto:Token: None 2024-04-13T22:15:21.331 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:21.331 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bVRNw3uRZuIvNNac2xCjg5G+wiU= 2024-04-13T22:15:21.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bVRNw3uRZuIvNNac2xCjg5G+wiU='} 2024-04-13T22:15:21.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a44c55a10de4d941-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.375 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.375 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.375 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.375 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.375 DEBUG:boto:Data: 2024-04-13T22:15:21.375 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.375 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.375 DEBUG:boto:Params: {} 2024-04-13T22:15:21.375 DEBUG:boto:Token: None 2024-04-13T22:15:21.376 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070a695c2cecbcf29-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.379 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:21.379 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:21.379 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.379 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:21.379 DEBUG:boto:Data: 2024-04-13T22:15:21.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.380 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.380 DEBUG:boto:Params: {} 2024-04-13T22:15:21.380 DEBUG:boto:Token: None 2024-04-13T22:15:21.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:21.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NM/bpR9rgUPXREx0tFqMSEXfnEs= 2024-04-13T22:15:21.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NM/bpR9rgUPXREx0tFqMSEXfnEs='} 2024-04-13T22:15:21.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d65eab0ce8e88522-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.385 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.385 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.385 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.385 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.385 DEBUG:boto:Data: 2024-04-13T22:15:21.386 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.386 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.386 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.386 DEBUG:boto:Params: {} 2024-04-13T22:15:21.386 DEBUG:boto:Token: None 2024-04-13T22:15:21.386 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.386 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020659d7b5cf61b14-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.390 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:21.390 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:21.390 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.390 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:21.390 DEBUG:boto:Data: 2024-04-13T22:15:21.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.390 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.390 DEBUG:boto:Params: {} 2024-04-13T22:15:21.391 DEBUG:boto:Token: None 2024-04-13T22:15:21.391 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:21.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t4MutV8K5XAiStJwUbiBj/oIRtg= 2024-04-13T22:15:21.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t4MutV8K5XAiStJwUbiBj/oIRtg='} 2024-04-13T22:15:21.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d882b39577b9d1f-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.435 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.435 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.435 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.435 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.435 DEBUG:boto:Data: 2024-04-13T22:15:21.435 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.435 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.435 DEBUG:boto:Params: {} 2024-04-13T22:15:21.435 DEBUG:boto:Token: None 2024-04-13T22:15:21.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008165f45313816943-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.440 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:21.440 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:21.440 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.440 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:21.441 DEBUG:boto:Data: 2024-04-13T22:15:21.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.441 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.441 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.441 DEBUG:boto:Params: {} 2024-04-13T22:15:21.441 DEBUG:boto:Token: None 2024-04-13T22:15:21.441 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:21.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TL3KVMYxxg0G+6rKzjkFBefTmZU= 2024-04-13T22:15:21.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TL3KVMYxxg0G+6rKzjkFBefTmZU='} 2024-04-13T22:15:21.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc828f84ed31edb5-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.446 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.447 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.447 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.447 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.447 DEBUG:boto:Data: 2024-04-13T22:15:21.447 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.447 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.447 DEBUG:boto:Params: {} 2024-04-13T22:15:21.447 DEBUG:boto:Token: None 2024-04-13T22:15:21.447 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082f1136cb6a274b7-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.451 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:21.451 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:21.451 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.451 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:21.451 DEBUG:boto:Data: 2024-04-13T22:15:21.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.452 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.452 DEBUG:boto:Params: {} 2024-04-13T22:15:21.452 DEBUG:boto:Token: None 2024-04-13T22:15:21.452 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:21.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Ke6wqBDA4MlV5VjMESG8So5Xic= 2024-04-13T22:15:21.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Ke6wqBDA4MlV5VjMESG8So5Xic='} 2024-04-13T22:15:21.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000280f0536c40311a3-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.495 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.495 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.495 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.495 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.495 DEBUG:boto:Data: 2024-04-13T22:15:21.495 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.495 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.495 DEBUG:boto:Params: {} 2024-04-13T22:15:21.495 DEBUG:boto:Token: None 2024-04-13T22:15:21.496 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2461facca7d2100-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.499 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:21.499 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:21.499 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.499 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:21.500 DEBUG:boto:Data: 2024-04-13T22:15:21.500 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.500 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.500 DEBUG:boto:Params: {} 2024-04-13T22:15:21.500 DEBUG:boto:Token: None 2024-04-13T22:15:21.500 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:21.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hCvDljDufUMtl+//tTTwsY2z1xU= 2024-04-13T22:15:21.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hCvDljDufUMtl+//tTTwsY2z1xU='} 2024-04-13T22:15:21.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063742864b040c8ce-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.505 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.505 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.505 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.506 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.506 DEBUG:boto:Data: 2024-04-13T22:15:21.506 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.506 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.506 DEBUG:boto:Params: {} 2024-04-13T22:15:21.506 DEBUG:boto:Token: None 2024-04-13T22:15:21.506 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.506 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5caa2cd9de4ef26-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.510 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:21.510 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:21.510 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.510 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:21.510 DEBUG:boto:Data: 2024-04-13T22:15:21.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.510 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.511 DEBUG:boto:Params: {} 2024-04-13T22:15:21.511 DEBUG:boto:Token: None 2024-04-13T22:15:21.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:21.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8AybdNWmybkl6dLS2m3sKITdwWY= 2024-04-13T22:15:21.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8AybdNWmybkl6dLS2m3sKITdwWY='} 2024-04-13T22:15:21.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5416376bad0210b-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.555 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.555 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.555 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.555 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.555 DEBUG:boto:Data: 2024-04-13T22:15:21.555 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.555 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.555 DEBUG:boto:Params: {} 2024-04-13T22:15:21.555 DEBUG:boto:Token: None 2024-04-13T22:15:21.556 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000738f4c67bb870cd7-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.559 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:21.559 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:21.559 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.560 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:21.560 DEBUG:boto:Data: 2024-04-13T22:15:21.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.560 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.560 DEBUG:boto:Params: {} 2024-04-13T22:15:21.560 DEBUG:boto:Token: None 2024-04-13T22:15:21.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:21.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:N+jE+kcrqldaFSg9rKYM2QVhEMk= 2024-04-13T22:15:21.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N+jE+kcrqldaFSg9rKYM2QVhEMk='} 2024-04-13T22:15:21.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb66964a4a92b083-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.566 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.567 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.567 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.567 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.567 DEBUG:boto:Data: 2024-04-13T22:15:21.567 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.567 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.567 DEBUG:boto:Params: {} 2024-04-13T22:15:21.567 DEBUG:boto:Token: None 2024-04-13T22:15:21.567 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f76cdd3c3ac5363-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.571 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:21.571 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:21.571 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.571 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:21.571 DEBUG:boto:Data: 2024-04-13T22:15:21.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.571 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.572 DEBUG:boto:Params: {} 2024-04-13T22:15:21.572 DEBUG:boto:Token: None 2024-04-13T22:15:21.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:21.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Boq+1jyjewMexEf9qKBGwuwOF1A= 2024-04-13T22:15:21.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Boq+1jyjewMexEf9qKBGwuwOF1A='} 2024-04-13T22:15:21.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003da47048f6fe92a0-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.615 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.615 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.615 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.615 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.615 DEBUG:boto:Data: 2024-04-13T22:15:21.615 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.615 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.615 DEBUG:boto:Params: {} 2024-04-13T22:15:21.616 DEBUG:boto:Token: None 2024-04-13T22:15:21.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000828091cf03c9643d-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.619 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:21.619 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:21.619 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.619 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:21.619 DEBUG:boto:Data: 2024-04-13T22:15:21.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.620 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.620 DEBUG:boto:Params: {} 2024-04-13T22:15:21.620 DEBUG:boto:Token: None 2024-04-13T22:15:21.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:21.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fekiEb5opBwN2Os3lcSLI9ThA+8= 2024-04-13T22:15:21.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fekiEb5opBwN2Os3lcSLI9ThA+8='} 2024-04-13T22:15:21.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d81902614382104-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:21.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:21.627 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:21.678 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:21.678 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:21.727 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:15:21.727 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-60 2024-04-13T22:15:21.740 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 53 2024-04-13T22:15:21.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:21.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:21.800 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:21.849 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:21.850 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:21.890 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:21.891 INFO:teuthology.orchestra.run.smithi132.stdout: "objs": [ 2024-04-13T22:15:21.891 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000060", 2024-04-13T22:15:21.891 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000000", 2024-04-13T22:15:21.891 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000011", 2024-04-13T22:15:21.891 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000001", 2024-04-13T22:15:21.891 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000033", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000065", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000005", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000016", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000048", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000015", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000026", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000030", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000062", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000009", 2024-04-13T22:15:21.892 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000040", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000024", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000012", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000002", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000013", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000045", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000006", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000017", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000007", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000039", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000027", 2024-04-13T22:15:21.893 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000010", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000021", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000053", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000020", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000004", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000036", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000003", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000014", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000019", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000057", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000018", 2024-04-13T22:15:21.894 INFO:teuthology.orchestra.run.smithi132.stdout: "obj00000000000000000008" 2024-04-13T22:15:21.895 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:21.895 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:21.896 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.896 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.896 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.896 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.896 DEBUG:boto:Data: 2024-04-13T22:15:21.896 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.896 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.896 DEBUG:boto:Params: {} 2024-04-13T22:15:21.897 DEBUG:boto:Token: None 2024-04-13T22:15:21.897 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.897 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b276820f3ad24b3a-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.901 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.901 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.901 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.901 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.901 DEBUG:boto:Data: 2024-04-13T22:15:21.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.901 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.901 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.901 DEBUG:boto:Params: {} 2024-04-13T22:15:21.902 DEBUG:boto:Token: None 2024-04-13T22:15:21.902 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:21.902 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r/UeEFhYTYgOa2MMT8rJTwoU//o= 2024-04-13T22:15:21.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r/UeEFhYTYgOa2MMT8rJTwoU//o='} 2024-04-13T22:15:21.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ae932945c4c62be-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.947 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.947 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.947 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.947 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.947 DEBUG:boto:Data: 2024-04-13T22:15:21.947 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.947 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.947 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.948 DEBUG:boto:Params: {} 2024-04-13T22:15:21.948 DEBUG:boto:Token: None 2024-04-13T22:15:21.948 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.948 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b3f9424c930d3d4-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.952 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:21.952 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:21.952 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.952 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:21.952 DEBUG:boto:Data: 2024-04-13T22:15:21.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.952 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.953 DEBUG:boto:Params: {} 2024-04-13T22:15:21.953 DEBUG:boto:Token: None 2024-04-13T22:15:21.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:21.953 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gmejM1BJF4UwI7hnrNapr1yOMv0= 2024-04-13T22:15:21.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gmejM1BJF4UwI7hnrNapr1yOMv0='} 2024-04-13T22:15:21.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edb075ec815de1ce-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.959 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:21.959 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:21.959 DEBUG:boto:Method: HEAD 2024-04-13T22:15:21.960 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:21.960 DEBUG:boto:Data: 2024-04-13T22:15:21.960 DEBUG:boto:Headers: {} 2024-04-13T22:15:21.960 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.960 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.960 DEBUG:boto:Params: {} 2024-04-13T22:15:21.960 DEBUG:boto:Token: None 2024-04-13T22:15:21.960 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/ 2024-04-13T22:15:21.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0= 2024-04-13T22:15:21.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7ru30QlFpGUhSBnIqRU/AOaON0=', 'Content-Length': '0'} 2024-04-13T22:15:21.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2a998650e05145b-00661b03f9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:21.964 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:21.964 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:21.964 DEBUG:boto:Method: PUT 2024-04-13T22:15:21.965 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:21.965 DEBUG:boto:Data: 2024-04-13T22:15:21.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:21.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:21.965 DEBUG:boto:Port: 8000 2024-04-13T22:15:21.965 DEBUG:boto:Params: {} 2024-04-13T22:15:21.965 DEBUG:boto:Token: None 2024-04-13T22:15:21.965 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:21 GMT /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:21.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YUHKT9ehgBhZEv+4ZWmqS8Y1zWs= 2024-04-13T22:15:21.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YUHKT9ehgBhZEv+4ZWmqS8Y1zWs='} 2024-04-13T22:15:22.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5cd2264c5e90bfd-00661b03f9-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.011 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.011 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.011 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.011 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.011 DEBUG:boto:Data: 2024-04-13T22:15:22.011 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.011 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.011 DEBUG:boto:Params: {} 2024-04-13T22:15:22.011 DEBUG:boto:Token: None 2024-04-13T22:15:22.012 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d342ec4f504866ab-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.015 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:22.015 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:22.015 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.015 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:22.016 DEBUG:boto:Data: 2024-04-13T22:15:22.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.016 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.016 DEBUG:boto:Params: {} 2024-04-13T22:15:22.016 DEBUG:boto:Token: None 2024-04-13T22:15:22.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:22.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+T9+5//+AL6hRRI9QJAEkAYjAAI= 2024-04-13T22:15:22.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+T9+5//+AL6hRRI9QJAEkAYjAAI='} 2024-04-13T22:15:22.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b663b81bc032485-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.022 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.023 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.023 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.023 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.023 DEBUG:boto:Data: 2024-04-13T22:15:22.023 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.023 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.023 DEBUG:boto:Params: {} 2024-04-13T22:15:22.023 DEBUG:boto:Token: None 2024-04-13T22:15:22.023 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b274ab4ec965b613-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.027 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:22.027 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:22.027 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.028 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:22.028 DEBUG:boto:Data: 2024-04-13T22:15:22.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.028 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.028 DEBUG:boto:Params: {} 2024-04-13T22:15:22.028 DEBUG:boto:Token: None 2024-04-13T22:15:22.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:22.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C55RNFAJ/sFk06N2uFZ6Wc/pBsg= 2024-04-13T22:15:22.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C55RNFAJ/sFk06N2uFZ6Wc/pBsg='} 2024-04-13T22:15:22.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000454f9911c457813f-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.071 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.071 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.071 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.071 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.071 DEBUG:boto:Data: 2024-04-13T22:15:22.071 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.071 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.071 DEBUG:boto:Params: {} 2024-04-13T22:15:22.072 DEBUG:boto:Token: None 2024-04-13T22:15:22.072 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000072c2d6e2e04b4cd-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.075 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:22.075 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:22.075 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.075 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:22.076 DEBUG:boto:Data: 2024-04-13T22:15:22.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.076 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.076 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.076 DEBUG:boto:Params: {} 2024-04-13T22:15:22.076 DEBUG:boto:Token: None 2024-04-13T22:15:22.076 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:22.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGA981CvV+vm4Eq2hzEIWuk1X24= 2024-04-13T22:15:22.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGA981CvV+vm4Eq2hzEIWuk1X24='} 2024-04-13T22:15:22.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a6b26dab7dd121e-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.083 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.083 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.083 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.083 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.083 DEBUG:boto:Data: 2024-04-13T22:15:22.083 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.084 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.084 DEBUG:boto:Params: {} 2024-04-13T22:15:22.084 DEBUG:boto:Token: None 2024-04-13T22:15:22.084 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.087 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d883f0559a22a7ca-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.088 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:22.088 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:22.088 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.088 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:22.088 DEBUG:boto:Data: 2024-04-13T22:15:22.088 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.088 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.088 DEBUG:boto:Params: {} 2024-04-13T22:15:22.088 DEBUG:boto:Token: None 2024-04-13T22:15:22.089 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:22.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TsI2+lsYlWubEaU+XlRIoFCOQIQ= 2024-04-13T22:15:22.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TsI2+lsYlWubEaU+XlRIoFCOQIQ='} 2024-04-13T22:15:22.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db8db6067b1f6312-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.131 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.131 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.131 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.131 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.131 DEBUG:boto:Data: 2024-04-13T22:15:22.131 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.131 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.131 DEBUG:boto:Params: {} 2024-04-13T22:15:22.132 DEBUG:boto:Token: None 2024-04-13T22:15:22.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002516728e351c7062-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.136 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:22.136 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:22.136 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.136 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:22.136 DEBUG:boto:Data: 2024-04-13T22:15:22.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.136 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.136 DEBUG:boto:Params: {} 2024-04-13T22:15:22.136 DEBUG:boto:Token: None 2024-04-13T22:15:22.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:22.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:seuwAelE/O69BYvMA3VNov3eo/M= 2024-04-13T22:15:22.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:seuwAelE/O69BYvMA3VNov3eo/M='} 2024-04-13T22:15:22.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092cbcb44b7ef7310-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.143 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.143 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.143 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.143 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.143 DEBUG:boto:Data: 2024-04-13T22:15:22.143 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.144 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.144 DEBUG:boto:Params: {} 2024-04-13T22:15:22.144 DEBUG:boto:Token: None 2024-04-13T22:15:22.144 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034d054fe5fb78689-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.148 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:22.148 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:22.148 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.148 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:22.148 DEBUG:boto:Data: 2024-04-13T22:15:22.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.148 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.148 DEBUG:boto:Params: {} 2024-04-13T22:15:22.149 DEBUG:boto:Token: None 2024-04-13T22:15:22.149 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:22.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HLw7qWJvyeFoL6DJUwYKiJrIaNc= 2024-04-13T22:15:22.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HLw7qWJvyeFoL6DJUwYKiJrIaNc='} 2024-04-13T22:15:22.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5d061af703c7709-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.191 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.191 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.191 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.191 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.191 DEBUG:boto:Data: 2024-04-13T22:15:22.191 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.191 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.191 DEBUG:boto:Params: {} 2024-04-13T22:15:22.192 DEBUG:boto:Token: None 2024-04-13T22:15:22.192 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fb4936d71a60519-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.196 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:22.196 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:22.196 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.196 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:22.196 DEBUG:boto:Data: 2024-04-13T22:15:22.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.196 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.197 DEBUG:boto:Params: {} 2024-04-13T22:15:22.197 DEBUG:boto:Token: None 2024-04-13T22:15:22.197 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:22.197 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p49xyRUu6cuePDbb2grcb9ZeSss= 2024-04-13T22:15:22.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p49xyRUu6cuePDbb2grcb9ZeSss='} 2024-04-13T22:15:22.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0016890d8edd2c3-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.203 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.203 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.203 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.203 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.203 DEBUG:boto:Data: 2024-04-13T22:15:22.204 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.204 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.204 DEBUG:boto:Params: {} 2024-04-13T22:15:22.204 DEBUG:boto:Token: None 2024-04-13T22:15:22.204 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072b543c7ea16adb7-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.208 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:22.208 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:22.208 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.208 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:22.208 DEBUG:boto:Data: 2024-04-13T22:15:22.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.209 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.209 DEBUG:boto:Params: {} 2024-04-13T22:15:22.209 DEBUG:boto:Token: None 2024-04-13T22:15:22.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:22.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dD3LJER/BCXnrv2zqLUCoRge6H0= 2024-04-13T22:15:22.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dD3LJER/BCXnrv2zqLUCoRge6H0='} 2024-04-13T22:15:22.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bda708f5cbe148d5-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.255 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.255 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.255 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.255 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.255 DEBUG:boto:Data: 2024-04-13T22:15:22.255 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.256 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.256 DEBUG:boto:Params: {} 2024-04-13T22:15:22.256 DEBUG:boto:Token: None 2024-04-13T22:15:22.256 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041bd9f9f097fcafd-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.260 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:22.260 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:22.260 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.260 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:22.260 DEBUG:boto:Data: 2024-04-13T22:15:22.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.261 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.261 DEBUG:boto:Params: {} 2024-04-13T22:15:22.261 DEBUG:boto:Token: None 2024-04-13T22:15:22.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:22.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:225OdWZMND7udfBc/wOmbnacbds= 2024-04-13T22:15:22.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:225OdWZMND7udfBc/wOmbnacbds='} 2024-04-13T22:15:22.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f96ce95954e8125a-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.267 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.267 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.268 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.268 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.268 DEBUG:boto:Data: 2024-04-13T22:15:22.268 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.268 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.268 DEBUG:boto:Params: {} 2024-04-13T22:15:22.268 DEBUG:boto:Token: None 2024-04-13T22:15:22.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b869053b0c1c6b3-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.272 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:22.272 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:22.272 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.272 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:22.273 DEBUG:boto:Data: 2024-04-13T22:15:22.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.273 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.273 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.273 DEBUG:boto:Params: {} 2024-04-13T22:15:22.273 DEBUG:boto:Token: None 2024-04-13T22:15:22.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:22.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ef6dFxAsRjLWc515SQSL+0MWpJU= 2024-04-13T22:15:22.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ef6dFxAsRjLWc515SQSL+0MWpJU='} 2024-04-13T22:15:22.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fc033e9e821180a-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.319 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.319 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.319 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.319 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.319 DEBUG:boto:Data: 2024-04-13T22:15:22.319 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.319 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.319 DEBUG:boto:Params: {} 2024-04-13T22:15:22.320 DEBUG:boto:Token: None 2024-04-13T22:15:22.320 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d4e6709b7655d9b-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.323 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:22.324 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:22.324 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.324 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:22.324 DEBUG:boto:Data: 2024-04-13T22:15:22.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.324 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.324 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.324 DEBUG:boto:Params: {} 2024-04-13T22:15:22.324 DEBUG:boto:Token: None 2024-04-13T22:15:22.324 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:22.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cgs9Ctm1wqZXj8vNZ9ZD7RlYpZY= 2024-04-13T22:15:22.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cgs9Ctm1wqZXj8vNZ9ZD7RlYpZY='} 2024-04-13T22:15:22.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b5f067709c827b7-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.331 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.331 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.331 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.331 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.331 DEBUG:boto:Data: 2024-04-13T22:15:22.331 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.331 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.331 DEBUG:boto:Params: {} 2024-04-13T22:15:22.332 DEBUG:boto:Token: None 2024-04-13T22:15:22.332 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000def0680676bcda8b-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.335 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:22.336 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:22.336 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.336 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:22.336 DEBUG:boto:Data: 2024-04-13T22:15:22.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.336 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.336 DEBUG:boto:Params: {} 2024-04-13T22:15:22.336 DEBUG:boto:Token: None 2024-04-13T22:15:22.337 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:22.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zWOeHd44wlflSyHSrUgqNQQPZEc= 2024-04-13T22:15:22.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zWOeHd44wlflSyHSrUgqNQQPZEc='} 2024-04-13T22:15:22.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b04cb13c34dfb15e-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.379 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.379 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.379 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.379 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.379 DEBUG:boto:Data: 2024-04-13T22:15:22.379 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.379 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.380 DEBUG:boto:Params: {} 2024-04-13T22:15:22.380 DEBUG:boto:Token: None 2024-04-13T22:15:22.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000840f105561b88a91-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.384 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:22.384 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:22.384 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.384 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:22.384 DEBUG:boto:Data: 2024-04-13T22:15:22.384 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.384 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.384 DEBUG:boto:Params: {} 2024-04-13T22:15:22.385 DEBUG:boto:Token: None 2024-04-13T22:15:22.385 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:22.385 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t/YNoxRFaSGAKxDV56gMk/Jl7Ws= 2024-04-13T22:15:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t/YNoxRFaSGAKxDV56gMk/Jl7Ws='} 2024-04-13T22:15:22.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ba28fb84c8a69d5-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.391 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.391 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.391 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.392 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.392 DEBUG:boto:Data: 2024-04-13T22:15:22.392 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.392 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.392 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.392 DEBUG:boto:Params: {} 2024-04-13T22:15:22.392 DEBUG:boto:Token: None 2024-04-13T22:15:22.392 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004a7e7a6c413700d-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.396 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:22.396 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:22.397 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.397 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:22.397 DEBUG:boto:Data: 2024-04-13T22:15:22.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.397 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.397 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.397 DEBUG:boto:Params: {} 2024-04-13T22:15:22.397 DEBUG:boto:Token: None 2024-04-13T22:15:22.397 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:22.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aNvcODkDGif+31cL643Hctsk9RU= 2024-04-13T22:15:22.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aNvcODkDGif+31cL643Hctsk9RU='} 2024-04-13T22:15:22.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eeb4ad33f4df7404-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.443 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.443 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.443 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.443 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.443 DEBUG:boto:Data: 2024-04-13T22:15:22.443 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.443 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.443 DEBUG:boto:Params: {} 2024-04-13T22:15:22.444 DEBUG:boto:Token: None 2024-04-13T22:15:22.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4a9e3f0a8c2bb85-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.447 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:22.447 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:22.448 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.448 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:22.448 DEBUG:boto:Data: 2024-04-13T22:15:22.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.448 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.448 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.448 DEBUG:boto:Params: {} 2024-04-13T22:15:22.448 DEBUG:boto:Token: None 2024-04-13T22:15:22.448 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:22.449 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ORlVsh4GbbCXK+HENGUnke1kNhI= 2024-04-13T22:15:22.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ORlVsh4GbbCXK+HENGUnke1kNhI='} 2024-04-13T22:15:22.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c38be9d11f1ae27b-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.455 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.455 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.455 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.455 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.455 DEBUG:boto:Data: 2024-04-13T22:15:22.455 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.455 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.455 DEBUG:boto:Params: {} 2024-04-13T22:15:22.456 DEBUG:boto:Token: None 2024-04-13T22:15:22.456 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e6f203e29a0b507-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.459 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:22.460 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:22.460 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.460 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:22.460 DEBUG:boto:Data: 2024-04-13T22:15:22.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.460 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.460 DEBUG:boto:Params: {} 2024-04-13T22:15:22.460 DEBUG:boto:Token: None 2024-04-13T22:15:22.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:22.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dorKHJkPyD36aybGI6vGbQHU94E= 2024-04-13T22:15:22.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dorKHJkPyD36aybGI6vGbQHU94E='} 2024-04-13T22:15:22.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7321694f60c855a-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.507 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.507 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.507 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.507 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.507 DEBUG:boto:Data: 2024-04-13T22:15:22.507 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.507 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.507 DEBUG:boto:Params: {} 2024-04-13T22:15:22.507 DEBUG:boto:Token: None 2024-04-13T22:15:22.508 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060bcc5b40fbded80-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.511 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:22.511 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:22.511 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.511 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:22.511 DEBUG:boto:Data: 2024-04-13T22:15:22.512 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.512 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.512 DEBUG:boto:Params: {} 2024-04-13T22:15:22.512 DEBUG:boto:Token: None 2024-04-13T22:15:22.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:22.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uRQf9CjXqvYMa1xAbZ5UDOD8iAc= 2024-04-13T22:15:22.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uRQf9CjXqvYMa1xAbZ5UDOD8iAc='} 2024-04-13T22:15:22.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003faefa2af299160c-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.519 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.519 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.519 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.519 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.519 DEBUG:boto:Data: 2024-04-13T22:15:22.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.519 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.520 DEBUG:boto:Params: {} 2024-04-13T22:15:22.520 DEBUG:boto:Token: None 2024-04-13T22:15:22.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000359fa65ab247619b-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.524 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:22.524 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:22.524 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.524 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:22.524 DEBUG:boto:Data: 2024-04-13T22:15:22.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.524 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.524 DEBUG:boto:Params: {} 2024-04-13T22:15:22.525 DEBUG:boto:Token: None 2024-04-13T22:15:22.525 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:22.525 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MNF9Hk+SkI0/WZuw7yVZMKTgons= 2024-04-13T22:15:22.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MNF9Hk+SkI0/WZuw7yVZMKTgons='} 2024-04-13T22:15:22.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa43b6495ddd0995-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.567 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.567 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.567 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.567 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.567 DEBUG:boto:Data: 2024-04-13T22:15:22.567 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.568 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.568 DEBUG:boto:Params: {} 2024-04-13T22:15:22.568 DEBUG:boto:Token: None 2024-04-13T22:15:22.568 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0c46b051b8d40f3-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.572 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:22.572 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:22.572 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.572 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:22.572 DEBUG:boto:Data: 2024-04-13T22:15:22.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.572 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.572 DEBUG:boto:Params: {} 2024-04-13T22:15:22.573 DEBUG:boto:Token: None 2024-04-13T22:15:22.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:22.573 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8elep1N3uNUKfOmnLN70Kr5bkNM= 2024-04-13T22:15:22.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8elep1N3uNUKfOmnLN70Kr5bkNM='} 2024-04-13T22:15:22.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cc8f647081dd449-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.578 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.578 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.578 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.578 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.578 DEBUG:boto:Data: 2024-04-13T22:15:22.579 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.579 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.579 DEBUG:boto:Params: {} 2024-04-13T22:15:22.579 DEBUG:boto:Token: None 2024-04-13T22:15:22.579 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.579 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d18a87b67593be0-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.582 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:22.583 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:22.583 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.583 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:22.583 DEBUG:boto:Data: 2024-04-13T22:15:22.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.583 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.583 DEBUG:boto:Params: {} 2024-04-13T22:15:22.583 DEBUG:boto:Token: None 2024-04-13T22:15:22.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:22.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ndxbg4RWlEcZtEaXHcuIJnVGvJk= 2024-04-13T22:15:22.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ndxbg4RWlEcZtEaXHcuIJnVGvJk='} 2024-04-13T22:15:22.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032f783e3640a41d6-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.627 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.627 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.627 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.627 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.627 DEBUG:boto:Data: 2024-04-13T22:15:22.627 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.627 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.628 DEBUG:boto:Params: {} 2024-04-13T22:15:22.628 DEBUG:boto:Token: None 2024-04-13T22:15:22.628 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c72d919be4b247e3-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.631 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:22.632 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:22.632 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.632 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:22.632 DEBUG:boto:Data: 2024-04-13T22:15:22.632 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.632 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.632 DEBUG:boto:Params: {} 2024-04-13T22:15:22.632 DEBUG:boto:Token: None 2024-04-13T22:15:22.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:22.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w3B/uE186SKlu5Najw6wGpoMIeY= 2024-04-13T22:15:22.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3B/uE186SKlu5Najw6wGpoMIeY='} 2024-04-13T22:15:22.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acdbf7c0075f9bff-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.637 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.637 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.638 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.638 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.638 DEBUG:boto:Data: 2024-04-13T22:15:22.638 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.638 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.638 DEBUG:boto:Params: {} 2024-04-13T22:15:22.638 DEBUG:boto:Token: None 2024-04-13T22:15:22.638 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d11bfc4c44357a98-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.642 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:22.642 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:22.642 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.642 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:22.642 DEBUG:boto:Data: 2024-04-13T22:15:22.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.643 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.643 DEBUG:boto:Params: {} 2024-04-13T22:15:22.643 DEBUG:boto:Token: None 2024-04-13T22:15:22.643 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:22.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0JIPLFVlGdJN1MoLkCWpFQ4+T5g= 2024-04-13T22:15:22.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0JIPLFVlGdJN1MoLkCWpFQ4+T5g='} 2024-04-13T22:15:22.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000988fbdaa279d7ad1-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.687 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.687 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.687 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.687 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.687 DEBUG:boto:Data: 2024-04-13T22:15:22.687 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.687 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.687 DEBUG:boto:Params: {} 2024-04-13T22:15:22.688 DEBUG:boto:Token: None 2024-04-13T22:15:22.688 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000679f694686f53469-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.691 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:22.692 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:22.692 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.692 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:22.692 DEBUG:boto:Data: 2024-04-13T22:15:22.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.692 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.692 DEBUG:boto:Params: {} 2024-04-13T22:15:22.692 DEBUG:boto:Token: None 2024-04-13T22:15:22.693 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:22.693 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZR/Smflj706j+zIhbIASjFxTBKU= 2024-04-13T22:15:22.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZR/Smflj706j+zIhbIASjFxTBKU='} 2024-04-13T22:15:22.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f633053b4670c45f-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.698 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.698 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.698 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.698 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.698 DEBUG:boto:Data: 2024-04-13T22:15:22.698 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.698 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.698 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.699 DEBUG:boto:Params: {} 2024-04-13T22:15:22.699 DEBUG:boto:Token: None 2024-04-13T22:15:22.699 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052ec1f0ccc25fc69-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.702 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:22.703 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:22.703 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.703 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:22.703 DEBUG:boto:Data: 2024-04-13T22:15:22.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.703 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.703 DEBUG:boto:Params: {} 2024-04-13T22:15:22.703 DEBUG:boto:Token: None 2024-04-13T22:15:22.703 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:22.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ddo+nHm6+bp6+ju8UlTHQOGnXAQ= 2024-04-13T22:15:22.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ddo+nHm6+bp6+ju8UlTHQOGnXAQ='} 2024-04-13T22:15:22.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000353c91ede22fb2b3-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.746 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.747 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.747 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.747 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.747 DEBUG:boto:Data: 2024-04-13T22:15:22.747 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.747 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.747 DEBUG:boto:Params: {} 2024-04-13T22:15:22.747 DEBUG:boto:Token: None 2024-04-13T22:15:22.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004001c946ea848a20-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.751 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:22.751 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:22.752 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.752 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:22.752 DEBUG:boto:Data: 2024-04-13T22:15:22.752 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.752 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.752 DEBUG:boto:Params: {} 2024-04-13T22:15:22.752 DEBUG:boto:Token: None 2024-04-13T22:15:22.752 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:22.753 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I/29j3dPWBdbR8isRqbTbuO75ak= 2024-04-13T22:15:22.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I/29j3dPWBdbR8isRqbTbuO75ak='} 2024-04-13T22:15:22.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042e3d0a869a193c4-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.758 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.759 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.759 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.759 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.759 DEBUG:boto:Data: 2024-04-13T22:15:22.759 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.759 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.759 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.759 DEBUG:boto:Params: {} 2024-04-13T22:15:22.759 DEBUG:boto:Token: None 2024-04-13T22:15:22.759 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.760 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042503dc83ad07e6d-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.763 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:22.763 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:22.763 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.763 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:22.763 DEBUG:boto:Data: 2024-04-13T22:15:22.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.764 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.764 DEBUG:boto:Params: {} 2024-04-13T22:15:22.764 DEBUG:boto:Token: None 2024-04-13T22:15:22.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:22.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKaWhuro2CBJ2OZINKpVrkVv3Xo= 2024-04-13T22:15:22.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKaWhuro2CBJ2OZINKpVrkVv3Xo='} 2024-04-13T22:15:22.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fb53f3de9c37148-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.807 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.807 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.807 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.807 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.807 DEBUG:boto:Data: 2024-04-13T22:15:22.807 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.807 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.807 DEBUG:boto:Params: {} 2024-04-13T22:15:22.807 DEBUG:boto:Token: None 2024-04-13T22:15:22.807 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017e6fec29d2385e3-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.811 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:22.811 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:22.812 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.812 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:22.812 DEBUG:boto:Data: 2024-04-13T22:15:22.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.812 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.812 DEBUG:boto:Params: {} 2024-04-13T22:15:22.812 DEBUG:boto:Token: None 2024-04-13T22:15:22.812 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:22.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uIeQlU2B49U3KhSbJ+uf57Je3Xg= 2024-04-13T22:15:22.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uIeQlU2B49U3KhSbJ+uf57Je3Xg='} 2024-04-13T22:15:22.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a2e38bcf866e333-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.819 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.819 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.819 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.819 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.819 DEBUG:boto:Data: 2024-04-13T22:15:22.819 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.819 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.819 DEBUG:boto:Params: {} 2024-04-13T22:15:22.819 DEBUG:boto:Token: None 2024-04-13T22:15:22.820 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000158a5df588142135-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.823 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:22.823 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:22.824 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.824 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:22.824 DEBUG:boto:Data: 2024-04-13T22:15:22.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.824 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.824 DEBUG:boto:Params: {} 2024-04-13T22:15:22.824 DEBUG:boto:Token: None 2024-04-13T22:15:22.824 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:22.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NpafMFKzwm84e5qnfrhFS9JUBt0= 2024-04-13T22:15:22.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NpafMFKzwm84e5qnfrhFS9JUBt0='} 2024-04-13T22:15:22.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047cb0394a38556a9-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.867 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.867 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.867 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.867 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.867 DEBUG:boto:Data: 2024-04-13T22:15:22.867 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.867 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.867 DEBUG:boto:Params: {} 2024-04-13T22:15:22.867 DEBUG:boto:Token: None 2024-04-13T22:15:22.868 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf82b53a9eba151-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.871 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:22.871 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:22.871 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.871 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:22.872 DEBUG:boto:Data: 2024-04-13T22:15:22.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.872 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.872 DEBUG:boto:Params: {} 2024-04-13T22:15:22.872 DEBUG:boto:Token: None 2024-04-13T22:15:22.872 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:22.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c7KUBvX2GBxBWdN9JVr2mqlepQ4= 2024-04-13T22:15:22.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c7KUBvX2GBxBWdN9JVr2mqlepQ4='} 2024-04-13T22:15:22.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000621d42d3d5a93884-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.879 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.879 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.879 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.879 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.879 DEBUG:boto:Data: 2024-04-13T22:15:22.879 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.880 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.880 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.880 DEBUG:boto:Params: {} 2024-04-13T22:15:22.880 DEBUG:boto:Token: None 2024-04-13T22:15:22.880 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026327c6a6042df1d-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.883 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:22.883 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:22.884 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.884 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:22.884 DEBUG:boto:Data: 2024-04-13T22:15:22.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.884 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.884 DEBUG:boto:Params: {} 2024-04-13T22:15:22.884 DEBUG:boto:Token: None 2024-04-13T22:15:22.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:22.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bp5ArQ0Vq6igQn1EiugrmufgLhc= 2024-04-13T22:15:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bp5ArQ0Vq6igQn1EiugrmufgLhc='} 2024-04-13T22:15:22.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6cd509b279c4c45-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.927 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:22.927 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:22.927 DEBUG:boto:Method: HEAD 2024-04-13T22:15:22.927 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:22.927 DEBUG:boto:Data: 2024-04-13T22:15:22.927 DEBUG:boto:Headers: {} 2024-04-13T22:15:22.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.927 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.927 DEBUG:boto:Params: {} 2024-04-13T22:15:22.928 DEBUG:boto:Token: None 2024-04-13T22:15:22.928 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/ 2024-04-13T22:15:22.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0= 2024-04-13T22:15:22.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GIvhLFP66rKaCSa/ctBhqGiEE0=', 'Content-Length': '0'} 2024-04-13T22:15:22.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005237b6a7c6941f7e-00661b03fa-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.931 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:22.931 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:22.932 DEBUG:boto:Method: PUT 2024-04-13T22:15:22.932 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:22.932 DEBUG:boto:Data: 2024-04-13T22:15:22.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:15:22.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:22.932 DEBUG:boto:Port: 8000 2024-04-13T22:15:22.932 DEBUG:boto:Params: {} 2024-04-13T22:15:22.932 DEBUG:boto:Token: None 2024-04-13T22:15:22.932 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:15:22 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:22.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DEdqqSXxVdFUiTy0wboLUkG9MiQ= 2024-04-13T22:15:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:15:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DEdqqSXxVdFUiTy0wboLUkG9MiQ='} 2024-04-13T22:15:22.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd8c7f9c4fed5c11-00661b03fa-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:22.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:22.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:15:22.939 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:15:22.991 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:22.991 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:23.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 2 2024-04-13T22:15:23.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 2 2024-04-13T22:15:23.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000001.29.6 2024-04-13T22:15:23.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 2 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 2 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 2 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.057+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.058+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.058+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.058+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 2 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.058+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000001.65.6 2024-04-13T22:15:23.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 2 2024-04-13T22:15:23.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position=00000000001.30.6 2024-04-13T22:15:23.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 2 2024-04-13T22:15:23.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position=00000000001.39.6 2024-04-13T22:15:23.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 2 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000001.86.6 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 2 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.061+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 2 2024-04-13T22:15:23.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.061+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position=00000000001.46.6 2024-04-13T22:15:23.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.109+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.109+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:23.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 0 2024-04-13T22:15:23.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:23.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.111+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 0 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: parsed entry: id=00000000001.33.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4/obj00000000000000000028 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: entry->timestamp=2024-04-13T22:15:17.447761+0000 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.33.6 op=0 op_state=1 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000028[0] 2024-04-13T22:15:23.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000001.33.6 2024-04-13T22:15:23.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000001 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:17.772266+0000 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000001 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket shard: 0 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.113+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket shard: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket shard: 0 2024-04-13T22:15:23.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.114+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.116+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: parsed entry: id=00000000001.33.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-13T22:15:23.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.116+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6/obj00000000000000000032 2024-04-13T22:15:23.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.116+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: entry->timestamp=2024-04-13T22:15:17.833063+0000 2024-04-13T22:15:23.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.116+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.33.6 op=0 op_state=1 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.116+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000032[0] 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.116+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position=00000000001.33.6 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: parsed entry: id=00000000001.65.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16/obj00000000000000000076 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: entry->timestamp=2024-04-13T22:15:17.587837+0000 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.65.6 op=0 op_state=1 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000076[0] 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position=00000000001.65.6 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11/obj00000000000000000021 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: entry->timestamp=2024-04-13T22:15:17.645705+0000 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.46.6 op=0 op_state=1 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000021[0] 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position=00000000001.46.6 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-13T22:15:23.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000013 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:17.842535+0000 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000013[0] 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket shard: 0 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.117+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: a sync operation returned error 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: backing out with sync_status=-5 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-13T22:15:23.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8/obj00000000000000000010 2024-04-13T22:15:23.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: entry->timestamp=2024-04-13T22:15:17.512792+0000 2024-04-13T22:15:23.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-13T22:15:23.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000010[0] 2024-04-13T22:15:23.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.118+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-13T22:15:23.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.123+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.123+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.123+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000028]: success 2024-04-13T22:15:23.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4:1 marker=00000000001.33.6 timestamp=2024-04-13T22:15:17.447761+0000 2024-04-13T22:15:23.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000013]: success 2024-04-13T22:15:23.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10:1 marker=00000000002.55.6 timestamp=2024-04-13T22:15:17.842535+0000 2024-04-13T22:15:23.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000032]: success 2024-04-13T22:15:23.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6:1 marker=00000000001.33.6 timestamp=2024-04-13T22:15:17.833063+0000 2024-04-13T22:15:23.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000010]: success 2024-04-13T22:15:23.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.128+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8:1 marker=00000000002.50.6 timestamp=2024-04-13T22:15:17.512792+0000 2024-04-13T22:15:23.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:23.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000016 2024-04-13T22:15:23.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: entry->timestamp=2024-04-13T22:15:17.578017+0000 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2/obj00000000000000000024 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: entry->timestamp=2024-04-13T22:15:17.462278+0000 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000007 2024-04-13T22:15:23.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:17.782920+0000 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.47.6 op=0 op_state=1 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000016 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000024[0] 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000007[0] 2024-04-13T22:15:23.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.153+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.155+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.155+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9/obj00000000000000000019 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.155+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: entry->timestamp=2024-04-13T22:15:17.523772+0000 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.155+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.52.6 op=0 op_state=1 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.155+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000019[0] 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.155+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position=00000000001.52.6 2024-04-13T22:15:23.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000000 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:17.711211+0000 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: : injecting data sync error on key=obj00000000000000000000 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.156+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000000 2024-04-13T22:15:23.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.158+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-13T22:15:23.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.158+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15/obj00000000000000000004 2024-04-13T22:15:23.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.158+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: entry->timestamp=2024-04-13T22:15:17.699738+0000 2024-04-13T22:15:23.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.158+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-13T22:15:23.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.158+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000004[0] 2024-04-13T22:15:23.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.158+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-13T22:15:23.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-13T22:15:23.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000026 2024-04-13T22:15:23.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: entry->timestamp=2024-04-13T22:15:17.396959+0000 2024-04-13T22:15:23.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: : injecting data sync error on key=obj00000000000000000026 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000030 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:17.635791+0000 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.51.6 op=0 op_state=1 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000026 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000030[0] 2024-04-13T22:15:23.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.160+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-13T22:15:23.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.162+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-13T22:15:23.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.162+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14/obj00000000000000000033 2024-04-13T22:15:23.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.162+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: entry->timestamp=2024-04-13T22:15:17.722335+0000 2024-04-13T22:15:23.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.162+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-13T22:15:23.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.162+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000033[0] 2024-04-13T22:15:23.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.162+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000001 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:17.772266+0000 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000001 2024-04-13T22:15:23.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000019]: success 2024-04-13T22:15:23.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.165+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9:1 marker=00000000001.52.6 timestamp=2024-04-13T22:15:17.523772+0000 2024-04-13T22:15:23.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.170+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000030]: success 2024-04-13T22:15:23.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.170+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12:1 marker=00000000001.51.6 timestamp=2024-04-13T22:15:17.635791+0000 2024-04-13T22:15:23.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.170+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000076]: success 2024-04-13T22:15:23.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.170+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16:1 marker=00000000001.65.6 timestamp=2024-04-13T22:15:17.587837+0000 2024-04-13T22:15:23.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.170+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000021]: success 2024-04-13T22:15:23.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.170+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11:1 marker=00000000001.46.6 timestamp=2024-04-13T22:15:17.645705+0000 2024-04-13T22:15:23.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.197+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: a sync operation returned error 2024-04-13T22:15:23.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.197+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: backing out with sync_status=-5 2024-04-13T22:15:23.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.197+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.197+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.197+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.198+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: a sync operation returned error 2024-04-13T22:15:23.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.198+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: backing out with sync_status=-5 2024-04-13T22:15:23.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.198+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.198+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.198+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:23.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.198+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.199+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.199+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.199+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.199+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000007]: success 2024-04-13T22:15:23.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.199+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0:1 marker=00000000001.47.6 timestamp=2024-04-13T22:15:17.782920+0000 2024-04-13T22:15:23.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.200+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000024]: success 2024-04-13T22:15:23.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.200+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2:1 marker=00000000001.45.6 timestamp=2024-04-13T22:15:17.462278+0000 2024-04-13T22:15:23.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.202+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: a sync operation returned error 2024-04-13T22:15:23.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.202+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: backing out with sync_status=-5 2024-04-13T22:15:23.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.202+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.202+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.203+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.204+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:23.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.204+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.204+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000004]: success 2024-04-13T22:15:23.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.204+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15:1 marker=00000000002.48.6 timestamp=2024-04-13T22:15:17.699738+0000 2024-04-13T22:15:23.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.207+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: a sync operation returned error 2024-04-13T22:15:23.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.207+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: backing out with sync_status=-5 2024-04-13T22:15:23.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.207+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.207+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.207+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.208+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.208+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.211+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000033]: success 2024-04-13T22:15:23.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.211+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14:1 marker=00000000001.45.6 timestamp=2024-04-13T22:15:17.722335+0000 2024-04-13T22:15:23.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.240+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:23.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.240+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000016 2024-04-13T22:15:23.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.240+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: entry->timestamp=2024-04-13T22:15:17.578017+0000 2024-04-13T22:15:23.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.240+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-13T22:15:23.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.240+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000016[0] 2024-04-13T22:15:23.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.240+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-13T22:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.242+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.242+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000000 2024-04-13T22:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.242+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:17.711211+0000 2024-04-13T22:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.242+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-13T22:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.242+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000000[0] 2024-04-13T22:15:23.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.242+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-13T22:15:23.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-13T22:15:23.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000026 2024-04-13T22:15:23.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: entry->timestamp=2024-04-13T22:15:17.396959+0000 2024-04-13T22:15:23.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-13T22:15:23.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000026[0] 2024-04-13T22:15:23.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-13T22:15:23.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000000]: success 2024-04-13T22:15:23.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.246+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5:1 marker=00000000002.54.6 timestamp=2024-04-13T22:15:17.711211+0000 2024-04-13T22:15:23.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.250+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:23.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.250+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000001 2024-04-13T22:15:23.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.250+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:17.772266+0000 2024-04-13T22:15:23.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.250+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-13T22:15:23.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.250+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000001[0] 2024-04-13T22:15:23.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.250+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-13T22:15:23.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.254+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000001]: success 2024-04-13T22:15:23.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.254+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7:1 marker=00000000002.47.6 timestamp=2024-04-13T22:15:17.772266+0000 2024-04-13T22:15:23.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.284+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000016]: success 2024-04-13T22:15:23.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.284+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1:1 marker=00000000001.40.6 timestamp=2024-04-13T22:15:17.578017+0000 2024-04-13T22:15:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.292+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000026]: success 2024-04-13T22:15:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.292+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13:1 marker=00000000001.43.6 timestamp=2024-04-13T22:15:17.396959+0000 2024-04-13T22:15:23.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:23.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.298+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:23.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.299+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 0 2024-04-13T22:15:23.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 0 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.300+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 0 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000010 2024-04-13T22:15:23.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: entry->timestamp=2024-04-13T22:15:18.531761+0000 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000010[0] 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.301+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000015 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:18.472763+0000 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.60.6 op=0 op_state=1 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000015[0] 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000002.60.6 2024-04-13T22:15:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6/obj00000000000000000030 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: entry->timestamp=2024-04-13T22:15:18.205430+0000 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000030[0] 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:23.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.302+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:23.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.303+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000021 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: entry->timestamp=2024-04-13T22:15:18.262755+0000 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.71.6 op=0 op_state=1 2024-04-13T22:15:23.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000021[0] 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000002.71.6 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000010]: success 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1:2 marker=00000000002.68.6 timestamp=2024-04-13T22:15:18.531761+0000 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11/obj00000000000000000014 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: entry->timestamp=2024-04-13T22:15:18.590345+0000 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.73.6 op=0 op_state=1 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16/obj00000000000000000011 2024-04-13T22:15:23.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: entry->timestamp=2024-04-13T22:15:18.520134+0000 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000014[0] 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position=00000000001.73.6 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000011[0] 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.305+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position=00000000002.91.6 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000009 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:18.457442+0000 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.47.6 op=0 op_state=1 2024-04-13T22:15:23.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000009[0] 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000030]: success 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.306+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6:2 marker=00000000002.50.6 timestamp=2024-04-13T22:15:18.205430+0000 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.307+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.307+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17/obj00000000000000000001 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.307+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: entry->timestamp=2024-04-13T22:15:18.407346+0000 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.307+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.58.6 op=0 op_state=1 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.307+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000001[0] 2024-04-13T22:15:23.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.307+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position=00000000002.58.6 2024-04-13T22:15:23.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-13T22:15:23.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15/obj00000000000000000006 2024-04-13T22:15:23.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: entry->timestamp=2024-04-13T22:15:18.251934+0000 2024-04-13T22:15:23.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-13T22:15:23.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000006[0] 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000007 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:18.323903+0000 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.38.6 op=0 op_state=1 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000002 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: entry->timestamp=2024-04-13T22:15:18.335228+0000 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.34.6 op=0 op_state=1 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000007[0] 2024-04-13T22:15:23.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.38.6 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000002[0] 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.308+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position=00000000002.34.6 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.309+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.309+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000012 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.309+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:18.137444+0000 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.309+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.309+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000012[0] 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.309+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000002.91.6 2024-04-13T22:15:23.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.310+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000021]: success 2024-04-13T22:15:23.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.310+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3:2 marker=00000000002.71.6 timestamp=2024-04-13T22:15:18.262755+0000 2024-04-13T22:15:23.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.313+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000007]: success 2024-04-13T22:15:23.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.313+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18:2 marker=00000000002.38.6 timestamp=2024-04-13T22:15:18.323903+0000 2024-04-13T22:15:23.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.315+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000001]: success 2024-04-13T22:15:23.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.315+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17:2 marker=00000000002.58.6 timestamp=2024-04-13T22:15:18.407346+0000 2024-04-13T22:15:23.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.318+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000011]: success 2024-04-13T22:15:23.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.318+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16:2 marker=00000000002.91.6 timestamp=2024-04-13T22:15:18.520134+0000 2024-04-13T22:15:23.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.319+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000012]: success 2024-04-13T22:15:23.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.319+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12:2 marker=00000000002.91.6 timestamp=2024-04-13T22:15:18.137444+0000 2024-04-13T22:15:23.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.321+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000002]: success 2024-04-13T22:15:23.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.321+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13:2 marker=00000000002.34.6 timestamp=2024-04-13T22:15:18.335228+0000 2024-04-13T22:15:23.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.321+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000006]: success 2024-04-13T22:15:23.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.321+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15:2 marker=00000000002.54.6 timestamp=2024-04-13T22:15:18.251934+0000 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000005 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:18.148367+0000 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000005[0] 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000002.61.6 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2/obj00000000000000000016 2024-04-13T22:15:23.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: entry->timestamp=2024-04-13T22:15:18.396379+0000 2024-04-13T22:15:23.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-04-13T22:15:23.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000016[0] 2024-04-13T22:15:23.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.340+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000002.52.6 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4/obj00000000000000000004 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: entry->timestamp=2024-04-13T22:15:18.346935+0000 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000004[0] 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-13T22:15:23.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000025 2024-04-13T22:15:23.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:18.195509+0000 2024-04-13T22:15:23.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.35.6 op=0 op_state=1 2024-04-13T22:15:23.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000025[0] 2024-04-13T22:15:23.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000001.35.6 2024-04-13T22:15:23.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000016]: success 2024-04-13T22:15:23.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.342+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2:2 marker=00000000002.52.6 timestamp=2024-04-13T22:15:18.396379+0000 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.343+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.343+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8/obj00000000000000000019 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.343+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: entry->timestamp=2024-04-13T22:15:18.579363+0000 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.343+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.40.6 op=0 op_state=1 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.343+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000019[0] 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.343+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position=00000000002.40.6 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.344+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000004]: success 2024-04-13T22:15:23.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.344+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4:2 marker=00000000002.65.6 timestamp=2024-04-13T22:15:18.346935+0000 2024-04-13T22:15:23.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.346+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000019]: success 2024-04-13T22:15:23.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.346+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8:2 marker=00000000002.40.6 timestamp=2024-04-13T22:15:18.579363+0000 2024-04-13T22:15:23.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.348+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:23.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.348+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14/obj00000000000000000000 2024-04-13T22:15:23.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.348+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: entry->timestamp=2024-04-13T22:15:18.273563+0000 2024-04-13T22:15:23.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.348+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.45.6 op=0 op_state=1 2024-04-13T22:15:23.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.348+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000000[0] 2024-04-13T22:15:23.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.348+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position=00000000002.45.6 2024-04-13T22:15:23.352 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.349+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000015]: success 2024-04-13T22:15:23.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.349+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7:2 marker=00000000002.60.6 timestamp=2024-04-13T22:15:18.472763+0000 2024-04-13T22:15:23.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.350+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000000]: success 2024-04-13T22:15:23.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.350+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14:2 marker=00000000002.45.6 timestamp=2024-04-13T22:15:18.273563+0000 2024-04-13T22:15:23.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.357+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000014]: success 2024-04-13T22:15:23.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.357+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11:2 marker=00000000001.73.6 timestamp=2024-04-13T22:15:18.590345+0000 2024-04-13T22:15:23.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.357+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000009]: success 2024-04-13T22:15:23.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.357+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10:2 marker=00000000001.47.6 timestamp=2024-04-13T22:15:18.457442+0000 2024-04-13T22:15:23.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.386+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000005]: success 2024-04-13T22:15:23.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.386+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0:2 marker=00000000002.61.6 timestamp=2024-04-13T22:15:18.148367+0000 2024-04-13T22:15:23.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.388+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000025]: success 2024-04-13T22:15:23.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.388+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5:2 marker=00000000001.35.6 timestamp=2024-04-13T22:15:18.195509+0000 2024-04-13T22:15:23.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.395+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:23.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 0 2024-04-13T22:15:23.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 0 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.396+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.397+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 0 2024-04-13T22:15:23.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.398+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket shard: 0 2024-04-13T22:15:23.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket shard: 0 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.399+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.400+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: parsed entry: id=00000000002.134.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.400+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9/obj00000000000000000007 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.400+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: entry->timestamp=2024-04-13T22:15:18.845087+0000 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.400+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.134.6 op=0 op_state=1 2024-04-13T22:15:23.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.400+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000007[0] 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.400+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position=00000000002.134.6 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6/obj00000000000000000032 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: entry->timestamp=2024-04-13T22:15:19.327899+0000 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000032[0] 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-13T22:15:23.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11/obj00000000000000000040 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: entry->timestamp=2024-04-13T22:15:19.354407+0000 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.72.6 op=0 op_state=1 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000002 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:19.142923+0000 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.401+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000040[0] 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position=00000000001.72.6 2024-04-13T22:15:23.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000009 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:18.903658+0000 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14/obj00000000000000000013 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: entry->timestamp=2024-04-13T22:15:18.892485+0000 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000002[0] 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000009[0] 2024-04-13T22:15:23.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000013[0] 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.402+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position=00000000002.98.6 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket shard: 0 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket shard: 0 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket shard: 0 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket shard: 0 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.403+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000007]: success 2024-04-13T22:15:23.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9:3 marker=00000000002.134.6 timestamp=2024-04-13T22:15:18.845087+0000 2024-04-13T22:15:23.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket shard: 0 2024-04-13T22:15:23.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket shard: 0 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket shard: 0 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.404+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000032]: success 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6:3 marker=00000000002.35.6 timestamp=2024-04-13T22:15:19.327899+0000 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16/obj00000000000000000033 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: entry->timestamp=2024-04-13T22:15:18.961908+0000 2024-04-13T22:15:23.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.96.6 op=0 op_state=1 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000033[0] 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.407+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position=00000000002.96.6 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000000 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:19.022288+0000 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.46.6 op=0 op_state=1 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000000[0] 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.46.6 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19/obj00000000000000000021 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: entry->timestamp=2024-04-13T22:15:19.082762+0000 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22/obj00000000000000000008 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: entry->timestamp=2024-04-13T22:15:19.203942+0000 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.45.6 op=0 op_state=1 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000002]: success 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7:3 marker=00000000002.55.6 timestamp=2024-04-13T22:15:19.142923+0000 2024-04-13T22:15:23.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000013]: success 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14:3 marker=00000000002.98.6 timestamp=2024-04-13T22:15:18.892485+0000 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000009]: success 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12:3 marker=00000000002.51.6 timestamp=2024-04-13T22:15:18.903658+0000 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21/obj00000000000000000055 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: entry->timestamp=2024-04-13T22:15:19.338310+0000 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000021[0] 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-13T22:15:23.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20/obj00000000000000000003 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: entry->timestamp=2024-04-13T22:15:19.213419+0000 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.408+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000008[0] 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position=00000000002.45.6 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000055[0] 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position=00000000001.71.6 2024-04-13T22:15:23.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000003[0] 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15/obj00000000000000000012 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: entry->timestamp=2024-04-13T22:15:19.259585+0000 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.79.6 op=0 op_state=1 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000012[0] 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.409+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position=00000000002.79.6 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.411+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000033]: success 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.411+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16:3 marker=00000000002.96.6 timestamp=2024-04-13T22:15:18.961908+0000 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.412+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000021]: success 2024-04-13T22:15:23.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.412+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19:3 marker=00000000002.39.6 timestamp=2024-04-13T22:15:19.082762+0000 2024-04-13T22:15:23.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.413+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000000]: success 2024-04-13T22:15:23.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.413+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18:3 marker=00000000002.46.6 timestamp=2024-04-13T22:15:19.022288+0000 2024-04-13T22:15:23.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.414+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000012]: success 2024-04-13T22:15:23.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.414+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15:3 marker=00000000002.79.6 timestamp=2024-04-13T22:15:19.259585+0000 2024-04-13T22:15:23.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.416+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000040]: success 2024-04-13T22:15:23.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.416+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11:3 marker=00000000001.72.6 timestamp=2024-04-13T22:15:19.354407+0000 2024-04-13T22:15:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.438+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2024-04-13T22:15:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.438+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000058 2024-04-13T22:15:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.438+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:19.280249+0000 2024-04-13T22:15:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.438+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.48.6 op=0 op_state=1 2024-04-13T22:15:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.438+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000058[0] 2024-04-13T22:15:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.438+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000001.48.6 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4/obj00000000000000000014 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: entry->timestamp=2024-04-13T22:15:19.153612+0000 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000014[0] 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2/obj00000000000000000011 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: entry->timestamp=2024-04-13T22:15:19.131690+0000 2024-04-13T22:15:23.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000046 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: entry->timestamp=2024-04-13T22:15:19.270861+0000 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000011[0] 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000046[0] 2024-04-13T22:15:23.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.439+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-13T22:15:23.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.440+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-13T22:15:23.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.440+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000050 2024-04-13T22:15:23.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.440+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:19.408150+0000 2024-04-13T22:15:23.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.440+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.60.6 op=0 op_state=1 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.440+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000050[0] 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.440+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000001.60.6 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.441+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.441+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8/obj00000000000000000039 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.441+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: entry->timestamp=2024-04-13T22:15:19.012582+0000 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.441+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.72.6 op=0 op_state=1 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.441+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000039[0] 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.441+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position=00000000001.72.6 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.442+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000011]: success 2024-04-13T22:15:23.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.442+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2:3 marker=00000000002.59.6 timestamp=2024-04-13T22:15:19.131690+0000 2024-04-13T22:15:23.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.443+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000014]: success 2024-04-13T22:15:23.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.443+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4:3 marker=00000000002.48.6 timestamp=2024-04-13T22:15:19.153612+0000 2024-04-13T22:15:23.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.444+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-13T22:15:23.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.444+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000026 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.444+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: entry->timestamp=2024-04-13T22:15:19.072007+0000 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.444+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.30.6 op=0 op_state=1 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.444+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000026[0] 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.444+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000002.30.6 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.445+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.445+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000045 2024-04-13T22:15:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.445+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: entry->timestamp=2024-04-13T22:15:18.951619+0000 2024-04-13T22:15:23.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.445+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.37.6 op=0 op_state=1 2024-04-13T22:15:23.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.445+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000045[0] 2024-04-13T22:15:23.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.445+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position=00000000001.37.6 2024-04-13T22:15:23.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.446+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000026]: success 2024-04-13T22:15:23.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.446+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1:3 marker=00000000002.30.6 timestamp=2024-04-13T22:15:19.072007+0000 2024-04-13T22:15:23.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.446+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000046]: success 2024-04-13T22:15:23.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.446+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3:3 marker=00000000001.45.6 timestamp=2024-04-13T22:15:19.270861+0000 2024-04-13T22:15:23.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.448+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000050]: success 2024-04-13T22:15:23.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.448+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5:3 marker=00000000001.60.6 timestamp=2024-04-13T22:15:19.408150+0000 2024-04-13T22:15:23.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.460+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000003]: success 2024-04-13T22:15:23.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.460+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20:3 marker=00000000002.66.6 timestamp=2024-04-13T22:15:19.213419+0000 2024-04-13T22:15:23.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.460+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000055]: success 2024-04-13T22:15:23.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.460+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21:3 marker=00000000001.71.6 timestamp=2024-04-13T22:15:19.338310+0000 2024-04-13T22:15:23.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.462+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000008]: success 2024-04-13T22:15:23.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.462+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22:3 marker=00000000002.45.6 timestamp=2024-04-13T22:15:19.203942+0000 2024-04-13T22:15:23.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.485+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000058]: success 2024-04-13T22:15:23.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.485+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0:3 marker=00000000001.48.6 timestamp=2024-04-13T22:15:19.280249+0000 2024-04-13T22:15:23.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.489+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000039]: success 2024-04-13T22:15:23.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.489+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8:3 marker=00000000001.72.6 timestamp=2024-04-13T22:15:19.012582+0000 2024-04-13T22:15:23.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.493+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000045]: success 2024-04-13T22:15:23.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.493+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13:3 marker=00000000001.37.6 timestamp=2024-04-13T22:15:18.951619+0000 2024-04-13T22:15:23.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.508+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.509+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 0 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 0 2024-04-13T22:15:23.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.510+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket shard: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 0 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.511+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 0 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.512+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket shard: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket shard: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket shard: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket shard: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket shard: 0 2024-04-13T22:15:23.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.513+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.514+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.514+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket shard: 0 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket shard: 0 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: sync status for source bucket shard: 0 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket shard: 0 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-04-13T22:15:23.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11/obj00000000000000000031 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: entry->timestamp=2024-04-13T22:15:19.690046+0000 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.37.6 op=0 op_state=1 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000015 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:20.345238+0000 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9/obj00000000000000000008 2024-04-13T22:15:23.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: entry->timestamp=2024-04-13T22:15:19.800354+0000 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8/obj00000000000000000017 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: entry->timestamp=2024-04-13T22:15:19.920415+0000 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000022 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:20.415744+0000 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-13T22:15:23.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000031[0] 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position=00000000001.37.6 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: : injecting data sync error on key=obj00000000000000000015 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000008[0] 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000017[0] 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position=00000000001.71.6 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.515+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-13T22:15:23.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000015 2024-04-13T22:15:23.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000022 2024-04-13T22:15:23.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: sync status for source bucket shard: 0 2024-04-13T22:15:23.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket shard: 0 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.516+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: sync status for source bucket shard: 0 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: sync status for source bucket shard: 0 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000013 2024-04-13T22:15:23.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: entry->timestamp=2024-04-13T22:15:19.980740+0000 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000013[0] 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000003 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:20.111735+0000 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2024-04-13T22:15:23.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.517+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000003 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.519+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket shard: 0 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.519+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21/obj00000000000000000001 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: entry->timestamp=2024-04-13T22:15:20.220084+0000 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.44.6 op=0 op_state=1 2024-04-13T22:15:23.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000001[0] 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position=00000000002.44.6 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23/obj00000000000000000025 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: entry->timestamp=2024-04-13T22:15:20.170421+0000 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.78.6 op=0 op_state=1 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22/obj00000000000000000009 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: entry->timestamp=2024-04-13T22:15:19.860547+0000 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-13T22:15:23.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000025[0] 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: listing bilog for incremental sync; position=00000000002.78.6 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000009[0] 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.520+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25/obj00000000000000000000 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: entry->timestamp=2024-04-13T22:15:20.041001+0000 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000000[0] 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-13T22:15:23.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27/obj00000000000000000030 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: entry->timestamp=2024-04-13T22:15:19.740297+0000 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000016 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: entry->timestamp=2024-04-13T22:15:20.053025+0000 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000030[0] 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-13T22:15:23.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000016[0] 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000008]: success 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.522+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9:4 marker=00000000002.54.6 timestamp=2024-04-13T22:15:19.800354+0000 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000013]: success 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13:4 marker=00000000002.62.6 timestamp=2024-04-13T22:15:19.980740+0000 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: a sync operation returned error 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: backing out with sync_status=-5 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000001]: success 2024-04-13T22:15:23.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.523+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21:4 marker=00000000002.44.6 timestamp=2024-04-13T22:15:20.220084+0000 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.525+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:entry[obj00000000000000000000]: success 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.525+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25:4 marker=00000000002.53.6 timestamp=2024-04-13T22:15:20.041001+0000 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.525+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: a sync operation returned error 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: backing out with sync_status=-5 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000025]: success 2024-04-13T22:15:23.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23:4 marker=00000000002.78.6 timestamp=2024-04-13T22:15:20.170421+0000 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000016]: success 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24:4 marker=00000000002.73.6 timestamp=2024-04-13T22:15:20.053025+0000 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000009]: success 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22:4 marker=00000000002.55.6 timestamp=2024-04-13T22:15:19.860547+0000 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000030]: success 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.526+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27:4 marker=00000000002.74.6 timestamp=2024-04-13T22:15:19.740297+0000 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.527+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.527+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.529+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.529+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:23.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.530+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000004 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: entry->timestamp=2024-04-13T22:15:20.234020+0000 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000002.80.6 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000059 2024-04-13T22:15:23.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: entry->timestamp=2024-04-13T22:15:19.993097+0000 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.51.6 op=0 op_state=1 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6/obj00000000000000000019 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: entry->timestamp=2024-04-13T22:15:20.284610+0000 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000004 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000059[0] 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-13T22:15:23.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000019[0] 2024-04-13T22:15:23.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.551+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-13T22:15:23.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-13T22:15:23.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000006 2024-04-13T22:15:23.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:20.296537+0000 2024-04-13T22:15:23.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000006[0] 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000002 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:19.752157+0000 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.552+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000002 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2/obj00000000000000000027 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: entry->timestamp=2024-04-13T22:15:19.679816+0000 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.34.6 op=0 op_state=1 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000027[0] 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000001.34.6 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4/obj00000000000000000011 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: entry->timestamp=2024-04-13T22:15:19.872000+0000 2024-04-13T22:15:23.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-13T22:15:23.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000011[0] 2024-04-13T22:15:23.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.553+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-13T22:15:23.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.554+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000019]: success 2024-04-13T22:15:23.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.554+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6:4 marker=00000000002.77.6 timestamp=2024-04-13T22:15:20.284610+0000 2024-04-13T22:15:23.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-04-13T22:15:23.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000049 2024-04-13T22:15:23.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:19.930432+0000 2024-04-13T22:15:23.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.76.6 op=0 op_state=1 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000049[0] 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000001.76.6 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000006]: success 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.555+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0:4 marker=00000000002.49.6 timestamp=2024-04-13T22:15:20.296537+0000 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14/obj00000000000000000020 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: entry->timestamp=2024-04-13T22:15:20.356471+0000 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.50.6 op=0 op_state=1 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000020[0] 2024-04-13T22:15:23.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000011]: success 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.556+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4:4 marker=00000000002.54.6 timestamp=2024-04-13T22:15:19.872000+0000 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17/obj00000000000000000012 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: entry->timestamp=2024-04-13T22:15:20.404311+0000 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000012[0] 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15/obj00000000000000000037 2024-04-13T22:15:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: entry->timestamp=2024-04-13T22:15:20.464041+0000 2024-04-13T22:15:23.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.51.6 op=0 op_state=1 2024-04-13T22:15:23.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000037[0] 2024-04-13T22:15:23.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.557+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-13T22:15:23.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: parsed entry: id=00000000001.100.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2024-04-13T22:15:23.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19/obj00000000000000000042 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: entry->timestamp=2024-04-13T22:15:20.160161+0000 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.100.6 op=0 op_state=1 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000042[0] 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position=00000000001.100.6 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000012]: success 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.559+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17:4 marker=00000000002.66.6 timestamp=2024-04-13T22:15:20.404311+0000 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.561+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.561+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16/obj00000000000000000005 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.561+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: entry->timestamp=2024-04-13T22:15:19.811855+0000 2024-04-13T22:15:23.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.561+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-04-13T22:15:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.561+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000005[0] 2024-04-13T22:15:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.561+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position=00000000002.76.6 2024-04-13T22:15:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: a sync operation returned error 2024-04-13T22:15:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28/obj00000000000000000021 2024-04-13T22:15:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: entry->timestamp=2024-04-13T22:15:20.100105+0000 2024-04-13T22:15:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-13T22:15:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000021[0] 2024-04-13T22:15:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-13T22:15:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: backing out with sync_status=-5 2024-04-13T22:15:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.562+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.563+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000005]: success 2024-04-13T22:15:23.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.563+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16:4 marker=00000000002.76.6 timestamp=2024-04-13T22:15:19.811855+0000 2024-04-13T22:15:23.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.565+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.565+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000021]: success 2024-04-13T22:15:23.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.565+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28:4 marker=00000000002.53.6 timestamp=2024-04-13T22:15:20.100105+0000 2024-04-13T22:15:23.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.565+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.565+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.566+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000037]: success 2024-04-13T22:15:23.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.566+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15:4 marker=00000000001.51.6 timestamp=2024-04-13T22:15:20.464041+0000 2024-04-13T22:15:23.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.567+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000042]: success 2024-04-13T22:15:23.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.567+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19:4 marker=00000000001.100.6 timestamp=2024-04-13T22:15:20.160161+0000 2024-04-13T22:15:23.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000031]: success 2024-04-13T22:15:23.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11:4 marker=00000000001.37.6 timestamp=2024-04-13T22:15:19.690046+0000 2024-04-13T22:15:23.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-13T22:15:23.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000022 2024-04-13T22:15:23.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:20.415744+0000 2024-04-13T22:15:23.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-13T22:15:23.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000022[0] 2024-04-13T22:15:23.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.569+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-13T22:15:23.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.572+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000017]: success 2024-04-13T22:15:23.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.572+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8:4 marker=00000000001.71.6 timestamp=2024-04-13T22:15:19.920415+0000 2024-04-13T22:15:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.573+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-13T22:15:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.573+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000003 2024-04-13T22:15:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.573+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:20.111735+0000 2024-04-13T22:15:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.573+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-13T22:15:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.573+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000003[0] 2024-04-13T22:15:23.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.573+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-13T22:15:23.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.575+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000003]: success 2024-04-13T22:15:23.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.575+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18:4 marker=00000000002.57.6 timestamp=2024-04-13T22:15:20.111735+0000 2024-04-13T22:15:23.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.594+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: a sync operation returned error 2024-04-13T22:15:23.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.594+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: backing out with sync_status=-5 2024-04-13T22:15:23.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.594+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.594+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.594+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.595+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:23.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.595+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.596+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: a sync operation returned error 2024-04-13T22:15:23.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.596+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: backing out with sync_status=-5 2024-04-13T22:15:23.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.596+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.596+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.597+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.598+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.598+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.600+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000059]: success 2024-04-13T22:15:23.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.600+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1:4 marker=00000000001.51.6 timestamp=2024-04-13T22:15:19.993097+0000 2024-04-13T22:15:23.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.602+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000027]: success 2024-04-13T22:15:23.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.602+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2:4 marker=00000000001.34.6 timestamp=2024-04-13T22:15:19.679816+0000 2024-04-13T22:15:23.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.604+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000020]: success 2024-04-13T22:15:23.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.604+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14:4 marker=00000000001.50.6 timestamp=2024-04-13T22:15:20.356471+0000 2024-04-13T22:15:23.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.605+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000049]: success 2024-04-13T22:15:23.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.605+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7:4 marker=00000000001.76.6 timestamp=2024-04-13T22:15:19.930432+0000 2024-04-13T22:15:23.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.609+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-13T22:15:23.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.609+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000015 2024-04-13T22:15:23.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.609+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:20.345238+0000 2024-04-13T22:15:23.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.609+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-13T22:15:23.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.609+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000015[0] 2024-04-13T22:15:23.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.609+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-13T22:15:23.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.611+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000015]: success 2024-04-13T22:15:23.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.611+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10:4 marker=00000000002.64.6 timestamp=2024-04-13T22:15:20.345238+0000 2024-04-13T22:15:23.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.616+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000022]: success 2024-04-13T22:15:23.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.616+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12:4 marker=00000000002.68.6 timestamp=2024-04-13T22:15:20.415744+0000 2024-04-13T22:15:23.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.639+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-13T22:15:23.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.639+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000004 2024-04-13T22:15:23.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.639+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: entry->timestamp=2024-04-13T22:15:20.234020+0000 2024-04-13T22:15:23.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.639+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-04-13T22:15:23.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.639+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000004[0] 2024-04-13T22:15:23.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.639+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000002.80.6 2024-04-13T22:15:23.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.640+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-13T22:15:23.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.640+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000002 2024-04-13T22:15:23.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.640+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:19.752157+0000 2024-04-13T22:15:23.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.640+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-13T22:15:23.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.640+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000002[0] 2024-04-13T22:15:23.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.640+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-13T22:15:23.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.641+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000004]: success 2024-04-13T22:15:23.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.641+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3:4 marker=00000000002.80.6 timestamp=2024-04-13T22:15:20.234020+0000 2024-04-13T22:15:23.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.641+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000002]: success 2024-04-13T22:15:23.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.641+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5:4 marker=00000000002.57.6 timestamp=2024-04-13T22:15:19.752157+0000 2024-04-13T22:15:23.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 0 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:23.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.687+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 0 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.688+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket shard: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.689+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000005 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: entry->timestamp=2024-04-13T22:15:20.742808+0000 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000005[0] 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000002.82.6 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.690+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket shard: 0 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket shard: 0 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket shard: 0 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.691+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket shard: 0 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000004 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:21.259734+0000 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-13T22:15:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8/obj00000000000000000022 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: entry->timestamp=2024-04-13T22:15:21.452516+0000 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000004[0] 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000022[0] 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.692+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.693+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: sync status for source bucket shard: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: sync status for source bucket shard: 0 2024-04-13T22:15:23.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: sync status for source bucket shard: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000005]: success 2024-04-13T22:15:23.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3:5 marker=00000000002.82.6 timestamp=2024-04-13T22:15:20.742808+0000 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000317 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:21.049756+0000 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4/obj00000000000000000056 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: entry->timestamp=2024-04-13T22:15:20.856786+0000 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.66.6 op=0 op_state=1 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: parsed entry: id=00000000002.107.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:23.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9/obj00000000000000000000 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: entry->timestamp=2024-04-13T22:15:21.572300+0000 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.107.6 op=0 op_state=1 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: : injecting data sync error on key=obj00000000000000000317 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000001.104.6 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000056[0] 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000001.66.6 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000000[0] 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.695+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position=00000000002.107.6 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.696+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.696+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000317 2024-04-13T22:15:23.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.696+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000004]: success 2024-04-13T22:15:23.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.696+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5:5 marker=00000000002.57.6 timestamp=2024-04-13T22:15:21.259734+0000 2024-04-13T22:15:23.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.696+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:entry[obj00000000000000000022]: success 2024-04-13T22:15:23.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.696+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8:5 marker=00000000002.67.6 timestamp=2024-04-13T22:15:21.452516+0000 2024-04-13T22:15:23.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: sync status for source bucket shard: 0 2024-04-13T22:15:23.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: sync status for source bucket shard: 0 2024-04-13T22:15:23.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16/obj00000000000000000017 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: entry->timestamp=2024-04-13T22:15:21.620710+0000 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15/obj00000000000000000007 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: entry->timestamp=2024-04-13T22:15:21.391582+0000 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000016 2024-04-13T22:15:23.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:21.272609+0000 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000017[0] 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000007[0] 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.698+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000016 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000011 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: entry->timestamp=2024-04-13T22:15:21.037900+0000 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.30.6 op=0 op_state=1 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position=00000000002.30.6 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27/obj00000000000000000061 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: entry->timestamp=2024-04-13T22:15:21.173534+0000 2024-04-13T22:15:23.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000011 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000061[0] 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.701+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28/obj00000000000000000050 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: entry->timestamp=2024-04-13T22:15:20.916897+0000 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: : injecting data sync error on key=obj00000000000000000050 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-13T22:15:23.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23/obj00000000000000000001 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: entry->timestamp=2024-04-13T22:15:20.987831+0000 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21/obj00000000000000000002 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: entry->timestamp=2024-04-13T22:15:21.511435+0000 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-13T22:15:23.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29/obj00000000000000000029 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: entry->timestamp=2024-04-13T22:15:21.320911+0000 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30/obj00000000000000000133 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: entry->timestamp=2024-04-13T22:15:20.976987+0000 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.70.6 op=0 op_state=1 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28/obj00000000000000000050 2024-04-13T22:15:23.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000001[0] 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000002[0] 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: : injecting data sync error on key=obj00000000000000000029 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: listing bilog for incremental sync; position=00000000001.86.6 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000133[0] 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: listing bilog for incremental sync; position=00000000001.70.6 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.703+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29/obj00000000000000000029 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.704+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000007]: success 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.704+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15:5 marker=00000000002.65.6 timestamp=2024-04-13T22:15:21.391582+0000 2024-04-13T22:15:23.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.704+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000000]: success 2024-04-13T22:15:23.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.704+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9:5 marker=00000000002.107.6 timestamp=2024-04-13T22:15:21.572300+0000 2024-04-13T22:15:23.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: a sync operation returned error 2024-04-13T22:15:23.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000017]: success 2024-04-13T22:15:23.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16:5 marker=00000000002.77.6 timestamp=2024-04-13T22:15:21.620710+0000 2024-04-13T22:15:23.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000001]: success 2024-04-13T22:15:23.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23:5 marker=00000000002.84.6 timestamp=2024-04-13T22:15:20.987831+0000 2024-04-13T22:15:23.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: a sync operation returned error 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: backing out with sync_status=-5 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: backing out with sync_status=-5 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000002]: success 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.707+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21:5 marker=00000000002.64.6 timestamp=2024-04-13T22:15:21.511435+0000 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.711+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000061]: success 2024-04-13T22:15:23.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.711+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27:5 marker=00000000001.43.6 timestamp=2024-04-13T22:15:21.173534+0000 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.711+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.712+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.712+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.712+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.712+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.712+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.714+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:entry[obj00000000000000000133]: success 2024-04-13T22:15:23.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.714+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30:5 marker=00000000001.70.6 timestamp=2024-04-13T22:15:20.976987+0000 2024-04-13T22:15:23.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-04-13T22:15:23.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2/obj00000000000000000467 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: entry->timestamp=2024-04-13T22:15:21.441815+0000 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.59.6 op=0 op_state=1 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000467[0] 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000001.59.6 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000019 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:20.792332+0000 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-13T22:15:23.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.731+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000019 2024-04-13T22:15:23.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11/obj00000000000000000009 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: entry->timestamp=2024-04-13T22:15:21.112085+0000 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000009[0] 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000271 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:21.331104+0000 2024-04-13T22:15:23.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.39.6 op=0 op_state=1 2024-04-13T22:15:23.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2024-04-13T22:15:23.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000001.39.6 2024-04-13T22:15:23.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.733+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000271 2024-04-13T22:15:23.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.735+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000009]: success 2024-04-13T22:15:23.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.735+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11:5 marker=00000000002.77.6 timestamp=2024-04-13T22:15:21.112085+0000 2024-04-13T22:15:23.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.736+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-13T22:15:23.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.736+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14/obj00000000000000000018 2024-04-13T22:15:23.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.736+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: entry->timestamp=2024-04-13T22:15:20.866984+0000 2024-04-13T22:15:23.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.736+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-13T22:15:23.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.736+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000018[0] 2024-04-13T22:15:23.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.736+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-13T22:15:23.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:21.100798+0000 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13/obj00000000000000000008 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: entry->timestamp=2024-04-13T22:15:21.161763+0000 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-13T22:15:23.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: parsed entry: id=00000000001.105.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17/obj00000000000000000162 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: entry->timestamp=2024-04-13T22:15:21.500565+0000 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.105.6 op=0 op_state=1 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000008[0] 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000162[0] 2024-04-13T22:15:23.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.737+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position=00000000001.105.6 2024-04-13T22:15:23.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.739+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:entry[obj00000000000000000008]: success 2024-04-13T22:15:23.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.739+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13:5 marker=00000000002.50.6 timestamp=2024-04-13T22:15:21.161763+0000 2024-04-13T22:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.741+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-13T22:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.741+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19/obj00000000000000000003 2024-04-13T22:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.741+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: entry->timestamp=2024-04-13T22:15:21.560821+0000 2024-04-13T22:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.741+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-13T22:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.741+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000003[0] 2024-04-13T22:15:23.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.741+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-13T22:15:23.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.742+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000003]: success 2024-04-13T22:15:23.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.742+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19:5 marker=00000000002.39.6 timestamp=2024-04-13T22:15:21.560821+0000 2024-04-13T22:15:23.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.743+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-13T22:15:23.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.743+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20/obj00000000000000000423 2024-04-13T22:15:23.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.743+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: entry->timestamp=2024-04-13T22:15:21.380541+0000 2024-04-13T22:15:23.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.743+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-04-13T22:15:23.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.743+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000423[0] 2024-04-13T22:15:23.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.743+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26/obj00000000000000000010 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: entry->timestamp=2024-04-13T22:15:20.929056+0000 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000010[0] 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22/obj00000000000000000012 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: entry->timestamp=2024-04-13T22:15:20.803886+0000 2024-04-13T22:15:23.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.60.6 op=0 op_state=1 2024-04-13T22:15:23.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000012[0] 2024-04-13T22:15:23.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.744+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position=00000000002.60.6 2024-04-13T22:15:23.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.746+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000012]: success 2024-04-13T22:15:23.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.746+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22:5 marker=00000000002.60.6 timestamp=2024-04-13T22:15:20.803886+0000 2024-04-13T22:15:23.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.746+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:entry[obj00000000000000000010]: success 2024-04-13T22:15:23.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.746+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26:5 marker=00000000002.65.6 timestamp=2024-04-13T22:15:20.929056+0000 2024-04-13T22:15:23.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.747+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: a sync operation returned error 2024-04-13T22:15:23.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.747+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: backing out with sync_status=-5 2024-04-13T22:15:23.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.747+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.747+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.747+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket shard: 0 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: a sync operation returned error 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: backing out with sync_status=-5 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000056]: success 2024-04-13T22:15:23.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.748+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4:5 marker=00000000001.66.6 timestamp=2024-04-13T22:15:20.856786+0000 2024-04-13T22:15:23.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.749+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: a sync operation returned error 2024-04-13T22:15:23.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.749+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: backing out with sync_status=-5 2024-04-13T22:15:23.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.749+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.749+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.749+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.750+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.750+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: sync status for source bucket shard: 0 2024-04-13T22:15:23.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.750+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.750+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket shard: 0 2024-04-13T22:15:23.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.750+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-04-13T22:15:23.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000317 2024-04-13T22:15:23.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:21.049756+0000 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000317[0] 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000001.104.6 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000016 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:21.272609+0000 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000016[0] 2024-04-13T22:15:23.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.755+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-13T22:15:23.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.757+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000016]: success 2024-04-13T22:15:23.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.757+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18:5 marker=00000000002.59.6 timestamp=2024-04-13T22:15:21.272609+0000 2024-04-13T22:15:23.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: a sync operation returned error 2024-04-13T22:15:23.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: backing out with sync_status=-5 2024-04-13T22:15:23.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:23.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.773+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: a sync operation returned error 2024-04-13T22:15:23.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: backing out with sync_status=-5 2024-04-13T22:15:23.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:23.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.775+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.778+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000467]: success 2024-04-13T22:15:23.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.778+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2:5 marker=00000000001.59.6 timestamp=2024-04-13T22:15:21.441815+0000 2024-04-13T22:15:23.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.781+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: a sync operation returned error 2024-04-13T22:15:23.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.781+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: backing out with sync_status=-5 2024-04-13T22:15:23.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.781+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.781+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.781+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.782+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.782+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.782+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000018]: success 2024-04-13T22:15:23.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.782+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14:5 marker=00000000002.68.6 timestamp=2024-04-13T22:15:20.866984+0000 2024-04-13T22:15:23.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.785+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000162]: success 2024-04-13T22:15:23.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.785+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17:5 marker=00000000001.105.6 timestamp=2024-04-13T22:15:21.500565+0000 2024-04-13T22:15:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.789+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000423]: success 2024-04-13T22:15:23.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.789+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20:5 marker=00000000001.56.6 timestamp=2024-04-13T22:15:21.380541+0000 2024-04-13T22:15:23.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.791+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-13T22:15:23.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.791+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000011 2024-04-13T22:15:23.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.791+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: entry->timestamp=2024-04-13T22:15:21.037900+0000 2024-04-13T22:15:23.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.791+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.30.6 op=0 op_state=1 2024-04-13T22:15:23.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.791+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000011[0] 2024-04-13T22:15:23.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.791+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position=00000000002.30.6 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29/obj00000000000000000029 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: entry->timestamp=2024-04-13T22:15:21.320911+0000 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28/obj00000000000000000050 2024-04-13T22:15:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: entry->timestamp=2024-04-13T22:15:20.916897+0000 2024-04-13T22:15:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.792+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-13T22:15:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.793+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000029[0] 2024-04-13T22:15:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.793+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: listing bilog for incremental sync; position=00000000001.86.6 2024-04-13T22:15:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.793+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000050[0] 2024-04-13T22:15:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.793+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-13T22:15:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.793+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000011]: success 2024-04-13T22:15:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.793+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24:5 marker=00000000002.30.6 timestamp=2024-04-13T22:15:21.037900+0000 2024-04-13T22:15:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.795+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:entry[obj00000000000000000050]: success 2024-04-13T22:15:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.795+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28:5 marker=00000000002.41.6 timestamp=2024-04-13T22:15:20.916897+0000 2024-04-13T22:15:23.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.802+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000317]: success 2024-04-13T22:15:23.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.802+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7:5 marker=00000000001.104.6 timestamp=2024-04-13T22:15:21.049756+0000 2024-04-13T22:15:23.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.815+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-13T22:15:23.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.815+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000019 2024-04-13T22:15:23.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.815+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:20.792332+0000 2024-04-13T22:15:23.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.815+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-13T22:15:23.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.815+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000019[0] 2024-04-13T22:15:23.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.815+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-13T22:15:23.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.816+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000019]: success 2024-04-13T22:15:23.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.816+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0:5 marker=00000000002.75.6 timestamp=2024-04-13T22:15:20.792332+0000 2024-04-13T22:15:23.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.818+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-04-13T22:15:23.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.818+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000271 2024-04-13T22:15:23.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.818+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:21.331104+0000 2024-04-13T22:15:23.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.818+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.39.6 op=0 op_state=1 2024-04-13T22:15:23.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.818+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000271[0] 2024-04-13T22:15:23.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.818+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000001.39.6 2024-04-13T22:15:23.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:21.100798+0000 2024-04-13T22:15:23.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-13T22:15:23.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-13T22:15:23.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-13T22:15:23.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.825+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.839+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000029]: success 2024-04-13T22:15:23.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.839+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29:5 marker=00000000001.86.6 timestamp=2024-04-13T22:15:21.320911+0000 2024-04-13T22:15:23.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.862+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000271]: success 2024-04-13T22:15:23.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.862+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12:5 marker=00000000001.39.6 timestamp=2024-04-13T22:15:21.331104+0000 2024-04-13T22:15:23.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: a sync operation returned error 2024-04-13T22:15:23.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: backing out with sync_status=-5 2024-04-13T22:15:23.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.867+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:21.100798+0000 2024-04-13T22:15:23.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-13T22:15:23.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-13T22:15:23.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-13T22:15:23.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:23.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.909+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: a sync operation returned error 2024-04-13T22:15:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: backing out with sync_status=-5 2024-04-13T22:15:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:23.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:23.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:23.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.951+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:23.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.993+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:23.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.993+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000021 2024-04-13T22:15:23.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.993+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:21.100798+0000 2024-04-13T22:15:23.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.993+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-13T22:15:23.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.993+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000021[0] 2024-04-13T22:15:23.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.993+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-13T22:15:23.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.995+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000021]: success 2024-04-13T22:15:23.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:23.995+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10:5 marker=00000000002.62.6 timestamp=2024-04-13T22:15:21.100798+0000 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: sync status for source bucket shard: 0 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: sync status for source bucket shard: 0 2024-04-13T22:15:24.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: sync status for source bucket shard: 0 2024-04-13T22:15:24.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.040+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: sync status for source bucket shard: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: sync status for source bucket shard: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.041+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: sync status for source bucket shard: 0 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: sync status for source bucket shard: 0 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: sync status for source bucket shard: 0 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.042+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: sync status for source bucket shard: 0 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:8]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: sync status for source bucket shard: 0 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: sync status for source bucket shard: 0 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.043+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1/obj00000000000000000000 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: entry->timestamp=2024-04-13T22:15:22.885018+0000 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3/obj00000000000000000001 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: entry->timestamp=2024-04-13T22:15:22.197466+0000 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000000[0] 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000001[0] 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: parsed entry: id=00000000001.84.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0/obj00000000000000000060 2024-04-13T22:15:24.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: entry->timestamp=2024-04-13T22:15:22.461106+0000 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.84.6 op=0 op_state=1 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: parsed entry: id=00000000001.89.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5/obj00000000000000000065 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: entry->timestamp=2024-04-13T22:15:22.583959+0000 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.89.6 op=0 op_state=1 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000060[0] 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: listing bilog for incremental sync; position=00000000001.84.6 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: parsed entry: id=00000000002.111.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6/obj00000000000000000005 2024-04-13T22:15:24.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: entry->timestamp=2024-04-13T22:15:21.953518+0000 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.111.6 op=0 op_state=1 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000065[0] 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: listing bilog for incremental sync; position=00000000001.89.6 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000005[0] 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.044+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: listing bilog for incremental sync; position=00000000002.111.6 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: parsed entry: id=00000000002.89.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4/obj00000000000000000033 2024-04-13T22:15:24.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: entry->timestamp=2024-04-13T22:15:22.261764+0000 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.89.6 op=0 op_state=1 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7/obj00000000000000000016 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: entry->timestamp=2024-04-13T22:15:22.512708+0000 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000033[0] 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: listing bilog for incremental sync; position=00000000002.89.6 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000016[0] 2024-04-13T22:15:24.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-13T22:15:24.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.045+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:24.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: sync status for source bucket shard: 0 2024-04-13T22:15:24.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: sync status for source bucket shard: 0 2024-04-13T22:15:24.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: sync status for source bucket shard: 0 2024-04-13T22:15:24.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: sync status for source bucket shard: 0 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: sync status for source bucket shard: 0 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:13]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.048+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: sync status for source bucket shard: 0 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: sync status for source bucket shard: 0 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: sync status for source bucket shard: 0 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:entry[obj00000000000000000001]: success 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.049+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:3:6 marker=00000000002.84.6 timestamp=2024-04-13T22:15:22.197466+0000 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: sync status for source bucket shard: 0 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: sync status for source bucket shard: 0 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket shard: 0 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: sync status for source bucket shard: 0 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: sync status for source bucket shard: 0 2024-04-13T22:15:24.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:entry[obj00000000000000000000]: success 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.050+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:1:6 marker=00000000002.53.6 timestamp=2024-04-13T22:15:22.885018+0000 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: sync status for source bucket shard: 0 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: sync status for source bucket shard: 0 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: sync status for source bucket shard: 0 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-13T22:15:24.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11/obj00000000000000000030 2024-04-13T22:15:24.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: entry->timestamp=2024-04-13T22:15:22.149207+0000 2024-04-13T22:15:24.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-13T22:15:24.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000030[0] 2024-04-13T22:15:24.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.051+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-13T22:15:24.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: sync status for source bucket shard: 0 2024-04-13T22:15:24.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: sync status for source bucket shard: 0 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:28]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16/obj00000000000000000012 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: entry->timestamp=2024-04-13T22:15:22.028905+0000 2024-04-13T22:15:24.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17/obj00000000000000000002 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: entry->timestamp=2024-04-13T22:15:22.703958+0000 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.60.6 op=0 op_state=1 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19/obj00000000000000000045 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.053+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: entry->timestamp=2024-04-13T22:15:22.764548+0000 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.63.6 op=0 op_state=1 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-13T22:15:24.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23/obj00000000000000000039 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: entry->timestamp=2024-04-13T22:15:22.324946+0000 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000012[0] 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: listing bilog for incremental sync; position=00000000002.80.6 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000002[0] 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: listing bilog for incremental sync; position=00000000002.60.6 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000045[0] 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: listing bilog for incremental sync; position=00000000002.63.6 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000039[0] 2024-04-13T22:15:24.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.054+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25/obj00000000000000000010 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: entry->timestamp=2024-04-13T22:15:22.449197+0000 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26/obj00000000000000000021 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: entry->timestamp=2024-04-13T22:15:21.902512+0000 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-13T22:15:24.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15/obj00000000000000000024 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: entry->timestamp=2024-04-13T22:15:21.966115+0000 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000010[0] 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: listing bilog for incremental sync; position=00000000002.81.6 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000021[0] 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000024[0] 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-13T22:15:24.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.055+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000027 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: entry->timestamp=2024-04-13T22:15:22.753014+0000 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: : injecting data sync error on key=obj00000000000000000027 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000027 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:entry[obj00000000000000000033]: success 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:4:6 marker=00000000002.89.6 timestamp=2024-04-13T22:15:22.261764+0000 2024-04-13T22:15:24.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:entry[obj00000000000000000005]: success 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.056+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:6:6 marker=00000000002.111.6 timestamp=2024-04-13T22:15:21.953518+0000 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34/obj00000000000000000057 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: entry->timestamp=2024-04-13T22:15:22.633001+0000 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.42.6 op=0 op_state=1 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.059+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000057[0] 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.059+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: listing bilog for incremental sync; position=00000000001.42.6 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:entry[obj00000000000000000016]: success 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:7:6 marker=00000000002.49.6 timestamp=2024-04-13T22:15:22.512708+0000 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-13T22:15:24.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29/obj00000000000000000004 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: entry->timestamp=2024-04-13T22:15:22.932927+0000 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.78.6 op=0 op_state=1 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:entry[obj00000000000000000030]: success 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:11:6 marker=00000000002.59.6 timestamp=2024-04-13T22:15:22.149207+0000 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000004[0] 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: listing bilog for incremental sync; position=00000000002.78.6 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36/obj00000000000000000008 2024-04-13T22:15:24.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: entry->timestamp=2024-04-13T22:15:22.016706+0000 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000008[0] 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30/obj00000000000000000036 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: entry->timestamp=2024-04-13T22:15:22.693218+0000 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.44.6 op=0 op_state=1 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000036[0] 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.060+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: listing bilog for incremental sync; position=00000000001.44.6 2024-04-13T22:15:24.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.062+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:entry[obj00000000000000000045]: success 2024-04-13T22:15:24.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.062+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:19:6 marker=00000000002.63.6 timestamp=2024-04-13T22:15:22.764548+0000 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:entry[obj00000000000000000039]: success 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:23:6 marker=00000000002.49.6 timestamp=2024-04-13T22:15:22.324946+0000 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:entry[obj00000000000000000012]: success 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:16:6 marker=00000000002.80.6 timestamp=2024-04-13T22:15:22.028905+0000 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: a sync operation returned error 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: backing out with sync_status=-5 2024-04-13T22:15:24.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:entry[obj00000000000000000024]: success 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.064+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:15:6 marker=00000000002.68.6 timestamp=2024-04-13T22:15:21.966115+0000 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:entry[obj00000000000000000065]: success 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:5:6 marker=00000000001.89.6 timestamp=2024-04-13T22:15:22.583959+0000 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:entry[obj00000000000000000002]: success 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:17:6 marker=00000000002.60.6 timestamp=2024-04-13T22:15:22.703958+0000 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:entry[obj00000000000000000004]: success 2024-04-13T22:15:24.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:29:6 marker=00000000002.78.6 timestamp=2024-04-13T22:15:22.932927+0000 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:entry[obj00000000000000000021]: success 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:26:6 marker=00000000002.85.6 timestamp=2024-04-13T22:15:21.902512+0000 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:entry[obj00000000000000000010]: success 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.066+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:25:6 marker=00000000002.81.6 timestamp=2024-04-13T22:15:22.449197+0000 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.068+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: sync status for source bucket shard: 0 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.068+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.069+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:entry[obj00000000000000000008]: success 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.069+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:36:6 marker=00000000002.73.6 timestamp=2024-04-13T22:15:22.016706+0000 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.070+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:entry[obj00000000000000000057]: success 2024-04-13T22:15:24.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.070+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:34:6 marker=00000000001.42.6 timestamp=2024-04-13T22:15:22.633001+0000 2024-04-13T22:15:24.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.083+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-13T22:15:24.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.083+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2/obj00000000000000000011 2024-04-13T22:15:24.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.083+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: entry->timestamp=2024-04-13T22:15:22.525291+0000 2024-04-13T22:15:24.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.083+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-13T22:15:24.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.083+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000011[0] 2024-04-13T22:15:24.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.083+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: parsed entry: id=00000000002.143.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9/obj00000000000000000015 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: entry->timestamp=2024-04-13T22:15:22.824861+0000 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.143.6 op=0 op_state=1 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000015[0] 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: listing bilog for incremental sync; position=00000000002.143.6 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:entry[obj00000000000000000011]: success 2024-04-13T22:15:24.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.085+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:2:6 marker=00000000002.51.6 timestamp=2024-04-13T22:15:22.525291+0000 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.086+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:entry[obj00000000000000000015]: success 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.086+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:9:6 marker=00000000002.143.6 timestamp=2024-04-13T22:15:22.824861+0000 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.087+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: parsed entry: id=00000000002.86.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.087+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10/obj00000000000000000026 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.087+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: entry->timestamp=2024-04-13T22:15:22.209594+0000 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.087+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.86.6 op=0 op_state=1 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.087+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000026[0] 2024-04-13T22:15:24.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.087+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: listing bilog for incremental sync; position=00000000002.86.6 2024-04-13T22:15:24.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.088+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:entry[obj00000000000000000026]: success 2024-04-13T22:15:24.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.088+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:10:6 marker=00000000002.86.6 timestamp=2024-04-13T22:15:22.209594+0000 2024-04-13T22:15:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-04-13T22:15:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000062 2024-04-13T22:15:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:22.573345+0000 2024-04-13T22:15:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.63.6 op=0 op_state=1 2024-04-13T22:15:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: : injecting data sync error on key=obj00000000000000000062 2024-04-13T22:15:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20/obj00000000000000000006 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: entry->timestamp=2024-04-13T22:15:22.385270+0000 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: parsed entry: id=00000000002.113.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14/obj00000000000000000040 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: entry->timestamp=2024-04-13T22:15:22.089265+0000 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.113.6 op=0 op_state=1 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:15:24.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 0 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: ERROR: failed to sync object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000062 2024-04-13T22:15:24.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000006[0] 2024-04-13T22:15:24.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-13T22:15:24.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000040[0] 2024-04-13T22:15:24.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.093+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: listing bilog for incremental sync; position=00000000002.113.6 2024-04-13T22:15:24.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.094+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: parsed entry: id=00000000002.113.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-13T22:15:24.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.094+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18/obj00000000000000000013 2024-04-13T22:15:24.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.094+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: entry->timestamp=2024-04-13T22:15:22.337381+0000 2024-04-13T22:15:24.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.094+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.113.6 op=0 op_state=1 2024-04-13T22:15:24.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.094+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000013[0] 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.094+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: listing bilog for incremental sync; position=00000000002.113.6 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.095+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.095+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22/obj00000000000000000007 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.095+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: entry->timestamp=2024-04-13T22:15:22.873094+0000 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.095+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.095+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000007[0] 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.095+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.096+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:entry[obj00000000000000000006]: success 2024-04-13T22:15:24.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.096+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:20:6 marker=00000000002.49.6 timestamp=2024-04-13T22:15:22.385270+0000 2024-04-13T22:15:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.096+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:entry[obj00000000000000000040]: success 2024-04-13T22:15:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.096+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:14:6 marker=00000000002.113.6 timestamp=2024-04-13T22:15:22.089265+0000 2024-04-13T22:15:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.096+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:entry[obj00000000000000000013]: success 2024-04-13T22:15:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.096+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:18:6 marker=00000000002.113.6 timestamp=2024-04-13T22:15:22.337381+0000 2024-04-13T22:15:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.097+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:entry[obj00000000000000000007]: success 2024-04-13T22:15:24.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.097+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:22:6 marker=00000000002.84.6 timestamp=2024-04-13T22:15:22.873094+0000 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21/obj00000000000000000017 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: entry->timestamp=2024-04-13T22:15:22.397860+0000 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000017[0] 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: listing bilog for incremental sync; position=00000000002.70.6 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: parsed entry: id=00000000002.141.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35/obj00000000000000000018 2024-04-13T22:15:24.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: entry->timestamp=2024-04-13T22:15:22.137251+0000 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.141.6 op=0 op_state=1 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33/obj00000000000000000019 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: entry->timestamp=2024-04-13T22:15:22.812839+0000 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000018[0] 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: listing bilog for incremental sync; position=00000000002.141.6 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000019[0] 2024-04-13T22:15:24.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.098+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32/obj00000000000000000014 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: entry->timestamp=2024-04-13T22:15:22.076707+0000 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000014[0] 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: listing bilog for incremental sync; position=00000000002.82.6 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: parsed entry: id=00000000002.86.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-04-13T22:15:24.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27/obj00000000000000000053 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: entry->timestamp=2024-04-13T22:15:22.643370+0000 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.86.6 op=0 op_state=1 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000053[0] 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: listing bilog for incremental sync; position=00000000002.86.6 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:entry[obj00000000000000000018]: success 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:35:6 marker=00000000002.141.6 timestamp=2024-04-13T22:15:22.137251+0000 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:entry[obj00000000000000000017]: success 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.100+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:21:6 marker=00000000002.70.6 timestamp=2024-04-13T22:15:22.397860+0000 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: parsed entry: id=00000000002.87.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-13T22:15:24.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31/obj00000000000000000003 2024-04-13T22:15:24.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: entry->timestamp=2024-04-13T22:15:22.273725+0000 2024-04-13T22:15:24.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.87.6 op=0 op_state=1 2024-04-13T22:15:24.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000003[0] 2024-04-13T22:15:24.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: listing bilog for incremental sync; position=00000000002.87.6 2024-04-13T22:15:24.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:entry[obj00000000000000000019]: success 2024-04-13T22:15:24.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.101+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:33:6 marker=00000000002.64.6 timestamp=2024-04-13T22:15:22.812839+0000 2024-04-13T22:15:24.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.103+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:entry[obj00000000000000000014]: success 2024-04-13T22:15:24.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.103+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:32:6 marker=00000000002.82.6 timestamp=2024-04-13T22:15:22.076707+0000 2024-04-13T22:15:24.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.104+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:entry[obj00000000000000000003]: success 2024-04-13T22:15:24.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.104+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:31:6 marker=00000000002.87.6 timestamp=2024-04-13T22:15:22.273725+0000 2024-04-13T22:15:24.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.109+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:entry[obj00000000000000000060]: success 2024-04-13T22:15:24.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.109+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:0:6 marker=00000000001.84.6 timestamp=2024-04-13T22:15:22.461106+0000 2024-04-13T22:15:24.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-13T22:15:24.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24/obj00000000000000000027 2024-04-13T22:15:24.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: entry->timestamp=2024-04-13T22:15:22.753014+0000 2024-04-13T22:15:24.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-13T22:15:24.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.110+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000027[0] 2024-04-13T22:15:24.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.110+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-13T22:15:24.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.112+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:entry[obj00000000000000000027]: success 2024-04-13T22:15:24.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:24:6 marker=00000000002.35.6 timestamp=2024-04-13T22:15:22.753014+0000 2024-04-13T22:15:24.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.112+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:entry[obj00000000000000000036]: success 2024-04-13T22:15:24.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.112+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:30:6 marker=00000000001.44.6 timestamp=2024-04-13T22:15:22.693218+0000 2024-04-13T22:15:24.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.135+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: a sync operation returned error 2024-04-13T22:15:24.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.135+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: backing out with sync_status=-5 2024-04-13T22:15:24.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.135+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: incremental sync on bucket failed, retcode=-5 2024-04-13T22:15:24.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.135+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: ERROR: incremental sync failed. error: -5 2024-04-13T22:15:24.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.135+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:15:24.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.136+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: sync status for source bucket shard: 0 2024-04-13T22:15:24.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.136+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position= 2024-04-13T22:15:24.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.146+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:entry[obj00000000000000000053]: success 2024-04-13T22:15:24.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.146+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:27:6 marker=00000000002.86.6 timestamp=2024-04-13T22:15:22.643370+0000 2024-04-13T22:15:24.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.177+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-04-13T22:15:24.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.177+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: syncing object: vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12/obj00000000000000000062 2024-04-13T22:15:24.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.177+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: entry->timestamp=2024-04-13T22:15:22.573345+0000 2024-04-13T22:15:24.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.177+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=00000000001.63.6 op=0 op_state=1 2024-04-13T22:15:24.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.177+0000 7f68669ee780 5 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-60[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23])/obj00000000000000000062[0] 2024-04-13T22:15:24.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.177+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-13T22:15:24.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.181+0000 7f68669ee780 10 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:entry[obj00000000000000000062]: success 2024-04-13T22:15:24.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:24.181+0000 7f68669ee780 20 RGW-SYNC:bucket[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23<-vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]:inc_sync[vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12]: updating marker marker_oid=bucket.sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23:12:6 marker=00000000001.63.6 timestamp=2024-04-13T22:15:22.573345+0000 2024-04-13T22:15:24.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:24.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:24.244 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:24.334 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:24.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:24.367 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000000", 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 20 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.368 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.885018Z", 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.369 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.15478116121656970309", 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000001", 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.370 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.197466Z", 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.371 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14988376006189765315", 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.372 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000002", 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 17 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.373 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.703958Z", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.374 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.3836101433500873395", 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000003", 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.273725Z", 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.376 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.5746650204239632394", 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000004", 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.378 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.932927Z", 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.379 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14811353583847300113", 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000005", 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.381 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.953518Z", 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.382 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17127319041533141454", 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.383 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000006", 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.384 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.385270Z", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.385 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7755919530006309333", 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000007", 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.386 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.873094Z", 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.387 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7069880467631716484", 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.388 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000008", 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.016706Z", 2024-04-13T22:15:24.389 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.11197702951894459525", 2024-04-13T22:15:24.390 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000009", 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.391 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.112085Z", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10845634058883959764", 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000010", 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.449197Z", 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.394 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14090612943538020987", 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000011", 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 19 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.396 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.525291Z", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.397 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12268850236110997909", 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000012", 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 18 2024-04-13T22:15:24.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.028905Z", 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.399 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4994378813364732223", 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.400 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000013", 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 16 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.401 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.337381Z", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.402 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12703723521048555870", 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000014", 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 20 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.403 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.076707Z", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.404 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8821187015964693022", 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.405 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000015", 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.406 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.824861Z", 2024-04-13T22:15:24.407 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.407 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.407 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.407 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.407 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.407 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.5173232533898614441", 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000016", 2024-04-13T22:15:24.408 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.409 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.512708Z", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4588880132702803468", 2024-04-13T22:15:24.410 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000017", 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 18 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.411 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.397860Z", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.412 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17200563315497989124", 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000018", 2024-04-13T22:15:24.413 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.137251Z", 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.414 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10577771645813879568", 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.415 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000019", 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 17 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.416 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.812839Z", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8804036705828397875", 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000020", 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.418 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.356471Z", 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.419 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4705904909990543783", 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000021", 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 17 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.421 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.902512Z", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.422 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8856665754985849534", 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000022", 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.423 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.452516Z", 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.424 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.2886531618942685603", 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000024", 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.426 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.966115Z", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.427 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.15406007716516793341", 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000025", 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:15:24.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.170421Z", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.429 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12850375388535219119", 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.430 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000026", 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.431 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.209594Z", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13665901446059608277", 2024-04-13T22:15:24.432 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000027", 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.433 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.753014Z", 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.434 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4819925447961842628", 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.435 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000028", 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:17.447761Z", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.2437340506098992606", 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000029", 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.320911Z", 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10148533215232773510", 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000030", 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.149207Z", 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.15406921731364648713", 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000031", 2024-04-13T22:15:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.690046Z", 2024-04-13T22:15:24.444 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13856565061865443664", 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.445 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000032", 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.446 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.327899Z", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.447 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.18354131229015616896", 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000033", 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.448 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.261764Z", 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17972996782814663258", 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.450 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000036", 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.451 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.693218Z", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17740529109285520479", 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000037", 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.453 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.464041Z", 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.454 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.2943420971841587037", 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000039", 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 22 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.456 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.324946Z", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.457 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.6583988288573941687", 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000040", 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.458 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.089265Z", 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.459 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.15820501204974134034", 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000042", 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.461 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.160161Z", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7192770534664534219", 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000045", 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.463 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.764548Z", 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.464 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.11508173957962625352", 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000046", 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.270861Z", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.467 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7806453896484284885", 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000049", 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.468 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.930432Z", 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.469 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16565704443855120682", 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000050", 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.471 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.916897Z", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.472 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17639601958886968824", 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000053", 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.643370Z", 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.474 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10993213753941392081", 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000055", 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.476 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.338310Z", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.477 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.9323512469789959020", 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000056", 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.856786Z", 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.479 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8982437759036759517", 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000057", 2024-04-13T22:15:24.480 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.633001Z", 2024-04-13T22:15:24.481 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12455821598552726527", 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.482 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000058", 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.483 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.280249Z", 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.484 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.3883728835990460631", 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000059", 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.485 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.993097Z", 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.486 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10116896204197237572", 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.487 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000060", 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.488 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.461106Z", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.489 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16659402800635872602", 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000061", 2024-04-13T22:15:24.490 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.173534Z", 2024-04-13T22:15:24.491 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14720596974119773750", 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.492 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000062", 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.493 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.573345Z", 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.494 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.921256908736549961", 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000065", 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.495 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.583959Z", 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.496 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.3672549033803137494", 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.497 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000076", 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.498 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:17.587837Z", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.499 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8284015201871699941", 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000133", 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.500 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.976987Z", 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.501 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7914582730454917603", 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000162", 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.503 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.500565Z", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.504 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7166397319972178126", 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000271", 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.331104Z", 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.506 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.11838931670661388609", 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.507 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000317", 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.508 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.049756Z", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17385570483830267274", 2024-04-13T22:15:24.509 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000423", 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:15:24.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.380541Z", 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.511 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.15446971843862889762", 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000467", 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 16 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.513 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.441815Z", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14736498731787611573", 2024-04-13T22:15:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.515 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:24.516 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:15:24.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:24.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:24.517 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-60 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:24.568 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:24.568 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000000", 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.885018Z", 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.603 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_MKT-JOwopSEa9I_4NNKMbKy9REQOuVW", 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000001", 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.197466Z", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_wIQ6dzz2IEa23X-lbfuIYbjHsbdqbmw", 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000002", 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.703958Z", 2024-04-13T22:15:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_hj1KRddQ5ojohl2Z8qCKOfgt1Lzz_7x", 2024-04-13T22:15:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000003", 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.273725Z", 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_f3OWb7Q1PhIwdFVhLxgMVVuqsSSGcry", 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000004", 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.932927Z", 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_IVIM7XAOqlAJuWhoKBk2RXezcMjpvwH", 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000005", 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.953518Z", 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.616 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_3JU5RNMREwXOc6xgnWLIewhKSKG9A2T", 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000006", 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.618 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.385270Z", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.619 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_n86JCDM1443rlzNrElKWbfEer2b_0tR", 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000007", 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.620 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.873094Z", 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.621 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "__PYz_dJPlWWaCEhgtsJH9CP4qMhPtkr", 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.622 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000008", 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.623 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.016706Z", 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.624 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_n5VFdcw0lmHUFjE4F626msU4-GgHUvX", 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000009", 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.625 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.112085Z", 2024-04-13T22:15:24.626 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_0LTidEuxMpEJpu9FxUpy4-NQgtktWgs", 2024-04-13T22:15:24.627 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000010", 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.449197Z", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_okblbv6i074dj6H9I4XfgaILKSRX_NB", 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000011", 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.631 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.525291Z", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_YdAYdkXx1slL84AmGzHcqSIIqMDTz-2", 2024-04-13T22:15:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000012", 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.028905Z", 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_gd1oMUDgb3g2WtlBc4wnrwSpeHjY-lb", 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000013", 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.337381Z", 2024-04-13T22:15:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_iwfHXjhfgQ74WsDOa4QD0ikOPDxgo74", 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000014", 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.076707Z", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_zo3rYlWqyiGs2MGsl7fHvLFCotKwxpl", 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000015", 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.824861Z", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_31-37F6BatbOiBXOkP-8S55W77OminT", 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000016", 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.512708Z", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_bvS0Gq7mG54YcnOSPXwtrkvzETw3n2P", 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000017", 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.397860Z", 2024-04-13T22:15:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_xeYmZItNxgLQZCNIVzrV9CYzk2BA_L6", 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000018", 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.137251Z", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_nf_c7XB0muGDS_6M2LTbTmf-fgumxY-", 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000019", 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.650 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.812839Z", 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.651 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_vns59UqijCwQaaDwKnJNbZUt-A6VzAO", 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.652 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000020", 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.653 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.356471Z", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.654 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_u1ovHvnEqAOqrc8j6C4krujvV3FOp73", 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000021", 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.655 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.902512Z", 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.656 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_RcRYeNIVvcOK9B2_1ZUUNWPv6kZoNlO", 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000022", 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.452516Z", 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.658 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_09g9hXgiES-yf08ESK4XIFCI_2LCX2W", 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000024", 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.966115Z", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_A_mEi4wjrjxoJvR_t4IHldcGVnbJGZ3", 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000025", 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.170421Z", 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_2sgkwQ8izTJuZ4LfYuv7xf8fh781UYu", 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000026", 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.209594Z", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_sbF7eppI42zmSFiP_OGRZ7Fa_gwOR2U", 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000027", 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.753014Z", 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_GYkMmMdpQI1CPxtAl88Bj3zZ-wuX76u", 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000028", 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:15:24.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:17.447761Z", 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_nRSXaMqfqmiHXLCP7nIRA3DkVvh9TJT", 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000029", 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.673 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.320911Z", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_2_iGCWHU4lDEx48kGPC7iMcmqEX7NW8", 2024-04-13T22:15:24.674 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000030", 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.149207Z", 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.676 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_K4E_O5Uq0IaS0EJiVwslErUnybbZ4R_", 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000031", 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.678 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.690046Z", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_H94nmhX59le9aEStUaWmamZLA39IoxR", 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000032", 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.680 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.327899Z", 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.681 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_gXnfagDw2ompUkGsO7SBl0mIoVb1ojy", 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000033", 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.683 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.261764Z", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_2UWKlxV-VkF2ghKlI9VHnSmr_gwOb2x", 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000036", 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.685 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.693218Z", 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.686 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_mCfw7k_6zQdz5bQV_2V8JgUTKQe1hzw", 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.687 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000037", 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.688 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.464041Z", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_Op9zsx2pAtDcUggM7Fpz0rNrZXs2tl6", 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.689 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000039", 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.690 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.324946Z", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_0tUoTars6QDITRHMx5PyFguZ6N27HvG", 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000040", 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.089265Z", 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.693 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_gPHQL48ZtGis6GGnhwiueu0LW1JsxrX", 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.694 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000042", 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.695 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.160161Z", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_-NZcKTH06C3kWS6jSuDtdd-LaU-5PjL", 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.696 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000045", 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.697 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.764548Z", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.698 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_3I_ofQC0MrXghJgaXBHVRdG9SW8yZmT", 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000046", 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.699 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.270861Z", 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.700 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_hgia2MRFVG1Hf3Mew8Pusc29cWJgMra", 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000049", 2024-04-13T22:15:24.701 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.930432Z", 2024-04-13T22:15:24.702 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_j8VnESc4r9yu-zbCb0K9-ncVrdg5bYm", 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.703 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000050", 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.704 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.916897Z", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.705 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_siKmneOCGuGvBQCojP5B1mzn63Ox6lN", 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.706 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000053", 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.707 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.643370Z", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.708 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_SxifazdBo9nZ9ZOv2BUKlFokPeGglfQ", 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000055", 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.709 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.338310Z", 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.710 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_KO0kNwYirus2dczI5S6P2_scA-Ue38U", 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.711 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000056", 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.712 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.856786Z", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_5v98ve_Yd3ku1RXouZdk5eGsPdmaxbL", 2024-04-13T22:15:24.713 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000057", 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.633001Z", 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.715 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.716 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_XBdNvToPy0NII-8jAK6ksZXt4o9we0r", 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000058", 2024-04-13T22:15:24.717 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.718 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.718 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.718 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.718 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.280249Z", 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.719 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_0eDO3Bq_hJAK7jD0_qMqcQAVTz7bO9z", 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000059", 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.721 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:19.993097Z", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.722 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_2qgPuZZboyHyR6DKPZv3EMjeqJThPtt", 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000060", 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.723 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.461106Z", 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.724 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_H6h6IQaSBnCRAazYt9fdgdCWmtCkoJP", 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000061", 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.726 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.173534Z", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.727 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_4oXRgqP8Ew2txwWLHqRxMlj5lnqNOl4", 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000062", 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.728 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.573345Z", 2024-04-13T22:15:24.729 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_MPC3TPLWA597D9wwU3eq0n-Ml-BOAmm", 2024-04-13T22:15:24.730 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000065", 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.731 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:22.583959Z", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.732 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_LY691dZyS1J14clpyNWmJ5rzMKD9ZB3", 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.733 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000076", 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.734 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:17.587837Z", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.735 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_-tgl44JYvtcAr7bGk66Nw1L0FwiJjWh", 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000133", 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.736 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:20.976987Z", 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.737 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_C_nH8OXMxGf_eaqBYan1OEf6we7Jir3", 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000162", 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:15:24.739 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.500565Z", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_q6PSpHRI0uN_vegVb27oD2bfMicQv3n", 2024-04-13T22:15:24.740 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000271", 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:15:24.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.331104Z", 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.742 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_V4cgz1myNN9vmKFzfD-YQTCYzE1g5zf", 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000317", 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.745 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.049756Z", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.746 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_bXNuO-RT8lQW7IbNLHm4kGgdfTPDLwH", 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000423", 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.380541Z", 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.748 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_lwALeShGFNa-VJ7zdNvi1jHZ_qmQLiQ", 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj00000000000000000467", 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:15:24.750 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:15:21.441815Z", 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:15:24.751 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_DgNcscFU2DnEpY04oM0xAsRjczbK6cf", 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:24.752 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:15:24.753 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:15:24.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:15:24.756 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:15:24.756 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:15:24.776 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:15:24.777 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:15:24.777 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:15:25.778 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:25.803 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:15:25.803 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:15:25.803 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:15:25.804 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:15:25.804 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-60 2024-04-13T22:15:25.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:15:25.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:15:25.805 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:15:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:25.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:25.943+0000 7f6e4e120780 1 bucket sync caught up with source: 2024-04-13T22:15:25.946 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.84.6, 00000000002.53.6, 00000000002.51.6, 00000000002.84.6, 00000000002.89.6, 00000000001.89.6, 00000000002.111.6, 00000000002.49.6, , 00000000002.143.6, 00000000002.86.6, 00000000002.59.6, 00000000001.63.6, , 00000000002.113.6, 00000000002.68.6, 00000000002.80.6, 00000000002.60.6, 00000000002.113.6, 00000000002.63.6, 00000000002.49.6, 00000000002.70.6, 00000000002.84.6, 00000000002.49.6, 00000000002.35.6, 00000000002.81.6, 00000000002.85.6, 00000000002.86.6, , 00000000002.78.6, 00000000001.44.6, 00000000002.87.6, 00000000002.82.6, 00000000002.64.6, 00000000001.42.6, 00000000002.141.6, 00000000002.73.6] 2024-04-13T22:15:25.946 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.84.6, 00000000002.53.6, 00000000002.51.6, 00000000002.84.6, 00000000002.89.6, 00000000001.89.6, 00000000002.111.6, 00000000002.49.6, , 00000000002.143.6, 00000000002.86.6, 00000000002.59.6, 00000000001.63.6, , 00000000002.113.6, 00000000002.68.6, 00000000002.80.6, 00000000002.60.6, 00000000002.113.6, 00000000002.63.6, 00000000002.49.6, 00000000002.70.6, 00000000002.84.6, 00000000002.49.6, 00000000002.35.6, 00000000002.81.6, 00000000002.85.6, 00000000002.86.6, , 00000000002.78.6, 00000000001.44.6, 00000000002.87.6, 00000000002.82.6, 00000000002.64.6, 00000000001.42.6, 00000000002.141.6, 00000000002.73.6] 2024-04-13T22:15:25.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:25.943+0000 7f6e4e120780 0 bucket checkpoint complete 2024-04-13T22:15:25.961 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-60 2024-04-13T22:15:25.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:15:25.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:15:25.962 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:15:26.016 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:26.016 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:26.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:26.048+0000 7f5359285780 1 waiting for incremental sync to catch up: 2024-04-13T22:15:26.051 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.1833.6, 00000000006.481.6, 00000000005.564.6, 00000000001.461.6, 00000000005.1021.6, 00000000008.2229.6, 00000000007.1251.6, 00000000001.950.6, 00000000007.426.6, 00000000007.1766.6, 00000000004.424.6] 2024-04-13T22:15:26.052 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.1833.6, 00000000006.481.6, 00000000005.564.6, 00000000010.479.6, 00000000005.1021.6, 00000000008.2229.6, 00000000007.1251.6, 00000000006.960.6, 00000000007.426.6, 00000000007.1766.6, 00000000004.424.6] 2024-04-13T22:15:31.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:31.049+0000 7f5359285780 1 bucket sync caught up with source: 2024-04-13T22:15:31.053 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000004.1833.6, 00000000006.481.6, 00000000005.564.6, 00000000010.479.6, 00000000005.1021.6, 00000000008.2229.6, 00000000007.1251.6, 00000000006.960.6, 00000000007.426.6, 00000000007.1766.6, 00000000004.424.6] 2024-04-13T22:15:31.053 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000004.1833.6, 00000000006.481.6, 00000000005.564.6, 00000000010.479.6, 00000000005.1021.6, 00000000008.2229.6, 00000000007.1251.6, 00000000006.960.6, 00000000007.426.6, 00000000007.1766.6, 00000000004.424.6] 2024-04-13T22:15:31.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:31.049+0000 7f5359285780 0 bucket checkpoint complete 2024-04-13T22:15:31.070 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-60 zones={a2, a1} 2024-04-13T22:15:31.070 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.070 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.070 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.070 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:31.070 DEBUG:boto:Data: 2024-04-13T22:15:31.070 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.071 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.071 DEBUG:boto:Params: {} 2024-04-13T22:15:31.071 DEBUG:boto:Token: None 2024-04-13T22:15:31.071 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HyO87cubXMu+s0+i4hpPNguGLCI= 2024-04-13T22:15:31.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HyO87cubXMu+s0+i4hpPNguGLCI=', 'Content-Length': '0'} 2024-04-13T22:15:31.072 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-13T22:15:31.072 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:15:31.084 DEBUG:boto:Token: None 2024-04-13T22:15:31.084 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HyO87cubXMu+s0+i4hpPNguGLCI= 2024-04-13T22:15:31.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HyO87cubXMu+s0+i4hpPNguGLCI=', 'Content-Length': '0'} 2024-04-13T22:15:31.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f014491c474a8022-00661b0403-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.091 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.091 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.091 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.091 DEBUG:boto:Path: /vwnfns-60/ 2024-04-13T22:15:31.091 DEBUG:boto:Data: 2024-04-13T22:15:31.091 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.091 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.091 DEBUG:boto:Params: {} 2024-04-13T22:15:31.092 DEBUG:boto:Token: None 2024-04-13T22:15:31.092 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HyO87cubXMu+s0+i4hpPNguGLCI= 2024-04-13T22:15:31.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HyO87cubXMu+s0+i4hpPNguGLCI=', 'Content-Length': '0'} 2024-04-13T22:15:31.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066dba0f19ed9cc0e-00661b0403-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.096 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:15:31.096 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.096 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.096 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.096 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.096 DEBUG:boto:Method: GET 2024-04-13T22:15:31.096 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.096 DEBUG:boto:Data: 2024-04-13T22:15:31.096 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.097 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.097 DEBUG:boto:Params: {} 2024-04-13T22:15:31.097 DEBUG:boto:Token: None 2024-04-13T22:15:31.097 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/?versions 2024-04-13T22:15:31.097 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw= 2024-04-13T22:15:31.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw=', 'Content-Length': '0'} 2024-04-13T22:15:31.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000740b070b2fcc6973-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.100 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-13T22:15:31.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-13T22:15:31.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-13T22:15:31.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-13T22:15:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-13T22:15:31.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-13T22:15:31.108 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:15:31.108 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.109 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.109 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.109 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.109 DEBUG:boto:Method: GET 2024-04-13T22:15:31.109 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.109 DEBUG:boto:Data: 2024-04-13T22:15:31.109 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.109 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.109 DEBUG:boto:Params: {} 2024-04-13T22:15:31.110 DEBUG:boto:Token: None 2024-04-13T22:15:31.110 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/?versions 2024-04-13T22:15:31.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw= 2024-04-13T22:15:31.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw=', 'Content-Length': '0'} 2024-04-13T22:15:31.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006276a2ff2bb547f3-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.115 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-13T22:15:31.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-13T22:15:31.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-13T22:15:31.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-13T22:15:31.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-13T22:15:31.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-13T22:15:31.122 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.123 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.123 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.123 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.123 DEBUG:boto:Method: GET 2024-04-13T22:15:31.123 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.123 DEBUG:boto:Data: 2024-04-13T22:15:31.123 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.123 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.123 DEBUG:boto:Params: {} 2024-04-13T22:15:31.124 DEBUG:boto:Token: None 2024-04-13T22:15:31.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/?versions 2024-04-13T22:15:31.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw= 2024-04-13T22:15:31.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw=', 'Content-Length': '0'} 2024-04-13T22:15:31.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3e0fced6093722-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.127 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.130 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.130 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.130 DEBUG:boto:path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.130 DEBUG:boto:auth_path=/vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.130 DEBUG:boto:Method: GET 2024-04-13T22:15:31.130 DEBUG:boto:Path: /vwnfns-60/?versions&max-keys=999 2024-04-13T22:15:31.130 DEBUG:boto:Data: 2024-04-13T22:15:31.130 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.131 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.131 DEBUG:boto:Params: {} 2024-04-13T22:15:31.131 DEBUG:boto:Token: None 2024-04-13T22:15:31.131 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/?versions 2024-04-13T22:15:31.131 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw= 2024-04-13T22:15:31.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVwpyh3Lk39cKjwzUJRIU1uougw=', 'Content-Length': '0'} 2024-04-13T22:15:31.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000327f570dc25c8e7-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.136 DEBUG:boto:b'vwnfns-60999falseobj00000000000000000000nulltrue2024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-13T22:15:31.140 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.140 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.140 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.141 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.141 DEBUG:boto:Method: GET 2024-04-13T22:15:31.141 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.141 DEBUG:boto:Data: 2024-04-13T22:15:31.141 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.141 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.141 DEBUG:boto:Params: {} 2024-04-13T22:15:31.141 DEBUG:boto:Token: None 2024-04-13T22:15:31.141 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok= 2024-04-13T22:15:31.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok=', 'Content-Length': '0'} 2024-04-13T22:15:31.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb63193060a4852-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.190 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.191 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.191 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.191 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.191 DEBUG:boto:Method: GET 2024-04-13T22:15:31.191 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.191 DEBUG:boto:Data: 2024-04-13T22:15:31.191 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.191 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.191 DEBUG:boto:Params: {} 2024-04-13T22:15:31.192 DEBUG:boto:Token: None 2024-04-13T22:15:31.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok= 2024-04-13T22:15:31.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok=', 'Content-Length': '0'} 2024-04-13T22:15:31.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aecc312e043381c8-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.194 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.194 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.194 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.195 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.195 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.195 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.195 DEBUG:boto:Data: 2024-04-13T22:15:31.195 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.195 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.195 DEBUG:boto:Params: {} 2024-04-13T22:15:31.195 DEBUG:boto:Token: None 2024-04-13T22:15:31.195 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7ar+liizJZ6MhIazzhUVelJLKNk= 2024-04-13T22:15:31.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7ar+liizJZ6MhIazzhUVelJLKNk=', 'Content-Length': '0'} 2024-04-13T22:15:31.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ae7b528d8d426c-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.197 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.197 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.198 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.198 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.198 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.198 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.198 DEBUG:boto:Data: 2024-04-13T22:15:31.198 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.198 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.198 DEBUG:boto:Params: {} 2024-04-13T22:15:31.198 DEBUG:boto:Token: None 2024-04-13T22:15:31.199 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7ar+liizJZ6MhIazzhUVelJLKNk= 2024-04-13T22:15:31.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7ar+liizJZ6MhIazzhUVelJLKNk=', 'Content-Length': '0'} 2024-04-13T22:15:31.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3905e4fc800f880-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-13T22:15:31.201 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.201 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.201 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.201 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.201 DEBUG:boto:Method: GET 2024-04-13T22:15:31.201 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.201 DEBUG:boto:Data: 2024-04-13T22:15:31.201 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.202 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.202 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.202 DEBUG:boto:Params: {} 2024-04-13T22:15:31.202 DEBUG:boto:Token: None 2024-04-13T22:15:31.202 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.202 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok= 2024-04-13T22:15:31.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok=', 'Content-Length': '0'} 2024-04-13T22:15:31.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4112eb9f7e4095e-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.204 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.204 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.204 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.204 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.205 DEBUG:boto:Method: GET 2024-04-13T22:15:31.205 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.205 DEBUG:boto:Data: 2024-04-13T22:15:31.205 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.205 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.205 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.205 DEBUG:boto:Params: {} 2024-04-13T22:15:31.205 DEBUG:boto:Token: None 2024-04-13T22:15:31.205 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000?versionId=null 2024-04-13T22:15:31.205 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok= 2024-04-13T22:15:31.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z15pNJlDH7z+dtwcdILIgUFQmok=', 'Content-Length': '0'} 2024-04-13T22:15:31.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080fca5d411e5bb21-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.208 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.208 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.208 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.208 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.208 DEBUG:boto:Method: GET 2024-04-13T22:15:31.208 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.208 DEBUG:boto:Data: 2024-04-13T22:15:31.208 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.208 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.208 DEBUG:boto:Params: {} 2024-04-13T22:15:31.209 DEBUG:boto:Token: None 2024-04-13T22:15:31.209 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a3c7fa9aad258b5-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.211 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.212 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.212 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.212 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.212 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.212 DEBUG:boto:Method: GET 2024-04-13T22:15:31.212 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.212 DEBUG:boto:Data: 2024-04-13T22:15:31.212 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.213 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.213 DEBUG:boto:Params: {} 2024-04-13T22:15:31.213 DEBUG:boto:Token: None 2024-04-13T22:15:31.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c4e56f6c53aee4-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.217 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.218 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.218 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.218 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.218 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.218 DEBUG:boto:Data: 2024-04-13T22:15:31.218 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.218 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.218 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.219 DEBUG:boto:Params: {} 2024-04-13T22:15:31.219 DEBUG:boto:Token: None 2024-04-13T22:15:31.219 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.219 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CwQkVM1Un4jVS6A4bZ6122VtM4s= 2024-04-13T22:15:31.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CwQkVM1Un4jVS6A4bZ6122VtM4s=', 'Content-Length': '0'} 2024-04-13T22:15:31.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a058f03bded4da-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.221 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.221 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.221 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.221 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.221 DEBUG:boto:Data: 2024-04-13T22:15:31.222 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.222 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.222 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.222 DEBUG:boto:Params: {} 2024-04-13T22:15:31.222 DEBUG:boto:Token: None 2024-04-13T22:15:31.222 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.222 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CwQkVM1Un4jVS6A4bZ6122VtM4s= 2024-04-13T22:15:31.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CwQkVM1Un4jVS6A4bZ6122VtM4s=', 'Content-Length': '0'} 2024-04-13T22:15:31.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ceeead01f38f119-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-13T22:15:31.224 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.224 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.225 DEBUG:boto:Method: GET 2024-04-13T22:15:31.225 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.225 DEBUG:boto:Data: 2024-04-13T22:15:31.225 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.225 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.225 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.225 DEBUG:boto:Params: {} 2024-04-13T22:15:31.225 DEBUG:boto:Token: None 2024-04-13T22:15:31.225 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.226 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FyLBxpTi3irofoitHYTzoiyHz48= 2024-04-13T22:15:31.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FyLBxpTi3irofoitHYTzoiyHz48=', 'Content-Length': '0'} 2024-04-13T22:15:31.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006034c93cd23dcbe9-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.228 DEBUG:boto:path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.228 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.228 DEBUG:boto:Method: GET 2024-04-13T22:15:31.228 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.228 DEBUG:boto:Data: 2024-04-13T22:15:31.228 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.228 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.229 DEBUG:boto:Params: {} 2024-04-13T22:15:31.229 DEBUG:boto:Token: None 2024-04-13T22:15:31.229 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000000 2024-04-13T22:15:31.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FyLBxpTi3irofoitHYTzoiyHz48= 2024-04-13T22:15:31.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FyLBxpTi3irofoitHYTzoiyHz48=', 'Content-Length': '0'} 2024-04-13T22:15:31.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e6048fb2d40f4c-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.231 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.231 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.231 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.231 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.232 DEBUG:boto:Method: GET 2024-04-13T22:15:31.232 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.232 DEBUG:boto:Data: 2024-04-13T22:15:31.232 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.232 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.232 DEBUG:boto:Params: {} 2024-04-13T22:15:31.232 DEBUG:boto:Token: None 2024-04-13T22:15:31.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d578f40219d6fa08-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.235 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.235 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.235 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.235 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.235 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.235 DEBUG:boto:Method: GET 2024-04-13T22:15:31.235 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000000 2024-04-13T22:15:31.236 DEBUG:boto:Data: 2024-04-13T22:15:31.236 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.236 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.236 DEBUG:boto:Params: {} 2024-04-13T22:15:31.236 DEBUG:boto:Token: None 2024-04-13T22:15:31.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b00ea6cde39a6353-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.286 DEBUG:boto:b'vwnfns-60obj000000000000000000001000falseobj000000000000000000002024-04-13T22:15:22.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-13T22:15:31.287 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.287 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.287 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.287 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.287 DEBUG:boto:Method: GET 2024-04-13T22:15:31.287 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.287 DEBUG:boto:Data: 2024-04-13T22:15:31.288 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.288 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.288 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.288 DEBUG:boto:Params: {} 2024-04-13T22:15:31.289 DEBUG:boto:Token: None 2024-04-13T22:15:31.289 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.289 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA= 2024-04-13T22:15:31.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA=', 'Content-Length': '0'} 2024-04-13T22:15:31.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea04482fd630ee67-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.292 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.292 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.292 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.292 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.292 DEBUG:boto:Method: GET 2024-04-13T22:15:31.292 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.292 DEBUG:boto:Data: 2024-04-13T22:15:31.293 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.293 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.293 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.293 DEBUG:boto:Params: {} 2024-04-13T22:15:31.293 DEBUG:boto:Token: None 2024-04-13T22:15:31.293 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.293 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA= 2024-04-13T22:15:31.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA=', 'Content-Length': '0'} 2024-04-13T22:15:31.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9b3fadf476ab55f-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.295 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.296 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.296 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.296 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.296 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.296 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.296 DEBUG:boto:Data: 2024-04-13T22:15:31.296 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.296 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.296 DEBUG:boto:Params: {} 2024-04-13T22:15:31.296 DEBUG:boto:Token: None 2024-04-13T22:15:31.297 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.297 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6grf0XdJnBFGHByhW3YRwpefYog= 2024-04-13T22:15:31.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6grf0XdJnBFGHByhW3YRwpefYog=', 'Content-Length': '0'} 2024-04-13T22:15:31.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96964de8718d5c8-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.299 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.299 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.299 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.299 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.299 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.299 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.299 DEBUG:boto:Data: 2024-04-13T22:15:31.299 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.300 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.300 DEBUG:boto:Params: {} 2024-04-13T22:15:31.300 DEBUG:boto:Token: None 2024-04-13T22:15:31.300 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6grf0XdJnBFGHByhW3YRwpefYog= 2024-04-13T22:15:31.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6grf0XdJnBFGHByhW3YRwpefYog=', 'Content-Length': '0'} 2024-04-13T22:15:31.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007869b78ee77f0017-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-13T22:15:31.302 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.302 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.303 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.303 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.303 DEBUG:boto:Method: GET 2024-04-13T22:15:31.303 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.303 DEBUG:boto:Data: 2024-04-13T22:15:31.303 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.303 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.303 DEBUG:boto:Params: {} 2024-04-13T22:15:31.303 DEBUG:boto:Token: None 2024-04-13T22:15:31.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA= 2024-04-13T22:15:31.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA=', 'Content-Length': '0'} 2024-04-13T22:15:31.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929794c370f36ceb-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.346 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.347 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.347 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.347 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.347 DEBUG:boto:Method: GET 2024-04-13T22:15:31.347 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.347 DEBUG:boto:Data: 2024-04-13T22:15:31.347 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.347 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.347 DEBUG:boto:Params: {} 2024-04-13T22:15:31.347 DEBUG:boto:Token: None 2024-04-13T22:15:31.348 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001?versionId=null 2024-04-13T22:15:31.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA= 2024-04-13T22:15:31.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G9GJLyXFC1DPLZnXRmKcVo1eflA=', 'Content-Length': '0'} 2024-04-13T22:15:31.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000769443d7a594a4aa-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.350 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.350 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.350 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.350 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.350 DEBUG:boto:Method: GET 2024-04-13T22:15:31.350 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.351 DEBUG:boto:Data: 2024-04-13T22:15:31.351 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.351 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.351 DEBUG:boto:Params: {} 2024-04-13T22:15:31.351 DEBUG:boto:Token: None 2024-04-13T22:15:31.351 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b679c41d8e909163-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.354 DEBUG:boto:b'vwnfns-60obj000000000000000000011000falseobj000000000000000000012024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.354 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.354 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.355 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.355 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.355 DEBUG:boto:Method: GET 2024-04-13T22:15:31.355 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.355 DEBUG:boto:Data: 2024-04-13T22:15:31.355 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.355 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.355 DEBUG:boto:Params: {} 2024-04-13T22:15:31.355 DEBUG:boto:Token: None 2024-04-13T22:15:31.356 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000078e1e7368aa3149-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.360 DEBUG:boto:b'vwnfns-60obj000000000000000000011000falseobj000000000000000000012024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.360 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.361 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.361 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.361 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.361 DEBUG:boto:Data: 2024-04-13T22:15:31.361 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.361 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.361 DEBUG:boto:Params: {} 2024-04-13T22:15:31.361 DEBUG:boto:Token: None 2024-04-13T22:15:31.361 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.362 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uc+Afzq1ynekQRjI9i+KMErIfwc= 2024-04-13T22:15:31.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uc+Afzq1ynekQRjI9i+KMErIfwc=', 'Content-Length': '0'} 2024-04-13T22:15:31.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b2b0e4142beb57c-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.364 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.364 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.364 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.364 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.364 DEBUG:boto:Data: 2024-04-13T22:15:31.364 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.364 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.364 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.364 DEBUG:boto:Params: {} 2024-04-13T22:15:31.364 DEBUG:boto:Token: None 2024-04-13T22:15:31.365 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uc+Afzq1ynekQRjI9i+KMErIfwc= 2024-04-13T22:15:31.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uc+Afzq1ynekQRjI9i+KMErIfwc=', 'Content-Length': '0'} 2024-04-13T22:15:31.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a49e5f0ea082ddf-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-13T22:15:31.367 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.367 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.367 DEBUG:boto:Method: GET 2024-04-13T22:15:31.367 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.367 DEBUG:boto:Data: 2024-04-13T22:15:31.367 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.367 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.367 DEBUG:boto:Params: {} 2024-04-13T22:15:31.367 DEBUG:boto:Token: None 2024-04-13T22:15:31.368 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VxFfe7oNxu8KZEqvzZI7xSgo+k8= 2024-04-13T22:15:31.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VxFfe7oNxu8KZEqvzZI7xSgo+k8=', 'Content-Length': '0'} 2024-04-13T22:15:31.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dddc322d4f573db6-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.370 DEBUG:boto:path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.370 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.370 DEBUG:boto:Method: GET 2024-04-13T22:15:31.370 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.370 DEBUG:boto:Data: 2024-04-13T22:15:31.370 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.370 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.371 DEBUG:boto:Params: {} 2024-04-13T22:15:31.371 DEBUG:boto:Token: None 2024-04-13T22:15:31.371 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000001 2024-04-13T22:15:31.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VxFfe7oNxu8KZEqvzZI7xSgo+k8= 2024-04-13T22:15:31.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VxFfe7oNxu8KZEqvzZI7xSgo+k8=', 'Content-Length': '0'} 2024-04-13T22:15:31.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134b98a342f61f34-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.414 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.414 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.415 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.415 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.415 DEBUG:boto:Method: GET 2024-04-13T22:15:31.415 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.415 DEBUG:boto:Data: 2024-04-13T22:15:31.415 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.415 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.415 DEBUG:boto:Params: {} 2024-04-13T22:15:31.415 DEBUG:boto:Token: None 2024-04-13T22:15:31.416 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce2aef0990ca66a-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.418 DEBUG:boto:b'vwnfns-60obj000000000000000000011000falseobj000000000000000000012024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.419 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.419 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.419 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.419 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.419 DEBUG:boto:Method: GET 2024-04-13T22:15:31.419 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000001 2024-04-13T22:15:31.419 DEBUG:boto:Data: 2024-04-13T22:15:31.419 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.419 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.419 DEBUG:boto:Params: {} 2024-04-13T22:15:31.420 DEBUG:boto:Token: None 2024-04-13T22:15:31.420 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.420 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e704261066b1315e-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.424 DEBUG:boto:b'vwnfns-60obj000000000000000000011000falseobj000000000000000000012024-04-13T22:15:22.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-13T22:15:31.424 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.424 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.424 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.425 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.425 DEBUG:boto:Method: GET 2024-04-13T22:15:31.425 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.425 DEBUG:boto:Data: 2024-04-13T22:15:31.425 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.425 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.425 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.425 DEBUG:boto:Params: {} 2024-04-13T22:15:31.425 DEBUG:boto:Token: None 2024-04-13T22:15:31.425 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.426 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k= 2024-04-13T22:15:31.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k=', 'Content-Length': '0'} 2024-04-13T22:15:31.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459827b3fa4de403-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.428 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.428 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.429 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.429 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.429 DEBUG:boto:Method: GET 2024-04-13T22:15:31.429 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.429 DEBUG:boto:Data: 2024-04-13T22:15:31.429 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.429 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.429 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.429 DEBUG:boto:Params: {} 2024-04-13T22:15:31.429 DEBUG:boto:Token: None 2024-04-13T22:15:31.429 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.430 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k= 2024-04-13T22:15:31.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k=', 'Content-Length': '0'} 2024-04-13T22:15:31.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e214a6e7eb6d277-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.432 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.432 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.432 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.432 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.432 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.432 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.432 DEBUG:boto:Data: 2024-04-13T22:15:31.433 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.433 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.433 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.433 DEBUG:boto:Params: {} 2024-04-13T22:15:31.433 DEBUG:boto:Token: None 2024-04-13T22:15:31.433 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.433 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3voMnItwJ93UwaQfAHKAYt5GMVY= 2024-04-13T22:15:31.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3voMnItwJ93UwaQfAHKAYt5GMVY=', 'Content-Length': '0'} 2024-04-13T22:15:31.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028badb7964534ab2-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.435 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.435 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.435 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.435 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.436 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.436 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.436 DEBUG:boto:Data: 2024-04-13T22:15:31.436 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.436 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.436 DEBUG:boto:Params: {} 2024-04-13T22:15:31.436 DEBUG:boto:Token: None 2024-04-13T22:15:31.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3voMnItwJ93UwaQfAHKAYt5GMVY= 2024-04-13T22:15:31.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3voMnItwJ93UwaQfAHKAYt5GMVY=', 'Content-Length': '0'} 2024-04-13T22:15:31.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e417d1ea1f61d2e-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-13T22:15:31.439 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.439 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.439 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.439 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.439 DEBUG:boto:Method: GET 2024-04-13T22:15:31.440 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.440 DEBUG:boto:Data: 2024-04-13T22:15:31.440 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.440 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.440 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.440 DEBUG:boto:Params: {} 2024-04-13T22:15:31.440 DEBUG:boto:Token: None 2024-04-13T22:15:31.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k= 2024-04-13T22:15:31.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k=', 'Content-Length': '0'} 2024-04-13T22:15:31.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3e5b383cb55c03-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.442 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.442 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.443 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.443 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.443 DEBUG:boto:Method: GET 2024-04-13T22:15:31.443 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.443 DEBUG:boto:Data: 2024-04-13T22:15:31.443 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.443 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.443 DEBUG:boto:Params: {} 2024-04-13T22:15:31.443 DEBUG:boto:Token: None 2024-04-13T22:15:31.444 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002?versionId=null 2024-04-13T22:15:31.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k= 2024-04-13T22:15:31.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QJl1uNlfSLRlq7g3DE6ccko1e7k=', 'Content-Length': '0'} 2024-04-13T22:15:31.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507935524953fa7d-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.486 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.487 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.487 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.487 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.487 DEBUG:boto:Method: GET 2024-04-13T22:15:31.487 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.487 DEBUG:boto:Data: 2024-04-13T22:15:31.487 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.487 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.488 DEBUG:boto:Params: {} 2024-04-13T22:15:31.488 DEBUG:boto:Token: None 2024-04-13T22:15:31.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ca738203a4bae8-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.491 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.494 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.494 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.494 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.494 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.494 DEBUG:boto:Method: GET 2024-04-13T22:15:31.494 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.494 DEBUG:boto:Data: 2024-04-13T22:15:31.494 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.494 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.494 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.495 DEBUG:boto:Params: {} 2024-04-13T22:15:31.495 DEBUG:boto:Token: None 2024-04-13T22:15:31.495 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a623cb64593f52-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.499 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.500 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.500 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.500 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.500 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.500 DEBUG:boto:Data: 2024-04-13T22:15:31.500 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.500 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.500 DEBUG:boto:Params: {} 2024-04-13T22:15:31.501 DEBUG:boto:Token: None 2024-04-13T22:15:31.501 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aTSvCNryFaR/7X1rKHnQGPh04G8= 2024-04-13T22:15:31.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aTSvCNryFaR/7X1rKHnQGPh04G8=', 'Content-Length': '0'} 2024-04-13T22:15:31.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703a7982dd16e43a-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.503 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.503 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.503 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.503 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.504 DEBUG:boto:Data: 2024-04-13T22:15:31.504 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.504 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.504 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.504 DEBUG:boto:Params: {} 2024-04-13T22:15:31.504 DEBUG:boto:Token: None 2024-04-13T22:15:31.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aTSvCNryFaR/7X1rKHnQGPh04G8= 2024-04-13T22:15:31.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aTSvCNryFaR/7X1rKHnQGPh04G8=', 'Content-Length': '0'} 2024-04-13T22:15:31.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094fcfe368739c39e-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-13T22:15:31.506 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.506 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.506 DEBUG:boto:Method: GET 2024-04-13T22:15:31.507 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.507 DEBUG:boto:Data: 2024-04-13T22:15:31.507 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.507 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.507 DEBUG:boto:Params: {} 2024-04-13T22:15:31.507 DEBUG:boto:Token: None 2024-04-13T22:15:31.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5DCUnJLMpPoZDwOgbKIzyz4kOZs= 2024-04-13T22:15:31.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5DCUnJLMpPoZDwOgbKIzyz4kOZs=', 'Content-Length': '0'} 2024-04-13T22:15:31.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1fdae20e39a912-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.554 DEBUG:boto:path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.555 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.555 DEBUG:boto:Method: GET 2024-04-13T22:15:31.555 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.555 DEBUG:boto:Data: 2024-04-13T22:15:31.555 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.555 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.555 DEBUG:boto:Params: {} 2024-04-13T22:15:31.555 DEBUG:boto:Token: None 2024-04-13T22:15:31.556 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000002 2024-04-13T22:15:31.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5DCUnJLMpPoZDwOgbKIzyz4kOZs= 2024-04-13T22:15:31.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5DCUnJLMpPoZDwOgbKIzyz4kOZs=', 'Content-Length': '0'} 2024-04-13T22:15:31.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17031d453f13a2f-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.558 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.558 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.558 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.558 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.558 DEBUG:boto:Method: GET 2024-04-13T22:15:31.558 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.559 DEBUG:boto:Data: 2024-04-13T22:15:31.559 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.559 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.559 DEBUG:boto:Params: {} 2024-04-13T22:15:31.559 DEBUG:boto:Token: None 2024-04-13T22:15:31.559 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9986fc3275a325-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.562 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.562 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.562 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.563 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.563 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.563 DEBUG:boto:Method: GET 2024-04-13T22:15:31.563 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000002 2024-04-13T22:15:31.563 DEBUG:boto:Data: 2024-04-13T22:15:31.563 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.563 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.563 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.563 DEBUG:boto:Params: {} 2024-04-13T22:15:31.563 DEBUG:boto:Token: None 2024-04-13T22:15:31.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000917dc2dd69138a0-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.568 DEBUG:boto:b'vwnfns-60obj000000000000000000021000falseobj000000000000000000022024-04-13T22:15:22.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-13T22:15:31.568 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.569 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.569 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.569 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.569 DEBUG:boto:Method: GET 2024-04-13T22:15:31.569 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.569 DEBUG:boto:Data: 2024-04-13T22:15:31.569 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.569 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.569 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.569 DEBUG:boto:Params: {} 2024-04-13T22:15:31.570 DEBUG:boto:Token: None 2024-04-13T22:15:31.570 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.570 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI= 2024-04-13T22:15:31.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI=', 'Content-Length': '0'} 2024-04-13T22:15:31.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000088798c65a5fc5a1-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.573 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.573 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.573 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.573 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.573 DEBUG:boto:Method: GET 2024-04-13T22:15:31.573 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.573 DEBUG:boto:Data: 2024-04-13T22:15:31.573 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.574 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.574 DEBUG:boto:Params: {} 2024-04-13T22:15:31.574 DEBUG:boto:Token: None 2024-04-13T22:15:31.574 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.574 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI= 2024-04-13T22:15:31.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI=', 'Content-Length': '0'} 2024-04-13T22:15:31.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d2aa2619f95b57-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.618 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.619 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.619 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.619 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.619 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.619 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.619 DEBUG:boto:Data: 2024-04-13T22:15:31.619 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.619 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.619 DEBUG:boto:Params: {} 2024-04-13T22:15:31.620 DEBUG:boto:Token: None 2024-04-13T22:15:31.620 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:blah8waiv3Hq/RXsl0NjVwxEYJo= 2024-04-13T22:15:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:blah8waiv3Hq/RXsl0NjVwxEYJo=', 'Content-Length': '0'} 2024-04-13T22:15:31.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc14fda5bbe95dd-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.622 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.622 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.622 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.622 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.622 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.623 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.623 DEBUG:boto:Data: 2024-04-13T22:15:31.623 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.623 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.623 DEBUG:boto:Params: {} 2024-04-13T22:15:31.623 DEBUG:boto:Token: None 2024-04-13T22:15:31.623 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:blah8waiv3Hq/RXsl0NjVwxEYJo= 2024-04-13T22:15:31.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:blah8waiv3Hq/RXsl0NjVwxEYJo=', 'Content-Length': '0'} 2024-04-13T22:15:31.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020dfd7430bdc2762-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-13T22:15:31.626 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.626 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.626 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.626 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.626 DEBUG:boto:Method: GET 2024-04-13T22:15:31.626 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.626 DEBUG:boto:Data: 2024-04-13T22:15:31.626 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.626 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.626 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.626 DEBUG:boto:Params: {} 2024-04-13T22:15:31.627 DEBUG:boto:Token: None 2024-04-13T22:15:31.627 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.627 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI= 2024-04-13T22:15:31.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI=', 'Content-Length': '0'} 2024-04-13T22:15:31.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd27ee7b1c97f802-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.629 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.629 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.629 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.629 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.629 DEBUG:boto:Method: GET 2024-04-13T22:15:31.629 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.629 DEBUG:boto:Data: 2024-04-13T22:15:31.630 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.630 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.630 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.630 DEBUG:boto:Params: {} 2024-04-13T22:15:31.630 DEBUG:boto:Token: None 2024-04-13T22:15:31.630 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003?versionId=null 2024-04-13T22:15:31.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI= 2024-04-13T22:15:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZRIZWEGZBN4o3l2kaFcw5XGEoRI=', 'Content-Length': '0'} 2024-04-13T22:15:31.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adefda66041a1316-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.632 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.632 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.632 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.632 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.633 DEBUG:boto:Method: GET 2024-04-13T22:15:31.633 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.633 DEBUG:boto:Data: 2024-04-13T22:15:31.633 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.633 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.633 DEBUG:boto:Params: {} 2024-04-13T22:15:31.633 DEBUG:boto:Token: None 2024-04-13T22:15:31.633 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031155c4e9e698fd5-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.636 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.637 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.637 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.637 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.637 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.637 DEBUG:boto:Method: GET 2024-04-13T22:15:31.637 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.637 DEBUG:boto:Data: 2024-04-13T22:15:31.637 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.637 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.637 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.638 DEBUG:boto:Params: {} 2024-04-13T22:15:31.638 DEBUG:boto:Token: None 2024-04-13T22:15:31.638 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3fc52f8a59b34d-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.642 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.642 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.642 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.643 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.643 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.643 DEBUG:boto:Data: 2024-04-13T22:15:31.643 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.643 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.643 DEBUG:boto:Params: {} 2024-04-13T22:15:31.643 DEBUG:boto:Token: None 2024-04-13T22:15:31.643 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GV0DBKb3uQjXlyxQc1NVxFSFGdw= 2024-04-13T22:15:31.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GV0DBKb3uQjXlyxQc1NVxFSFGdw=', 'Content-Length': '0'} 2024-04-13T22:15:31.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e28e19aa343e864-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.645 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.646 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.646 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.646 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.646 DEBUG:boto:Data: 2024-04-13T22:15:31.646 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.646 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.646 DEBUG:boto:Params: {} 2024-04-13T22:15:31.646 DEBUG:boto:Token: None 2024-04-13T22:15:31.647 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GV0DBKb3uQjXlyxQc1NVxFSFGdw= 2024-04-13T22:15:31.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GV0DBKb3uQjXlyxQc1NVxFSFGdw=', 'Content-Length': '0'} 2024-04-13T22:15:31.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef625ba4875a868-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-13T22:15:31.649 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.649 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.649 DEBUG:boto:Method: GET 2024-04-13T22:15:31.649 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.649 DEBUG:boto:Data: 2024-04-13T22:15:31.649 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.650 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.650 DEBUG:boto:Params: {} 2024-04-13T22:15:31.650 DEBUG:boto:Token: None 2024-04-13T22:15:31.650 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.650 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7vAeYL/QVRxboQumcu9IQk2Evtg= 2024-04-13T22:15:31.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7vAeYL/QVRxboQumcu9IQk2Evtg=', 'Content-Length': '0'} 2024-04-13T22:15:31.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d156595232ae2eea-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.652 DEBUG:boto:path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.652 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.652 DEBUG:boto:Method: GET 2024-04-13T22:15:31.652 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.652 DEBUG:boto:Data: 2024-04-13T22:15:31.653 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.653 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.653 DEBUG:boto:Params: {} 2024-04-13T22:15:31.653 DEBUG:boto:Token: None 2024-04-13T22:15:31.653 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000003 2024-04-13T22:15:31.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7vAeYL/QVRxboQumcu9IQk2Evtg= 2024-04-13T22:15:31.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7vAeYL/QVRxboQumcu9IQk2Evtg=', 'Content-Length': '0'} 2024-04-13T22:15:31.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae01a1d3228787f-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.656 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.656 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.656 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.656 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.656 DEBUG:boto:Method: GET 2024-04-13T22:15:31.656 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.656 DEBUG:boto:Data: 2024-04-13T22:15:31.656 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.656 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.657 DEBUG:boto:Params: {} 2024-04-13T22:15:31.657 DEBUG:boto:Token: None 2024-04-13T22:15:31.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a677afb25d839764-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.659 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.660 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.660 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.660 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.660 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.660 DEBUG:boto:Method: GET 2024-04-13T22:15:31.660 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000003 2024-04-13T22:15:31.660 DEBUG:boto:Data: 2024-04-13T22:15:31.660 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.660 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.661 DEBUG:boto:Params: {} 2024-04-13T22:15:31.661 DEBUG:boto:Token: None 2024-04-13T22:15:31.661 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.661 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f59fa18dafd42b21-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.665 DEBUG:boto:b'vwnfns-60obj000000000000000000031000falseobj000000000000000000032024-04-13T22:15:22.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-13T22:15:31.666 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.666 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.666 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.666 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.666 DEBUG:boto:Method: GET 2024-04-13T22:15:31.666 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.666 DEBUG:boto:Data: 2024-04-13T22:15:31.667 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.667 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.667 DEBUG:boto:Params: {} 2024-04-13T22:15:31.667 DEBUG:boto:Token: None 2024-04-13T22:15:31.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo= 2024-04-13T22:15:31.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo=', 'Content-Length': '0'} 2024-04-13T22:15:31.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d04ee4de99754f-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.710 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.711 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.711 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.711 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.711 DEBUG:boto:Method: GET 2024-04-13T22:15:31.711 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.711 DEBUG:boto:Data: 2024-04-13T22:15:31.711 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.712 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.712 DEBUG:boto:Params: {} 2024-04-13T22:15:31.712 DEBUG:boto:Token: None 2024-04-13T22:15:31.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo= 2024-04-13T22:15:31.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo=', 'Content-Length': '0'} 2024-04-13T22:15:31.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a712c41765c76ddb-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.715 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.715 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.715 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.715 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.715 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.715 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.716 DEBUG:boto:Data: 2024-04-13T22:15:31.716 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.716 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.716 DEBUG:boto:Params: {} 2024-04-13T22:15:31.716 DEBUG:boto:Token: None 2024-04-13T22:15:31.716 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8//zpHOE1ZL4d6QW6FSyTb2vlAQ= 2024-04-13T22:15:31.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8//zpHOE1ZL4d6QW6FSyTb2vlAQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0c16eb4139e270-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.719 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.719 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.719 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.719 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.719 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.719 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.719 DEBUG:boto:Data: 2024-04-13T22:15:31.719 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.719 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.720 DEBUG:boto:Params: {} 2024-04-13T22:15:31.720 DEBUG:boto:Token: None 2024-04-13T22:15:31.720 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8//zpHOE1ZL4d6QW6FSyTb2vlAQ= 2024-04-13T22:15:31.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8//zpHOE1ZL4d6QW6FSyTb2vlAQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000769db789fe846dcd-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-13T22:15:31.722 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.722 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.722 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.722 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.723 DEBUG:boto:Method: GET 2024-04-13T22:15:31.723 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.723 DEBUG:boto:Data: 2024-04-13T22:15:31.723 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.723 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.723 DEBUG:boto:Params: {} 2024-04-13T22:15:31.723 DEBUG:boto:Token: None 2024-04-13T22:15:31.723 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo= 2024-04-13T22:15:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo=', 'Content-Length': '0'} 2024-04-13T22:15:31.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099843371068c1aac-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.725 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.726 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.726 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.726 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.726 DEBUG:boto:Method: GET 2024-04-13T22:15:31.726 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.726 DEBUG:boto:Data: 2024-04-13T22:15:31.726 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.726 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.726 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.727 DEBUG:boto:Params: {} 2024-04-13T22:15:31.727 DEBUG:boto:Token: None 2024-04-13T22:15:31.727 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004?versionId=null 2024-04-13T22:15:31.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo= 2024-04-13T22:15:31.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w29I0COTex9qF658IFCdro8kBYo=', 'Content-Length': '0'} 2024-04-13T22:15:31.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000845363906c0fcae5-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.770 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.771 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.771 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.771 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.771 DEBUG:boto:Method: GET 2024-04-13T22:15:31.771 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.771 DEBUG:boto:Data: 2024-04-13T22:15:31.771 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.771 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.772 DEBUG:boto:Params: {} 2024-04-13T22:15:31.772 DEBUG:boto:Token: None 2024-04-13T22:15:31.772 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041562a33fa81d6ff-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.774 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.775 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.775 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.775 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.775 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.775 DEBUG:boto:Method: GET 2024-04-13T22:15:31.776 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.776 DEBUG:boto:Data: 2024-04-13T22:15:31.776 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.776 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.776 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.776 DEBUG:boto:Params: {} 2024-04-13T22:15:31.776 DEBUG:boto:Token: None 2024-04-13T22:15:31.776 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.777 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbce0a8d590c129-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.781 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.781 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.781 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.781 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.782 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.782 DEBUG:boto:Data: 2024-04-13T22:15:31.782 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.782 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.782 DEBUG:boto:Params: {} 2024-04-13T22:15:31.782 DEBUG:boto:Token: None 2024-04-13T22:15:31.782 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qsd3saMgDzk54+gJZ00h/jCjhNE= 2024-04-13T22:15:31.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qsd3saMgDzk54+gJZ00h/jCjhNE=', 'Content-Length': '0'} 2024-04-13T22:15:31.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008802dfd7c96251d8-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.785 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.785 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.785 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.785 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.785 DEBUG:boto:Data: 2024-04-13T22:15:31.785 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.785 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.786 DEBUG:boto:Params: {} 2024-04-13T22:15:31.786 DEBUG:boto:Token: None 2024-04-13T22:15:31.786 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qsd3saMgDzk54+gJZ00h/jCjhNE= 2024-04-13T22:15:31.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qsd3saMgDzk54+gJZ00h/jCjhNE=', 'Content-Length': '0'} 2024-04-13T22:15:31.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b61ecb3868ca4f-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-13T22:15:31.788 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.788 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.788 DEBUG:boto:Method: GET 2024-04-13T22:15:31.788 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.788 DEBUG:boto:Data: 2024-04-13T22:15:31.788 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.789 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.789 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.789 DEBUG:boto:Params: {} 2024-04-13T22:15:31.789 DEBUG:boto:Token: None 2024-04-13T22:15:31.789 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bn4fDUBupmO9mBiPsfaERwBTHWk= 2024-04-13T22:15:31.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bn4fDUBupmO9mBiPsfaERwBTHWk=', 'Content-Length': '0'} 2024-04-13T22:15:31.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561f6bb2f93319ad-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.834 DEBUG:boto:path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.835 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.835 DEBUG:boto:Method: GET 2024-04-13T22:15:31.835 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.835 DEBUG:boto:Data: 2024-04-13T22:15:31.835 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.835 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.835 DEBUG:boto:Params: {} 2024-04-13T22:15:31.835 DEBUG:boto:Token: None 2024-04-13T22:15:31.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000004 2024-04-13T22:15:31.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bn4fDUBupmO9mBiPsfaERwBTHWk= 2024-04-13T22:15:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bn4fDUBupmO9mBiPsfaERwBTHWk=', 'Content-Length': '0'} 2024-04-13T22:15:31.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8061d0792cc756-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.838 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.838 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.838 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.838 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.838 DEBUG:boto:Method: GET 2024-04-13T22:15:31.838 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.839 DEBUG:boto:Data: 2024-04-13T22:15:31.839 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.839 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.839 DEBUG:boto:Params: {} 2024-04-13T22:15:31.839 DEBUG:boto:Token: None 2024-04-13T22:15:31.839 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.839 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d489928e55c6b330-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.842 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.842 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.842 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.843 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.843 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.843 DEBUG:boto:Method: GET 2024-04-13T22:15:31.843 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000004 2024-04-13T22:15:31.843 DEBUG:boto:Data: 2024-04-13T22:15:31.843 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.843 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.843 DEBUG:boto:Params: {} 2024-04-13T22:15:31.843 DEBUG:boto:Token: None 2024-04-13T22:15:31.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b227392c919b104a-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.847 DEBUG:boto:b'vwnfns-60obj000000000000000000041000falseobj000000000000000000042024-04-13T22:15:22.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-13T22:15:31.848 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.848 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.848 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.848 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.848 DEBUG:boto:Method: GET 2024-04-13T22:15:31.848 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.848 DEBUG:boto:Data: 2024-04-13T22:15:31.849 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.849 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.849 DEBUG:boto:Params: {} 2024-04-13T22:15:31.849 DEBUG:boto:Token: None 2024-04-13T22:15:31.849 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws= 2024-04-13T22:15:31.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws=', 'Content-Length': '0'} 2024-04-13T22:15:31.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6d373890d7b3dd-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.851 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.851 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.851 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.852 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.852 DEBUG:boto:Method: GET 2024-04-13T22:15:31.852 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.852 DEBUG:boto:Data: 2024-04-13T22:15:31.852 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.852 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.852 DEBUG:boto:Params: {} 2024-04-13T22:15:31.852 DEBUG:boto:Token: None 2024-04-13T22:15:31.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws= 2024-04-13T22:15:31.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws=', 'Content-Length': '0'} 2024-04-13T22:15:31.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003267dc8d37e2e067-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.901 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.901 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.901 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.901 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.902 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.902 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.902 DEBUG:boto:Data: 2024-04-13T22:15:31.902 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.902 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.902 DEBUG:boto:Params: {} 2024-04-13T22:15:31.902 DEBUG:boto:Token: None 2024-04-13T22:15:31.902 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VFhZ001/3uv5XWTSdFt11FtPy+Y= 2024-04-13T22:15:31.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VFhZ001/3uv5XWTSdFt11FtPy+Y=', 'Content-Length': '0'} 2024-04-13T22:15:31.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f8e42276b05421-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.905 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.905 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.905 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.905 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.905 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.905 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.905 DEBUG:boto:Data: 2024-04-13T22:15:31.905 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.905 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.906 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.906 DEBUG:boto:Params: {} 2024-04-13T22:15:31.906 DEBUG:boto:Token: None 2024-04-13T22:15:31.906 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.906 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VFhZ001/3uv5XWTSdFt11FtPy+Y= 2024-04-13T22:15:31.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VFhZ001/3uv5XWTSdFt11FtPy+Y=', 'Content-Length': '0'} 2024-04-13T22:15:31.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c212fb65d024f365-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-13T22:15:31.908 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.908 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.908 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.908 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.908 DEBUG:boto:Method: GET 2024-04-13T22:15:31.908 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.909 DEBUG:boto:Data: 2024-04-13T22:15:31.909 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.909 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.909 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.909 DEBUG:boto:Params: {} 2024-04-13T22:15:31.909 DEBUG:boto:Token: None 2024-04-13T22:15:31.909 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws= 2024-04-13T22:15:31.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws=', 'Content-Length': '0'} 2024-04-13T22:15:31.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b11455029779ec7-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.911 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.911 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.911 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.911 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.911 DEBUG:boto:Method: GET 2024-04-13T22:15:31.911 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.912 DEBUG:boto:Data: 2024-04-13T22:15:31.912 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.912 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.912 DEBUG:boto:Params: {} 2024-04-13T22:15:31.912 DEBUG:boto:Token: None 2024-04-13T22:15:31.912 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005?versionId=null 2024-04-13T22:15:31.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws= 2024-04-13T22:15:31.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TVfME9La5GefJHzt6itBNt82Iws=', 'Content-Length': '0'} 2024-04-13T22:15:31.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1cae1659618b17-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.914 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.915 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.915 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.915 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.915 DEBUG:boto:Method: GET 2024-04-13T22:15:31.915 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.915 DEBUG:boto:Data: 2024-04-13T22:15:31.915 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.915 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.915 DEBUG:boto:Params: {} 2024-04-13T22:15:31.915 DEBUG:boto:Token: None 2024-04-13T22:15:31.916 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d50605bfabe45f-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.918 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.919 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.919 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.919 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.919 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.919 DEBUG:boto:Method: GET 2024-04-13T22:15:31.920 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.920 DEBUG:boto:Data: 2024-04-13T22:15:31.920 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.920 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.920 DEBUG:boto:Params: {} 2024-04-13T22:15:31.920 DEBUG:boto:Token: None 2024-04-13T22:15:31.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b4ae7e27851b5ae-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.925 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.925 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.925 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.926 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.926 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.926 DEBUG:boto:Data: 2024-04-13T22:15:31.926 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.926 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.926 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.926 DEBUG:boto:Params: {} 2024-04-13T22:15:31.926 DEBUG:boto:Token: None 2024-04-13T22:15:31.926 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.927 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gBK/WzxL/ZKb0L9JrwfR3wiZnUk= 2024-04-13T22:15:31.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gBK/WzxL/ZKb0L9JrwfR3wiZnUk=', 'Content-Length': '0'} 2024-04-13T22:15:31.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3aff94cdd72eab3-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.928 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.928 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.929 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.929 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.929 DEBUG:boto:Data: 2024-04-13T22:15:31.929 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.929 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.929 DEBUG:boto:Params: {} 2024-04-13T22:15:31.929 DEBUG:boto:Token: None 2024-04-13T22:15:31.929 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gBK/WzxL/ZKb0L9JrwfR3wiZnUk= 2024-04-13T22:15:31.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gBK/WzxL/ZKb0L9JrwfR3wiZnUk=', 'Content-Length': '0'} 2024-04-13T22:15:31.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a05c400d8f77738-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-13T22:15:31.931 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.931 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.931 DEBUG:boto:Method: GET 2024-04-13T22:15:31.932 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.932 DEBUG:boto:Data: 2024-04-13T22:15:31.932 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.932 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.932 DEBUG:boto:Params: {} 2024-04-13T22:15:31.932 DEBUG:boto:Token: None 2024-04-13T22:15:31.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3lHEzMdDpgOshD8Hs0bqn1HUe8c= 2024-04-13T22:15:31.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3lHEzMdDpgOshD8Hs0bqn1HUe8c=', 'Content-Length': '0'} 2024-04-13T22:15:31.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745e123c4687fa98-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.974 DEBUG:boto:path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.975 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.975 DEBUG:boto:Method: GET 2024-04-13T22:15:31.975 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.975 DEBUG:boto:Data: 2024-04-13T22:15:31.975 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.975 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.975 DEBUG:boto:Params: {} 2024-04-13T22:15:31.975 DEBUG:boto:Token: None 2024-04-13T22:15:31.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000005 2024-04-13T22:15:31.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3lHEzMdDpgOshD8Hs0bqn1HUe8c= 2024-04-13T22:15:31.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3lHEzMdDpgOshD8Hs0bqn1HUe8c=', 'Content-Length': '0'} 2024-04-13T22:15:31.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f17d4f59edd7c7db-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.978 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.978 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.978 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.978 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.978 DEBUG:boto:Method: GET 2024-04-13T22:15:31.978 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.979 DEBUG:boto:Data: 2024-04-13T22:15:31.979 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.979 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.979 DEBUG:boto:Params: {} 2024-04-13T22:15:31.979 DEBUG:boto:Token: None 2024-04-13T22:15:31.979 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.979 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b906ec4c06fe667-00661b0403-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.983 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.983 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:31.983 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:31.983 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.983 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.983 DEBUG:boto:Method: GET 2024-04-13T22:15:31.984 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000005 2024-04-13T22:15:31.984 DEBUG:boto:Data: 2024-04-13T22:15:31.984 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.984 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.984 DEBUG:boto:Params: {} 2024-04-13T22:15:31.984 DEBUG:boto:Token: None 2024-04-13T22:15:31.984 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/ 2024-04-13T22:15:31.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ= 2024-04-13T22:15:31.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8xIn0vXeq6n+wWGrW0V/0AR8ZBQ=', 'Content-Length': '0'} 2024-04-13T22:15:31.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4c42a8f827e722d-00661b0403-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.988 DEBUG:boto:b'vwnfns-60obj000000000000000000051000falseobj000000000000000000052024-04-13T22:15:21.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:31.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-13T22:15:31.989 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:31.989 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:31.989 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.989 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.989 DEBUG:boto:Method: GET 2024-04-13T22:15:31.990 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.990 DEBUG:boto:Data: 2024-04-13T22:15:31.990 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.990 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.990 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.990 DEBUG:boto:Params: {} 2024-04-13T22:15:31.990 DEBUG:boto:Token: None 2024-04-13T22:15:31.990 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3g53Rpa2WVDQ9Ti2JtLXWonV650= 2024-04-13T22:15:31.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3g53Rpa2WVDQ9Ti2JtLXWonV650=', 'Content-Length': '0'} 2024-04-13T22:15:31.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c6da1d45192d72-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.993 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:31.993 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:31.993 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.993 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.993 DEBUG:boto:Method: GET 2024-04-13T22:15:31.994 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.994 DEBUG:boto:Data: 2024-04-13T22:15:31.994 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:31.994 DEBUG:boto:Port: 8000 2024-04-13T22:15:31.994 DEBUG:boto:Params: {} 2024-04-13T22:15:31.994 DEBUG:boto:Token: None 2024-04-13T22:15:31.994 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3g53Rpa2WVDQ9Ti2JtLXWonV650= 2024-04-13T22:15:31.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3g53Rpa2WVDQ9Ti2JtLXWonV650=', 'Content-Length': '0'} 2024-04-13T22:15:31.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c601a3ae13631ff-00661b0403-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:31.996 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:31.997 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:31.997 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.997 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.997 DEBUG:boto:Method: HEAD 2024-04-13T22:15:31.997 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.997 DEBUG:boto:Data: 2024-04-13T22:15:31.997 DEBUG:boto:Headers: {} 2024-04-13T22:15:31.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:31.997 DEBUG:boto:Port: 8001 2024-04-13T22:15:31.997 DEBUG:boto:Params: {} 2024-04-13T22:15:31.998 DEBUG:boto:Token: None 2024-04-13T22:15:31.998 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:31 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:31.998 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NPOzg1J5DxvpxvPBnuaGciYh+Xw= 2024-04-13T22:15:31.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NPOzg1J5DxvpxvPBnuaGciYh+Xw=', 'Content-Length': '0'} 2024-04-13T22:15:32.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cabddb0bc55e16-00661b0403-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.000 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.000 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.000 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.000 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.000 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.000 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.000 DEBUG:boto:Data: 2024-04-13T22:15:32.001 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.001 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.001 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.001 DEBUG:boto:Params: {} 2024-04-13T22:15:32.001 DEBUG:boto:Token: None 2024-04-13T22:15:32.001 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GyvBvGAHcQqawKr3tBNdbhfgeUQ= 2024-04-13T22:15:32.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GyvBvGAHcQqawKr3tBNdbhfgeUQ=', 'Content-Length': '0'} 2024-04-13T22:15:32.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac369c63b9f912b-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-13T22:15:32.003 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.004 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.004 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.004 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.004 DEBUG:boto:Method: GET 2024-04-13T22:15:32.004 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.004 DEBUG:boto:Data: 2024-04-13T22:15:32.004 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.004 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.004 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.004 DEBUG:boto:Params: {} 2024-04-13T22:15:32.005 DEBUG:boto:Token: None 2024-04-13T22:15:32.005 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.005 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xnfWZhgGrlzYjb2R+0fk7FVTinw= 2024-04-13T22:15:32.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xnfWZhgGrlzYjb2R+0fk7FVTinw=', 'Content-Length': '0'} 2024-04-13T22:15:32.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009514bfe3363346b9-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.007 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.007 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.007 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.007 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.007 DEBUG:boto:Method: GET 2024-04-13T22:15:32.007 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.007 DEBUG:boto:Data: 2024-04-13T22:15:32.008 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.008 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.008 DEBUG:boto:Params: {} 2024-04-13T22:15:32.008 DEBUG:boto:Token: None 2024-04-13T22:15:32.008 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006?versionId=null 2024-04-13T22:15:32.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xnfWZhgGrlzYjb2R+0fk7FVTinw= 2024-04-13T22:15:32.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xnfWZhgGrlzYjb2R+0fk7FVTinw=', 'Content-Length': '0'} 2024-04-13T22:15:32.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e25f4c94377093-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.010 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.010 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.010 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.011 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.011 DEBUG:boto:Method: GET 2024-04-13T22:15:32.011 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.011 DEBUG:boto:Data: 2024-04-13T22:15:32.011 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.011 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.011 DEBUG:boto:Params: {} 2024-04-13T22:15:32.011 DEBUG:boto:Token: None 2024-04-13T22:15:32.011 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07c6c1caf4c97f4-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.054 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.055 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.055 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.055 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.055 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.055 DEBUG:boto:Method: GET 2024-04-13T22:15:32.055 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.055 DEBUG:boto:Data: 2024-04-13T22:15:32.055 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.055 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.055 DEBUG:boto:Params: {} 2024-04-13T22:15:32.056 DEBUG:boto:Token: None 2024-04-13T22:15:32.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007127bdf48bb2c27c-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.060 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.060 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.060 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.060 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.060 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.060 DEBUG:boto:Data: 2024-04-13T22:15:32.060 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.061 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.061 DEBUG:boto:Params: {} 2024-04-13T22:15:32.061 DEBUG:boto:Token: None 2024-04-13T22:15:32.061 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k2Y7VOo1fThvvIqiN0xnKk75qLQ= 2024-04-13T22:15:32.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k2Y7VOo1fThvvIqiN0xnKk75qLQ=', 'Content-Length': '0'} 2024-04-13T22:15:32.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009730a74199a9440e-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.063 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.063 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.063 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.063 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.063 DEBUG:boto:Data: 2024-04-13T22:15:32.063 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.063 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.064 DEBUG:boto:Params: {} 2024-04-13T22:15:32.064 DEBUG:boto:Token: None 2024-04-13T22:15:32.064 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.064 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k2Y7VOo1fThvvIqiN0xnKk75qLQ= 2024-04-13T22:15:32.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k2Y7VOo1fThvvIqiN0xnKk75qLQ=', 'Content-Length': '0'} 2024-04-13T22:15:32.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b15f20faf369678-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-13T22:15:32.066 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.066 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.066 DEBUG:boto:Method: GET 2024-04-13T22:15:32.066 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.066 DEBUG:boto:Data: 2024-04-13T22:15:32.066 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.066 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.067 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.067 DEBUG:boto:Params: {} 2024-04-13T22:15:32.067 DEBUG:boto:Token: None 2024-04-13T22:15:32.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHO9M5GozNfWrwSjSzQb9UJG2DM= 2024-04-13T22:15:32.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHO9M5GozNfWrwSjSzQb9UJG2DM=', 'Content-Length': '0'} 2024-04-13T22:15:32.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003107ec3e3a2eb982-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.069 DEBUG:boto:path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.069 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.069 DEBUG:boto:Method: GET 2024-04-13T22:15:32.069 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.069 DEBUG:boto:Data: 2024-04-13T22:15:32.069 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.069 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.069 DEBUG:boto:Params: {} 2024-04-13T22:15:32.070 DEBUG:boto:Token: None 2024-04-13T22:15:32.070 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000006 2024-04-13T22:15:32.070 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sHO9M5GozNfWrwSjSzQb9UJG2DM= 2024-04-13T22:15:32.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sHO9M5GozNfWrwSjSzQb9UJG2DM=', 'Content-Length': '0'} 2024-04-13T22:15:32.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a833192bc4a942ca-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.114 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.115 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.115 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.115 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.115 DEBUG:boto:Method: GET 2024-04-13T22:15:32.115 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.115 DEBUG:boto:Data: 2024-04-13T22:15:32.115 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.115 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.116 DEBUG:boto:Params: {} 2024-04-13T22:15:32.116 DEBUG:boto:Token: None 2024-04-13T22:15:32.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df0d9fe9f2c86cec-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.119 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.119 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.119 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.119 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.119 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.119 DEBUG:boto:Method: GET 2024-04-13T22:15:32.119 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000006 2024-04-13T22:15:32.120 DEBUG:boto:Data: 2024-04-13T22:15:32.120 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.120 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.120 DEBUG:boto:Params: {} 2024-04-13T22:15:32.120 DEBUG:boto:Token: None 2024-04-13T22:15:32.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da460070ea8ad42-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.124 DEBUG:boto:b'vwnfns-60obj000000000000000000061000falseobj000000000000000000062024-04-13T22:15:22.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-13T22:15:32.124 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.125 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.125 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.125 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.125 DEBUG:boto:Method: GET 2024-04-13T22:15:32.125 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.125 DEBUG:boto:Data: 2024-04-13T22:15:32.125 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.125 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.125 DEBUG:boto:Params: {} 2024-04-13T22:15:32.125 DEBUG:boto:Token: None 2024-04-13T22:15:32.125 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4= 2024-04-13T22:15:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4=', 'Content-Length': '0'} 2024-04-13T22:15:32.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb885232977afb5b-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.170 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.170 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.170 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.171 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.171 DEBUG:boto:Method: GET 2024-04-13T22:15:32.171 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.171 DEBUG:boto:Data: 2024-04-13T22:15:32.171 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.171 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.171 DEBUG:boto:Params: {} 2024-04-13T22:15:32.171 DEBUG:boto:Token: None 2024-04-13T22:15:32.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4= 2024-04-13T22:15:32.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4=', 'Content-Length': '0'} 2024-04-13T22:15:32.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b9100690b54fa7-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.174 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.174 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.174 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.174 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.174 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.174 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.174 DEBUG:boto:Data: 2024-04-13T22:15:32.175 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.175 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.175 DEBUG:boto:Params: {} 2024-04-13T22:15:32.175 DEBUG:boto:Token: None 2024-04-13T22:15:32.175 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PJxY6tfxDMtB+EJUoXPbSGP53Yc= 2024-04-13T22:15:32.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PJxY6tfxDMtB+EJUoXPbSGP53Yc=', 'Content-Length': '0'} 2024-04-13T22:15:32.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6b4c368905a3bb-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.177 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.177 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.177 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.177 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.178 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.178 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.178 DEBUG:boto:Data: 2024-04-13T22:15:32.178 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.178 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.178 DEBUG:boto:Params: {} 2024-04-13T22:15:32.178 DEBUG:boto:Token: None 2024-04-13T22:15:32.178 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PJxY6tfxDMtB+EJUoXPbSGP53Yc= 2024-04-13T22:15:32.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PJxY6tfxDMtB+EJUoXPbSGP53Yc=', 'Content-Length': '0'} 2024-04-13T22:15:32.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc7053767364e28-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-13T22:15:32.180 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.181 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.181 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.181 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.181 DEBUG:boto:Method: GET 2024-04-13T22:15:32.181 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.181 DEBUG:boto:Data: 2024-04-13T22:15:32.181 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.181 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.181 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.181 DEBUG:boto:Params: {} 2024-04-13T22:15:32.181 DEBUG:boto:Token: None 2024-04-13T22:15:32.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4= 2024-04-13T22:15:32.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4=', 'Content-Length': '0'} 2024-04-13T22:15:32.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af0560d2582da38-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.184 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.184 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.184 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.184 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.184 DEBUG:boto:Method: GET 2024-04-13T22:15:32.184 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.184 DEBUG:boto:Data: 2024-04-13T22:15:32.184 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.184 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.184 DEBUG:boto:Params: {} 2024-04-13T22:15:32.185 DEBUG:boto:Token: None 2024-04-13T22:15:32.185 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007?versionId=null 2024-04-13T22:15:32.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4= 2024-04-13T22:15:32.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:miUNH4Vvf3bIdlswv+5zyqi6gu4=', 'Content-Length': '0'} 2024-04-13T22:15:32.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a164b670307abf1c-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.187 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.187 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.187 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.187 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.187 DEBUG:boto:Method: GET 2024-04-13T22:15:32.187 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.187 DEBUG:boto:Data: 2024-04-13T22:15:32.187 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.188 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.188 DEBUG:boto:Params: {} 2024-04-13T22:15:32.188 DEBUG:boto:Token: None 2024-04-13T22:15:32.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca0d3db42e6a51c5-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.191 DEBUG:boto:b'vwnfns-60obj000000000000000000071000falseobj000000000000000000072024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.191 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.192 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.192 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.192 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.192 DEBUG:boto:Method: GET 2024-04-13T22:15:32.192 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.192 DEBUG:boto:Data: 2024-04-13T22:15:32.192 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.192 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.192 DEBUG:boto:Params: {} 2024-04-13T22:15:32.192 DEBUG:boto:Token: None 2024-04-13T22:15:32.193 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b2ec43a0a22f3c-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.196 DEBUG:boto:b'vwnfns-60obj000000000000000000071000falseobj000000000000000000072024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.197 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.197 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.197 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.197 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.197 DEBUG:boto:Data: 2024-04-13T22:15:32.197 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.197 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.198 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.198 DEBUG:boto:Params: {} 2024-04-13T22:15:32.198 DEBUG:boto:Token: None 2024-04-13T22:15:32.198 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f5QOv0vJGpb8SeBZzdVKCTV/S10= 2024-04-13T22:15:32.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f5QOv0vJGpb8SeBZzdVKCTV/S10=', 'Content-Length': '0'} 2024-04-13T22:15:32.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9c9981fbd21d98b-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.200 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.200 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.200 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.200 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.200 DEBUG:boto:Data: 2024-04-13T22:15:32.201 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.201 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.201 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.201 DEBUG:boto:Params: {} 2024-04-13T22:15:32.201 DEBUG:boto:Token: None 2024-04-13T22:15:32.201 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.201 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f5QOv0vJGpb8SeBZzdVKCTV/S10= 2024-04-13T22:15:32.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f5QOv0vJGpb8SeBZzdVKCTV/S10=', 'Content-Length': '0'} 2024-04-13T22:15:32.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c8dc9076f077d8-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-13T22:15:32.203 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.203 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.203 DEBUG:boto:Method: GET 2024-04-13T22:15:32.203 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.204 DEBUG:boto:Data: 2024-04-13T22:15:32.204 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.204 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.204 DEBUG:boto:Params: {} 2024-04-13T22:15:32.204 DEBUG:boto:Token: None 2024-04-13T22:15:32.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pMxPTMcRXR/v0tvi0hohkmjmepo= 2024-04-13T22:15:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pMxPTMcRXR/v0tvi0hohkmjmepo=', 'Content-Length': '0'} 2024-04-13T22:15:32.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044a55adc754d2b3-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.207 DEBUG:boto:path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.207 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.207 DEBUG:boto:Method: GET 2024-04-13T22:15:32.207 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.207 DEBUG:boto:Data: 2024-04-13T22:15:32.207 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.207 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.207 DEBUG:boto:Params: {} 2024-04-13T22:15:32.207 DEBUG:boto:Token: None 2024-04-13T22:15:32.207 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000007 2024-04-13T22:15:32.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pMxPTMcRXR/v0tvi0hohkmjmepo= 2024-04-13T22:15:32.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pMxPTMcRXR/v0tvi0hohkmjmepo=', 'Content-Length': '0'} 2024-04-13T22:15:32.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000811bf35d9f9f8180-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.250 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.250 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.250 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.251 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.251 DEBUG:boto:Method: GET 2024-04-13T22:15:32.251 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.251 DEBUG:boto:Data: 2024-04-13T22:15:32.251 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.251 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.251 DEBUG:boto:Params: {} 2024-04-13T22:15:32.251 DEBUG:boto:Token: None 2024-04-13T22:15:32.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98c3a6ee4727c99-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.254 DEBUG:boto:b'vwnfns-60obj000000000000000000071000falseobj000000000000000000072024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.254 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.254 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.254 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.254 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.255 DEBUG:boto:Method: GET 2024-04-13T22:15:32.255 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000007 2024-04-13T22:15:32.255 DEBUG:boto:Data: 2024-04-13T22:15:32.255 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.255 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.255 DEBUG:boto:Params: {} 2024-04-13T22:15:32.255 DEBUG:boto:Token: None 2024-04-13T22:15:32.255 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6151a4d6e931174-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.259 DEBUG:boto:b'vwnfns-60obj000000000000000000071000falseobj000000000000000000072024-04-13T22:15:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-13T22:15:32.260 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.260 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.260 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.260 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.260 DEBUG:boto:Method: GET 2024-04-13T22:15:32.260 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.260 DEBUG:boto:Data: 2024-04-13T22:15:32.261 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.261 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.261 DEBUG:boto:Params: {} 2024-04-13T22:15:32.261 DEBUG:boto:Token: None 2024-04-13T22:15:32.261 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0= 2024-04-13T22:15:32.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0=', 'Content-Length': '0'} 2024-04-13T22:15:32.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc3ecfd08e9bf53-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.264 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.264 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.264 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.264 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.264 DEBUG:boto:Method: GET 2024-04-13T22:15:32.264 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.264 DEBUG:boto:Data: 2024-04-13T22:15:32.264 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.264 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.264 DEBUG:boto:Params: {} 2024-04-13T22:15:32.265 DEBUG:boto:Token: None 2024-04-13T22:15:32.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0= 2024-04-13T22:15:32.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0=', 'Content-Length': '0'} 2024-04-13T22:15:32.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584e7ce9309cf92c-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.267 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.267 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.267 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.267 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.267 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.267 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.267 DEBUG:boto:Data: 2024-04-13T22:15:32.267 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.268 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.268 DEBUG:boto:Params: {} 2024-04-13T22:15:32.268 DEBUG:boto:Token: None 2024-04-13T22:15:32.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zJf7PMQQQx10QnK/3m8AQ+pPP+s= 2024-04-13T22:15:32.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zJf7PMQQQx10QnK/3m8AQ+pPP+s=', 'Content-Length': '0'} 2024-04-13T22:15:32.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5cea869cdeebae9-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.270 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.270 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.270 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.270 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.270 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.271 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.271 DEBUG:boto:Data: 2024-04-13T22:15:32.271 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.271 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.271 DEBUG:boto:Params: {} 2024-04-13T22:15:32.271 DEBUG:boto:Token: None 2024-04-13T22:15:32.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zJf7PMQQQx10QnK/3m8AQ+pPP+s= 2024-04-13T22:15:32.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zJf7PMQQQx10QnK/3m8AQ+pPP+s=', 'Content-Length': '0'} 2024-04-13T22:15:32.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a600314c154adb6e-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-13T22:15:32.273 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.273 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.274 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.274 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.274 DEBUG:boto:Method: GET 2024-04-13T22:15:32.274 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.274 DEBUG:boto:Data: 2024-04-13T22:15:32.274 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.274 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.274 DEBUG:boto:Params: {} 2024-04-13T22:15:32.274 DEBUG:boto:Token: None 2024-04-13T22:15:32.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0= 2024-04-13T22:15:32.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0=', 'Content-Length': '0'} 2024-04-13T22:15:32.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e26c2484cbb343-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.318 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.318 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.319 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.319 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.319 DEBUG:boto:Method: GET 2024-04-13T22:15:32.319 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.319 DEBUG:boto:Data: 2024-04-13T22:15:32.319 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.319 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.319 DEBUG:boto:Params: {} 2024-04-13T22:15:32.319 DEBUG:boto:Token: None 2024-04-13T22:15:32.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008?versionId=null 2024-04-13T22:15:32.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0= 2024-04-13T22:15:32.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T2dQzYdymgJgdZARfaHkiJnMxq0=', 'Content-Length': '0'} 2024-04-13T22:15:32.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067461b19ba23940b-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.322 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.322 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.322 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.322 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.323 DEBUG:boto:Method: GET 2024-04-13T22:15:32.323 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.323 DEBUG:boto:Data: 2024-04-13T22:15:32.323 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.323 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.323 DEBUG:boto:Params: {} 2024-04-13T22:15:32.323 DEBUG:boto:Token: None 2024-04-13T22:15:32.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a05b64e71f347b8-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.326 DEBUG:boto:b'vwnfns-60obj000000000000000000081000falseobj000000000000000000082024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.326 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.327 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.327 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.327 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.327 DEBUG:boto:Method: GET 2024-04-13T22:15:32.327 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.327 DEBUG:boto:Data: 2024-04-13T22:15:32.327 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.328 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.328 DEBUG:boto:Params: {} 2024-04-13T22:15:32.328 DEBUG:boto:Token: None 2024-04-13T22:15:32.328 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049f7f216c28ed078-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.332 DEBUG:boto:b'vwnfns-60obj000000000000000000081000falseobj000000000000000000082024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.332 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.333 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.333 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.333 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.333 DEBUG:boto:Data: 2024-04-13T22:15:32.333 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.333 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.333 DEBUG:boto:Params: {} 2024-04-13T22:15:32.333 DEBUG:boto:Token: None 2024-04-13T22:15:32.334 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9/hyBZgVtOXDu0nSN3FFkY4XR3g= 2024-04-13T22:15:32.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9/hyBZgVtOXDu0nSN3FFkY4XR3g=', 'Content-Length': '0'} 2024-04-13T22:15:32.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087dec8a74715cc2e-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.335 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.336 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.336 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.336 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.336 DEBUG:boto:Data: 2024-04-13T22:15:32.336 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.336 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.336 DEBUG:boto:Params: {} 2024-04-13T22:15:32.336 DEBUG:boto:Token: None 2024-04-13T22:15:32.336 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9/hyBZgVtOXDu0nSN3FFkY4XR3g= 2024-04-13T22:15:32.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9/hyBZgVtOXDu0nSN3FFkY4XR3g=', 'Content-Length': '0'} 2024-04-13T22:15:32.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33857d7a33f6fd0-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-13T22:15:32.339 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.339 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.339 DEBUG:boto:Method: GET 2024-04-13T22:15:32.339 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.339 DEBUG:boto:Data: 2024-04-13T22:15:32.339 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.339 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.339 DEBUG:boto:Params: {} 2024-04-13T22:15:32.339 DEBUG:boto:Token: None 2024-04-13T22:15:32.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tn1WpcUQA8M+HJDxR90lVQeiMLQ= 2024-04-13T22:15:32.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tn1WpcUQA8M+HJDxR90lVQeiMLQ=', 'Content-Length': '0'} 2024-04-13T22:15:32.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a2f5aa5ad3a6ca-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.341 DEBUG:boto:path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.342 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.342 DEBUG:boto:Method: GET 2024-04-13T22:15:32.342 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.342 DEBUG:boto:Data: 2024-04-13T22:15:32.342 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.342 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.342 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.342 DEBUG:boto:Params: {} 2024-04-13T22:15:32.342 DEBUG:boto:Token: None 2024-04-13T22:15:32.342 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000008 2024-04-13T22:15:32.343 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tn1WpcUQA8M+HJDxR90lVQeiMLQ= 2024-04-13T22:15:32.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tn1WpcUQA8M+HJDxR90lVQeiMLQ=', 'Content-Length': '0'} 2024-04-13T22:15:32.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d8843fea44b67e-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.344 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.345 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.345 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.345 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.345 DEBUG:boto:Method: GET 2024-04-13T22:15:32.345 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.345 DEBUG:boto:Data: 2024-04-13T22:15:32.345 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.345 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.345 DEBUG:boto:Params: {} 2024-04-13T22:15:32.346 DEBUG:boto:Token: None 2024-04-13T22:15:32.346 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.346 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ebfe2737497f62-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.349 DEBUG:boto:b'vwnfns-60obj000000000000000000081000falseobj000000000000000000082024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.349 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.349 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.349 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.349 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.349 DEBUG:boto:Method: GET 2024-04-13T22:15:32.349 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000008 2024-04-13T22:15:32.349 DEBUG:boto:Data: 2024-04-13T22:15:32.350 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.350 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.350 DEBUG:boto:Params: {} 2024-04-13T22:15:32.350 DEBUG:boto:Token: None 2024-04-13T22:15:32.350 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.350 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afafcc283e4c3cf5-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.354 DEBUG:boto:b'vwnfns-60obj000000000000000000081000falseobj000000000000000000082024-04-13T22:15:22.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-13T22:15:32.354 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.354 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.354 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.355 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.355 DEBUG:boto:Method: GET 2024-04-13T22:15:32.355 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.355 DEBUG:boto:Data: 2024-04-13T22:15:32.355 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.355 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.355 DEBUG:boto:Params: {} 2024-04-13T22:15:32.355 DEBUG:boto:Token: None 2024-04-13T22:15:32.355 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.355 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM= 2024-04-13T22:15:32.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM=', 'Content-Length': '0'} 2024-04-13T22:15:32.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8aed1c81807c77-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.358 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.359 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.359 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.359 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.359 DEBUG:boto:Method: GET 2024-04-13T22:15:32.360 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.360 DEBUG:boto:Data: 2024-04-13T22:15:32.360 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.360 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.360 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.360 DEBUG:boto:Params: {} 2024-04-13T22:15:32.360 DEBUG:boto:Token: None 2024-04-13T22:15:32.360 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM= 2024-04-13T22:15:32.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM=', 'Content-Length': '0'} 2024-04-13T22:15:32.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dce0f6e2f7778ab-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.362 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.362 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.363 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.363 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.363 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.363 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.363 DEBUG:boto:Data: 2024-04-13T22:15:32.363 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.363 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.363 DEBUG:boto:Params: {} 2024-04-13T22:15:32.363 DEBUG:boto:Token: None 2024-04-13T22:15:32.364 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dagJQW7eawPfUAvGmTeWmT+Om5A= 2024-04-13T22:15:32.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dagJQW7eawPfUAvGmTeWmT+Om5A=', 'Content-Length': '0'} 2024-04-13T22:15:32.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edce485accab415e-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.366 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.366 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.366 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.366 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.366 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.366 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.366 DEBUG:boto:Data: 2024-04-13T22:15:32.366 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.366 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.366 DEBUG:boto:Params: {} 2024-04-13T22:15:32.367 DEBUG:boto:Token: None 2024-04-13T22:15:32.367 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dagJQW7eawPfUAvGmTeWmT+Om5A= 2024-04-13T22:15:32.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dagJQW7eawPfUAvGmTeWmT+Om5A=', 'Content-Length': '0'} 2024-04-13T22:15:32.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e933fb1f04ce81-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-13T22:15:32.369 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.369 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.369 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.369 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.369 DEBUG:boto:Method: GET 2024-04-13T22:15:32.369 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.369 DEBUG:boto:Data: 2024-04-13T22:15:32.369 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.369 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.369 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.370 DEBUG:boto:Params: {} 2024-04-13T22:15:32.370 DEBUG:boto:Token: None 2024-04-13T22:15:32.370 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.370 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM= 2024-04-13T22:15:32.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM=', 'Content-Length': '0'} 2024-04-13T22:15:32.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee3eb386f434c89-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.372 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.372 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.372 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.372 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.372 DEBUG:boto:Method: GET 2024-04-13T22:15:32.372 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.372 DEBUG:boto:Data: 2024-04-13T22:15:32.373 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.373 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.373 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.373 DEBUG:boto:Params: {} 2024-04-13T22:15:32.373 DEBUG:boto:Token: None 2024-04-13T22:15:32.373 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009?versionId=null 2024-04-13T22:15:32.373 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM= 2024-04-13T22:15:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EZg/dCiSQCJguaKZ1qcWHzCpkRM=', 'Content-Length': '0'} 2024-04-13T22:15:32.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e093b7ef81d67239-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.418 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.435 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.435 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.436 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.436 DEBUG:boto:Method: GET 2024-04-13T22:15:32.436 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.436 DEBUG:boto:Data: 2024-04-13T22:15:32.436 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.436 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.436 DEBUG:boto:Params: {} 2024-04-13T22:15:32.437 DEBUG:boto:Token: None 2024-04-13T22:15:32.437 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b18720517a2f0a-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.440 DEBUG:boto:b'vwnfns-60obj000000000000000000091000falseobj000000000000000000092024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.440 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.440 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.440 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.441 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.441 DEBUG:boto:Method: GET 2024-04-13T22:15:32.441 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.441 DEBUG:boto:Data: 2024-04-13T22:15:32.441 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.441 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.441 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.441 DEBUG:boto:Params: {} 2024-04-13T22:15:32.442 DEBUG:boto:Token: None 2024-04-13T22:15:32.442 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.442 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000967df14ff2e61708-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.446 DEBUG:boto:b'vwnfns-60obj000000000000000000091000falseobj000000000000000000092024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.446 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.446 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.446 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.447 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.447 DEBUG:boto:Data: 2024-04-13T22:15:32.447 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.447 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.447 DEBUG:boto:Params: {} 2024-04-13T22:15:32.447 DEBUG:boto:Token: None 2024-04-13T22:15:32.447 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3X4tDuBSMfTPlIaCbjexLyImTd8= 2024-04-13T22:15:32.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3X4tDuBSMfTPlIaCbjexLyImTd8=', 'Content-Length': '0'} 2024-04-13T22:15:32.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa3115077008b0d-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.450 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.450 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.450 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.450 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.450 DEBUG:boto:Data: 2024-04-13T22:15:32.450 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.451 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.451 DEBUG:boto:Params: {} 2024-04-13T22:15:32.451 DEBUG:boto:Token: None 2024-04-13T22:15:32.451 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3X4tDuBSMfTPlIaCbjexLyImTd8= 2024-04-13T22:15:32.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3X4tDuBSMfTPlIaCbjexLyImTd8=', 'Content-Length': '0'} 2024-04-13T22:15:32.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e50e40458105e1ad-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-13T22:15:32.453 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.453 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.454 DEBUG:boto:Method: GET 2024-04-13T22:15:32.454 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.454 DEBUG:boto:Data: 2024-04-13T22:15:32.454 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.454 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.454 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.454 DEBUG:boto:Params: {} 2024-04-13T22:15:32.454 DEBUG:boto:Token: None 2024-04-13T22:15:32.454 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rKL3I1cCZgS7pdWK7anquV6vGCc= 2024-04-13T22:15:32.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rKL3I1cCZgS7pdWK7anquV6vGCc=', 'Content-Length': '0'} 2024-04-13T22:15:32.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff27d94d25bb50c3-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.456 DEBUG:boto:path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.457 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.457 DEBUG:boto:Method: GET 2024-04-13T22:15:32.457 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.457 DEBUG:boto:Data: 2024-04-13T22:15:32.457 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.457 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.457 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.457 DEBUG:boto:Params: {} 2024-04-13T22:15:32.457 DEBUG:boto:Token: None 2024-04-13T22:15:32.457 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000009 2024-04-13T22:15:32.458 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rKL3I1cCZgS7pdWK7anquV6vGCc= 2024-04-13T22:15:32.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rKL3I1cCZgS7pdWK7anquV6vGCc=', 'Content-Length': '0'} 2024-04-13T22:15:32.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eefd746ec7e5360-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.460 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.460 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.460 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.460 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.460 DEBUG:boto:Method: GET 2024-04-13T22:15:32.460 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.460 DEBUG:boto:Data: 2024-04-13T22:15:32.460 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.461 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.461 DEBUG:boto:Params: {} 2024-04-13T22:15:32.461 DEBUG:boto:Token: None 2024-04-13T22:15:32.462 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.462 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff4da1b18632ab77-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.464 DEBUG:boto:b'vwnfns-60obj000000000000000000091000falseobj000000000000000000092024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.465 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.465 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.465 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.465 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.465 DEBUG:boto:Method: GET 2024-04-13T22:15:32.465 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000009 2024-04-13T22:15:32.465 DEBUG:boto:Data: 2024-04-13T22:15:32.465 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.465 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.466 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.466 DEBUG:boto:Params: {} 2024-04-13T22:15:32.466 DEBUG:boto:Token: None 2024-04-13T22:15:32.466 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.466 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000173da4d4c32afe0f-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.470 DEBUG:boto:b'vwnfns-60obj000000000000000000091000falseobj000000000000000000092024-04-13T22:15:21.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-13T22:15:32.470 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.470 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.471 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.471 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.471 DEBUG:boto:Method: GET 2024-04-13T22:15:32.471 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.471 DEBUG:boto:Data: 2024-04-13T22:15:32.471 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.471 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.471 DEBUG:boto:Params: {} 2024-04-13T22:15:32.471 DEBUG:boto:Token: None 2024-04-13T22:15:32.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY= 2024-04-13T22:15:32.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY=', 'Content-Length': '0'} 2024-04-13T22:15:32.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34f4799669103fc-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.474 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.474 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.474 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.474 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.474 DEBUG:boto:Method: GET 2024-04-13T22:15:32.474 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.474 DEBUG:boto:Data: 2024-04-13T22:15:32.475 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.475 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.475 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.475 DEBUG:boto:Params: {} 2024-04-13T22:15:32.475 DEBUG:boto:Token: None 2024-04-13T22:15:32.475 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.475 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY= 2024-04-13T22:15:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY=', 'Content-Length': '0'} 2024-04-13T22:15:32.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000909050e2a31da8cc-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.478 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.478 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.478 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.478 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.478 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.478 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.478 DEBUG:boto:Data: 2024-04-13T22:15:32.478 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.478 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.478 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.478 DEBUG:boto:Params: {} 2024-04-13T22:15:32.479 DEBUG:boto:Token: None 2024-04-13T22:15:32.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NQ+0DMr5P7DKdXiJuJ2HKEkm1U8= 2024-04-13T22:15:32.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NQ+0DMr5P7DKdXiJuJ2HKEkm1U8=', 'Content-Length': '0'} 2024-04-13T22:15:32.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ef9884fa4f77e1-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.481 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.481 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.481 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.481 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.481 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.482 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.482 DEBUG:boto:Data: 2024-04-13T22:15:32.482 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.482 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.482 DEBUG:boto:Params: {} 2024-04-13T22:15:32.482 DEBUG:boto:Token: None 2024-04-13T22:15:32.482 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.482 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NQ+0DMr5P7DKdXiJuJ2HKEkm1U8= 2024-04-13T22:15:32.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NQ+0DMr5P7DKdXiJuJ2HKEkm1U8=', 'Content-Length': '0'} 2024-04-13T22:15:32.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004073ff50fd2f4779-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-13T22:15:32.485 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.485 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.485 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.485 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.485 DEBUG:boto:Method: GET 2024-04-13T22:15:32.485 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.485 DEBUG:boto:Data: 2024-04-13T22:15:32.485 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.486 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.486 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.486 DEBUG:boto:Params: {} 2024-04-13T22:15:32.486 DEBUG:boto:Token: None 2024-04-13T22:15:32.486 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY= 2024-04-13T22:15:32.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY=', 'Content-Length': '0'} 2024-04-13T22:15:32.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af1eb3efd4f69431-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.488 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.489 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.489 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.489 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.489 DEBUG:boto:Method: GET 2024-04-13T22:15:32.489 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.489 DEBUG:boto:Data: 2024-04-13T22:15:32.489 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.489 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.489 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.490 DEBUG:boto:Params: {} 2024-04-13T22:15:32.490 DEBUG:boto:Token: None 2024-04-13T22:15:32.490 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010?versionId=null 2024-04-13T22:15:32.490 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY= 2024-04-13T22:15:32.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKn+ePQpyga3V4wFIYJ0SBNXEhY=', 'Content-Length': '0'} 2024-04-13T22:15:32.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af6fa1ead622d012-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.492 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.492 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.492 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.492 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.493 DEBUG:boto:Method: GET 2024-04-13T22:15:32.493 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.493 DEBUG:boto:Data: 2024-04-13T22:15:32.493 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.493 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.493 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.493 DEBUG:boto:Params: {} 2024-04-13T22:15:32.493 DEBUG:boto:Token: None 2024-04-13T22:15:32.493 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.493 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1aa537363f2ab41-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.496 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.496 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.496 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.497 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.497 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.497 DEBUG:boto:Method: GET 2024-04-13T22:15:32.497 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.497 DEBUG:boto:Data: 2024-04-13T22:15:32.497 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.497 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.497 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.497 DEBUG:boto:Params: {} 2024-04-13T22:15:32.497 DEBUG:boto:Token: None 2024-04-13T22:15:32.497 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.498 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb06237db3f3900b-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.501 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.502 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.502 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.502 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.502 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.502 DEBUG:boto:Data: 2024-04-13T22:15:32.502 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.502 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.502 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.502 DEBUG:boto:Params: {} 2024-04-13T22:15:32.503 DEBUG:boto:Token: None 2024-04-13T22:15:32.503 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.503 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LcoJLeQ9wDAJghLhh7noeWLocAs= 2024-04-13T22:15:32.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LcoJLeQ9wDAJghLhh7noeWLocAs=', 'Content-Length': '0'} 2024-04-13T22:15:32.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005744733040bc7da1-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.505 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.505 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.505 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.505 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.506 DEBUG:boto:Data: 2024-04-13T22:15:32.506 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.506 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.506 DEBUG:boto:Params: {} 2024-04-13T22:15:32.507 DEBUG:boto:Token: None 2024-04-13T22:15:32.507 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LcoJLeQ9wDAJghLhh7noeWLocAs= 2024-04-13T22:15:32.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LcoJLeQ9wDAJghLhh7noeWLocAs=', 'Content-Length': '0'} 2024-04-13T22:15:32.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082995cf7bae9d78b-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-13T22:15:32.509 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.509 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.509 DEBUG:boto:Method: GET 2024-04-13T22:15:32.509 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.509 DEBUG:boto:Data: 2024-04-13T22:15:32.510 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.510 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.510 DEBUG:boto:Params: {} 2024-04-13T22:15:32.510 DEBUG:boto:Token: None 2024-04-13T22:15:32.510 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.510 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/sqwjgdIOXIXKrfBPNBCKkdD9+c= 2024-04-13T22:15:32.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/sqwjgdIOXIXKrfBPNBCKkdD9+c=', 'Content-Length': '0'} 2024-04-13T22:15:32.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3cf35af6261cd64-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.513 DEBUG:boto:path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.513 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.513 DEBUG:boto:Method: GET 2024-04-13T22:15:32.513 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.513 DEBUG:boto:Data: 2024-04-13T22:15:32.514 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.514 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.514 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.514 DEBUG:boto:Params: {} 2024-04-13T22:15:32.514 DEBUG:boto:Token: None 2024-04-13T22:15:32.515 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000010 2024-04-13T22:15:32.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/sqwjgdIOXIXKrfBPNBCKkdD9+c= 2024-04-13T22:15:32.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/sqwjgdIOXIXKrfBPNBCKkdD9+c=', 'Content-Length': '0'} 2024-04-13T22:15:32.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008531e2db0783f878-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.517 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.518 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.518 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.518 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.518 DEBUG:boto:Method: GET 2024-04-13T22:15:32.518 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.519 DEBUG:boto:Data: 2024-04-13T22:15:32.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.519 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.519 DEBUG:boto:Params: {} 2024-04-13T22:15:32.520 DEBUG:boto:Token: None 2024-04-13T22:15:32.520 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c034348545fa9315-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.523 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.524 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.524 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.524 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.524 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.525 DEBUG:boto:Method: GET 2024-04-13T22:15:32.525 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000010 2024-04-13T22:15:32.525 DEBUG:boto:Data: 2024-04-13T22:15:32.525 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.525 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.526 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.526 DEBUG:boto:Params: {} 2024-04-13T22:15:32.526 DEBUG:boto:Token: None 2024-04-13T22:15:32.526 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000022d6cc3dfc9bee4-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.531 DEBUG:boto:b'vwnfns-60obj000000000000000000101000falseobj000000000000000000102024-04-13T22:15:22.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-13T22:15:32.531 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.532 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.532 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.532 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.532 DEBUG:boto:Method: GET 2024-04-13T22:15:32.532 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.533 DEBUG:boto:Data: 2024-04-13T22:15:32.533 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.533 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.533 DEBUG:boto:Params: {} 2024-04-13T22:15:32.533 DEBUG:boto:Token: None 2024-04-13T22:15:32.534 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA= 2024-04-13T22:15:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA=', 'Content-Length': '0'} 2024-04-13T22:15:32.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13a05e16bcadaa9-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.582 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.583 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.583 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.583 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.583 DEBUG:boto:Method: GET 2024-04-13T22:15:32.583 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.583 DEBUG:boto:Data: 2024-04-13T22:15:32.583 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.583 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.583 DEBUG:boto:Params: {} 2024-04-13T22:15:32.584 DEBUG:boto:Token: None 2024-04-13T22:15:32.584 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA= 2024-04-13T22:15:32.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA=', 'Content-Length': '0'} 2024-04-13T22:15:32.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2f8f59261c5716-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.586 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.587 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.587 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.587 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.587 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.587 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.587 DEBUG:boto:Data: 2024-04-13T22:15:32.587 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.588 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.588 DEBUG:boto:Params: {} 2024-04-13T22:15:32.588 DEBUG:boto:Token: None 2024-04-13T22:15:32.588 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:16SbVJl+2HSaupURIdRMvDbJoQ4= 2024-04-13T22:15:32.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:16SbVJl+2HSaupURIdRMvDbJoQ4=', 'Content-Length': '0'} 2024-04-13T22:15:32.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000247221b6f03f0c94-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.590 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.590 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.590 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.590 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.591 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.591 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.591 DEBUG:boto:Data: 2024-04-13T22:15:32.591 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.591 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.591 DEBUG:boto:Params: {} 2024-04-13T22:15:32.591 DEBUG:boto:Token: None 2024-04-13T22:15:32.592 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:16SbVJl+2HSaupURIdRMvDbJoQ4= 2024-04-13T22:15:32.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:16SbVJl+2HSaupURIdRMvDbJoQ4=', 'Content-Length': '0'} 2024-04-13T22:15:32.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a289a8d96d3d45a-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-13T22:15:32.594 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.594 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.594 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.594 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.595 DEBUG:boto:Method: GET 2024-04-13T22:15:32.595 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.595 DEBUG:boto:Data: 2024-04-13T22:15:32.595 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.595 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.595 DEBUG:boto:Params: {} 2024-04-13T22:15:32.595 DEBUG:boto:Token: None 2024-04-13T22:15:32.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA= 2024-04-13T22:15:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA=', 'Content-Length': '0'} 2024-04-13T22:15:32.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004503907e45f98bce-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.597 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.598 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.598 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.598 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.598 DEBUG:boto:Method: GET 2024-04-13T22:15:32.598 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.598 DEBUG:boto:Data: 2024-04-13T22:15:32.598 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.598 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.599 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.599 DEBUG:boto:Params: {} 2024-04-13T22:15:32.599 DEBUG:boto:Token: None 2024-04-13T22:15:32.599 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011?versionId=null 2024-04-13T22:15:32.599 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA= 2024-04-13T22:15:32.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qczOwfLGp0+t31Ccc/PVsBrIimA=', 'Content-Length': '0'} 2024-04-13T22:15:32.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063104438759e4fb0-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.602 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.602 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.602 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.602 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.603 DEBUG:boto:Method: GET 2024-04-13T22:15:32.603 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.603 DEBUG:boto:Data: 2024-04-13T22:15:32.603 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.604 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.604 DEBUG:boto:Params: {} 2024-04-13T22:15:32.604 DEBUG:boto:Token: None 2024-04-13T22:15:32.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052f4806d18664231-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.607 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.608 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.608 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.608 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.608 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.608 DEBUG:boto:Method: GET 2024-04-13T22:15:32.608 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.608 DEBUG:boto:Data: 2024-04-13T22:15:32.609 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.609 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.609 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.609 DEBUG:boto:Params: {} 2024-04-13T22:15:32.609 DEBUG:boto:Token: None 2024-04-13T22:15:32.610 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.610 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000765497887e006893-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.614 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.614 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.615 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.615 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.615 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.615 DEBUG:boto:Data: 2024-04-13T22:15:32.615 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.615 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.615 DEBUG:boto:Params: {} 2024-04-13T22:15:32.615 DEBUG:boto:Token: None 2024-04-13T22:15:32.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EeVo0VmYvrPQgR9F4l7aKnuRs60= 2024-04-13T22:15:32.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EeVo0VmYvrPQgR9F4l7aKnuRs60=', 'Content-Length': '0'} 2024-04-13T22:15:32.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c378a1a73bbd687-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.617 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.617 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.618 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.618 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.618 DEBUG:boto:Data: 2024-04-13T22:15:32.618 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.618 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.618 DEBUG:boto:Params: {} 2024-04-13T22:15:32.618 DEBUG:boto:Token: None 2024-04-13T22:15:32.618 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EeVo0VmYvrPQgR9F4l7aKnuRs60= 2024-04-13T22:15:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EeVo0VmYvrPQgR9F4l7aKnuRs60=', 'Content-Length': '0'} 2024-04-13T22:15:32.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002681981d724dbbd2-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-13T22:15:32.620 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.620 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.621 DEBUG:boto:Method: GET 2024-04-13T22:15:32.621 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.621 DEBUG:boto:Data: 2024-04-13T22:15:32.621 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.621 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.621 DEBUG:boto:Params: {} 2024-04-13T22:15:32.621 DEBUG:boto:Token: None 2024-04-13T22:15:32.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.621 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qepTo94Rhrcs5GkB4PK3IC5eDd8= 2024-04-13T22:15:32.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qepTo94Rhrcs5GkB4PK3IC5eDd8=', 'Content-Length': '0'} 2024-04-13T22:15:32.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c473f9632227199d-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.623 DEBUG:boto:path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.623 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.623 DEBUG:boto:Method: GET 2024-04-13T22:15:32.623 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.623 DEBUG:boto:Data: 2024-04-13T22:15:32.624 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.624 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.624 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.624 DEBUG:boto:Params: {} 2024-04-13T22:15:32.624 DEBUG:boto:Token: None 2024-04-13T22:15:32.624 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000011 2024-04-13T22:15:32.624 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qepTo94Rhrcs5GkB4PK3IC5eDd8= 2024-04-13T22:15:32.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qepTo94Rhrcs5GkB4PK3IC5eDd8=', 'Content-Length': '0'} 2024-04-13T22:15:32.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb2071cc31a478b-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.626 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.626 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.626 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.626 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.627 DEBUG:boto:Method: GET 2024-04-13T22:15:32.627 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.627 DEBUG:boto:Data: 2024-04-13T22:15:32.627 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.627 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.627 DEBUG:boto:Params: {} 2024-04-13T22:15:32.627 DEBUG:boto:Token: None 2024-04-13T22:15:32.627 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.627 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f5868f10d43b52d-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.670 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.671 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.671 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.671 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.671 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.671 DEBUG:boto:Method: GET 2024-04-13T22:15:32.671 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000011 2024-04-13T22:15:32.671 DEBUG:boto:Data: 2024-04-13T22:15:32.672 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.672 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.672 DEBUG:boto:Params: {} 2024-04-13T22:15:32.672 DEBUG:boto:Token: None 2024-04-13T22:15:32.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a5ae269204635c-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.676 DEBUG:boto:b'vwnfns-60obj000000000000000000111000falseobj000000000000000000112024-04-13T22:15:22.525Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-13T22:15:32.676 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.676 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.676 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.677 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.677 DEBUG:boto:Method: GET 2024-04-13T22:15:32.677 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.677 DEBUG:boto:Data: 2024-04-13T22:15:32.677 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.677 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.677 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.677 DEBUG:boto:Params: {} 2024-04-13T22:15:32.677 DEBUG:boto:Token: None 2024-04-13T22:15:32.677 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.678 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0= 2024-04-13T22:15:32.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0=', 'Content-Length': '0'} 2024-04-13T22:15:32.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a9410c61363e0cc-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.680 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.680 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.680 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.680 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.680 DEBUG:boto:Method: GET 2024-04-13T22:15:32.680 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.680 DEBUG:boto:Data: 2024-04-13T22:15:32.681 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.681 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.681 DEBUG:boto:Params: {} 2024-04-13T22:15:32.681 DEBUG:boto:Token: None 2024-04-13T22:15:32.681 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.681 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0= 2024-04-13T22:15:32.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0=', 'Content-Length': '0'} 2024-04-13T22:15:32.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02ad7ad98c386aa-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.683 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.683 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.683 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.683 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.683 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.683 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.683 DEBUG:boto:Data: 2024-04-13T22:15:32.684 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.684 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.684 DEBUG:boto:Params: {} 2024-04-13T22:15:32.684 DEBUG:boto:Token: None 2024-04-13T22:15:32.684 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uQR3tToK9dFk45k0fwbfhebQNIE= 2024-04-13T22:15:32.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uQR3tToK9dFk45k0fwbfhebQNIE=', 'Content-Length': '0'} 2024-04-13T22:15:32.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a7c4ff824dd5f47-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.686 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.686 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.686 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.686 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.686 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.687 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.687 DEBUG:boto:Data: 2024-04-13T22:15:32.687 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.687 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.687 DEBUG:boto:Params: {} 2024-04-13T22:15:32.687 DEBUG:boto:Token: None 2024-04-13T22:15:32.687 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uQR3tToK9dFk45k0fwbfhebQNIE= 2024-04-13T22:15:32.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uQR3tToK9dFk45k0fwbfhebQNIE=', 'Content-Length': '0'} 2024-04-13T22:15:32.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d4413660bb0d87-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-13T22:15:32.689 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.689 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.690 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.690 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.690 DEBUG:boto:Method: GET 2024-04-13T22:15:32.690 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.690 DEBUG:boto:Data: 2024-04-13T22:15:32.690 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.690 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.690 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.690 DEBUG:boto:Params: {} 2024-04-13T22:15:32.690 DEBUG:boto:Token: None 2024-04-13T22:15:32.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0= 2024-04-13T22:15:32.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0=', 'Content-Length': '0'} 2024-04-13T22:15:32.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6d58f06a222c46-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.692 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.692 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.693 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.693 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.693 DEBUG:boto:Method: GET 2024-04-13T22:15:32.693 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.693 DEBUG:boto:Data: 2024-04-13T22:15:32.693 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.693 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.693 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.693 DEBUG:boto:Params: {} 2024-04-13T22:15:32.693 DEBUG:boto:Token: None 2024-04-13T22:15:32.693 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012?versionId=null 2024-04-13T22:15:32.694 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0= 2024-04-13T22:15:32.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nWPEcbC1TKsYWdOLXH0PHVUHih0=', 'Content-Length': '0'} 2024-04-13T22:15:32.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d35aed2e9fae19a-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.738 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.739 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.739 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.739 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.739 DEBUG:boto:Method: GET 2024-04-13T22:15:32.739 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.739 DEBUG:boto:Data: 2024-04-13T22:15:32.739 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.739 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.739 DEBUG:boto:Params: {} 2024-04-13T22:15:32.740 DEBUG:boto:Token: None 2024-04-13T22:15:32.740 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c02043060bac85e-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.742 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.743 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.743 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.743 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.743 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.743 DEBUG:boto:Method: GET 2024-04-13T22:15:32.743 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.744 DEBUG:boto:Data: 2024-04-13T22:15:32.744 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.744 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.744 DEBUG:boto:Params: {} 2024-04-13T22:15:32.744 DEBUG:boto:Token: None 2024-04-13T22:15:32.744 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2726d7a031d0f5-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.748 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.749 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.749 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.749 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.749 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.749 DEBUG:boto:Data: 2024-04-13T22:15:32.749 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.749 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.749 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.750 DEBUG:boto:Params: {} 2024-04-13T22:15:32.750 DEBUG:boto:Token: None 2024-04-13T22:15:32.750 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.750 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LvXHgral6NYGcLaZ5tIg6gP4d0k= 2024-04-13T22:15:32.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LvXHgral6NYGcLaZ5tIg6gP4d0k=', 'Content-Length': '0'} 2024-04-13T22:15:32.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e284d8af756446a-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.752 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.752 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.752 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.752 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.752 DEBUG:boto:Data: 2024-04-13T22:15:32.752 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.753 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.753 DEBUG:boto:Params: {} 2024-04-13T22:15:32.753 DEBUG:boto:Token: None 2024-04-13T22:15:32.753 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.753 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LvXHgral6NYGcLaZ5tIg6gP4d0k= 2024-04-13T22:15:32.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LvXHgral6NYGcLaZ5tIg6gP4d0k=', 'Content-Length': '0'} 2024-04-13T22:15:32.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fd3b9148aebae8e-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-13T22:15:32.755 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.755 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.755 DEBUG:boto:Method: GET 2024-04-13T22:15:32.755 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.755 DEBUG:boto:Data: 2024-04-13T22:15:32.756 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.756 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.756 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.756 DEBUG:boto:Params: {} 2024-04-13T22:15:32.756 DEBUG:boto:Token: None 2024-04-13T22:15:32.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KbBp3KIvShgbyeC+TvKM/8I/qKE= 2024-04-13T22:15:32.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KbBp3KIvShgbyeC+TvKM/8I/qKE=', 'Content-Length': '0'} 2024-04-13T22:15:32.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be2e7e7144277193-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.758 DEBUG:boto:path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.758 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.758 DEBUG:boto:Method: GET 2024-04-13T22:15:32.758 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.758 DEBUG:boto:Data: 2024-04-13T22:15:32.758 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.758 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.759 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.759 DEBUG:boto:Params: {} 2024-04-13T22:15:32.759 DEBUG:boto:Token: None 2024-04-13T22:15:32.759 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000012 2024-04-13T22:15:32.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KbBp3KIvShgbyeC+TvKM/8I/qKE= 2024-04-13T22:15:32.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KbBp3KIvShgbyeC+TvKM/8I/qKE=', 'Content-Length': '0'} 2024-04-13T22:15:32.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23a60eb247bad0e-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.802 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.802 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.803 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.803 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.803 DEBUG:boto:Method: GET 2024-04-13T22:15:32.803 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.803 DEBUG:boto:Data: 2024-04-13T22:15:32.803 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.803 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.803 DEBUG:boto:Params: {} 2024-04-13T22:15:32.803 DEBUG:boto:Token: None 2024-04-13T22:15:32.803 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0783615e694838-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.806 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.806 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.806 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.807 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.807 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.807 DEBUG:boto:Method: GET 2024-04-13T22:15:32.807 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000012 2024-04-13T22:15:32.807 DEBUG:boto:Data: 2024-04-13T22:15:32.807 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.807 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.807 DEBUG:boto:Params: {} 2024-04-13T22:15:32.807 DEBUG:boto:Token: None 2024-04-13T22:15:32.807 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d302a22857c10223-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.811 DEBUG:boto:b'vwnfns-60obj000000000000000000121000falseobj000000000000000000122024-04-13T22:15:22.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-13T22:15:32.812 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.812 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.812 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.812 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.812 DEBUG:boto:Method: GET 2024-04-13T22:15:32.812 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.812 DEBUG:boto:Data: 2024-04-13T22:15:32.812 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.813 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.813 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.813 DEBUG:boto:Params: {} 2024-04-13T22:15:32.813 DEBUG:boto:Token: None 2024-04-13T22:15:32.813 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY= 2024-04-13T22:15:32.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY=', 'Content-Length': '0'} 2024-04-13T22:15:32.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681a7d06a5a42a67-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.816 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.816 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.816 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.816 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.816 DEBUG:boto:Method: GET 2024-04-13T22:15:32.816 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.816 DEBUG:boto:Data: 2024-04-13T22:15:32.816 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.817 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.817 DEBUG:boto:Params: {} 2024-04-13T22:15:32.817 DEBUG:boto:Token: None 2024-04-13T22:15:32.817 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.817 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY= 2024-04-13T22:15:32.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY=', 'Content-Length': '0'} 2024-04-13T22:15:32.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1c7a0228ae5fef-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.819 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.819 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.819 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.819 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.819 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.819 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.819 DEBUG:boto:Data: 2024-04-13T22:15:32.820 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.820 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.820 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.820 DEBUG:boto:Params: {} 2024-04-13T22:15:32.820 DEBUG:boto:Token: None 2024-04-13T22:15:32.820 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Imt5ic7HsKENVN8v0g9rn6M/BXI= 2024-04-13T22:15:32.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Imt5ic7HsKENVN8v0g9rn6M/BXI=', 'Content-Length': '0'} 2024-04-13T22:15:32.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c886e2197cb3c60c-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.822 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.822 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.822 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.822 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.822 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.822 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.822 DEBUG:boto:Data: 2024-04-13T22:15:32.823 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.823 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.823 DEBUG:boto:Params: {} 2024-04-13T22:15:32.823 DEBUG:boto:Token: None 2024-04-13T22:15:32.823 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Imt5ic7HsKENVN8v0g9rn6M/BXI= 2024-04-13T22:15:32.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Imt5ic7HsKENVN8v0g9rn6M/BXI=', 'Content-Length': '0'} 2024-04-13T22:15:32.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c73ece68be3c78-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-13T22:15:32.825 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.825 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.825 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.825 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.825 DEBUG:boto:Method: GET 2024-04-13T22:15:32.825 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.825 DEBUG:boto:Data: 2024-04-13T22:15:32.826 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.826 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.826 DEBUG:boto:Params: {} 2024-04-13T22:15:32.826 DEBUG:boto:Token: None 2024-04-13T22:15:32.826 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.826 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY= 2024-04-13T22:15:32.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY=', 'Content-Length': '0'} 2024-04-13T22:15:32.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745dd5de27922bc6-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.870 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.870 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.870 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.871 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.871 DEBUG:boto:Method: GET 2024-04-13T22:15:32.871 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.871 DEBUG:boto:Data: 2024-04-13T22:15:32.871 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.871 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.871 DEBUG:boto:Params: {} 2024-04-13T22:15:32.871 DEBUG:boto:Token: None 2024-04-13T22:15:32.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013?versionId=null 2024-04-13T22:15:32.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY= 2024-04-13T22:15:32.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C43GlKGWGkmZxEBGHfLAqEBeYmY=', 'Content-Length': '0'} 2024-04-13T22:15:32.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049dde41365db3a80-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.874 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.874 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.874 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.874 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.874 DEBUG:boto:Method: GET 2024-04-13T22:15:32.874 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.874 DEBUG:boto:Data: 2024-04-13T22:15:32.874 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.874 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.875 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.875 DEBUG:boto:Params: {} 2024-04-13T22:15:32.875 DEBUG:boto:Token: None 2024-04-13T22:15:32.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3c2cae2a44a0b7-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.877 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.878 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.878 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.878 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.878 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.878 DEBUG:boto:Method: GET 2024-04-13T22:15:32.878 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.878 DEBUG:boto:Data: 2024-04-13T22:15:32.878 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.879 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.879 DEBUG:boto:Params: {} 2024-04-13T22:15:32.879 DEBUG:boto:Token: None 2024-04-13T22:15:32.879 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.879 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d233eafbd44448f2-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.883 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.883 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.883 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.883 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.883 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.883 DEBUG:boto:Data: 2024-04-13T22:15:32.884 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.884 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.884 DEBUG:boto:Params: {} 2024-04-13T22:15:32.884 DEBUG:boto:Token: None 2024-04-13T22:15:32.884 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HbrcUte5jOGbkQVWcVq6lb7T2FY= 2024-04-13T22:15:32.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HbrcUte5jOGbkQVWcVq6lb7T2FY=', 'Content-Length': '0'} 2024-04-13T22:15:32.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a65a7b8a66c9f0-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.886 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.886 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.886 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.886 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.886 DEBUG:boto:Data: 2024-04-13T22:15:32.886 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.886 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.887 DEBUG:boto:Params: {} 2024-04-13T22:15:32.887 DEBUG:boto:Token: None 2024-04-13T22:15:32.887 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HbrcUte5jOGbkQVWcVq6lb7T2FY= 2024-04-13T22:15:32.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HbrcUte5jOGbkQVWcVq6lb7T2FY=', 'Content-Length': '0'} 2024-04-13T22:15:32.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000703bc5970768737-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-13T22:15:32.889 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.889 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.889 DEBUG:boto:Method: GET 2024-04-13T22:15:32.890 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.890 DEBUG:boto:Data: 2024-04-13T22:15:32.890 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.890 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.890 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.890 DEBUG:boto:Params: {} 2024-04-13T22:15:32.890 DEBUG:boto:Token: None 2024-04-13T22:15:32.890 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.891 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YMykmmTD6bXjg+qzJA2zYiQk5wU= 2024-04-13T22:15:32.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YMykmmTD6bXjg+qzJA2zYiQk5wU=', 'Content-Length': '0'} 2024-04-13T22:15:32.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644c1e6af1933b0d-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.892 DEBUG:boto:path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.892 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.893 DEBUG:boto:Method: GET 2024-04-13T22:15:32.893 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.893 DEBUG:boto:Data: 2024-04-13T22:15:32.893 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.893 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.893 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.893 DEBUG:boto:Params: {} 2024-04-13T22:15:32.893 DEBUG:boto:Token: None 2024-04-13T22:15:32.893 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000013 2024-04-13T22:15:32.894 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YMykmmTD6bXjg+qzJA2zYiQk5wU= 2024-04-13T22:15:32.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YMykmmTD6bXjg+qzJA2zYiQk5wU=', 'Content-Length': '0'} 2024-04-13T22:15:32.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e9e6383c6dda1f-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.896 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.896 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.896 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.896 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.896 DEBUG:boto:Method: GET 2024-04-13T22:15:32.896 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.896 DEBUG:boto:Data: 2024-04-13T22:15:32.896 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.897 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.897 DEBUG:boto:Params: {} 2024-04-13T22:15:32.897 DEBUG:boto:Token: None 2024-04-13T22:15:32.897 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.897 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000314411c512c5a7c6-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.900 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.901 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.901 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.901 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.901 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.901 DEBUG:boto:Method: GET 2024-04-13T22:15:32.901 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000013 2024-04-13T22:15:32.902 DEBUG:boto:Data: 2024-04-13T22:15:32.902 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.902 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.902 DEBUG:boto:Params: {} 2024-04-13T22:15:32.902 DEBUG:boto:Token: None 2024-04-13T22:15:32.903 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000013c46b5b39b93e1-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.907 DEBUG:boto:b'vwnfns-60obj000000000000000000131000falseobj000000000000000000132024-04-13T22:15:22.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-13T22:15:32.908 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.908 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.908 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.908 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.908 DEBUG:boto:Method: GET 2024-04-13T22:15:32.908 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.909 DEBUG:boto:Data: 2024-04-13T22:15:32.909 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.909 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.909 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.909 DEBUG:boto:Params: {} 2024-04-13T22:15:32.909 DEBUG:boto:Token: None 2024-04-13T22:15:32.910 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.910 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg= 2024-04-13T22:15:32.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg=', 'Content-Length': '0'} 2024-04-13T22:15:32.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001973f4a331b4bd85-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.912 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.912 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.912 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.912 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.912 DEBUG:boto:Method: GET 2024-04-13T22:15:32.912 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.912 DEBUG:boto:Data: 2024-04-13T22:15:32.912 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.912 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.913 DEBUG:boto:Params: {} 2024-04-13T22:15:32.913 DEBUG:boto:Token: None 2024-04-13T22:15:32.913 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg= 2024-04-13T22:15:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg=', 'Content-Length': '0'} 2024-04-13T22:15:32.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005917efa98090657a-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.915 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.915 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.915 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.915 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.915 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.915 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.915 DEBUG:boto:Data: 2024-04-13T22:15:32.916 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.916 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.916 DEBUG:boto:Params: {} 2024-04-13T22:15:32.916 DEBUG:boto:Token: None 2024-04-13T22:15:32.916 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QmiU4alC9MADoG4B7qp0kvMTtE0= 2024-04-13T22:15:32.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QmiU4alC9MADoG4B7qp0kvMTtE0=', 'Content-Length': '0'} 2024-04-13T22:15:32.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db165c9045d8247c-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.918 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.918 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.919 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.919 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.919 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.919 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.919 DEBUG:boto:Data: 2024-04-13T22:15:32.920 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.920 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.920 DEBUG:boto:Params: {} 2024-04-13T22:15:32.920 DEBUG:boto:Token: None 2024-04-13T22:15:32.921 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QmiU4alC9MADoG4B7qp0kvMTtE0= 2024-04-13T22:15:32.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QmiU4alC9MADoG4B7qp0kvMTtE0=', 'Content-Length': '0'} 2024-04-13T22:15:32.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f030a0ffd0755282-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-13T22:15:32.925 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.926 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.926 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.926 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.926 DEBUG:boto:Method: GET 2024-04-13T22:15:32.926 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.926 DEBUG:boto:Data: 2024-04-13T22:15:32.926 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.926 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.926 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.926 DEBUG:boto:Params: {} 2024-04-13T22:15:32.927 DEBUG:boto:Token: None 2024-04-13T22:15:32.927 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.927 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg= 2024-04-13T22:15:32.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg=', 'Content-Length': '0'} 2024-04-13T22:15:32.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a961bf2afb16fe5c-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.929 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.929 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.929 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.929 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.929 DEBUG:boto:Method: GET 2024-04-13T22:15:32.929 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.929 DEBUG:boto:Data: 2024-04-13T22:15:32.929 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.930 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.930 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.930 DEBUG:boto:Params: {} 2024-04-13T22:15:32.930 DEBUG:boto:Token: None 2024-04-13T22:15:32.930 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014?versionId=null 2024-04-13T22:15:32.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg= 2024-04-13T22:15:32.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0pZ8w1QGPLBmwPILMV+qdZHZakg=', 'Content-Length': '0'} 2024-04-13T22:15:32.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf70cb3b4ec4604-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.975 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.975 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.975 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:32.975 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:32.976 DEBUG:boto:Method: GET 2024-04-13T22:15:32.976 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:32.976 DEBUG:boto:Data: 2024-04-13T22:15:32.976 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.977 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.977 DEBUG:boto:Params: {} 2024-04-13T22:15:32.977 DEBUG:boto:Token: None 2024-04-13T22:15:32.977 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092dcaab45f2617dc-00661b0404-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.979 DEBUG:boto:b'vwnfns-60obj000000000000000000141000falseobj000000000000000000142024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.980 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:32.980 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:32.980 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:32.980 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:32.980 DEBUG:boto:Method: GET 2024-04-13T22:15:32.980 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:32.980 DEBUG:boto:Data: 2024-04-13T22:15:32.981 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.981 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.981 DEBUG:boto:Params: {} 2024-04-13T22:15:32.981 DEBUG:boto:Token: None 2024-04-13T22:15:32.981 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/ 2024-04-13T22:15:32.981 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U= 2024-04-13T22:15:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cR74gEsyCz6oE13o0cvdFTkri0U=', 'Content-Length': '0'} 2024-04-13T22:15:32.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000003d31e56238dd16-00661b0404-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.986 DEBUG:boto:b'vwnfns-60obj000000000000000000141000falseobj000000000000000000142024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:32.986 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.986 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.986 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.986 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.986 DEBUG:boto:Data: 2024-04-13T22:15:32.986 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.987 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.987 DEBUG:boto:Params: {} 2024-04-13T22:15:32.987 DEBUG:boto:Token: None 2024-04-13T22:15:32.987 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nPn9aSsYvwNTbHoUNly2eCtX5II= 2024-04-13T22:15:32.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nPn9aSsYvwNTbHoUNly2eCtX5II=', 'Content-Length': '0'} 2024-04-13T22:15:32.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a36070b0138d57a5-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.989 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.989 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.989 DEBUG:boto:Method: HEAD 2024-04-13T22:15:32.989 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.989 DEBUG:boto:Data: 2024-04-13T22:15:32.989 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.989 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:32.989 DEBUG:boto:Port: 8000 2024-04-13T22:15:32.990 DEBUG:boto:Params: {} 2024-04-13T22:15:32.990 DEBUG:boto:Token: None 2024-04-13T22:15:32.990 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.990 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nPn9aSsYvwNTbHoUNly2eCtX5II= 2024-04-13T22:15:32.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nPn9aSsYvwNTbHoUNly2eCtX5II=', 'Content-Length': '0'} 2024-04-13T22:15:32.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b99ffac28593944-00661b0404-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:32.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-13T22:15:32.992 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.992 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.992 DEBUG:boto:Method: GET 2024-04-13T22:15:32.992 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.992 DEBUG:boto:Data: 2024-04-13T22:15:32.993 DEBUG:boto:Headers: {} 2024-04-13T22:15:32.993 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:32.993 DEBUG:boto:Port: 8001 2024-04-13T22:15:32.993 DEBUG:boto:Params: {} 2024-04-13T22:15:32.993 DEBUG:boto:Token: None 2024-04-13T22:15:32.993 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:32 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:32.993 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ENcovwwnXfvJyJwnVd24sKfIKh4= 2024-04-13T22:15:32.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ENcovwwnXfvJyJwnVd24sKfIKh4=', 'Content-Length': '0'} 2024-04-13T22:15:32.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc246c6da620253-00661b0404-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.038 DEBUG:boto:path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:33.039 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000014 2024-04-13T22:15:33.039 DEBUG:boto:Method: GET 2024-04-13T22:15:33.039 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:33.039 DEBUG:boto:Data: 2024-04-13T22:15:33.039 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.039 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.039 DEBUG:boto:Params: {} 2024-04-13T22:15:33.039 DEBUG:boto:Token: None 2024-04-13T22:15:33.040 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000014 2024-04-13T22:15:33.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tVTQOMtLlYmcnns5jSW5ROzgyQc= 2024-04-13T22:15:33.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tVTQOMtLlYmcnns5jSW5ROzgyQc=', 'Content-Length': '0'} 2024-04-13T22:15:33.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc5723be48a8930-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.042 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.042 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.042 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:33.042 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:33.043 DEBUG:boto:Method: GET 2024-04-13T22:15:33.043 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:33.043 DEBUG:boto:Data: 2024-04-13T22:15:33.043 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.043 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.043 DEBUG:boto:Params: {} 2024-04-13T22:15:33.043 DEBUG:boto:Token: None 2024-04-13T22:15:33.044 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf3d26663281422-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.046 DEBUG:boto:b'vwnfns-60obj000000000000000000141000falseobj000000000000000000142024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.047 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.047 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.047 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:33.047 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:33.047 DEBUG:boto:Method: GET 2024-04-13T22:15:33.048 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000014 2024-04-13T22:15:33.048 DEBUG:boto:Data: 2024-04-13T22:15:33.048 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.048 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.048 DEBUG:boto:Params: {} 2024-04-13T22:15:33.048 DEBUG:boto:Token: None 2024-04-13T22:15:33.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aca9b415c037cd02-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.052 DEBUG:boto:b'vwnfns-60obj000000000000000000141000falseobj000000000000000000142024-04-13T22:15:22.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-13T22:15:33.053 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.053 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.053 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.053 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.053 DEBUG:boto:Method: GET 2024-04-13T22:15:33.053 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.053 DEBUG:boto:Data: 2024-04-13T22:15:33.053 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.053 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.054 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.054 DEBUG:boto:Params: {} 2024-04-13T22:15:33.054 DEBUG:boto:Token: None 2024-04-13T22:15:33.054 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.054 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo= 2024-04-13T22:15:33.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo=', 'Content-Length': '0'} 2024-04-13T22:15:33.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee55f0605edca400-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.056 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.056 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.056 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.057 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.057 DEBUG:boto:Method: GET 2024-04-13T22:15:33.057 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.057 DEBUG:boto:Data: 2024-04-13T22:15:33.057 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.057 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.058 DEBUG:boto:Params: {} 2024-04-13T22:15:33.058 DEBUG:boto:Token: None 2024-04-13T22:15:33.058 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.058 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo= 2024-04-13T22:15:33.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo=', 'Content-Length': '0'} 2024-04-13T22:15:33.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe6e7c6e15d4aad-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.102 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.103 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.103 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.103 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.103 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.103 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.103 DEBUG:boto:Data: 2024-04-13T22:15:33.103 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.103 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.103 DEBUG:boto:Params: {} 2024-04-13T22:15:33.104 DEBUG:boto:Token: None 2024-04-13T22:15:33.104 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.104 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXKnTWMYckBJpj8kdfp5OJpIxyE= 2024-04-13T22:15:33.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXKnTWMYckBJpj8kdfp5OJpIxyE=', 'Content-Length': '0'} 2024-04-13T22:15:33.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000331096a03fe3b90-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.106 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.106 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.106 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.106 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.106 DEBUG:boto:Data: 2024-04-13T22:15:33.107 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.107 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.107 DEBUG:boto:Params: {} 2024-04-13T22:15:33.107 DEBUG:boto:Token: None 2024-04-13T22:15:33.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXKnTWMYckBJpj8kdfp5OJpIxyE= 2024-04-13T22:15:33.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXKnTWMYckBJpj8kdfp5OJpIxyE=', 'Content-Length': '0'} 2024-04-13T22:15:33.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071329b7d1e33a57c-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-13T22:15:33.109 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.109 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.109 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.109 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.110 DEBUG:boto:Method: GET 2024-04-13T22:15:33.110 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.110 DEBUG:boto:Data: 2024-04-13T22:15:33.110 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.110 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.110 DEBUG:boto:Params: {} 2024-04-13T22:15:33.110 DEBUG:boto:Token: None 2024-04-13T22:15:33.110 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo= 2024-04-13T22:15:33.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo=', 'Content-Length': '0'} 2024-04-13T22:15:33.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e591ca43f77d97-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.113 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.113 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.113 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.113 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.113 DEBUG:boto:Method: GET 2024-04-13T22:15:33.113 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.113 DEBUG:boto:Data: 2024-04-13T22:15:33.113 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.113 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.114 DEBUG:boto:Params: {} 2024-04-13T22:15:33.114 DEBUG:boto:Token: None 2024-04-13T22:15:33.114 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015?versionId=null 2024-04-13T22:15:33.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo= 2024-04-13T22:15:33.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h88AqCmaZE+VasVFQjAjfZ3TOTo=', 'Content-Length': '0'} 2024-04-13T22:15:33.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a90b7c23bde029f-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.116 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.116 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.116 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.116 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.116 DEBUG:boto:Method: GET 2024-04-13T22:15:33.116 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.116 DEBUG:boto:Data: 2024-04-13T22:15:33.116 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.117 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.117 DEBUG:boto:Params: {} 2024-04-13T22:15:33.117 DEBUG:boto:Token: None 2024-04-13T22:15:33.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe59d0077b975ab-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.120 DEBUG:boto:b'vwnfns-60obj000000000000000000151000falseobj000000000000000000152024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.120 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.120 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.120 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.120 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.120 DEBUG:boto:Method: GET 2024-04-13T22:15:33.121 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.121 DEBUG:boto:Data: 2024-04-13T22:15:33.121 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.121 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.121 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.121 DEBUG:boto:Params: {} 2024-04-13T22:15:33.121 DEBUG:boto:Token: None 2024-04-13T22:15:33.121 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050c4348bbf9d348-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.125 DEBUG:boto:b'vwnfns-60obj000000000000000000151000falseobj000000000000000000152024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.125 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.125 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.125 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.125 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.125 DEBUG:boto:Data: 2024-04-13T22:15:33.126 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.126 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.126 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.126 DEBUG:boto:Params: {} 2024-04-13T22:15:33.126 DEBUG:boto:Token: None 2024-04-13T22:15:33.126 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wo3E4bOy8nXbTdk7ovT86jTDrlA= 2024-04-13T22:15:33.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wo3E4bOy8nXbTdk7ovT86jTDrlA=', 'Content-Length': '0'} 2024-04-13T22:15:33.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d63dccf390daf3-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.128 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.128 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.128 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.128 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.128 DEBUG:boto:Data: 2024-04-13T22:15:33.129 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.129 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.129 DEBUG:boto:Params: {} 2024-04-13T22:15:33.129 DEBUG:boto:Token: None 2024-04-13T22:15:33.129 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wo3E4bOy8nXbTdk7ovT86jTDrlA= 2024-04-13T22:15:33.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wo3E4bOy8nXbTdk7ovT86jTDrlA=', 'Content-Length': '0'} 2024-04-13T22:15:33.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ac4405609c3a02-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-13T22:15:33.131 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.131 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.132 DEBUG:boto:Method: GET 2024-04-13T22:15:33.132 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.132 DEBUG:boto:Data: 2024-04-13T22:15:33.132 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.132 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.132 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.132 DEBUG:boto:Params: {} 2024-04-13T22:15:33.132 DEBUG:boto:Token: None 2024-04-13T22:15:33.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UsDf9G8L7dHzMi8lu12sHEe81pw= 2024-04-13T22:15:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UsDf9G8L7dHzMi8lu12sHEe81pw=', 'Content-Length': '0'} 2024-04-13T22:15:33.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33634feeacc41aa-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.174 DEBUG:boto:path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.174 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.175 DEBUG:boto:Method: GET 2024-04-13T22:15:33.175 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.175 DEBUG:boto:Data: 2024-04-13T22:15:33.175 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.175 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.175 DEBUG:boto:Params: {} 2024-04-13T22:15:33.175 DEBUG:boto:Token: None 2024-04-13T22:15:33.175 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000015 2024-04-13T22:15:33.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UsDf9G8L7dHzMi8lu12sHEe81pw= 2024-04-13T22:15:33.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UsDf9G8L7dHzMi8lu12sHEe81pw=', 'Content-Length': '0'} 2024-04-13T22:15:33.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f604c5d59176f7-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.178 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.178 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.178 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.178 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.178 DEBUG:boto:Method: GET 2024-04-13T22:15:33.178 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.178 DEBUG:boto:Data: 2024-04-13T22:15:33.178 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.179 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.179 DEBUG:boto:Params: {} 2024-04-13T22:15:33.179 DEBUG:boto:Token: None 2024-04-13T22:15:33.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000310dc4acb99fd4f9-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.181 DEBUG:boto:b'vwnfns-60obj000000000000000000151000falseobj000000000000000000152024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.182 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.182 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.182 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.182 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.182 DEBUG:boto:Method: GET 2024-04-13T22:15:33.182 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000015 2024-04-13T22:15:33.182 DEBUG:boto:Data: 2024-04-13T22:15:33.183 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.183 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.183 DEBUG:boto:Params: {} 2024-04-13T22:15:33.183 DEBUG:boto:Token: None 2024-04-13T22:15:33.183 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e62e1a8820eb018d-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.187 DEBUG:boto:b'vwnfns-60obj000000000000000000151000falseobj000000000000000000152024-04-13T22:15:22.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-13T22:15:33.187 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.187 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.187 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.187 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.188 DEBUG:boto:Method: GET 2024-04-13T22:15:33.188 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.188 DEBUG:boto:Data: 2024-04-13T22:15:33.188 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.188 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.188 DEBUG:boto:Params: {} 2024-04-13T22:15:33.188 DEBUG:boto:Token: None 2024-04-13T22:15:33.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg= 2024-04-13T22:15:33.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg=', 'Content-Length': '0'} 2024-04-13T22:15:33.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34a3d4450d38efd-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.191 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.191 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.191 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.191 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.191 DEBUG:boto:Method: GET 2024-04-13T22:15:33.191 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.191 DEBUG:boto:Data: 2024-04-13T22:15:33.192 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.192 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.192 DEBUG:boto:Params: {} 2024-04-13T22:15:33.192 DEBUG:boto:Token: None 2024-04-13T22:15:33.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg= 2024-04-13T22:15:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg=', 'Content-Length': '0'} 2024-04-13T22:15:33.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aebfa69895cc972-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.234 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.235 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.235 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.235 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.235 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.235 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.235 DEBUG:boto:Data: 2024-04-13T22:15:33.235 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.235 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.235 DEBUG:boto:Params: {} 2024-04-13T22:15:33.236 DEBUG:boto:Token: None 2024-04-13T22:15:33.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qk83t16X+940N0rTbqO5vf5aXlM= 2024-04-13T22:15:33.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qk83t16X+940N0rTbqO5vf5aXlM=', 'Content-Length': '0'} 2024-04-13T22:15:33.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012203d4dbcb0f6ca-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.238 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.238 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.238 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.238 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.238 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.238 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.238 DEBUG:boto:Data: 2024-04-13T22:15:33.239 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.239 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.239 DEBUG:boto:Params: {} 2024-04-13T22:15:33.239 DEBUG:boto:Token: None 2024-04-13T22:15:33.239 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qk83t16X+940N0rTbqO5vf5aXlM= 2024-04-13T22:15:33.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qk83t16X+940N0rTbqO5vf5aXlM=', 'Content-Length': '0'} 2024-04-13T22:15:33.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051fc30223f88ec2a-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-13T22:15:33.241 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.241 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.241 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.241 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.242 DEBUG:boto:Method: GET 2024-04-13T22:15:33.242 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.242 DEBUG:boto:Data: 2024-04-13T22:15:33.242 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.242 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.242 DEBUG:boto:Params: {} 2024-04-13T22:15:33.242 DEBUG:boto:Token: None 2024-04-13T22:15:33.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg= 2024-04-13T22:15:33.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg=', 'Content-Length': '0'} 2024-04-13T22:15:33.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd401a92825b9e2-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.244 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.244 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.244 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.245 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.245 DEBUG:boto:Method: GET 2024-04-13T22:15:33.245 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.245 DEBUG:boto:Data: 2024-04-13T22:15:33.245 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.245 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.245 DEBUG:boto:Params: {} 2024-04-13T22:15:33.245 DEBUG:boto:Token: None 2024-04-13T22:15:33.245 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016?versionId=null 2024-04-13T22:15:33.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg= 2024-04-13T22:15:33.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:21hdeIHFo9NpbnUqz5/AAm9HAHg=', 'Content-Length': '0'} 2024-04-13T22:15:33.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce7d319fc09b2d0-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.251 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.251 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.251 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.251 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.251 DEBUG:boto:Method: GET 2024-04-13T22:15:33.251 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.251 DEBUG:boto:Data: 2024-04-13T22:15:33.251 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.252 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.252 DEBUG:boto:Params: {} 2024-04-13T22:15:33.252 DEBUG:boto:Token: None 2024-04-13T22:15:33.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c91cf953f5f8e9c-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.254 DEBUG:boto:b'vwnfns-60obj000000000000000000161000falseobj000000000000000000162024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.255 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.255 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.255 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.255 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.255 DEBUG:boto:Method: GET 2024-04-13T22:15:33.255 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.255 DEBUG:boto:Data: 2024-04-13T22:15:33.255 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.256 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.256 DEBUG:boto:Params: {} 2024-04-13T22:15:33.256 DEBUG:boto:Token: None 2024-04-13T22:15:33.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000887cb65ec5616a5b-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.260 DEBUG:boto:b'vwnfns-60obj000000000000000000161000falseobj000000000000000000162024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.260 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.260 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.260 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.260 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.260 DEBUG:boto:Data: 2024-04-13T22:15:33.260 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.260 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.261 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.261 DEBUG:boto:Params: {} 2024-04-13T22:15:33.261 DEBUG:boto:Token: None 2024-04-13T22:15:33.261 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Ql5hL1W330tVfWwIlLOCtqD9XQ= 2024-04-13T22:15:33.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Ql5hL1W330tVfWwIlLOCtqD9XQ=', 'Content-Length': '0'} 2024-04-13T22:15:33.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb53de32b10c45d-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.263 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.263 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.263 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.263 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.263 DEBUG:boto:Data: 2024-04-13T22:15:33.263 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.263 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.263 DEBUG:boto:Params: {} 2024-04-13T22:15:33.264 DEBUG:boto:Token: None 2024-04-13T22:15:33.264 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Ql5hL1W330tVfWwIlLOCtqD9XQ= 2024-04-13T22:15:33.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Ql5hL1W330tVfWwIlLOCtqD9XQ=', 'Content-Length': '0'} 2024-04-13T22:15:33.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b730ae9d5d6fdb1f-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-13T22:15:33.266 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.266 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.266 DEBUG:boto:Method: GET 2024-04-13T22:15:33.266 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.266 DEBUG:boto:Data: 2024-04-13T22:15:33.266 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.266 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.266 DEBUG:boto:Params: {} 2024-04-13T22:15:33.267 DEBUG:boto:Token: None 2024-04-13T22:15:33.267 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hn5JfNnzSnGdQJgXHY21Jdc7iew= 2024-04-13T22:15:33.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hn5JfNnzSnGdQJgXHY21Jdc7iew=', 'Content-Length': '0'} 2024-04-13T22:15:33.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018caf69006d750b4-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.310 DEBUG:boto:path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.311 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.311 DEBUG:boto:Method: GET 2024-04-13T22:15:33.311 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.311 DEBUG:boto:Data: 2024-04-13T22:15:33.311 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.311 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.311 DEBUG:boto:Params: {} 2024-04-13T22:15:33.312 DEBUG:boto:Token: None 2024-04-13T22:15:33.312 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000016 2024-04-13T22:15:33.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hn5JfNnzSnGdQJgXHY21Jdc7iew= 2024-04-13T22:15:33.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hn5JfNnzSnGdQJgXHY21Jdc7iew=', 'Content-Length': '0'} 2024-04-13T22:15:33.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5a2f2af6387b7d-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.314 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.314 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.314 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.314 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.314 DEBUG:boto:Method: GET 2024-04-13T22:15:33.315 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.315 DEBUG:boto:Data: 2024-04-13T22:15:33.315 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.315 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.315 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.315 DEBUG:boto:Params: {} 2024-04-13T22:15:33.315 DEBUG:boto:Token: None 2024-04-13T22:15:33.315 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eba93303b4279d1-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.318 DEBUG:boto:b'vwnfns-60obj000000000000000000161000falseobj000000000000000000162024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.318 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.318 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.318 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.318 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.319 DEBUG:boto:Method: GET 2024-04-13T22:15:33.319 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000016 2024-04-13T22:15:33.319 DEBUG:boto:Data: 2024-04-13T22:15:33.319 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.319 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.319 DEBUG:boto:Params: {} 2024-04-13T22:15:33.319 DEBUG:boto:Token: None 2024-04-13T22:15:33.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cce0e2784d0162d-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.323 DEBUG:boto:b'vwnfns-60obj000000000000000000161000falseobj000000000000000000162024-04-13T22:15:22.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-13T22:15:33.324 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.324 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.324 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.324 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.324 DEBUG:boto:Method: GET 2024-04-13T22:15:33.325 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.325 DEBUG:boto:Data: 2024-04-13T22:15:33.325 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.325 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.325 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.325 DEBUG:boto:Params: {} 2024-04-13T22:15:33.325 DEBUG:boto:Token: None 2024-04-13T22:15:33.325 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.325 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8= 2024-04-13T22:15:33.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8=', 'Content-Length': '0'} 2024-04-13T22:15:33.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e3396bc413f811-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.328 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.328 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.328 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.328 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.328 DEBUG:boto:Method: GET 2024-04-13T22:15:33.329 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.329 DEBUG:boto:Data: 2024-04-13T22:15:33.329 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.329 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.329 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.329 DEBUG:boto:Params: {} 2024-04-13T22:15:33.329 DEBUG:boto:Token: None 2024-04-13T22:15:33.329 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.330 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8= 2024-04-13T22:15:33.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8=', 'Content-Length': '0'} 2024-04-13T22:15:33.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e43b087a2e7627a-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.374 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.375 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.375 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.375 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.375 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.375 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.375 DEBUG:boto:Data: 2024-04-13T22:15:33.375 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.375 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.375 DEBUG:boto:Params: {} 2024-04-13T22:15:33.376 DEBUG:boto:Token: None 2024-04-13T22:15:33.376 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vTLAiIf1tECuug+MUQdFMZnCzOc= 2024-04-13T22:15:33.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vTLAiIf1tECuug+MUQdFMZnCzOc=', 'Content-Length': '0'} 2024-04-13T22:15:33.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e39a21edad5ca2-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.378 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.378 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.378 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.378 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.378 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.378 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.379 DEBUG:boto:Data: 2024-04-13T22:15:33.379 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.379 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.379 DEBUG:boto:Params: {} 2024-04-13T22:15:33.379 DEBUG:boto:Token: None 2024-04-13T22:15:33.379 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.379 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vTLAiIf1tECuug+MUQdFMZnCzOc= 2024-04-13T22:15:33.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vTLAiIf1tECuug+MUQdFMZnCzOc=', 'Content-Length': '0'} 2024-04-13T22:15:33.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908cf9fdecaf9d3c-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-13T22:15:33.381 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.381 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.381 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.382 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.382 DEBUG:boto:Method: GET 2024-04-13T22:15:33.382 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.382 DEBUG:boto:Data: 2024-04-13T22:15:33.382 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.382 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.382 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.382 DEBUG:boto:Params: {} 2024-04-13T22:15:33.382 DEBUG:boto:Token: None 2024-04-13T22:15:33.382 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8= 2024-04-13T22:15:33.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8=', 'Content-Length': '0'} 2024-04-13T22:15:33.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0f8dd53213af0f-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.384 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.384 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.385 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.385 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.385 DEBUG:boto:Method: GET 2024-04-13T22:15:33.385 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.385 DEBUG:boto:Data: 2024-04-13T22:15:33.385 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.385 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.385 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.385 DEBUG:boto:Params: {} 2024-04-13T22:15:33.385 DEBUG:boto:Token: None 2024-04-13T22:15:33.386 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017?versionId=null 2024-04-13T22:15:33.386 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8= 2024-04-13T22:15:33.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZF8CBcBBeOcpoT3Ulxw+zgnw3o8=', 'Content-Length': '0'} 2024-04-13T22:15:33.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d5ccce699aee68c-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.388 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.388 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.388 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.388 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.388 DEBUG:boto:Method: GET 2024-04-13T22:15:33.388 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.388 DEBUG:boto:Data: 2024-04-13T22:15:33.388 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.388 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.389 DEBUG:boto:Params: {} 2024-04-13T22:15:33.389 DEBUG:boto:Token: None 2024-04-13T22:15:33.389 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.389 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078f993232e717718-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.391 DEBUG:boto:b'vwnfns-60obj000000000000000000171000falseobj000000000000000000172024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.392 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.392 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.392 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.392 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.392 DEBUG:boto:Method: GET 2024-04-13T22:15:33.392 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.392 DEBUG:boto:Data: 2024-04-13T22:15:33.392 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.392 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.393 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.393 DEBUG:boto:Params: {} 2024-04-13T22:15:33.393 DEBUG:boto:Token: None 2024-04-13T22:15:33.393 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.393 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c57b85054d8908-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.398 DEBUG:boto:b'vwnfns-60obj000000000000000000171000falseobj000000000000000000172024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.398 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.398 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.398 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.398 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.398 DEBUG:boto:Data: 2024-04-13T22:15:33.398 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.399 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.399 DEBUG:boto:Params: {} 2024-04-13T22:15:33.399 DEBUG:boto:Token: None 2024-04-13T22:15:33.399 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.399 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xB2Wjz8IGuyB0WrDucuRPLGO+I8= 2024-04-13T22:15:33.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xB2Wjz8IGuyB0WrDucuRPLGO+I8=', 'Content-Length': '0'} 2024-04-13T22:15:33.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7043ca55245a561-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.401 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.401 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.401 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.401 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.401 DEBUG:boto:Data: 2024-04-13T22:15:33.401 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.402 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.402 DEBUG:boto:Params: {} 2024-04-13T22:15:33.402 DEBUG:boto:Token: None 2024-04-13T22:15:33.402 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.402 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xB2Wjz8IGuyB0WrDucuRPLGO+I8= 2024-04-13T22:15:33.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xB2Wjz8IGuyB0WrDucuRPLGO+I8=', 'Content-Length': '0'} 2024-04-13T22:15:33.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1af172c7590c64-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-13T22:15:33.404 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.404 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.405 DEBUG:boto:Method: GET 2024-04-13T22:15:33.405 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.405 DEBUG:boto:Data: 2024-04-13T22:15:33.405 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.405 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.405 DEBUG:boto:Params: {} 2024-04-13T22:15:33.405 DEBUG:boto:Token: None 2024-04-13T22:15:33.405 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c702CWxHXD0dvqvHZfbf8Q7waRM= 2024-04-13T22:15:33.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c702CWxHXD0dvqvHZfbf8Q7waRM=', 'Content-Length': '0'} 2024-04-13T22:15:33.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001316d8f8b5a27835-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.407 DEBUG:boto:path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.407 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.407 DEBUG:boto:Method: GET 2024-04-13T22:15:33.408 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.408 DEBUG:boto:Data: 2024-04-13T22:15:33.408 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.408 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.408 DEBUG:boto:Params: {} 2024-04-13T22:15:33.408 DEBUG:boto:Token: None 2024-04-13T22:15:33.408 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000017 2024-04-13T22:15:33.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c702CWxHXD0dvqvHZfbf8Q7waRM= 2024-04-13T22:15:33.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c702CWxHXD0dvqvHZfbf8Q7waRM=', 'Content-Length': '0'} 2024-04-13T22:15:33.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab445915fb5abf39-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.411 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.411 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.411 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.411 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.411 DEBUG:boto:Method: GET 2024-04-13T22:15:33.411 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.411 DEBUG:boto:Data: 2024-04-13T22:15:33.411 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.411 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.411 DEBUG:boto:Params: {} 2024-04-13T22:15:33.412 DEBUG:boto:Token: None 2024-04-13T22:15:33.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503cdcf656ab5ba7-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.458 DEBUG:boto:b'vwnfns-60obj000000000000000000171000falseobj000000000000000000172024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.459 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.459 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.459 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.459 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.459 DEBUG:boto:Method: GET 2024-04-13T22:15:33.460 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000017 2024-04-13T22:15:33.460 DEBUG:boto:Data: 2024-04-13T22:15:33.460 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.460 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.460 DEBUG:boto:Params: {} 2024-04-13T22:15:33.460 DEBUG:boto:Token: None 2024-04-13T22:15:33.460 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4eebd1257c59391-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.465 DEBUG:boto:b'vwnfns-60obj000000000000000000171000falseobj000000000000000000172024-04-13T22:15:22.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-13T22:15:33.465 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.465 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.465 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.465 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.465 DEBUG:boto:Method: GET 2024-04-13T22:15:33.466 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.466 DEBUG:boto:Data: 2024-04-13T22:15:33.466 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.466 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.466 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.466 DEBUG:boto:Params: {} 2024-04-13T22:15:33.466 DEBUG:boto:Token: None 2024-04-13T22:15:33.466 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc= 2024-04-13T22:15:33.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc=', 'Content-Length': '0'} 2024-04-13T22:15:33.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3bb2e469cf6375-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.469 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.469 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.469 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.469 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.469 DEBUG:boto:Method: GET 2024-04-13T22:15:33.469 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.469 DEBUG:boto:Data: 2024-04-13T22:15:33.469 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.470 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.470 DEBUG:boto:Params: {} 2024-04-13T22:15:33.470 DEBUG:boto:Token: None 2024-04-13T22:15:33.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc= 2024-04-13T22:15:33.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc=', 'Content-Length': '0'} 2024-04-13T22:15:33.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8ad85bd6db679c-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.514 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.515 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.515 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.515 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.515 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.515 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.515 DEBUG:boto:Data: 2024-04-13T22:15:33.515 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.516 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.516 DEBUG:boto:Params: {} 2024-04-13T22:15:33.516 DEBUG:boto:Token: None 2024-04-13T22:15:33.516 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXYo61jOKMwHVDLLr8SLMaeVjmQ= 2024-04-13T22:15:33.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXYo61jOKMwHVDLLr8SLMaeVjmQ=', 'Content-Length': '0'} 2024-04-13T22:15:33.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d7265e1cd5ea85-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.518 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.518 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.518 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.518 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.519 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.519 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.519 DEBUG:boto:Data: 2024-04-13T22:15:33.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.519 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.519 DEBUG:boto:Params: {} 2024-04-13T22:15:33.519 DEBUG:boto:Token: None 2024-04-13T22:15:33.519 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXYo61jOKMwHVDLLr8SLMaeVjmQ= 2024-04-13T22:15:33.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXYo61jOKMwHVDLLr8SLMaeVjmQ=', 'Content-Length': '0'} 2024-04-13T22:15:33.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf8f119d957540a-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-13T22:15:33.521 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.521 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.522 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.522 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.522 DEBUG:boto:Method: GET 2024-04-13T22:15:33.522 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.522 DEBUG:boto:Data: 2024-04-13T22:15:33.522 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.522 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.522 DEBUG:boto:Params: {} 2024-04-13T22:15:33.523 DEBUG:boto:Token: None 2024-04-13T22:15:33.523 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc= 2024-04-13T22:15:33.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc=', 'Content-Length': '0'} 2024-04-13T22:15:33.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc32b6d69a3c5181-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.525 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.525 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.525 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.525 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.525 DEBUG:boto:Method: GET 2024-04-13T22:15:33.525 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.525 DEBUG:boto:Data: 2024-04-13T22:15:33.525 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.526 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.526 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.526 DEBUG:boto:Params: {} 2024-04-13T22:15:33.526 DEBUG:boto:Token: None 2024-04-13T22:15:33.526 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018?versionId=null 2024-04-13T22:15:33.526 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc= 2024-04-13T22:15:33.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pisc/sm+M8x5/yuf20sWMvpMilc=', 'Content-Length': '0'} 2024-04-13T22:15:33.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3d4228b692991e-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.528 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.528 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.528 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.529 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.529 DEBUG:boto:Method: GET 2024-04-13T22:15:33.529 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.529 DEBUG:boto:Data: 2024-04-13T22:15:33.529 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.529 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.529 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.529 DEBUG:boto:Params: {} 2024-04-13T22:15:33.529 DEBUG:boto:Token: None 2024-04-13T22:15:33.530 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.530 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85867aee6a0460c-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.532 DEBUG:boto:b'vwnfns-60obj000000000000000000181000falseobj000000000000000000182024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.532 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.533 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.533 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.533 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.533 DEBUG:boto:Method: GET 2024-04-13T22:15:33.533 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.533 DEBUG:boto:Data: 2024-04-13T22:15:33.533 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.533 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.534 DEBUG:boto:Params: {} 2024-04-13T22:15:33.534 DEBUG:boto:Token: None 2024-04-13T22:15:33.534 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423ae27046e60ee7-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.538 DEBUG:boto:b'vwnfns-60obj000000000000000000181000falseobj000000000000000000182024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.538 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.538 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.538 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.538 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.539 DEBUG:boto:Data: 2024-04-13T22:15:33.539 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.539 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.539 DEBUG:boto:Params: {} 2024-04-13T22:15:33.539 DEBUG:boto:Token: None 2024-04-13T22:15:33.539 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jpvf2FMa1p2qp+3/Abm0tOrNfk8= 2024-04-13T22:15:33.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jpvf2FMa1p2qp+3/Abm0tOrNfk8=', 'Content-Length': '0'} 2024-04-13T22:15:33.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d38297e2f792048-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.541 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.541 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.541 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.542 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.542 DEBUG:boto:Data: 2024-04-13T22:15:33.542 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.542 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.542 DEBUG:boto:Params: {} 2024-04-13T22:15:33.542 DEBUG:boto:Token: None 2024-04-13T22:15:33.542 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jpvf2FMa1p2qp+3/Abm0tOrNfk8= 2024-04-13T22:15:33.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jpvf2FMa1p2qp+3/Abm0tOrNfk8=', 'Content-Length': '0'} 2024-04-13T22:15:33.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16ed546c13062d6-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-13T22:15:33.545 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.545 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.545 DEBUG:boto:Method: GET 2024-04-13T22:15:33.545 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.545 DEBUG:boto:Data: 2024-04-13T22:15:33.545 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.545 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.545 DEBUG:boto:Params: {} 2024-04-13T22:15:33.546 DEBUG:boto:Token: None 2024-04-13T22:15:33.546 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jxyt7je/wy+EPA9SSmDJpybtdOI= 2024-04-13T22:15:33.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jxyt7je/wy+EPA9SSmDJpybtdOI=', 'Content-Length': '0'} 2024-04-13T22:15:33.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d78851db111ed2-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.548 DEBUG:boto:path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.548 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.548 DEBUG:boto:Method: GET 2024-04-13T22:15:33.548 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.548 DEBUG:boto:Data: 2024-04-13T22:15:33.548 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.549 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.549 DEBUG:boto:Params: {} 2024-04-13T22:15:33.549 DEBUG:boto:Token: None 2024-04-13T22:15:33.549 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000018 2024-04-13T22:15:33.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jxyt7je/wy+EPA9SSmDJpybtdOI= 2024-04-13T22:15:33.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jxyt7je/wy+EPA9SSmDJpybtdOI=', 'Content-Length': '0'} 2024-04-13T22:15:33.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08292a176ca1b7d-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.551 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.551 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.552 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.552 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.552 DEBUG:boto:Method: GET 2024-04-13T22:15:33.552 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.552 DEBUG:boto:Data: 2024-04-13T22:15:33.552 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.553 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.553 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.553 DEBUG:boto:Params: {} 2024-04-13T22:15:33.553 DEBUG:boto:Token: None 2024-04-13T22:15:33.553 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.553 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007540ef021f14dd17-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.555 DEBUG:boto:b'vwnfns-60obj000000000000000000181000falseobj000000000000000000182024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.556 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.556 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.556 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.556 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.556 DEBUG:boto:Method: GET 2024-04-13T22:15:33.556 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000018 2024-04-13T22:15:33.556 DEBUG:boto:Data: 2024-04-13T22:15:33.557 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.557 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.557 DEBUG:boto:Params: {} 2024-04-13T22:15:33.557 DEBUG:boto:Token: None 2024-04-13T22:15:33.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7ee3533b01818f3-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.561 DEBUG:boto:b'vwnfns-60obj000000000000000000181000falseobj000000000000000000182024-04-13T22:15:22.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-13T22:15:33.561 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.562 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.562 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.562 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.562 DEBUG:boto:Method: GET 2024-04-13T22:15:33.562 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.562 DEBUG:boto:Data: 2024-04-13T22:15:33.562 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.562 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.562 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.563 DEBUG:boto:Params: {} 2024-04-13T22:15:33.563 DEBUG:boto:Token: None 2024-04-13T22:15:33.563 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.563 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4= 2024-04-13T22:15:33.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4=', 'Content-Length': '0'} 2024-04-13T22:15:33.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b7aabd89767d0b1-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.565 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.565 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.566 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.566 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.566 DEBUG:boto:Method: GET 2024-04-13T22:15:33.566 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.566 DEBUG:boto:Data: 2024-04-13T22:15:33.566 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.566 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.566 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.566 DEBUG:boto:Params: {} 2024-04-13T22:15:33.566 DEBUG:boto:Token: None 2024-04-13T22:15:33.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4= 2024-04-13T22:15:33.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4=', 'Content-Length': '0'} 2024-04-13T22:15:33.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c3d29e4d64bfd2-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.610 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.611 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.611 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.611 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.611 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.611 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.611 DEBUG:boto:Data: 2024-04-13T22:15:33.611 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.611 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.611 DEBUG:boto:Params: {} 2024-04-13T22:15:33.612 DEBUG:boto:Token: None 2024-04-13T22:15:33.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A/ytzN7ahXTm7Zqn/K3ccWIg100= 2024-04-13T22:15:33.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A/ytzN7ahXTm7Zqn/K3ccWIg100=', 'Content-Length': '0'} 2024-04-13T22:15:33.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784ea5378548ec30-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.614 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.614 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.614 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.614 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.614 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.615 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.615 DEBUG:boto:Data: 2024-04-13T22:15:33.615 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.615 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.615 DEBUG:boto:Params: {} 2024-04-13T22:15:33.615 DEBUG:boto:Token: None 2024-04-13T22:15:33.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A/ytzN7ahXTm7Zqn/K3ccWIg100= 2024-04-13T22:15:33.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A/ytzN7ahXTm7Zqn/K3ccWIg100=', 'Content-Length': '0'} 2024-04-13T22:15:33.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e101a584e27d4d-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-13T22:15:33.618 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.618 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.618 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.618 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.618 DEBUG:boto:Method: GET 2024-04-13T22:15:33.618 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.618 DEBUG:boto:Data: 2024-04-13T22:15:33.618 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.619 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.619 DEBUG:boto:Params: {} 2024-04-13T22:15:33.619 DEBUG:boto:Token: None 2024-04-13T22:15:33.619 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4= 2024-04-13T22:15:33.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4=', 'Content-Length': '0'} 2024-04-13T22:15:33.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7baa72f7f85620f-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.621 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.621 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.621 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.621 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.621 DEBUG:boto:Method: GET 2024-04-13T22:15:33.622 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.622 DEBUG:boto:Data: 2024-04-13T22:15:33.622 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.622 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.622 DEBUG:boto:Params: {} 2024-04-13T22:15:33.622 DEBUG:boto:Token: None 2024-04-13T22:15:33.622 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019?versionId=null 2024-04-13T22:15:33.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4= 2024-04-13T22:15:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G5GMbqRxhFo3+rMxMdWPnTk/ZE4=', 'Content-Length': '0'} 2024-04-13T22:15:33.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000532904de875a9fb6-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.624 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.625 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.625 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.625 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.625 DEBUG:boto:Method: GET 2024-04-13T22:15:33.625 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.625 DEBUG:boto:Data: 2024-04-13T22:15:33.625 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.625 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.625 DEBUG:boto:Params: {} 2024-04-13T22:15:33.626 DEBUG:boto:Token: None 2024-04-13T22:15:33.626 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.626 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f63cff5fe453c076-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.628 DEBUG:boto:b'vwnfns-60obj000000000000000000191000falseobj000000000000000000192024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.629 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.629 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.629 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.629 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.629 DEBUG:boto:Method: GET 2024-04-13T22:15:33.629 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.629 DEBUG:boto:Data: 2024-04-13T22:15:33.629 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.629 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.630 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.630 DEBUG:boto:Params: {} 2024-04-13T22:15:33.630 DEBUG:boto:Token: None 2024-04-13T22:15:33.630 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb356c2073f27cc-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.634 DEBUG:boto:b'vwnfns-60obj000000000000000000191000falseobj000000000000000000192024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.634 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.634 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.634 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.634 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.634 DEBUG:boto:Data: 2024-04-13T22:15:33.635 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.635 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.635 DEBUG:boto:Params: {} 2024-04-13T22:15:33.635 DEBUG:boto:Token: None 2024-04-13T22:15:33.635 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0S30g6kjaGEqgmfw+ewLb8pniuM= 2024-04-13T22:15:33.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0S30g6kjaGEqgmfw+ewLb8pniuM=', 'Content-Length': '0'} 2024-04-13T22:15:33.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad27806544218afe-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.637 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.637 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.637 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.637 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.637 DEBUG:boto:Data: 2024-04-13T22:15:33.638 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.638 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.638 DEBUG:boto:Params: {} 2024-04-13T22:15:33.638 DEBUG:boto:Token: None 2024-04-13T22:15:33.638 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0S30g6kjaGEqgmfw+ewLb8pniuM= 2024-04-13T22:15:33.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0S30g6kjaGEqgmfw+ewLb8pniuM=', 'Content-Length': '0'} 2024-04-13T22:15:33.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7acf325986e4986-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-13T22:15:33.640 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.640 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.640 DEBUG:boto:Method: GET 2024-04-13T22:15:33.641 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.641 DEBUG:boto:Data: 2024-04-13T22:15:33.641 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.641 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.641 DEBUG:boto:Params: {} 2024-04-13T22:15:33.641 DEBUG:boto:Token: None 2024-04-13T22:15:33.641 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.641 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x3vK6Lr4WqU/7S+BPrFE8Ptyd3o= 2024-04-13T22:15:33.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x3vK6Lr4WqU/7S+BPrFE8Ptyd3o=', 'Content-Length': '0'} 2024-04-13T22:15:33.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c948a5ac5bcee45-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.643 DEBUG:boto:path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.643 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.643 DEBUG:boto:Method: GET 2024-04-13T22:15:33.644 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.644 DEBUG:boto:Data: 2024-04-13T22:15:33.644 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.644 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.644 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.644 DEBUG:boto:Params: {} 2024-04-13T22:15:33.644 DEBUG:boto:Token: None 2024-04-13T22:15:33.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000019 2024-04-13T22:15:33.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x3vK6Lr4WqU/7S+BPrFE8Ptyd3o= 2024-04-13T22:15:33.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x3vK6Lr4WqU/7S+BPrFE8Ptyd3o=', 'Content-Length': '0'} 2024-04-13T22:15:33.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e12d8d4b669c83-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.646 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.646 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.647 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.647 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.647 DEBUG:boto:Method: GET 2024-04-13T22:15:33.647 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.647 DEBUG:boto:Data: 2024-04-13T22:15:33.647 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.647 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.647 DEBUG:boto:Params: {} 2024-04-13T22:15:33.648 DEBUG:boto:Token: None 2024-04-13T22:15:33.648 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b80a5ce6809491c-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.690 DEBUG:boto:b'vwnfns-60obj000000000000000000191000falseobj000000000000000000192024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.691 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.691 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.691 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.691 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.691 DEBUG:boto:Method: GET 2024-04-13T22:15:33.691 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000019 2024-04-13T22:15:33.691 DEBUG:boto:Data: 2024-04-13T22:15:33.692 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.692 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.692 DEBUG:boto:Params: {} 2024-04-13T22:15:33.692 DEBUG:boto:Token: None 2024-04-13T22:15:33.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000007a7ec75248bca9-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.696 DEBUG:boto:b'vwnfns-60obj000000000000000000191000falseobj000000000000000000192024-04-13T22:15:22.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-13T22:15:33.696 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.697 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.697 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.697 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.697 DEBUG:boto:Method: GET 2024-04-13T22:15:33.697 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.697 DEBUG:boto:Data: 2024-04-13T22:15:33.697 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.697 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.697 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.697 DEBUG:boto:Params: {} 2024-04-13T22:15:33.697 DEBUG:boto:Token: None 2024-04-13T22:15:33.698 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA= 2024-04-13T22:15:33.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA=', 'Content-Length': '0'} 2024-04-13T22:15:33.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cdad534725d3693-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.701 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.701 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.701 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.701 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.701 DEBUG:boto:Method: GET 2024-04-13T22:15:33.701 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.701 DEBUG:boto:Data: 2024-04-13T22:15:33.701 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.701 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.701 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.702 DEBUG:boto:Params: {} 2024-04-13T22:15:33.702 DEBUG:boto:Token: None 2024-04-13T22:15:33.702 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA= 2024-04-13T22:15:33.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA=', 'Content-Length': '0'} 2024-04-13T22:15:33.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2621e7a662f75e-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.704 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.704 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.704 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.704 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.704 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.704 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.704 DEBUG:boto:Data: 2024-04-13T22:15:33.705 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.705 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.705 DEBUG:boto:Params: {} 2024-04-13T22:15:33.705 DEBUG:boto:Token: None 2024-04-13T22:15:33.705 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MWAATFmSKaetPZ6Bu56QVQbb4X8= 2024-04-13T22:15:33.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MWAATFmSKaetPZ6Bu56QVQbb4X8=', 'Content-Length': '0'} 2024-04-13T22:15:33.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ea8f9f3f4ee751-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.707 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.707 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.707 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.707 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.707 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.707 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.708 DEBUG:boto:Data: 2024-04-13T22:15:33.708 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.708 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.708 DEBUG:boto:Params: {} 2024-04-13T22:15:33.708 DEBUG:boto:Token: None 2024-04-13T22:15:33.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MWAATFmSKaetPZ6Bu56QVQbb4X8= 2024-04-13T22:15:33.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MWAATFmSKaetPZ6Bu56QVQbb4X8=', 'Content-Length': '0'} 2024-04-13T22:15:33.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423b77c11379ab52-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-13T22:15:33.710 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.710 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.711 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.711 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.711 DEBUG:boto:Method: GET 2024-04-13T22:15:33.711 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.711 DEBUG:boto:Data: 2024-04-13T22:15:33.711 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.711 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.711 DEBUG:boto:Params: {} 2024-04-13T22:15:33.711 DEBUG:boto:Token: None 2024-04-13T22:15:33.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA= 2024-04-13T22:15:33.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA=', 'Content-Length': '0'} 2024-04-13T22:15:33.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004482c129b6a55f5d-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.713 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.714 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.714 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.714 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.714 DEBUG:boto:Method: GET 2024-04-13T22:15:33.714 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.714 DEBUG:boto:Data: 2024-04-13T22:15:33.714 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.714 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.714 DEBUG:boto:Params: {} 2024-04-13T22:15:33.715 DEBUG:boto:Token: None 2024-04-13T22:15:33.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020?versionId=null 2024-04-13T22:15:33.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA= 2024-04-13T22:15:33.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWbp4ZEcbB7eOsWC/MttC8v5CnA=', 'Content-Length': '0'} 2024-04-13T22:15:33.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6e52cb56446df0-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.717 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.717 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.717 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.717 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.717 DEBUG:boto:Method: GET 2024-04-13T22:15:33.717 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.717 DEBUG:boto:Data: 2024-04-13T22:15:33.717 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.717 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.717 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.717 DEBUG:boto:Params: {} 2024-04-13T22:15:33.718 DEBUG:boto:Token: None 2024-04-13T22:15:33.718 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9446b6070bd2ccd-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.720 DEBUG:boto:b'vwnfns-60obj000000000000000000201000falseobj000000000000000000202024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.721 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.721 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.721 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.721 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.721 DEBUG:boto:Method: GET 2024-04-13T22:15:33.721 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.721 DEBUG:boto:Data: 2024-04-13T22:15:33.721 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.721 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.721 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.722 DEBUG:boto:Params: {} 2024-04-13T22:15:33.722 DEBUG:boto:Token: None 2024-04-13T22:15:33.722 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.722 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac41fe53903afc84-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.726 DEBUG:boto:b'vwnfns-60obj000000000000000000201000falseobj000000000000000000202024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.726 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.726 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.726 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.726 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.727 DEBUG:boto:Data: 2024-04-13T22:15:33.727 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.727 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.727 DEBUG:boto:Params: {} 2024-04-13T22:15:33.727 DEBUG:boto:Token: None 2024-04-13T22:15:33.727 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/vcltqXkvzj8aiPUqlNp7LE5CdI= 2024-04-13T22:15:33.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/vcltqXkvzj8aiPUqlNp7LE5CdI=', 'Content-Length': '0'} 2024-04-13T22:15:33.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008518663f756260eb-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.729 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.729 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.729 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.729 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.729 DEBUG:boto:Data: 2024-04-13T22:15:33.729 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.730 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.730 DEBUG:boto:Params: {} 2024-04-13T22:15:33.730 DEBUG:boto:Token: None 2024-04-13T22:15:33.730 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.730 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/vcltqXkvzj8aiPUqlNp7LE5CdI= 2024-04-13T22:15:33.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/vcltqXkvzj8aiPUqlNp7LE5CdI=', 'Content-Length': '0'} 2024-04-13T22:15:33.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da0d7d7bef1fa6b-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-13T22:15:33.732 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.732 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.732 DEBUG:boto:Method: GET 2024-04-13T22:15:33.732 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.732 DEBUG:boto:Data: 2024-04-13T22:15:33.732 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.733 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.733 DEBUG:boto:Params: {} 2024-04-13T22:15:33.733 DEBUG:boto:Token: None 2024-04-13T22:15:33.733 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.733 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fNACQSZNWESZ3QH/e/8gitkDM6o= 2024-04-13T22:15:33.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fNACQSZNWESZ3QH/e/8gitkDM6o=', 'Content-Length': '0'} 2024-04-13T22:15:33.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad16dfecbd353cf-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.735 DEBUG:boto:path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.735 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.735 DEBUG:boto:Method: GET 2024-04-13T22:15:33.735 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.735 DEBUG:boto:Data: 2024-04-13T22:15:33.735 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.735 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.735 DEBUG:boto:Params: {} 2024-04-13T22:15:33.736 DEBUG:boto:Token: None 2024-04-13T22:15:33.736 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000020 2024-04-13T22:15:33.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fNACQSZNWESZ3QH/e/8gitkDM6o= 2024-04-13T22:15:33.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fNACQSZNWESZ3QH/e/8gitkDM6o=', 'Content-Length': '0'} 2024-04-13T22:15:33.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a0bc9c04fa691fb-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.738 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.738 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.738 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.738 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.738 DEBUG:boto:Method: GET 2024-04-13T22:15:33.738 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.738 DEBUG:boto:Data: 2024-04-13T22:15:33.739 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.739 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.739 DEBUG:boto:Params: {} 2024-04-13T22:15:33.739 DEBUG:boto:Token: None 2024-04-13T22:15:33.739 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ce6720346fdd6d-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.782 DEBUG:boto:b'vwnfns-60obj000000000000000000201000falseobj000000000000000000202024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.783 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.783 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.783 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.783 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.783 DEBUG:boto:Method: GET 2024-04-13T22:15:33.783 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000020 2024-04-13T22:15:33.783 DEBUG:boto:Data: 2024-04-13T22:15:33.784 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.784 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.784 DEBUG:boto:Params: {} 2024-04-13T22:15:33.784 DEBUG:boto:Token: None 2024-04-13T22:15:33.784 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9564d377c7927f6-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.788 DEBUG:boto:b'vwnfns-60obj000000000000000000201000falseobj000000000000000000202024-04-13T22:15:20.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-13T22:15:33.789 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.789 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.789 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.789 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.789 DEBUG:boto:Method: GET 2024-04-13T22:15:33.789 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.789 DEBUG:boto:Data: 2024-04-13T22:15:33.790 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.790 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.790 DEBUG:boto:Params: {} 2024-04-13T22:15:33.790 DEBUG:boto:Token: None 2024-04-13T22:15:33.790 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.790 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI= 2024-04-13T22:15:33.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI=', 'Content-Length': '0'} 2024-04-13T22:15:33.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdede6bf43dbd3ed-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.792 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.792 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.793 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.793 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.793 DEBUG:boto:Method: GET 2024-04-13T22:15:33.793 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.793 DEBUG:boto:Data: 2024-04-13T22:15:33.793 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.793 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.793 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.793 DEBUG:boto:Params: {} 2024-04-13T22:15:33.793 DEBUG:boto:Token: None 2024-04-13T22:15:33.794 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.794 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI= 2024-04-13T22:15:33.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI=', 'Content-Length': '0'} 2024-04-13T22:15:33.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005318bb83ad1bda57-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.838 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.839 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.839 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.839 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.839 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.839 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.839 DEBUG:boto:Data: 2024-04-13T22:15:33.839 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.839 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.839 DEBUG:boto:Params: {} 2024-04-13T22:15:33.840 DEBUG:boto:Token: None 2024-04-13T22:15:33.840 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RFg4PJ5YnEWYH2nydW8w+j3gMOs= 2024-04-13T22:15:33.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RFg4PJ5YnEWYH2nydW8w+j3gMOs=', 'Content-Length': '0'} 2024-04-13T22:15:33.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5ae16873a176b1-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.842 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.842 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.842 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.842 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.842 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.842 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.842 DEBUG:boto:Data: 2024-04-13T22:15:33.842 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.843 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.843 DEBUG:boto:Params: {} 2024-04-13T22:15:33.843 DEBUG:boto:Token: None 2024-04-13T22:15:33.843 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.843 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RFg4PJ5YnEWYH2nydW8w+j3gMOs= 2024-04-13T22:15:33.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RFg4PJ5YnEWYH2nydW8w+j3gMOs=', 'Content-Length': '0'} 2024-04-13T22:15:33.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e7b671db571b10-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-13T22:15:33.845 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.845 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.845 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.845 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.845 DEBUG:boto:Method: GET 2024-04-13T22:15:33.845 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.845 DEBUG:boto:Data: 2024-04-13T22:15:33.846 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.846 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.846 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.846 DEBUG:boto:Params: {} 2024-04-13T22:15:33.846 DEBUG:boto:Token: None 2024-04-13T22:15:33.846 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.846 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI= 2024-04-13T22:15:33.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI=', 'Content-Length': '0'} 2024-04-13T22:15:33.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f46bd541d5d3c5-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.848 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.849 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.849 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.849 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.849 DEBUG:boto:Method: GET 2024-04-13T22:15:33.849 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.849 DEBUG:boto:Data: 2024-04-13T22:15:33.849 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.849 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.849 DEBUG:boto:Params: {} 2024-04-13T22:15:33.850 DEBUG:boto:Token: None 2024-04-13T22:15:33.850 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021?versionId=null 2024-04-13T22:15:33.850 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI= 2024-04-13T22:15:33.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78LncAoh037vx+2VTcdpyF+fswI=', 'Content-Length': '0'} 2024-04-13T22:15:33.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419985ae85ca39d3-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.852 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.852 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.852 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.852 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.852 DEBUG:boto:Method: GET 2024-04-13T22:15:33.852 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.852 DEBUG:boto:Data: 2024-04-13T22:15:33.852 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.853 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.853 DEBUG:boto:Params: {} 2024-04-13T22:15:33.853 DEBUG:boto:Token: None 2024-04-13T22:15:33.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009589639c87b245aa-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.856 DEBUG:boto:b'vwnfns-60obj000000000000000000211000falseobj000000000000000000212024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.856 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.856 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.856 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.856 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.856 DEBUG:boto:Method: GET 2024-04-13T22:15:33.857 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.857 DEBUG:boto:Data: 2024-04-13T22:15:33.857 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.857 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.857 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.857 DEBUG:boto:Params: {} 2024-04-13T22:15:33.857 DEBUG:boto:Token: None 2024-04-13T22:15:33.858 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.858 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a67956c03a6121-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.861 DEBUG:boto:b'vwnfns-60obj000000000000000000211000falseobj000000000000000000212024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.862 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.862 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.862 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.862 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.862 DEBUG:boto:Data: 2024-04-13T22:15:33.862 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.862 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.862 DEBUG:boto:Params: {} 2024-04-13T22:15:33.863 DEBUG:boto:Token: None 2024-04-13T22:15:33.863 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8StsIs+/uBTmJopAkGG/n0qqP+0= 2024-04-13T22:15:33.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8StsIs+/uBTmJopAkGG/n0qqP+0=', 'Content-Length': '0'} 2024-04-13T22:15:33.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a12c53c645bb7da-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.864 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.865 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.865 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.865 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.865 DEBUG:boto:Data: 2024-04-13T22:15:33.865 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.865 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.865 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.865 DEBUG:boto:Params: {} 2024-04-13T22:15:33.865 DEBUG:boto:Token: None 2024-04-13T22:15:33.865 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.866 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8StsIs+/uBTmJopAkGG/n0qqP+0= 2024-04-13T22:15:33.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8StsIs+/uBTmJopAkGG/n0qqP+0=', 'Content-Length': '0'} 2024-04-13T22:15:33.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9a694a3c85ab34-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-13T22:15:33.867 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.868 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.868 DEBUG:boto:Method: GET 2024-04-13T22:15:33.868 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.868 DEBUG:boto:Data: 2024-04-13T22:15:33.868 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.868 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.868 DEBUG:boto:Params: {} 2024-04-13T22:15:33.868 DEBUG:boto:Token: None 2024-04-13T22:15:33.869 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Op8hVulHt0a01g6k8vi9Cd9B4Xw= 2024-04-13T22:15:33.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Op8hVulHt0a01g6k8vi9Cd9B4Xw=', 'Content-Length': '0'} 2024-04-13T22:15:33.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033c3eb55b80fc141-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.871 DEBUG:boto:path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.871 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.871 DEBUG:boto:Method: GET 2024-04-13T22:15:33.871 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.871 DEBUG:boto:Data: 2024-04-13T22:15:33.871 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.871 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.872 DEBUG:boto:Params: {} 2024-04-13T22:15:33.872 DEBUG:boto:Token: None 2024-04-13T22:15:33.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000021 2024-04-13T22:15:33.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Op8hVulHt0a01g6k8vi9Cd9B4Xw= 2024-04-13T22:15:33.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Op8hVulHt0a01g6k8vi9Cd9B4Xw=', 'Content-Length': '0'} 2024-04-13T22:15:33.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2a0190c6f1c1c0f-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.874 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.874 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.874 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.874 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.874 DEBUG:boto:Method: GET 2024-04-13T22:15:33.874 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.874 DEBUG:boto:Data: 2024-04-13T22:15:33.875 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.875 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.875 DEBUG:boto:Params: {} 2024-04-13T22:15:33.875 DEBUG:boto:Token: None 2024-04-13T22:15:33.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72a9f02a1b3fd79-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.918 DEBUG:boto:b'vwnfns-60obj000000000000000000211000falseobj000000000000000000212024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.919 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.919 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.919 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.919 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.919 DEBUG:boto:Method: GET 2024-04-13T22:15:33.919 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000021 2024-04-13T22:15:33.919 DEBUG:boto:Data: 2024-04-13T22:15:33.920 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.920 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.920 DEBUG:boto:Params: {} 2024-04-13T22:15:33.920 DEBUG:boto:Token: None 2024-04-13T22:15:33.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffef49a96de1637-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.924 DEBUG:boto:b'vwnfns-60obj000000000000000000211000falseobj000000000000000000212024-04-13T22:15:21.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-13T22:15:33.925 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.925 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.925 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.925 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.925 DEBUG:boto:Method: GET 2024-04-13T22:15:33.925 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.925 DEBUG:boto:Data: 2024-04-13T22:15:33.925 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.926 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.926 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.926 DEBUG:boto:Params: {} 2024-04-13T22:15:33.926 DEBUG:boto:Token: None 2024-04-13T22:15:33.926 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0= 2024-04-13T22:15:33.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0=', 'Content-Length': '0'} 2024-04-13T22:15:33.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068003f8e3099bbf7-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.928 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.928 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.928 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.929 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.929 DEBUG:boto:Method: GET 2024-04-13T22:15:33.929 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.929 DEBUG:boto:Data: 2024-04-13T22:15:33.929 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.929 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.929 DEBUG:boto:Params: {} 2024-04-13T22:15:33.929 DEBUG:boto:Token: None 2024-04-13T22:15:33.929 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0= 2024-04-13T22:15:33.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0=', 'Content-Length': '0'} 2024-04-13T22:15:33.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816f4446da1c202f-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.931 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.932 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.932 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.932 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.932 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.932 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.932 DEBUG:boto:Data: 2024-04-13T22:15:33.932 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.932 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.932 DEBUG:boto:Params: {} 2024-04-13T22:15:33.932 DEBUG:boto:Token: None 2024-04-13T22:15:33.933 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DHSvbgclk7s9NcN8W5n4yXMkWeo= 2024-04-13T22:15:33.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DHSvbgclk7s9NcN8W5n4yXMkWeo=', 'Content-Length': '0'} 2024-04-13T22:15:33.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59aa5afe1806077-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.934 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.934 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.935 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.935 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.935 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.935 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.935 DEBUG:boto:Data: 2024-04-13T22:15:33.935 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.935 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.935 DEBUG:boto:Params: {} 2024-04-13T22:15:33.935 DEBUG:boto:Token: None 2024-04-13T22:15:33.936 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.936 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DHSvbgclk7s9NcN8W5n4yXMkWeo= 2024-04-13T22:15:33.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DHSvbgclk7s9NcN8W5n4yXMkWeo=', 'Content-Length': '0'} 2024-04-13T22:15:33.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000695d7f09cd0a4990-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-13T22:15:33.938 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.938 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.938 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.938 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.938 DEBUG:boto:Method: GET 2024-04-13T22:15:33.938 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.938 DEBUG:boto:Data: 2024-04-13T22:15:33.938 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.938 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.939 DEBUG:boto:Params: {} 2024-04-13T22:15:33.939 DEBUG:boto:Token: None 2024-04-13T22:15:33.939 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0= 2024-04-13T22:15:33.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0=', 'Content-Length': '0'} 2024-04-13T22:15:33.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036af6d41939ed4d2-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.941 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.941 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.941 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.941 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.941 DEBUG:boto:Method: GET 2024-04-13T22:15:33.941 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.942 DEBUG:boto:Data: 2024-04-13T22:15:33.942 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.942 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.942 DEBUG:boto:Params: {} 2024-04-13T22:15:33.942 DEBUG:boto:Token: None 2024-04-13T22:15:33.942 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022?versionId=null 2024-04-13T22:15:33.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0= 2024-04-13T22:15:33.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1ycYAcRjY+rg7QpYFBY0Q3xAkm0=', 'Content-Length': '0'} 2024-04-13T22:15:33.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f80d5ea08c937d-00661b0405-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.986 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.987 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.987 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:33.987 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:33.987 DEBUG:boto:Method: GET 2024-04-13T22:15:33.987 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:33.987 DEBUG:boto:Data: 2024-04-13T22:15:33.987 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.987 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.987 DEBUG:boto:Params: {} 2024-04-13T22:15:33.988 DEBUG:boto:Token: None 2024-04-13T22:15:33.988 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053514c9a2fb44e92-00661b0405-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.990 DEBUG:boto:b'vwnfns-60obj000000000000000000221000falseobj000000000000000000222024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.991 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:33.991 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:33.991 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:33.991 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:33.991 DEBUG:boto:Method: GET 2024-04-13T22:15:33.991 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:33.991 DEBUG:boto:Data: 2024-04-13T22:15:33.991 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.992 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:33.992 DEBUG:boto:Port: 8000 2024-04-13T22:15:33.992 DEBUG:boto:Params: {} 2024-04-13T22:15:33.992 DEBUG:boto:Token: None 2024-04-13T22:15:33.992 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/ 2024-04-13T22:15:33.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w= 2024-04-13T22:15:33.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a6JTX3fuWBxmcTSfPfSpOYvVW4w=', 'Content-Length': '0'} 2024-04-13T22:15:33.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cd54f897d5dc010-00661b0405-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:33.996 DEBUG:boto:b'vwnfns-60obj000000000000000000221000falseobj000000000000000000222024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:33.997 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.997 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.997 DEBUG:boto:Method: HEAD 2024-04-13T22:15:33.997 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.997 DEBUG:boto:Data: 2024-04-13T22:15:33.997 DEBUG:boto:Headers: {} 2024-04-13T22:15:33.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:33.997 DEBUG:boto:Port: 8001 2024-04-13T22:15:33.997 DEBUG:boto:Params: {} 2024-04-13T22:15:33.997 DEBUG:boto:Token: None 2024-04-13T22:15:33.998 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:33 GMT /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:33.998 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qy1+WCMqLsknQzOF/isc8GwHtIc= 2024-04-13T22:15:33.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qy1+WCMqLsknQzOF/isc8GwHtIc=', 'Content-Length': '0'} 2024-04-13T22:15:33.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e0f6031aeaee92-00661b0405-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.000 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.000 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.000 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.000 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.000 DEBUG:boto:Data: 2024-04-13T22:15:34.000 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.000 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.000 DEBUG:boto:Params: {} 2024-04-13T22:15:34.000 DEBUG:boto:Token: None 2024-04-13T22:15:34.001 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uJfL7EuuWr3R3BNxI3UWYMGbKcU= 2024-04-13T22:15:34.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uJfL7EuuWr3R3BNxI3UWYMGbKcU=', 'Content-Length': '0'} 2024-04-13T22:15:34.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707809af6fa956d6-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-13T22:15:34.003 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.003 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.003 DEBUG:boto:Method: GET 2024-04-13T22:15:34.003 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.003 DEBUG:boto:Data: 2024-04-13T22:15:34.003 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.003 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.003 DEBUG:boto:Params: {} 2024-04-13T22:15:34.003 DEBUG:boto:Token: None 2024-04-13T22:15:34.004 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y87Mu3D8EO1Vxaep7yKlVJH7M+8= 2024-04-13T22:15:34.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y87Mu3D8EO1Vxaep7yKlVJH7M+8=', 'Content-Length': '0'} 2024-04-13T22:15:34.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032abf77da55b7a0c-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.006 DEBUG:boto:path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.006 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.006 DEBUG:boto:Method: GET 2024-04-13T22:15:34.006 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.006 DEBUG:boto:Data: 2024-04-13T22:15:34.006 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.006 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.006 DEBUG:boto:Params: {} 2024-04-13T22:15:34.007 DEBUG:boto:Token: None 2024-04-13T22:15:34.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000022 2024-04-13T22:15:34.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y87Mu3D8EO1Vxaep7yKlVJH7M+8= 2024-04-13T22:15:34.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y87Mu3D8EO1Vxaep7yKlVJH7M+8=', 'Content-Length': '0'} 2024-04-13T22:15:34.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a5b334ea9c5470-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.050 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.051 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.051 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:34.051 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:34.051 DEBUG:boto:Method: GET 2024-04-13T22:15:34.051 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:34.051 DEBUG:boto:Data: 2024-04-13T22:15:34.051 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.051 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.051 DEBUG:boto:Params: {} 2024-04-13T22:15:34.052 DEBUG:boto:Token: None 2024-04-13T22:15:34.052 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000919655df836891fb-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.055 DEBUG:boto:b'vwnfns-60obj000000000000000000221000falseobj000000000000000000222024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.055 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.055 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.055 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:34.055 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:34.055 DEBUG:boto:Method: GET 2024-04-13T22:15:34.055 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000022 2024-04-13T22:15:34.056 DEBUG:boto:Data: 2024-04-13T22:15:34.056 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.056 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.056 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.056 DEBUG:boto:Params: {} 2024-04-13T22:15:34.056 DEBUG:boto:Token: None 2024-04-13T22:15:34.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7318fbf310fd62-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.060 DEBUG:boto:b'vwnfns-60obj000000000000000000221000falseobj000000000000000000222024-04-13T22:15:21.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-13T22:15:34.060 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.060 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.061 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.061 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.061 DEBUG:boto:Method: GET 2024-04-13T22:15:34.061 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.061 DEBUG:boto:Data: 2024-04-13T22:15:34.061 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.061 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.061 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.061 DEBUG:boto:Params: {} 2024-04-13T22:15:34.061 DEBUG:boto:Token: None 2024-04-13T22:15:34.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.062 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y= 2024-04-13T22:15:34.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4348d423ba9a40-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.106 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.107 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.107 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.107 DEBUG:boto:Method: GET 2024-04-13T22:15:34.107 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.107 DEBUG:boto:Data: 2024-04-13T22:15:34.107 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.107 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.107 DEBUG:boto:Params: {} 2024-04-13T22:15:34.108 DEBUG:boto:Token: None 2024-04-13T22:15:34.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y= 2024-04-13T22:15:34.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dac474f3c4bb229-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.109 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.110 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.110 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.110 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.110 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.110 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.110 DEBUG:boto:Data: 2024-04-13T22:15:34.110 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.111 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.111 DEBUG:boto:Params: {} 2024-04-13T22:15:34.111 DEBUG:boto:Token: None 2024-04-13T22:15:34.111 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f2P5t+rJ1c9ylpWWVTrwGtlZcfQ= 2024-04-13T22:15:34.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f2P5t+rJ1c9ylpWWVTrwGtlZcfQ=', 'Content-Length': '0'} 2024-04-13T22:15:34.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f925d87a62bb149b-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.113 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.113 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.113 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.113 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.113 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.113 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.113 DEBUG:boto:Data: 2024-04-13T22:15:34.113 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.114 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.114 DEBUG:boto:Params: {} 2024-04-13T22:15:34.114 DEBUG:boto:Token: None 2024-04-13T22:15:34.114 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f2P5t+rJ1c9ylpWWVTrwGtlZcfQ= 2024-04-13T22:15:34.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f2P5t+rJ1c9ylpWWVTrwGtlZcfQ=', 'Content-Length': '0'} 2024-04-13T22:15:34.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e84d1b329edbd7-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-13T22:15:34.116 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.116 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.116 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.116 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.116 DEBUG:boto:Method: GET 2024-04-13T22:15:34.116 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.117 DEBUG:boto:Data: 2024-04-13T22:15:34.117 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.117 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.117 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.117 DEBUG:boto:Params: {} 2024-04-13T22:15:34.117 DEBUG:boto:Token: None 2024-04-13T22:15:34.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y= 2024-04-13T22:15:34.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a4bdc946be0107-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.119 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.119 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.119 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.119 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.120 DEBUG:boto:Method: GET 2024-04-13T22:15:34.120 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.120 DEBUG:boto:Data: 2024-04-13T22:15:34.120 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.120 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.120 DEBUG:boto:Params: {} 2024-04-13T22:15:34.120 DEBUG:boto:Token: None 2024-04-13T22:15:34.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024?versionId=null 2024-04-13T22:15:34.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y= 2024-04-13T22:15:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ngkheq+eDrP6/y62BKFJQUTwz8Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae660b9ba172ad86-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.122 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.122 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.123 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.123 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.123 DEBUG:boto:Method: GET 2024-04-13T22:15:34.123 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.123 DEBUG:boto:Data: 2024-04-13T22:15:34.123 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.123 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.123 DEBUG:boto:Params: {} 2024-04-13T22:15:34.123 DEBUG:boto:Token: None 2024-04-13T22:15:34.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d73f1dbe847a81d7-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.126 DEBUG:boto:b'vwnfns-60obj000000000000000000241000falseobj000000000000000000242024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.126 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.126 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.126 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.126 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.126 DEBUG:boto:Method: GET 2024-04-13T22:15:34.126 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.126 DEBUG:boto:Data: 2024-04-13T22:15:34.126 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.127 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.127 DEBUG:boto:Params: {} 2024-04-13T22:15:34.127 DEBUG:boto:Token: None 2024-04-13T22:15:34.127 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.127 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd16ae5743c90188-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.130 DEBUG:boto:b'vwnfns-60obj000000000000000000241000falseobj000000000000000000242024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.131 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.131 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.131 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.131 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.131 DEBUG:boto:Data: 2024-04-13T22:15:34.131 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.131 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.132 DEBUG:boto:Params: {} 2024-04-13T22:15:34.132 DEBUG:boto:Token: None 2024-04-13T22:15:34.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+lFVwdB5KYHZ1pLUOjMZGxzdf1I= 2024-04-13T22:15:34.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+lFVwdB5KYHZ1pLUOjMZGxzdf1I=', 'Content-Length': '0'} 2024-04-13T22:15:34.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c760e69e382308c-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.134 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.134 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.134 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.134 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.134 DEBUG:boto:Data: 2024-04-13T22:15:34.134 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.134 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.135 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.135 DEBUG:boto:Params: {} 2024-04-13T22:15:34.135 DEBUG:boto:Token: None 2024-04-13T22:15:34.135 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+lFVwdB5KYHZ1pLUOjMZGxzdf1I= 2024-04-13T22:15:34.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+lFVwdB5KYHZ1pLUOjMZGxzdf1I=', 'Content-Length': '0'} 2024-04-13T22:15:34.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000796e9b95b3988029-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-13T22:15:34.137 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.137 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.137 DEBUG:boto:Method: GET 2024-04-13T22:15:34.137 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.137 DEBUG:boto:Data: 2024-04-13T22:15:34.137 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.138 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.138 DEBUG:boto:Params: {} 2024-04-13T22:15:34.138 DEBUG:boto:Token: None 2024-04-13T22:15:34.138 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.138 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1smDzsRpAA0OQg3IZKfE7FcMWj4= 2024-04-13T22:15:34.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1smDzsRpAA0OQg3IZKfE7FcMWj4=', 'Content-Length': '0'} 2024-04-13T22:15:34.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22a21ffef9645e1-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.140 DEBUG:boto:path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.140 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.140 DEBUG:boto:Method: GET 2024-04-13T22:15:34.140 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.140 DEBUG:boto:Data: 2024-04-13T22:15:34.140 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.140 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.140 DEBUG:boto:Params: {} 2024-04-13T22:15:34.141 DEBUG:boto:Token: None 2024-04-13T22:15:34.141 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000024 2024-04-13T22:15:34.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1smDzsRpAA0OQg3IZKfE7FcMWj4= 2024-04-13T22:15:34.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1smDzsRpAA0OQg3IZKfE7FcMWj4=', 'Content-Length': '0'} 2024-04-13T22:15:34.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009989bdcf5df47583-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.186 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.187 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.187 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.187 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.187 DEBUG:boto:Method: GET 2024-04-13T22:15:34.187 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.187 DEBUG:boto:Data: 2024-04-13T22:15:34.187 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.187 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.187 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.187 DEBUG:boto:Params: {} 2024-04-13T22:15:34.187 DEBUG:boto:Token: None 2024-04-13T22:15:34.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089781aa1325a6e16-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.190 DEBUG:boto:b'vwnfns-60obj000000000000000000241000falseobj000000000000000000242024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.191 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.191 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.191 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.191 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.191 DEBUG:boto:Method: GET 2024-04-13T22:15:34.191 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000024 2024-04-13T22:15:34.191 DEBUG:boto:Data: 2024-04-13T22:15:34.191 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.192 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.192 DEBUG:boto:Params: {} 2024-04-13T22:15:34.192 DEBUG:boto:Token: None 2024-04-13T22:15:34.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26774bde2a871a4-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.196 DEBUG:boto:b'vwnfns-60obj000000000000000000241000falseobj000000000000000000242024-04-13T22:15:21.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-13T22:15:34.196 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.196 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.196 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.196 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.196 DEBUG:boto:Method: GET 2024-04-13T22:15:34.197 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.197 DEBUG:boto:Data: 2024-04-13T22:15:34.197 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.197 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.197 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.197 DEBUG:boto:Params: {} 2024-04-13T22:15:34.197 DEBUG:boto:Token: None 2024-04-13T22:15:34.197 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE= 2024-04-13T22:15:34.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE=', 'Content-Length': '0'} 2024-04-13T22:15:34.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c769d9e44f1c04f0-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.242 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.242 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.243 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.243 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.243 DEBUG:boto:Method: GET 2024-04-13T22:15:34.243 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.243 DEBUG:boto:Data: 2024-04-13T22:15:34.243 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.243 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.243 DEBUG:boto:Params: {} 2024-04-13T22:15:34.244 DEBUG:boto:Token: None 2024-04-13T22:15:34.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE= 2024-04-13T22:15:34.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE=', 'Content-Length': '0'} 2024-04-13T22:15:34.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca62f169a2262514-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.246 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.246 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.246 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.247 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.247 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.247 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.247 DEBUG:boto:Data: 2024-04-13T22:15:34.247 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.247 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.247 DEBUG:boto:Params: {} 2024-04-13T22:15:34.247 DEBUG:boto:Token: None 2024-04-13T22:15:34.247 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xzbORdVC0jlABzHj0QSa/8lgbgI= 2024-04-13T22:15:34.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xzbORdVC0jlABzHj0QSa/8lgbgI=', 'Content-Length': '0'} 2024-04-13T22:15:34.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b256371bac7172-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.249 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.250 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.250 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.250 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.250 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.250 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.250 DEBUG:boto:Data: 2024-04-13T22:15:34.251 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.251 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.251 DEBUG:boto:Params: {} 2024-04-13T22:15:34.251 DEBUG:boto:Token: None 2024-04-13T22:15:34.251 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xzbORdVC0jlABzHj0QSa/8lgbgI= 2024-04-13T22:15:34.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xzbORdVC0jlABzHj0QSa/8lgbgI=', 'Content-Length': '0'} 2024-04-13T22:15:34.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fbf46566e4af77e-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-13T22:15:34.253 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.253 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.253 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.253 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.253 DEBUG:boto:Method: GET 2024-04-13T22:15:34.254 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.254 DEBUG:boto:Data: 2024-04-13T22:15:34.254 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.254 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.254 DEBUG:boto:Params: {} 2024-04-13T22:15:34.254 DEBUG:boto:Token: None 2024-04-13T22:15:34.254 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE= 2024-04-13T22:15:34.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE=', 'Content-Length': '0'} 2024-04-13T22:15:34.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e00b6a529abed2d-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.256 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.256 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.257 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.257 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.257 DEBUG:boto:Method: GET 2024-04-13T22:15:34.257 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.257 DEBUG:boto:Data: 2024-04-13T22:15:34.257 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.257 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.257 DEBUG:boto:Params: {} 2024-04-13T22:15:34.257 DEBUG:boto:Token: None 2024-04-13T22:15:34.258 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025?versionId=null 2024-04-13T22:15:34.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE= 2024-04-13T22:15:34.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PiyJLBWvwgllxaumsfrfltwviFE=', 'Content-Length': '0'} 2024-04-13T22:15:34.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089e53a00a8f35ea1-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.260 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.260 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.260 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.260 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.260 DEBUG:boto:Method: GET 2024-04-13T22:15:34.260 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.260 DEBUG:boto:Data: 2024-04-13T22:15:34.261 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.261 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.261 DEBUG:boto:Params: {} 2024-04-13T22:15:34.261 DEBUG:boto:Token: None 2024-04-13T22:15:34.261 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1c775dc55208e5-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.264 DEBUG:boto:b'vwnfns-60obj000000000000000000251000falseobj000000000000000000252024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.264 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.264 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.264 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.264 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.264 DEBUG:boto:Method: GET 2024-04-13T22:15:34.264 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.265 DEBUG:boto:Data: 2024-04-13T22:15:34.265 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.265 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.265 DEBUG:boto:Params: {} 2024-04-13T22:15:34.265 DEBUG:boto:Token: None 2024-04-13T22:15:34.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ca3910a7ab4ba3-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.269 DEBUG:boto:b'vwnfns-60obj000000000000000000251000falseobj000000000000000000252024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.269 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.270 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.270 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.270 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.270 DEBUG:boto:Data: 2024-04-13T22:15:34.270 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.270 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.270 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.270 DEBUG:boto:Params: {} 2024-04-13T22:15:34.270 DEBUG:boto:Token: None 2024-04-13T22:15:34.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rieUsvjGJq8ddyO0eGcwsagiuQw= 2024-04-13T22:15:34.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rieUsvjGJq8ddyO0eGcwsagiuQw=', 'Content-Length': '0'} 2024-04-13T22:15:34.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771166ee52b58ed1-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.273 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.273 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.273 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.273 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.273 DEBUG:boto:Data: 2024-04-13T22:15:34.273 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.273 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.273 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.273 DEBUG:boto:Params: {} 2024-04-13T22:15:34.273 DEBUG:boto:Token: None 2024-04-13T22:15:34.274 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rieUsvjGJq8ddyO0eGcwsagiuQw= 2024-04-13T22:15:34.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rieUsvjGJq8ddyO0eGcwsagiuQw=', 'Content-Length': '0'} 2024-04-13T22:15:34.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020c5ce9eb1e9bbe-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-13T22:15:34.276 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.276 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.276 DEBUG:boto:Method: GET 2024-04-13T22:15:34.276 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.276 DEBUG:boto:Data: 2024-04-13T22:15:34.276 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.276 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.276 DEBUG:boto:Params: {} 2024-04-13T22:15:34.276 DEBUG:boto:Token: None 2024-04-13T22:15:34.276 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.277 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKMxdAIxhuQYWVkdfG2dPjbemRg= 2024-04-13T22:15:34.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKMxdAIxhuQYWVkdfG2dPjbemRg=', 'Content-Length': '0'} 2024-04-13T22:15:34.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007259ba38783c7b5e-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.279 DEBUG:boto:path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.279 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.279 DEBUG:boto:Method: GET 2024-04-13T22:15:34.279 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.279 DEBUG:boto:Data: 2024-04-13T22:15:34.279 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.280 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.280 DEBUG:boto:Params: {} 2024-04-13T22:15:34.280 DEBUG:boto:Token: None 2024-04-13T22:15:34.280 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000025 2024-04-13T22:15:34.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZKMxdAIxhuQYWVkdfG2dPjbemRg= 2024-04-13T22:15:34.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZKMxdAIxhuQYWVkdfG2dPjbemRg=', 'Content-Length': '0'} 2024-04-13T22:15:34.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9475c2672cc180c-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.282 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.282 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.282 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.282 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.282 DEBUG:boto:Method: GET 2024-04-13T22:15:34.282 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.282 DEBUG:boto:Data: 2024-04-13T22:15:34.282 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.282 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.283 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.283 DEBUG:boto:Params: {} 2024-04-13T22:15:34.283 DEBUG:boto:Token: None 2024-04-13T22:15:34.283 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dfe273cf98c5184-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.326 DEBUG:boto:b'vwnfns-60obj000000000000000000251000falseobj000000000000000000252024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.327 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.327 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.327 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.327 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.327 DEBUG:boto:Method: GET 2024-04-13T22:15:34.327 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000025 2024-04-13T22:15:34.327 DEBUG:boto:Data: 2024-04-13T22:15:34.327 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.328 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.328 DEBUG:boto:Params: {} 2024-04-13T22:15:34.328 DEBUG:boto:Token: None 2024-04-13T22:15:34.328 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f49fa6c66955ea-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.332 DEBUG:boto:b'vwnfns-60obj000000000000000000251000falseobj000000000000000000252024-04-13T22:15:20.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-13T22:15:34.332 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.332 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.333 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.333 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.333 DEBUG:boto:Method: GET 2024-04-13T22:15:34.333 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.333 DEBUG:boto:Data: 2024-04-13T22:15:34.333 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.333 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.333 DEBUG:boto:Params: {} 2024-04-13T22:15:34.333 DEBUG:boto:Token: None 2024-04-13T22:15:34.333 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns= 2024-04-13T22:15:34.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns=', 'Content-Length': '0'} 2024-04-13T22:15:34.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1fe780cab00e378-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.336 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.336 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.336 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.336 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.336 DEBUG:boto:Method: GET 2024-04-13T22:15:34.336 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.336 DEBUG:boto:Data: 2024-04-13T22:15:34.336 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.336 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.336 DEBUG:boto:Params: {} 2024-04-13T22:15:34.337 DEBUG:boto:Token: None 2024-04-13T22:15:34.337 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns= 2024-04-13T22:15:34.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns=', 'Content-Length': '0'} 2024-04-13T22:15:34.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4ea99ec38b2ec3-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.382 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.382 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.383 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.383 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.383 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.383 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.383 DEBUG:boto:Data: 2024-04-13T22:15:34.383 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.383 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.383 DEBUG:boto:Params: {} 2024-04-13T22:15:34.383 DEBUG:boto:Token: None 2024-04-13T22:15:34.384 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f+da3+3kiiw9mCoMWDbK9Rh1Rmg= 2024-04-13T22:15:34.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f+da3+3kiiw9mCoMWDbK9Rh1Rmg=', 'Content-Length': '0'} 2024-04-13T22:15:34.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bd95239a2002a5-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.386 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.386 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.386 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.386 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.386 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.386 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.386 DEBUG:boto:Data: 2024-04-13T22:15:34.387 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.387 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.387 DEBUG:boto:Params: {} 2024-04-13T22:15:34.387 DEBUG:boto:Token: None 2024-04-13T22:15:34.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f+da3+3kiiw9mCoMWDbK9Rh1Rmg= 2024-04-13T22:15:34.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f+da3+3kiiw9mCoMWDbK9Rh1Rmg=', 'Content-Length': '0'} 2024-04-13T22:15:34.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a18de2ca519b50-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-13T22:15:34.389 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.389 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.389 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.390 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.390 DEBUG:boto:Method: GET 2024-04-13T22:15:34.390 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.390 DEBUG:boto:Data: 2024-04-13T22:15:34.390 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.390 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.390 DEBUG:boto:Params: {} 2024-04-13T22:15:34.390 DEBUG:boto:Token: None 2024-04-13T22:15:34.390 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns= 2024-04-13T22:15:34.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns=', 'Content-Length': '0'} 2024-04-13T22:15:34.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a79b797b036db7-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.392 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.393 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.393 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.393 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.393 DEBUG:boto:Method: GET 2024-04-13T22:15:34.393 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.393 DEBUG:boto:Data: 2024-04-13T22:15:34.393 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.393 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.393 DEBUG:boto:Params: {} 2024-04-13T22:15:34.394 DEBUG:boto:Token: None 2024-04-13T22:15:34.394 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026?versionId=null 2024-04-13T22:15:34.394 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns= 2024-04-13T22:15:34.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PJJuZzx0O5JgYteEzJfsiCeB9ns=', 'Content-Length': '0'} 2024-04-13T22:15:34.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e208238e008fab08-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.396 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.396 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.396 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.396 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.396 DEBUG:boto:Method: GET 2024-04-13T22:15:34.396 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.396 DEBUG:boto:Data: 2024-04-13T22:15:34.397 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.397 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.397 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.397 DEBUG:boto:Params: {} 2024-04-13T22:15:34.397 DEBUG:boto:Token: None 2024-04-13T22:15:34.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3bb6f7efe3f90f8-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.399 DEBUG:boto:b'vwnfns-60obj000000000000000000261000falseobj000000000000000000262024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.400 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.400 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.400 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.400 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.400 DEBUG:boto:Method: GET 2024-04-13T22:15:34.400 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.400 DEBUG:boto:Data: 2024-04-13T22:15:34.401 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.401 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.401 DEBUG:boto:Params: {} 2024-04-13T22:15:34.401 DEBUG:boto:Token: None 2024-04-13T22:15:34.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a7c9d60cc8f3264-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.405 DEBUG:boto:b'vwnfns-60obj000000000000000000261000falseobj000000000000000000262024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.405 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.406 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.406 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.406 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.406 DEBUG:boto:Data: 2024-04-13T22:15:34.406 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.406 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.406 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.406 DEBUG:boto:Params: {} 2024-04-13T22:15:34.406 DEBUG:boto:Token: None 2024-04-13T22:15:34.407 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.407 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uGyjl8K6Qlug/g9NpR+K5hNVMZQ= 2024-04-13T22:15:34.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uGyjl8K6Qlug/g9NpR+K5hNVMZQ=', 'Content-Length': '0'} 2024-04-13T22:15:34.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1ba419d0f830f3-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.408 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.408 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.409 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.409 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.409 DEBUG:boto:Data: 2024-04-13T22:15:34.409 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.409 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.409 DEBUG:boto:Params: {} 2024-04-13T22:15:34.409 DEBUG:boto:Token: None 2024-04-13T22:15:34.409 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uGyjl8K6Qlug/g9NpR+K5hNVMZQ= 2024-04-13T22:15:34.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uGyjl8K6Qlug/g9NpR+K5hNVMZQ=', 'Content-Length': '0'} 2024-04-13T22:15:34.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf0efba685590ff-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-13T22:15:34.412 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.412 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.412 DEBUG:boto:Method: GET 2024-04-13T22:15:34.412 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.412 DEBUG:boto:Data: 2024-04-13T22:15:34.412 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.412 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.412 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.412 DEBUG:boto:Params: {} 2024-04-13T22:15:34.412 DEBUG:boto:Token: None 2024-04-13T22:15:34.413 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:76J/cBl5AgGcG+2/HEOKZEEJn50= 2024-04-13T22:15:34.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:76J/cBl5AgGcG+2/HEOKZEEJn50=', 'Content-Length': '0'} 2024-04-13T22:15:34.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008594930ea4354240-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.415 DEBUG:boto:path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.415 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.415 DEBUG:boto:Method: GET 2024-04-13T22:15:34.415 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.415 DEBUG:boto:Data: 2024-04-13T22:15:34.415 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.416 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.416 DEBUG:boto:Params: {} 2024-04-13T22:15:34.416 DEBUG:boto:Token: None 2024-04-13T22:15:34.416 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000026 2024-04-13T22:15:34.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:76J/cBl5AgGcG+2/HEOKZEEJn50= 2024-04-13T22:15:34.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:76J/cBl5AgGcG+2/HEOKZEEJn50=', 'Content-Length': '0'} 2024-04-13T22:15:34.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a978d051cf89ef5-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.458 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.459 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.459 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.459 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.459 DEBUG:boto:Method: GET 2024-04-13T22:15:34.459 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.459 DEBUG:boto:Data: 2024-04-13T22:15:34.459 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.459 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.459 DEBUG:boto:Params: {} 2024-04-13T22:15:34.460 DEBUG:boto:Token: None 2024-04-13T22:15:34.460 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025aa4303038291cc-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.462 DEBUG:boto:b'vwnfns-60obj000000000000000000261000falseobj000000000000000000262024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.463 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.463 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.463 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.463 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.463 DEBUG:boto:Method: GET 2024-04-13T22:15:34.463 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000026 2024-04-13T22:15:34.463 DEBUG:boto:Data: 2024-04-13T22:15:34.463 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.464 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.464 DEBUG:boto:Params: {} 2024-04-13T22:15:34.464 DEBUG:boto:Token: None 2024-04-13T22:15:34.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27fb23df1230acc-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.468 DEBUG:boto:b'vwnfns-60obj000000000000000000261000falseobj000000000000000000262024-04-13T22:15:22.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-13T22:15:34.469 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.469 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.469 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.469 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.469 DEBUG:boto:Method: GET 2024-04-13T22:15:34.469 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.469 DEBUG:boto:Data: 2024-04-13T22:15:34.469 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.470 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.470 DEBUG:boto:Params: {} 2024-04-13T22:15:34.470 DEBUG:boto:Token: None 2024-04-13T22:15:34.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg= 2024-04-13T22:15:34.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg=', 'Content-Length': '0'} 2024-04-13T22:15:34.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e2bf41256d3a57-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.514 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.514 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.515 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.515 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.515 DEBUG:boto:Method: GET 2024-04-13T22:15:34.515 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.515 DEBUG:boto:Data: 2024-04-13T22:15:34.515 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.515 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.516 DEBUG:boto:Params: {} 2024-04-13T22:15:34.516 DEBUG:boto:Token: None 2024-04-13T22:15:34.516 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg= 2024-04-13T22:15:34.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg=', 'Content-Length': '0'} 2024-04-13T22:15:34.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1da549ea26d5b87-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.518 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.518 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.519 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.519 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.519 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.519 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.519 DEBUG:boto:Data: 2024-04-13T22:15:34.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.519 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.519 DEBUG:boto:Params: {} 2024-04-13T22:15:34.519 DEBUG:boto:Token: None 2024-04-13T22:15:34.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pRmQHT8+/iksCGcfQKB/zSfDLM4= 2024-04-13T22:15:34.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pRmQHT8+/iksCGcfQKB/zSfDLM4=', 'Content-Length': '0'} 2024-04-13T22:15:34.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73c944437655691-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.521 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.522 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.522 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.522 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.522 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.522 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.522 DEBUG:boto:Data: 2024-04-13T22:15:34.522 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.522 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.523 DEBUG:boto:Params: {} 2024-04-13T22:15:34.523 DEBUG:boto:Token: None 2024-04-13T22:15:34.523 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pRmQHT8+/iksCGcfQKB/zSfDLM4= 2024-04-13T22:15:34.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pRmQHT8+/iksCGcfQKB/zSfDLM4=', 'Content-Length': '0'} 2024-04-13T22:15:34.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bb383495e48865-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-13T22:15:34.525 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.525 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.525 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.526 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.526 DEBUG:boto:Method: GET 2024-04-13T22:15:34.526 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.526 DEBUG:boto:Data: 2024-04-13T22:15:34.526 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.526 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.526 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.527 DEBUG:boto:Params: {} 2024-04-13T22:15:34.527 DEBUG:boto:Token: None 2024-04-13T22:15:34.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg= 2024-04-13T22:15:34.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg=', 'Content-Length': '0'} 2024-04-13T22:15:34.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7b78ec8892ffec-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.529 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.529 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.530 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.530 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.530 DEBUG:boto:Method: GET 2024-04-13T22:15:34.530 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.530 DEBUG:boto:Data: 2024-04-13T22:15:34.530 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.530 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.530 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.530 DEBUG:boto:Params: {} 2024-04-13T22:15:34.531 DEBUG:boto:Token: None 2024-04-13T22:15:34.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027?versionId=null 2024-04-13T22:15:34.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg= 2024-04-13T22:15:34.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6N9KnLl9vyUwNnQHhV49OUuhaNg=', 'Content-Length': '0'} 2024-04-13T22:15:34.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693d8b8dea35b53d-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.533 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.533 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.533 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.534 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.534 DEBUG:boto:Method: GET 2024-04-13T22:15:34.534 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.534 DEBUG:boto:Data: 2024-04-13T22:15:34.534 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.534 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.534 DEBUG:boto:Params: {} 2024-04-13T22:15:34.534 DEBUG:boto:Token: None 2024-04-13T22:15:34.535 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa28f2b7441593b-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.537 DEBUG:boto:b'vwnfns-60obj000000000000000000271000falseobj000000000000000000272024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.537 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.538 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.538 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.538 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.538 DEBUG:boto:Method: GET 2024-04-13T22:15:34.538 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.538 DEBUG:boto:Data: 2024-04-13T22:15:34.538 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.538 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.538 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.538 DEBUG:boto:Params: {} 2024-04-13T22:15:34.539 DEBUG:boto:Token: None 2024-04-13T22:15:34.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250f84b4a76562fc-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.543 DEBUG:boto:b'vwnfns-60obj000000000000000000271000falseobj000000000000000000272024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.543 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.543 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.543 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.543 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.543 DEBUG:boto:Data: 2024-04-13T22:15:34.543 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.544 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.544 DEBUG:boto:Params: {} 2024-04-13T22:15:34.544 DEBUG:boto:Token: None 2024-04-13T22:15:34.544 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/GYf+Sk+1Mp7XwkF9Nuon7CI2hQ= 2024-04-13T22:15:34.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/GYf+Sk+1Mp7XwkF9Nuon7CI2hQ=', 'Content-Length': '0'} 2024-04-13T22:15:34.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095bcb9a6db908468-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.546 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.546 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.546 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.546 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.546 DEBUG:boto:Data: 2024-04-13T22:15:34.547 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.547 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.547 DEBUG:boto:Params: {} 2024-04-13T22:15:34.547 DEBUG:boto:Token: None 2024-04-13T22:15:34.547 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/GYf+Sk+1Mp7XwkF9Nuon7CI2hQ= 2024-04-13T22:15:34.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/GYf+Sk+1Mp7XwkF9Nuon7CI2hQ=', 'Content-Length': '0'} 2024-04-13T22:15:34.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5feff9e1886858a-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-13T22:15:34.550 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.550 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.550 DEBUG:boto:Method: GET 2024-04-13T22:15:34.550 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.550 DEBUG:boto:Data: 2024-04-13T22:15:34.550 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.550 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.550 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.551 DEBUG:boto:Params: {} 2024-04-13T22:15:34.551 DEBUG:boto:Token: None 2024-04-13T22:15:34.551 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tBp9VD4OZBHjNfHs7a2/+EPKtjo= 2024-04-13T22:15:34.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tBp9VD4OZBHjNfHs7a2/+EPKtjo=', 'Content-Length': '0'} 2024-04-13T22:15:34.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030200ca44fbc89b6-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.553 DEBUG:boto:path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.553 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.553 DEBUG:boto:Method: GET 2024-04-13T22:15:34.553 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.553 DEBUG:boto:Data: 2024-04-13T22:15:34.553 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.553 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.553 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.554 DEBUG:boto:Params: {} 2024-04-13T22:15:34.554 DEBUG:boto:Token: None 2024-04-13T22:15:34.554 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000027 2024-04-13T22:15:34.554 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tBp9VD4OZBHjNfHs7a2/+EPKtjo= 2024-04-13T22:15:34.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tBp9VD4OZBHjNfHs7a2/+EPKtjo=', 'Content-Length': '0'} 2024-04-13T22:15:34.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001657c90d6143659-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.556 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.556 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.556 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.556 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.556 DEBUG:boto:Method: GET 2024-04-13T22:15:34.556 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.556 DEBUG:boto:Data: 2024-04-13T22:15:34.557 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.557 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.557 DEBUG:boto:Params: {} 2024-04-13T22:15:34.557 DEBUG:boto:Token: None 2024-04-13T22:15:34.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d0c2a46bf26fd61-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.560 DEBUG:boto:b'vwnfns-60obj000000000000000000271000falseobj000000000000000000272024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.560 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.560 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.561 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.561 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.561 DEBUG:boto:Method: GET 2024-04-13T22:15:34.561 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000027 2024-04-13T22:15:34.561 DEBUG:boto:Data: 2024-04-13T22:15:34.561 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.561 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.562 DEBUG:boto:Params: {} 2024-04-13T22:15:34.562 DEBUG:boto:Token: None 2024-04-13T22:15:34.562 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c95d358ce8af87-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.566 DEBUG:boto:b'vwnfns-60obj000000000000000000271000falseobj000000000000000000272024-04-13T22:15:22.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-13T22:15:34.566 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.566 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.566 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.567 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.567 DEBUG:boto:Method: GET 2024-04-13T22:15:34.567 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.567 DEBUG:boto:Data: 2024-04-13T22:15:34.567 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.567 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.567 DEBUG:boto:Params: {} 2024-04-13T22:15:34.567 DEBUG:boto:Token: None 2024-04-13T22:15:34.568 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY= 2024-04-13T22:15:34.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY=', 'Content-Length': '0'} 2024-04-13T22:15:34.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee9551455fc76c8-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.570 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.570 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.570 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.570 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.570 DEBUG:boto:Method: GET 2024-04-13T22:15:34.571 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.571 DEBUG:boto:Data: 2024-04-13T22:15:34.571 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.571 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.571 DEBUG:boto:Params: {} 2024-04-13T22:15:34.571 DEBUG:boto:Token: None 2024-04-13T22:15:34.571 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY= 2024-04-13T22:15:34.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY=', 'Content-Length': '0'} 2024-04-13T22:15:34.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020476f1fe283e202-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.574 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.574 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.574 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.574 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.574 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.574 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.574 DEBUG:boto:Data: 2024-04-13T22:15:34.574 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.575 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.575 DEBUG:boto:Params: {} 2024-04-13T22:15:34.575 DEBUG:boto:Token: None 2024-04-13T22:15:34.575 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cv+q775LtYGpEztTuw5UXw8GGZk= 2024-04-13T22:15:34.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cv+q775LtYGpEztTuw5UXw8GGZk=', 'Content-Length': '0'} 2024-04-13T22:15:34.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b73f21be468705b8-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.577 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.577 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.577 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.577 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.578 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.578 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.578 DEBUG:boto:Data: 2024-04-13T22:15:34.578 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.578 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.578 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.578 DEBUG:boto:Params: {} 2024-04-13T22:15:34.578 DEBUG:boto:Token: None 2024-04-13T22:15:34.578 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.579 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cv+q775LtYGpEztTuw5UXw8GGZk= 2024-04-13T22:15:34.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cv+q775LtYGpEztTuw5UXw8GGZk=', 'Content-Length': '0'} 2024-04-13T22:15:34.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f8c03f7848cd90-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-13T22:15:34.580 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.580 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.581 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.581 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.581 DEBUG:boto:Method: GET 2024-04-13T22:15:34.581 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.581 DEBUG:boto:Data: 2024-04-13T22:15:34.581 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.581 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.581 DEBUG:boto:Params: {} 2024-04-13T22:15:34.581 DEBUG:boto:Token: None 2024-04-13T22:15:34.582 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY= 2024-04-13T22:15:34.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY=', 'Content-Length': '0'} 2024-04-13T22:15:34.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82adeabac592bd1-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.584 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.584 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.584 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.584 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.584 DEBUG:boto:Method: GET 2024-04-13T22:15:34.584 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.584 DEBUG:boto:Data: 2024-04-13T22:15:34.585 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.585 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.585 DEBUG:boto:Params: {} 2024-04-13T22:15:34.585 DEBUG:boto:Token: None 2024-04-13T22:15:34.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028?versionId=null 2024-04-13T22:15:34.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY= 2024-04-13T22:15:34.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Et8kl81aEcI7bdEAPkl7xeADRQY=', 'Content-Length': '0'} 2024-04-13T22:15:34.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000863e0320898aeb5-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.587 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.587 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.587 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.587 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.587 DEBUG:boto:Method: GET 2024-04-13T22:15:34.588 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.588 DEBUG:boto:Data: 2024-04-13T22:15:34.588 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.588 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.588 DEBUG:boto:Params: {} 2024-04-13T22:15:34.588 DEBUG:boto:Token: None 2024-04-13T22:15:34.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f236641bd8f70ead-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.591 DEBUG:boto:b'vwnfns-60obj000000000000000000281000falseobj000000000000000000282024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.592 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.592 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.592 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.592 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.592 DEBUG:boto:Method: GET 2024-04-13T22:15:34.592 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.592 DEBUG:boto:Data: 2024-04-13T22:15:34.592 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.592 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.593 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.593 DEBUG:boto:Params: {} 2024-04-13T22:15:34.593 DEBUG:boto:Token: None 2024-04-13T22:15:34.593 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.593 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed704604164c0ed7-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.638 DEBUG:boto:b'vwnfns-60obj000000000000000000281000falseobj000000000000000000282024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.639 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.639 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.639 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.639 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.639 DEBUG:boto:Data: 2024-04-13T22:15:34.640 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.640 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.640 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.640 DEBUG:boto:Params: {} 2024-04-13T22:15:34.640 DEBUG:boto:Token: None 2024-04-13T22:15:34.640 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aqUc7dX6chCrw9gq3jArzpxv23k= 2024-04-13T22:15:34.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aqUc7dX6chCrw9gq3jArzpxv23k=', 'Content-Length': '0'} 2024-04-13T22:15:34.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115dc37880217dcb-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.642 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.643 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.643 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.643 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.643 DEBUG:boto:Data: 2024-04-13T22:15:34.643 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.643 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.643 DEBUG:boto:Params: {} 2024-04-13T22:15:34.643 DEBUG:boto:Token: None 2024-04-13T22:15:34.644 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aqUc7dX6chCrw9gq3jArzpxv23k= 2024-04-13T22:15:34.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aqUc7dX6chCrw9gq3jArzpxv23k=', 'Content-Length': '0'} 2024-04-13T22:15:34.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033211a5e3810b28f-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-13T22:15:34.646 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.646 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.646 DEBUG:boto:Method: GET 2024-04-13T22:15:34.646 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.646 DEBUG:boto:Data: 2024-04-13T22:15:34.646 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.647 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.647 DEBUG:boto:Params: {} 2024-04-13T22:15:34.647 DEBUG:boto:Token: None 2024-04-13T22:15:34.647 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tQCamgu2xEHlUlOrYmAubHlyrs4= 2024-04-13T22:15:34.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tQCamgu2xEHlUlOrYmAubHlyrs4=', 'Content-Length': '0'} 2024-04-13T22:15:34.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026fca0f98b66dfe8-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.649 DEBUG:boto:path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.649 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.649 DEBUG:boto:Method: GET 2024-04-13T22:15:34.650 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.650 DEBUG:boto:Data: 2024-04-13T22:15:34.650 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.650 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.650 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.650 DEBUG:boto:Params: {} 2024-04-13T22:15:34.650 DEBUG:boto:Token: None 2024-04-13T22:15:34.650 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000028 2024-04-13T22:15:34.650 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tQCamgu2xEHlUlOrYmAubHlyrs4= 2024-04-13T22:15:34.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tQCamgu2xEHlUlOrYmAubHlyrs4=', 'Content-Length': '0'} 2024-04-13T22:15:34.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca0b00a988884d3a-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.652 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.652 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.653 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.653 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.653 DEBUG:boto:Method: GET 2024-04-13T22:15:34.653 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.653 DEBUG:boto:Data: 2024-04-13T22:15:34.653 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.653 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.653 DEBUG:boto:Params: {} 2024-04-13T22:15:34.653 DEBUG:boto:Token: None 2024-04-13T22:15:34.654 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.654 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3568e6aa1e97605-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.657 DEBUG:boto:b'vwnfns-60obj000000000000000000281000falseobj000000000000000000282024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.657 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.658 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.658 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.658 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.658 DEBUG:boto:Method: GET 2024-04-13T22:15:34.658 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000028 2024-04-13T22:15:34.658 DEBUG:boto:Data: 2024-04-13T22:15:34.658 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.658 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.658 DEBUG:boto:Params: {} 2024-04-13T22:15:34.659 DEBUG:boto:Token: None 2024-04-13T22:15:34.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053a89f37f8390a98-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.663 DEBUG:boto:b'vwnfns-60obj000000000000000000281000falseobj000000000000000000282024-04-13T22:15:17.447Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-13T22:15:34.663 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.663 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.664 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.664 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.664 DEBUG:boto:Method: GET 2024-04-13T22:15:34.664 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.664 DEBUG:boto:Data: 2024-04-13T22:15:34.664 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.664 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.664 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.665 DEBUG:boto:Params: {} 2024-04-13T22:15:34.665 DEBUG:boto:Token: None 2024-04-13T22:15:34.665 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.665 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y= 2024-04-13T22:15:34.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fde476dd50084d-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.667 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.668 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.668 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.668 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.668 DEBUG:boto:Method: GET 2024-04-13T22:15:34.668 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.668 DEBUG:boto:Data: 2024-04-13T22:15:34.668 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.668 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.668 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.668 DEBUG:boto:Params: {} 2024-04-13T22:15:34.669 DEBUG:boto:Token: None 2024-04-13T22:15:34.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y= 2024-04-13T22:15:34.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006837d4b2b1fc863d-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.671 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.671 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.671 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.671 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.671 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.671 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.671 DEBUG:boto:Data: 2024-04-13T22:15:34.671 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.672 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.672 DEBUG:boto:Params: {} 2024-04-13T22:15:34.672 DEBUG:boto:Token: None 2024-04-13T22:15:34.672 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kJ9NfkPTWS77K3g9e+CKV61j+Zg= 2024-04-13T22:15:34.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kJ9NfkPTWS77K3g9e+CKV61j+Zg=', 'Content-Length': '0'} 2024-04-13T22:15:34.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df2fde5c02ad953-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.674 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.674 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.674 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.675 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.675 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.675 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.675 DEBUG:boto:Data: 2024-04-13T22:15:34.675 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.675 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.675 DEBUG:boto:Params: {} 2024-04-13T22:15:34.675 DEBUG:boto:Token: None 2024-04-13T22:15:34.676 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kJ9NfkPTWS77K3g9e+CKV61j+Zg= 2024-04-13T22:15:34.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kJ9NfkPTWS77K3g9e+CKV61j+Zg=', 'Content-Length': '0'} 2024-04-13T22:15:34.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053755f68afc169e8-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-13T22:15:34.678 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.678 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.678 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.678 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.678 DEBUG:boto:Method: GET 2024-04-13T22:15:34.679 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.679 DEBUG:boto:Data: 2024-04-13T22:15:34.679 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.679 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.679 DEBUG:boto:Params: {} 2024-04-13T22:15:34.679 DEBUG:boto:Token: None 2024-04-13T22:15:34.679 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.679 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y= 2024-04-13T22:15:34.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000168a0acb276d9a18-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.722 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.723 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.723 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.723 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.723 DEBUG:boto:Method: GET 2024-04-13T22:15:34.723 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.723 DEBUG:boto:Data: 2024-04-13T22:15:34.723 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.723 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.724 DEBUG:boto:Params: {} 2024-04-13T22:15:34.724 DEBUG:boto:Token: None 2024-04-13T22:15:34.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029?versionId=null 2024-04-13T22:15:34.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y= 2024-04-13T22:15:34.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WxK0w45TPwbvsSn3B4gPCPU0u4Y=', 'Content-Length': '0'} 2024-04-13T22:15:34.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a3455b1dafc3e93-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.726 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.726 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.726 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.727 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.727 DEBUG:boto:Method: GET 2024-04-13T22:15:34.727 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.727 DEBUG:boto:Data: 2024-04-13T22:15:34.727 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.727 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.727 DEBUG:boto:Params: {} 2024-04-13T22:15:34.727 DEBUG:boto:Token: None 2024-04-13T22:15:34.727 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa3ec96c7767a69-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.730 DEBUG:boto:b'vwnfns-60obj000000000000000000291000falseobj000000000000000000292024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.730 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.730 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.730 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.731 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.731 DEBUG:boto:Method: GET 2024-04-13T22:15:34.731 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.731 DEBUG:boto:Data: 2024-04-13T22:15:34.731 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.731 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.731 DEBUG:boto:Params: {} 2024-04-13T22:15:34.731 DEBUG:boto:Token: None 2024-04-13T22:15:34.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006919620a5e836a95-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.735 DEBUG:boto:b'vwnfns-60obj000000000000000000291000falseobj000000000000000000292024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.735 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.736 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.736 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.736 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.736 DEBUG:boto:Data: 2024-04-13T22:15:34.736 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.736 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.736 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.736 DEBUG:boto:Params: {} 2024-04-13T22:15:34.736 DEBUG:boto:Token: None 2024-04-13T22:15:34.737 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W+J52MeyX/jQWy82qZPkD2asc2c= 2024-04-13T22:15:34.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W+J52MeyX/jQWy82qZPkD2asc2c=', 'Content-Length': '0'} 2024-04-13T22:15:34.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25714289c8057eb-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.739 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.739 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.739 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.739 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.739 DEBUG:boto:Data: 2024-04-13T22:15:34.739 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.739 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.739 DEBUG:boto:Params: {} 2024-04-13T22:15:34.739 DEBUG:boto:Token: None 2024-04-13T22:15:34.740 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W+J52MeyX/jQWy82qZPkD2asc2c= 2024-04-13T22:15:34.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W+J52MeyX/jQWy82qZPkD2asc2c=', 'Content-Length': '0'} 2024-04-13T22:15:34.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc42d7701f93a651-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-13T22:15:34.741 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.741 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.742 DEBUG:boto:Method: GET 2024-04-13T22:15:34.742 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.742 DEBUG:boto:Data: 2024-04-13T22:15:34.742 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.742 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.742 DEBUG:boto:Params: {} 2024-04-13T22:15:34.742 DEBUG:boto:Token: None 2024-04-13T22:15:34.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W/YCsWy4Vj5LP3dmGM8mmHm+XHI= 2024-04-13T22:15:34.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W/YCsWy4Vj5LP3dmGM8mmHm+XHI=', 'Content-Length': '0'} 2024-04-13T22:15:34.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094adb834df7cc25e-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.744 DEBUG:boto:path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.744 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.744 DEBUG:boto:Method: GET 2024-04-13T22:15:34.745 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.745 DEBUG:boto:Data: 2024-04-13T22:15:34.745 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.745 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.745 DEBUG:boto:Params: {} 2024-04-13T22:15:34.745 DEBUG:boto:Token: None 2024-04-13T22:15:34.745 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000029 2024-04-13T22:15:34.745 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W/YCsWy4Vj5LP3dmGM8mmHm+XHI= 2024-04-13T22:15:34.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W/YCsWy4Vj5LP3dmGM8mmHm+XHI=', 'Content-Length': '0'} 2024-04-13T22:15:34.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c8bf965611773f5-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.747 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.747 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.747 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.747 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.748 DEBUG:boto:Method: GET 2024-04-13T22:15:34.748 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.748 DEBUG:boto:Data: 2024-04-13T22:15:34.748 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.748 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.748 DEBUG:boto:Params: {} 2024-04-13T22:15:34.748 DEBUG:boto:Token: None 2024-04-13T22:15:34.748 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8affaf6d066cc2-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.751 DEBUG:boto:b'vwnfns-60obj000000000000000000291000falseobj000000000000000000292024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.751 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.751 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.751 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.751 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.752 DEBUG:boto:Method: GET 2024-04-13T22:15:34.752 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000029 2024-04-13T22:15:34.752 DEBUG:boto:Data: 2024-04-13T22:15:34.752 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.752 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.752 DEBUG:boto:Params: {} 2024-04-13T22:15:34.752 DEBUG:boto:Token: None 2024-04-13T22:15:34.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ecd642801caa01-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.756 DEBUG:boto:b'vwnfns-60obj000000000000000000291000falseobj000000000000000000292024-04-13T22:15:21.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-13T22:15:34.756 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.756 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.756 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.757 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.757 DEBUG:boto:Method: GET 2024-04-13T22:15:34.757 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.757 DEBUG:boto:Data: 2024-04-13T22:15:34.757 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.757 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.757 DEBUG:boto:Params: {} 2024-04-13T22:15:34.757 DEBUG:boto:Token: None 2024-04-13T22:15:34.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk= 2024-04-13T22:15:34.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk=', 'Content-Length': '0'} 2024-04-13T22:15:34.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000175466166ea35f95-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.806 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.807 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.807 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.807 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.807 DEBUG:boto:Method: GET 2024-04-13T22:15:34.807 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.807 DEBUG:boto:Data: 2024-04-13T22:15:34.807 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.807 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.807 DEBUG:boto:Params: {} 2024-04-13T22:15:34.808 DEBUG:boto:Token: None 2024-04-13T22:15:34.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk= 2024-04-13T22:15:34.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk=', 'Content-Length': '0'} 2024-04-13T22:15:34.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5401fab14c165b7-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.811 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.811 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.811 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.811 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.811 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.811 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.811 DEBUG:boto:Data: 2024-04-13T22:15:34.811 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.812 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.812 DEBUG:boto:Params: {} 2024-04-13T22:15:34.812 DEBUG:boto:Token: None 2024-04-13T22:15:34.812 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H90ee5AtZiak+VrKaId4krvv5os= 2024-04-13T22:15:34.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H90ee5AtZiak+VrKaId4krvv5os=', 'Content-Length': '0'} 2024-04-13T22:15:34.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043024ff08d947859-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.814 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.814 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.814 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.815 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.815 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.815 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.815 DEBUG:boto:Data: 2024-04-13T22:15:34.815 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.815 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.815 DEBUG:boto:Params: {} 2024-04-13T22:15:34.815 DEBUG:boto:Token: None 2024-04-13T22:15:34.816 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H90ee5AtZiak+VrKaId4krvv5os= 2024-04-13T22:15:34.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H90ee5AtZiak+VrKaId4krvv5os=', 'Content-Length': '0'} 2024-04-13T22:15:34.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aae2871b50a11e9-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-13T22:15:34.818 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.818 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.818 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.818 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.818 DEBUG:boto:Method: GET 2024-04-13T22:15:34.819 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.819 DEBUG:boto:Data: 2024-04-13T22:15:34.819 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.819 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.819 DEBUG:boto:Params: {} 2024-04-13T22:15:34.819 DEBUG:boto:Token: None 2024-04-13T22:15:34.820 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk= 2024-04-13T22:15:34.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk=', 'Content-Length': '0'} 2024-04-13T22:15:34.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc37ee122701692a-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.822 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.822 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.822 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.822 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.822 DEBUG:boto:Method: GET 2024-04-13T22:15:34.822 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.822 DEBUG:boto:Data: 2024-04-13T22:15:34.822 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.823 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.823 DEBUG:boto:Params: {} 2024-04-13T22:15:34.823 DEBUG:boto:Token: None 2024-04-13T22:15:34.823 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030?versionId=null 2024-04-13T22:15:34.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk= 2024-04-13T22:15:34.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+r/3FrA3QKhnB9Jf+PYx5CE0KWk=', 'Content-Length': '0'} 2024-04-13T22:15:34.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000204a01dd3459454c-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.866 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.867 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.867 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.867 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.867 DEBUG:boto:Method: GET 2024-04-13T22:15:34.867 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.867 DEBUG:boto:Data: 2024-04-13T22:15:34.867 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.867 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.868 DEBUG:boto:Params: {} 2024-04-13T22:15:34.868 DEBUG:boto:Token: None 2024-04-13T22:15:34.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9fdfde93c5db48f-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.870 DEBUG:boto:b'vwnfns-60obj000000000000000000301000falseobj000000000000000000302024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.871 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.871 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.871 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.871 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.871 DEBUG:boto:Method: GET 2024-04-13T22:15:34.871 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.871 DEBUG:boto:Data: 2024-04-13T22:15:34.872 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.872 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.872 DEBUG:boto:Params: {} 2024-04-13T22:15:34.872 DEBUG:boto:Token: None 2024-04-13T22:15:34.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f8e203bf9a1cd7-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.876 DEBUG:boto:b'vwnfns-60obj000000000000000000301000falseobj000000000000000000302024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.876 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.876 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.876 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.876 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.876 DEBUG:boto:Data: 2024-04-13T22:15:34.877 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.877 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.877 DEBUG:boto:Params: {} 2024-04-13T22:15:34.877 DEBUG:boto:Token: None 2024-04-13T22:15:34.877 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DGIFlU4sK2NekIhWU51yWOkz+mI= 2024-04-13T22:15:34.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DGIFlU4sK2NekIhWU51yWOkz+mI=', 'Content-Length': '0'} 2024-04-13T22:15:34.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad36f911b35ab26b-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.879 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.879 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.879 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.879 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.879 DEBUG:boto:Data: 2024-04-13T22:15:34.879 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.880 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.880 DEBUG:boto:Params: {} 2024-04-13T22:15:34.880 DEBUG:boto:Token: None 2024-04-13T22:15:34.880 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DGIFlU4sK2NekIhWU51yWOkz+mI= 2024-04-13T22:15:34.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DGIFlU4sK2NekIhWU51yWOkz+mI=', 'Content-Length': '0'} 2024-04-13T22:15:34.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc890a1edfce3ca9-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-13T22:15:34.882 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.882 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.882 DEBUG:boto:Method: GET 2024-04-13T22:15:34.882 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.882 DEBUG:boto:Data: 2024-04-13T22:15:34.882 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.882 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.883 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.883 DEBUG:boto:Params: {} 2024-04-13T22:15:34.883 DEBUG:boto:Token: None 2024-04-13T22:15:34.883 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ycdHdVGz5hy4KmBK9wWaK1QGYMg= 2024-04-13T22:15:34.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ycdHdVGz5hy4KmBK9wWaK1QGYMg=', 'Content-Length': '0'} 2024-04-13T22:15:34.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83851acb2163f39-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.926 DEBUG:boto:path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.926 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.927 DEBUG:boto:Method: GET 2024-04-13T22:15:34.927 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.927 DEBUG:boto:Data: 2024-04-13T22:15:34.927 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.927 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.927 DEBUG:boto:Params: {} 2024-04-13T22:15:34.927 DEBUG:boto:Token: None 2024-04-13T22:15:34.927 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000030 2024-04-13T22:15:34.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ycdHdVGz5hy4KmBK9wWaK1QGYMg= 2024-04-13T22:15:34.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ycdHdVGz5hy4KmBK9wWaK1QGYMg=', 'Content-Length': '0'} 2024-04-13T22:15:34.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24233e927b6a482-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.929 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.929 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.930 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.930 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.930 DEBUG:boto:Method: GET 2024-04-13T22:15:34.930 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.930 DEBUG:boto:Data: 2024-04-13T22:15:34.930 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.930 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.930 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.930 DEBUG:boto:Params: {} 2024-04-13T22:15:34.930 DEBUG:boto:Token: None 2024-04-13T22:15:34.931 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.931 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000130ef5b5aa0f3c29-00661b0406-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.933 DEBUG:boto:b'vwnfns-60obj000000000000000000301000falseobj000000000000000000302024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.933 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:34.933 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:34.934 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.934 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.934 DEBUG:boto:Method: GET 2024-04-13T22:15:34.934 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000030 2024-04-13T22:15:34.934 DEBUG:boto:Data: 2024-04-13T22:15:34.934 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.934 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.934 DEBUG:boto:Params: {} 2024-04-13T22:15:34.934 DEBUG:boto:Token: None 2024-04-13T22:15:34.935 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/ 2024-04-13T22:15:34.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38= 2024-04-13T22:15:34.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C645f2o5bTXqKE3Z9X0ZuKJMz38=', 'Content-Length': '0'} 2024-04-13T22:15:34.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f0333b95a804f7-00661b0406-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.938 DEBUG:boto:b'vwnfns-60obj000000000000000000301000falseobj000000000000000000302024-04-13T22:15:22.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:34.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-13T22:15:34.939 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.939 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.939 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.939 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.939 DEBUG:boto:Method: GET 2024-04-13T22:15:34.939 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.939 DEBUG:boto:Data: 2024-04-13T22:15:34.940 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.940 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.940 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.940 DEBUG:boto:Params: {} 2024-04-13T22:15:34.940 DEBUG:boto:Token: None 2024-04-13T22:15:34.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY= 2024-04-13T22:15:34.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY=', 'Content-Length': '0'} 2024-04-13T22:15:34.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000237a3c9cce5a426c-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.942 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.943 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.943 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.943 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.943 DEBUG:boto:Method: GET 2024-04-13T22:15:34.943 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.943 DEBUG:boto:Data: 2024-04-13T22:15:34.943 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.943 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.943 DEBUG:boto:Params: {} 2024-04-13T22:15:34.944 DEBUG:boto:Token: None 2024-04-13T22:15:34.944 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.944 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY= 2024-04-13T22:15:34.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY=', 'Content-Length': '0'} 2024-04-13T22:15:34.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c823995415a831-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.986 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.987 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.987 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.987 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.987 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.987 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.987 DEBUG:boto:Data: 2024-04-13T22:15:34.987 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.987 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.987 DEBUG:boto:Params: {} 2024-04-13T22:15:34.988 DEBUG:boto:Token: None 2024-04-13T22:15:34.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Ow655xNtPB18INPhKJnuRYGTQE= 2024-04-13T22:15:34.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Ow655xNtPB18INPhKJnuRYGTQE=', 'Content-Length': '0'} 2024-04-13T22:15:34.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6e368afbc85ad9-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.990 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.990 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.990 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.990 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.990 DEBUG:boto:Method: HEAD 2024-04-13T22:15:34.990 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.991 DEBUG:boto:Data: 2024-04-13T22:15:34.991 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.991 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.991 DEBUG:boto:Params: {} 2024-04-13T22:15:34.991 DEBUG:boto:Token: None 2024-04-13T22:15:34.991 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Ow655xNtPB18INPhKJnuRYGTQE= 2024-04-13T22:15:34.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Ow655xNtPB18INPhKJnuRYGTQE=', 'Content-Length': '0'} 2024-04-13T22:15:34.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1f65af081eae4d-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-13T22:15:34.994 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.994 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.994 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.994 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.994 DEBUG:boto:Method: GET 2024-04-13T22:15:34.994 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.994 DEBUG:boto:Data: 2024-04-13T22:15:34.994 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:34.994 DEBUG:boto:Port: 8001 2024-04-13T22:15:34.995 DEBUG:boto:Params: {} 2024-04-13T22:15:34.995 DEBUG:boto:Token: None 2024-04-13T22:15:34.995 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY= 2024-04-13T22:15:34.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY=', 'Content-Length': '0'} 2024-04-13T22:15:34.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbfe2ea26959707e-00661b0406-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:34.997 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.997 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:34.997 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.997 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.997 DEBUG:boto:Method: GET 2024-04-13T22:15:34.997 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.998 DEBUG:boto:Data: 2024-04-13T22:15:34.998 DEBUG:boto:Headers: {} 2024-04-13T22:15:34.998 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:34.998 DEBUG:boto:Port: 8000 2024-04-13T22:15:34.998 DEBUG:boto:Params: {} 2024-04-13T22:15:34.998 DEBUG:boto:Token: None 2024-04-13T22:15:34.998 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:34 GMT /vwnfns-60/obj00000000000000000031?versionId=null 2024-04-13T22:15:34.998 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY= 2024-04-13T22:15:34.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ar+45xfd+1DLv0RWAKpYszDltTY=', 'Content-Length': '0'} 2024-04-13T22:15:35.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000760e50acf36d9a58-00661b0406-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.000 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.000 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.000 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.000 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.001 DEBUG:boto:Method: GET 2024-04-13T22:15:35.001 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.001 DEBUG:boto:Data: 2024-04-13T22:15:35.001 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.001 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.001 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.001 DEBUG:boto:Params: {} 2024-04-13T22:15:35.001 DEBUG:boto:Token: None 2024-04-13T22:15:35.001 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085fe09bbba6c17ae-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.004 DEBUG:boto:b'vwnfns-60obj000000000000000000311000falseobj000000000000000000312024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.004 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.004 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.005 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.005 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.005 DEBUG:boto:Method: GET 2024-04-13T22:15:35.005 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.005 DEBUG:boto:Data: 2024-04-13T22:15:35.005 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.005 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.005 DEBUG:boto:Params: {} 2024-04-13T22:15:35.005 DEBUG:boto:Token: None 2024-04-13T22:15:35.006 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce64f10f5094fad-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.009 DEBUG:boto:b'vwnfns-60obj000000000000000000311000falseobj000000000000000000312024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.010 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.010 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.010 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.010 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.010 DEBUG:boto:Data: 2024-04-13T22:15:35.010 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.011 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.011 DEBUG:boto:Params: {} 2024-04-13T22:15:35.011 DEBUG:boto:Token: None 2024-04-13T22:15:35.011 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9jFGyxVfeekXtnqSg/RpSw9iDrQ= 2024-04-13T22:15:35.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9jFGyxVfeekXtnqSg/RpSw9iDrQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e05b7570a78c3c49-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.013 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.013 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.013 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.013 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.013 DEBUG:boto:Data: 2024-04-13T22:15:35.013 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.014 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.014 DEBUG:boto:Params: {} 2024-04-13T22:15:35.014 DEBUG:boto:Token: None 2024-04-13T22:15:35.014 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.014 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9jFGyxVfeekXtnqSg/RpSw9iDrQ= 2024-04-13T22:15:35.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9jFGyxVfeekXtnqSg/RpSw9iDrQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ee73c5642cda05-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-13T22:15:35.016 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.016 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.016 DEBUG:boto:Method: GET 2024-04-13T22:15:35.016 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.016 DEBUG:boto:Data: 2024-04-13T22:15:35.017 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.017 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.017 DEBUG:boto:Params: {} 2024-04-13T22:15:35.017 DEBUG:boto:Token: None 2024-04-13T22:15:35.017 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ngmBwg83THahZrv91VoMOWcfhVY= 2024-04-13T22:15:35.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ngmBwg83THahZrv91VoMOWcfhVY=', 'Content-Length': '0'} 2024-04-13T22:15:35.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ec305a0107d146-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.019 DEBUG:boto:path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.019 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.019 DEBUG:boto:Method: GET 2024-04-13T22:15:35.019 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.020 DEBUG:boto:Data: 2024-04-13T22:15:35.020 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.020 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.020 DEBUG:boto:Params: {} 2024-04-13T22:15:35.020 DEBUG:boto:Token: None 2024-04-13T22:15:35.020 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000031 2024-04-13T22:15:35.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ngmBwg83THahZrv91VoMOWcfhVY= 2024-04-13T22:15:35.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ngmBwg83THahZrv91VoMOWcfhVY=', 'Content-Length': '0'} 2024-04-13T22:15:35.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057221f5825e8b90e-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.022 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.022 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.023 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.023 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.023 DEBUG:boto:Method: GET 2024-04-13T22:15:35.023 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.023 DEBUG:boto:Data: 2024-04-13T22:15:35.023 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.023 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.023 DEBUG:boto:Params: {} 2024-04-13T22:15:35.024 DEBUG:boto:Token: None 2024-04-13T22:15:35.024 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e334f669c76af22f-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.066 DEBUG:boto:b'vwnfns-60obj000000000000000000311000falseobj000000000000000000312024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.067 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.067 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.067 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.067 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.067 DEBUG:boto:Method: GET 2024-04-13T22:15:35.068 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000031 2024-04-13T22:15:35.068 DEBUG:boto:Data: 2024-04-13T22:15:35.068 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.068 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.068 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.068 DEBUG:boto:Params: {} 2024-04-13T22:15:35.068 DEBUG:boto:Token: None 2024-04-13T22:15:35.069 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.069 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ed8e3e4878e4e2-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.073 DEBUG:boto:b'vwnfns-60obj000000000000000000311000falseobj000000000000000000312024-04-13T22:15:19.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-13T22:15:35.074 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.074 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.074 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.074 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.074 DEBUG:boto:Method: GET 2024-04-13T22:15:35.074 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.074 DEBUG:boto:Data: 2024-04-13T22:15:35.075 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.075 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.075 DEBUG:boto:Params: {} 2024-04-13T22:15:35.075 DEBUG:boto:Token: None 2024-04-13T22:15:35.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ= 2024-04-13T22:15:35.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e62e885259550f9-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.078 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.078 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.078 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.079 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.079 DEBUG:boto:Method: GET 2024-04-13T22:15:35.079 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.079 DEBUG:boto:Data: 2024-04-13T22:15:35.079 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.079 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.079 DEBUG:boto:Params: {} 2024-04-13T22:15:35.079 DEBUG:boto:Token: None 2024-04-13T22:15:35.079 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ= 2024-04-13T22:15:35.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a80189c35b2064-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.082 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.082 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.082 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.082 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.082 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.082 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.082 DEBUG:boto:Data: 2024-04-13T22:15:35.082 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.083 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.083 DEBUG:boto:Params: {} 2024-04-13T22:15:35.083 DEBUG:boto:Token: None 2024-04-13T22:15:35.083 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VEdireb6MqwBfS1mUWRoHivLDNM= 2024-04-13T22:15:35.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VEdireb6MqwBfS1mUWRoHivLDNM=', 'Content-Length': '0'} 2024-04-13T22:15:35.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d72e01fcbe7964b-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.085 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.085 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.085 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.085 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.086 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.086 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.086 DEBUG:boto:Data: 2024-04-13T22:15:35.086 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.086 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.086 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.086 DEBUG:boto:Params: {} 2024-04-13T22:15:35.086 DEBUG:boto:Token: None 2024-04-13T22:15:35.087 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.087 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VEdireb6MqwBfS1mUWRoHivLDNM= 2024-04-13T22:15:35.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VEdireb6MqwBfS1mUWRoHivLDNM=', 'Content-Length': '0'} 2024-04-13T22:15:35.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091da189643a28027-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-13T22:15:35.089 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.089 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.089 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.089 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.089 DEBUG:boto:Method: GET 2024-04-13T22:15:35.089 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.089 DEBUG:boto:Data: 2024-04-13T22:15:35.089 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.090 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.090 DEBUG:boto:Params: {} 2024-04-13T22:15:35.090 DEBUG:boto:Token: None 2024-04-13T22:15:35.090 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.090 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ= 2024-04-13T22:15:35.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028a409e2b07a6ff4-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.092 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.092 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.092 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.092 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.092 DEBUG:boto:Method: GET 2024-04-13T22:15:35.093 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.093 DEBUG:boto:Data: 2024-04-13T22:15:35.093 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.093 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.093 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.093 DEBUG:boto:Params: {} 2024-04-13T22:15:35.093 DEBUG:boto:Token: None 2024-04-13T22:15:35.093 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032?versionId=null 2024-04-13T22:15:35.094 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ= 2024-04-13T22:15:35.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9gaq38oFVkesGCYNCiqfw8DY1sQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2ff32e034d10fb9-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.096 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.096 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.096 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.096 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.096 DEBUG:boto:Method: GET 2024-04-13T22:15:35.097 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.097 DEBUG:boto:Data: 2024-04-13T22:15:35.097 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.097 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.097 DEBUG:boto:Params: {} 2024-04-13T22:15:35.097 DEBUG:boto:Token: None 2024-04-13T22:15:35.097 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.097 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019f83efc89fdc6c6-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.100 DEBUG:boto:b'vwnfns-60obj000000000000000000321000falseobj000000000000000000322024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.100 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.100 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.100 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.100 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.101 DEBUG:boto:Method: GET 2024-04-13T22:15:35.101 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.101 DEBUG:boto:Data: 2024-04-13T22:15:35.101 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.101 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.101 DEBUG:boto:Params: {} 2024-04-13T22:15:35.101 DEBUG:boto:Token: None 2024-04-13T22:15:35.101 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7fb4f05482b729-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.105 DEBUG:boto:b'vwnfns-60obj000000000000000000321000falseobj000000000000000000322024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.106 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.106 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.106 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.106 DEBUG:boto:Data: 2024-04-13T22:15:35.106 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.107 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.107 DEBUG:boto:Params: {} 2024-04-13T22:15:35.107 DEBUG:boto:Token: None 2024-04-13T22:15:35.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5nC4bIdjuRAJ1y9xC79soRZ4ScY= 2024-04-13T22:15:35.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5nC4bIdjuRAJ1y9xC79soRZ4ScY=', 'Content-Length': '0'} 2024-04-13T22:15:35.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000386b56323c237f44-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.109 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.109 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.109 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.109 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.109 DEBUG:boto:Data: 2024-04-13T22:15:35.110 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.110 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.110 DEBUG:boto:Params: {} 2024-04-13T22:15:35.110 DEBUG:boto:Token: None 2024-04-13T22:15:35.110 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5nC4bIdjuRAJ1y9xC79soRZ4ScY= 2024-04-13T22:15:35.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5nC4bIdjuRAJ1y9xC79soRZ4ScY=', 'Content-Length': '0'} 2024-04-13T22:15:35.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000badbcdfcfdf4cefd-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-13T22:15:35.112 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.113 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.113 DEBUG:boto:Method: GET 2024-04-13T22:15:35.113 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.113 DEBUG:boto:Data: 2024-04-13T22:15:35.113 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.113 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.113 DEBUG:boto:Params: {} 2024-04-13T22:15:35.113 DEBUG:boto:Token: None 2024-04-13T22:15:35.114 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cX8d2hBoyNbjPP5GFHxkCOEZNt0= 2024-04-13T22:15:35.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cX8d2hBoyNbjPP5GFHxkCOEZNt0=', 'Content-Length': '0'} 2024-04-13T22:15:35.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f6cca93a14fe5f-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.116 DEBUG:boto:path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.116 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.116 DEBUG:boto:Method: GET 2024-04-13T22:15:35.116 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.116 DEBUG:boto:Data: 2024-04-13T22:15:35.116 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.116 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.117 DEBUG:boto:Params: {} 2024-04-13T22:15:35.117 DEBUG:boto:Token: None 2024-04-13T22:15:35.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000032 2024-04-13T22:15:35.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cX8d2hBoyNbjPP5GFHxkCOEZNt0= 2024-04-13T22:15:35.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cX8d2hBoyNbjPP5GFHxkCOEZNt0=', 'Content-Length': '0'} 2024-04-13T22:15:35.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855a3ed2ba3e34ac-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.119 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.119 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.119 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.119 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.119 DEBUG:boto:Method: GET 2024-04-13T22:15:35.120 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.120 DEBUG:boto:Data: 2024-04-13T22:15:35.120 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.120 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.120 DEBUG:boto:Params: {} 2024-04-13T22:15:35.120 DEBUG:boto:Token: None 2024-04-13T22:15:35.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed1c5b5e0d10286-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.123 DEBUG:boto:b'vwnfns-60obj000000000000000000321000falseobj000000000000000000322024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.123 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.123 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.123 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.124 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.124 DEBUG:boto:Method: GET 2024-04-13T22:15:35.124 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000032 2024-04-13T22:15:35.124 DEBUG:boto:Data: 2024-04-13T22:15:35.124 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.124 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.124 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.124 DEBUG:boto:Params: {} 2024-04-13T22:15:35.124 DEBUG:boto:Token: None 2024-04-13T22:15:35.125 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.125 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aeabb3b85aa4104-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.170 DEBUG:boto:b'vwnfns-60obj000000000000000000321000falseobj000000000000000000322024-04-13T22:15:19.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-13T22:15:35.171 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.171 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.171 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.171 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.171 DEBUG:boto:Method: GET 2024-04-13T22:15:35.172 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.172 DEBUG:boto:Data: 2024-04-13T22:15:35.172 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.172 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.172 DEBUG:boto:Params: {} 2024-04-13T22:15:35.173 DEBUG:boto:Token: None 2024-04-13T22:15:35.173 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk= 2024-04-13T22:15:35.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk=', 'Content-Length': '0'} 2024-04-13T22:15:35.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e442ec553a4b8bd7-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.175 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.175 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.176 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.176 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.176 DEBUG:boto:Method: GET 2024-04-13T22:15:35.176 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.176 DEBUG:boto:Data: 2024-04-13T22:15:35.176 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.176 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.176 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.176 DEBUG:boto:Params: {} 2024-04-13T22:15:35.176 DEBUG:boto:Token: None 2024-04-13T22:15:35.177 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.177 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk= 2024-04-13T22:15:35.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk=', 'Content-Length': '0'} 2024-04-13T22:15:35.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6277dd83cd2a00e-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.178 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.179 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.179 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.179 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.179 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.179 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.179 DEBUG:boto:Data: 2024-04-13T22:15:35.179 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.179 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.179 DEBUG:boto:Params: {} 2024-04-13T22:15:35.179 DEBUG:boto:Token: None 2024-04-13T22:15:35.180 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.180 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:08dGHRPDsQxAvgqaYhjHCWVkQ9w= 2024-04-13T22:15:35.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:08dGHRPDsQxAvgqaYhjHCWVkQ9w=', 'Content-Length': '0'} 2024-04-13T22:15:35.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c22d0b152ff5cb-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.181 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.182 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.182 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.182 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.182 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.182 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.182 DEBUG:boto:Data: 2024-04-13T22:15:35.182 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.182 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.182 DEBUG:boto:Params: {} 2024-04-13T22:15:35.182 DEBUG:boto:Token: None 2024-04-13T22:15:35.183 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:08dGHRPDsQxAvgqaYhjHCWVkQ9w= 2024-04-13T22:15:35.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:08dGHRPDsQxAvgqaYhjHCWVkQ9w=', 'Content-Length': '0'} 2024-04-13T22:15:35.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001706df7dac5a6db1-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-13T22:15:35.185 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.185 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.185 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.185 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.185 DEBUG:boto:Method: GET 2024-04-13T22:15:35.185 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.185 DEBUG:boto:Data: 2024-04-13T22:15:35.185 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.185 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.185 DEBUG:boto:Params: {} 2024-04-13T22:15:35.186 DEBUG:boto:Token: None 2024-04-13T22:15:35.186 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk= 2024-04-13T22:15:35.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk=', 'Content-Length': '0'} 2024-04-13T22:15:35.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca82309ca88582f2-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.230 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.231 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.231 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.231 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.231 DEBUG:boto:Method: GET 2024-04-13T22:15:35.231 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.231 DEBUG:boto:Data: 2024-04-13T22:15:35.231 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.231 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.231 DEBUG:boto:Params: {} 2024-04-13T22:15:35.232 DEBUG:boto:Token: None 2024-04-13T22:15:35.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033?versionId=null 2024-04-13T22:15:35.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk= 2024-04-13T22:15:35.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9UColJ1blC8mn+CpzVGKMfQQNVk=', 'Content-Length': '0'} 2024-04-13T22:15:35.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b571e63fbd4f0d-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.234 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.234 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.234 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.234 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.234 DEBUG:boto:Method: GET 2024-04-13T22:15:35.234 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.235 DEBUG:boto:Data: 2024-04-13T22:15:35.235 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.235 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.235 DEBUG:boto:Params: {} 2024-04-13T22:15:35.235 DEBUG:boto:Token: None 2024-04-13T22:15:35.235 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b132e4d843d57e9-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.238 DEBUG:boto:b'vwnfns-60obj000000000000000000331000falseobj000000000000000000332024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.238 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.238 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.238 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.239 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.239 DEBUG:boto:Method: GET 2024-04-13T22:15:35.239 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.239 DEBUG:boto:Data: 2024-04-13T22:15:35.239 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.239 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.239 DEBUG:boto:Params: {} 2024-04-13T22:15:35.239 DEBUG:boto:Token: None 2024-04-13T22:15:35.239 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab189d8c0dbeeb28-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.243 DEBUG:boto:b'vwnfns-60obj000000000000000000331000falseobj000000000000000000332024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.244 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.244 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.244 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.244 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.244 DEBUG:boto:Data: 2024-04-13T22:15:35.244 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.244 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.244 DEBUG:boto:Params: {} 2024-04-13T22:15:35.244 DEBUG:boto:Token: None 2024-04-13T22:15:35.244 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.245 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DC40CADNGnIoTnfF/mPDiir25qE= 2024-04-13T22:15:35.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DC40CADNGnIoTnfF/mPDiir25qE=', 'Content-Length': '0'} 2024-04-13T22:15:35.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eefb2c8d9c8538c-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.246 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.246 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.247 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.247 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.247 DEBUG:boto:Data: 2024-04-13T22:15:35.247 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.247 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.247 DEBUG:boto:Params: {} 2024-04-13T22:15:35.247 DEBUG:boto:Token: None 2024-04-13T22:15:35.247 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DC40CADNGnIoTnfF/mPDiir25qE= 2024-04-13T22:15:35.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DC40CADNGnIoTnfF/mPDiir25qE=', 'Content-Length': '0'} 2024-04-13T22:15:35.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000570a31cce06fac22-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-13T22:15:35.249 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.249 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.250 DEBUG:boto:Method: GET 2024-04-13T22:15:35.250 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.250 DEBUG:boto:Data: 2024-04-13T22:15:35.250 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.250 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.250 DEBUG:boto:Params: {} 2024-04-13T22:15:35.250 DEBUG:boto:Token: None 2024-04-13T22:15:35.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CoCjqfxAiop2XMOlxseFhFAzSDw= 2024-04-13T22:15:35.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CoCjqfxAiop2XMOlxseFhFAzSDw=', 'Content-Length': '0'} 2024-04-13T22:15:35.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea71533042575e7c-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.252 DEBUG:boto:path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.253 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.253 DEBUG:boto:Method: GET 2024-04-13T22:15:35.253 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.253 DEBUG:boto:Data: 2024-04-13T22:15:35.253 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.253 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.253 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.253 DEBUG:boto:Params: {} 2024-04-13T22:15:35.253 DEBUG:boto:Token: None 2024-04-13T22:15:35.253 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000033 2024-04-13T22:15:35.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CoCjqfxAiop2XMOlxseFhFAzSDw= 2024-04-13T22:15:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CoCjqfxAiop2XMOlxseFhFAzSDw=', 'Content-Length': '0'} 2024-04-13T22:15:35.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3280e69c934fad-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.302 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.303 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.303 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.303 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.303 DEBUG:boto:Method: GET 2024-04-13T22:15:35.303 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.303 DEBUG:boto:Data: 2024-04-13T22:15:35.303 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.303 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.303 DEBUG:boto:Params: {} 2024-04-13T22:15:35.304 DEBUG:boto:Token: None 2024-04-13T22:15:35.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908bd56e56a59d40-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.306 DEBUG:boto:b'vwnfns-60obj000000000000000000331000falseobj000000000000000000332024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.307 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.307 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.307 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.307 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.307 DEBUG:boto:Method: GET 2024-04-13T22:15:35.307 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000033 2024-04-13T22:15:35.307 DEBUG:boto:Data: 2024-04-13T22:15:35.307 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.308 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.308 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.308 DEBUG:boto:Params: {} 2024-04-13T22:15:35.308 DEBUG:boto:Token: None 2024-04-13T22:15:35.308 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a8fdadb4a9c279-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.312 DEBUG:boto:b'vwnfns-60obj000000000000000000331000falseobj000000000000000000332024-04-13T22:15:22.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-13T22:15:35.312 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.312 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.313 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.313 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.313 DEBUG:boto:Method: GET 2024-04-13T22:15:35.313 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.313 DEBUG:boto:Data: 2024-04-13T22:15:35.313 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.313 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.313 DEBUG:boto:Params: {} 2024-04-13T22:15:35.313 DEBUG:boto:Token: None 2024-04-13T22:15:35.314 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg= 2024-04-13T22:15:35.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg=', 'Content-Length': '0'} 2024-04-13T22:15:35.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c5f4ec218e4bb2-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.316 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.316 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.316 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.316 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.316 DEBUG:boto:Method: GET 2024-04-13T22:15:35.316 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.317 DEBUG:boto:Data: 2024-04-13T22:15:35.317 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.317 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.317 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.317 DEBUG:boto:Params: {} 2024-04-13T22:15:35.317 DEBUG:boto:Token: None 2024-04-13T22:15:35.317 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.317 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg= 2024-04-13T22:15:35.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg=', 'Content-Length': '0'} 2024-04-13T22:15:35.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e90b94724a6ef3c-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.319 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.319 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.319 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.319 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.319 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.319 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.320 DEBUG:boto:Data: 2024-04-13T22:15:35.320 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.320 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.320 DEBUG:boto:Params: {} 2024-04-13T22:15:35.320 DEBUG:boto:Token: None 2024-04-13T22:15:35.320 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vU/LdYdrPxY6/1fSSa1LOvTCG9g= 2024-04-13T22:15:35.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vU/LdYdrPxY6/1fSSa1LOvTCG9g=', 'Content-Length': '0'} 2024-04-13T22:15:35.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efc54286401394cf-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.322 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.322 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.322 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.322 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.323 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.323 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.323 DEBUG:boto:Data: 2024-04-13T22:15:35.323 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.323 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.323 DEBUG:boto:Params: {} 2024-04-13T22:15:35.323 DEBUG:boto:Token: None 2024-04-13T22:15:35.323 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vU/LdYdrPxY6/1fSSa1LOvTCG9g= 2024-04-13T22:15:35.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vU/LdYdrPxY6/1fSSa1LOvTCG9g=', 'Content-Length': '0'} 2024-04-13T22:15:35.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffd5e9006a1ba25-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-13T22:15:35.325 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.325 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.325 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.325 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.325 DEBUG:boto:Method: GET 2024-04-13T22:15:35.326 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.326 DEBUG:boto:Data: 2024-04-13T22:15:35.326 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.326 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.326 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.326 DEBUG:boto:Params: {} 2024-04-13T22:15:35.326 DEBUG:boto:Token: None 2024-04-13T22:15:35.326 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg= 2024-04-13T22:15:35.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg=', 'Content-Length': '0'} 2024-04-13T22:15:35.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e38eca63d8eb0d0-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.328 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.328 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.328 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.328 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.328 DEBUG:boto:Method: GET 2024-04-13T22:15:35.328 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.329 DEBUG:boto:Data: 2024-04-13T22:15:35.329 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.329 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.329 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.329 DEBUG:boto:Params: {} 2024-04-13T22:15:35.329 DEBUG:boto:Token: None 2024-04-13T22:15:35.329 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036?versionId=null 2024-04-13T22:15:35.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg= 2024-04-13T22:15:35.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PqsMaMxFAUYBgi0lsnjOoD629pg=', 'Content-Length': '0'} 2024-04-13T22:15:35.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ffaa577014cbaa-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.374 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.375 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.375 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.375 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.375 DEBUG:boto:Method: GET 2024-04-13T22:15:35.375 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.375 DEBUG:boto:Data: 2024-04-13T22:15:35.375 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.375 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.375 DEBUG:boto:Params: {} 2024-04-13T22:15:35.376 DEBUG:boto:Token: None 2024-04-13T22:15:35.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b940766b899995-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.378 DEBUG:boto:b'vwnfns-60obj000000000000000000361000falseobj000000000000000000362024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.379 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.379 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.379 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.379 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.379 DEBUG:boto:Method: GET 2024-04-13T22:15:35.379 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.379 DEBUG:boto:Data: 2024-04-13T22:15:35.381 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.381 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.381 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.381 DEBUG:boto:Params: {} 2024-04-13T22:15:35.382 DEBUG:boto:Token: None 2024-04-13T22:15:35.382 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.382 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078720799aadbc44e-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.386 DEBUG:boto:b'vwnfns-60obj000000000000000000361000falseobj000000000000000000362024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.386 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.386 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.386 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.386 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.386 DEBUG:boto:Data: 2024-04-13T22:15:35.387 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.387 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.387 DEBUG:boto:Params: {} 2024-04-13T22:15:35.387 DEBUG:boto:Token: None 2024-04-13T22:15:35.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fY3Ty4+Wm8bYlwMG+dwaQK5xaZY= 2024-04-13T22:15:35.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fY3Ty4+Wm8bYlwMG+dwaQK5xaZY=', 'Content-Length': '0'} 2024-04-13T22:15:35.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099067fe38f54cac4-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.389 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.389 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.389 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.389 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.389 DEBUG:boto:Data: 2024-04-13T22:15:35.389 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.389 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.389 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.390 DEBUG:boto:Params: {} 2024-04-13T22:15:35.390 DEBUG:boto:Token: None 2024-04-13T22:15:35.390 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.390 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fY3Ty4+Wm8bYlwMG+dwaQK5xaZY= 2024-04-13T22:15:35.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fY3Ty4+Wm8bYlwMG+dwaQK5xaZY=', 'Content-Length': '0'} 2024-04-13T22:15:35.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051d673d9e547658e-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-13T22:15:35.392 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.392 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.392 DEBUG:boto:Method: GET 2024-04-13T22:15:35.392 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.392 DEBUG:boto:Data: 2024-04-13T22:15:35.392 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.392 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.393 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.393 DEBUG:boto:Params: {} 2024-04-13T22:15:35.393 DEBUG:boto:Token: None 2024-04-13T22:15:35.393 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.393 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AXaM2xleJr3PhOr/SSipndcqa0o= 2024-04-13T22:15:35.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AXaM2xleJr3PhOr/SSipndcqa0o=', 'Content-Length': '0'} 2024-04-13T22:15:35.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b044a2e214b00b-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.438 DEBUG:boto:path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.439 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.439 DEBUG:boto:Method: GET 2024-04-13T22:15:35.439 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.439 DEBUG:boto:Data: 2024-04-13T22:15:35.439 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.439 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.439 DEBUG:boto:Params: {} 2024-04-13T22:15:35.439 DEBUG:boto:Token: None 2024-04-13T22:15:35.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000036 2024-04-13T22:15:35.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AXaM2xleJr3PhOr/SSipndcqa0o= 2024-04-13T22:15:35.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AXaM2xleJr3PhOr/SSipndcqa0o=', 'Content-Length': '0'} 2024-04-13T22:15:35.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65f557c4050d7de-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.442 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.442 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.442 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.442 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.442 DEBUG:boto:Method: GET 2024-04-13T22:15:35.442 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.442 DEBUG:boto:Data: 2024-04-13T22:15:35.442 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.443 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.443 DEBUG:boto:Params: {} 2024-04-13T22:15:35.443 DEBUG:boto:Token: None 2024-04-13T22:15:35.443 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f0c93c10942402-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.445 DEBUG:boto:b'vwnfns-60obj000000000000000000361000falseobj000000000000000000362024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.446 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.446 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.446 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.446 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.446 DEBUG:boto:Method: GET 2024-04-13T22:15:35.446 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000036 2024-04-13T22:15:35.446 DEBUG:boto:Data: 2024-04-13T22:15:35.446 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.447 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.447 DEBUG:boto:Params: {} 2024-04-13T22:15:35.447 DEBUG:boto:Token: None 2024-04-13T22:15:35.447 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a110400bdfa14dd-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.450 DEBUG:boto:b'vwnfns-60obj000000000000000000361000falseobj000000000000000000362024-04-13T22:15:22.693Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-13T22:15:35.451 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.451 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.451 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.451 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.451 DEBUG:boto:Method: GET 2024-04-13T22:15:35.451 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.451 DEBUG:boto:Data: 2024-04-13T22:15:35.452 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.452 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.452 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.452 DEBUG:boto:Params: {} 2024-04-13T22:15:35.452 DEBUG:boto:Token: None 2024-04-13T22:15:35.452 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs= 2024-04-13T22:15:35.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs=', 'Content-Length': '0'} 2024-04-13T22:15:35.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606e3ba9c866a6a6-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.454 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.455 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.455 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.455 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.455 DEBUG:boto:Method: GET 2024-04-13T22:15:35.455 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.455 DEBUG:boto:Data: 2024-04-13T22:15:35.455 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.455 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.455 DEBUG:boto:Params: {} 2024-04-13T22:15:35.455 DEBUG:boto:Token: None 2024-04-13T22:15:35.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs= 2024-04-13T22:15:35.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs=', 'Content-Length': '0'} 2024-04-13T22:15:35.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f16cf84a8c96cfc-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.458 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.458 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.458 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.458 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.458 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.458 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.458 DEBUG:boto:Data: 2024-04-13T22:15:35.459 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.459 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.459 DEBUG:boto:Params: {} 2024-04-13T22:15:35.459 DEBUG:boto:Token: None 2024-04-13T22:15:35.459 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AOA94hPjeyVAySb+dyMG27nZbPc= 2024-04-13T22:15:35.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AOA94hPjeyVAySb+dyMG27nZbPc=', 'Content-Length': '0'} 2024-04-13T22:15:35.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc81b63b7de8344-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.461 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.461 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.461 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.462 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.462 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.462 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.462 DEBUG:boto:Data: 2024-04-13T22:15:35.462 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.462 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.462 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.462 DEBUG:boto:Params: {} 2024-04-13T22:15:35.462 DEBUG:boto:Token: None 2024-04-13T22:15:35.462 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.463 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AOA94hPjeyVAySb+dyMG27nZbPc= 2024-04-13T22:15:35.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AOA94hPjeyVAySb+dyMG27nZbPc=', 'Content-Length': '0'} 2024-04-13T22:15:35.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e220932288b8e98e-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-13T22:15:35.465 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.465 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.465 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.465 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.465 DEBUG:boto:Method: GET 2024-04-13T22:15:35.465 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.465 DEBUG:boto:Data: 2024-04-13T22:15:35.465 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.466 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.466 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.466 DEBUG:boto:Params: {} 2024-04-13T22:15:35.466 DEBUG:boto:Token: None 2024-04-13T22:15:35.466 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.466 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs= 2024-04-13T22:15:35.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs=', 'Content-Length': '0'} 2024-04-13T22:15:35.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42ce1c48c83964d-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.468 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.468 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.468 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.469 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.469 DEBUG:boto:Method: GET 2024-04-13T22:15:35.469 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.469 DEBUG:boto:Data: 2024-04-13T22:15:35.469 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.469 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.469 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.469 DEBUG:boto:Params: {} 2024-04-13T22:15:35.469 DEBUG:boto:Token: None 2024-04-13T22:15:35.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037?versionId=null 2024-04-13T22:15:35.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs= 2024-04-13T22:15:35.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b0jZaQ3aakH01EHno+LOr7VNzvs=', 'Content-Length': '0'} 2024-04-13T22:15:35.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab9eb18a50bbad1d-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.472 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.472 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.472 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.472 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.473 DEBUG:boto:Method: GET 2024-04-13T22:15:35.473 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.473 DEBUG:boto:Data: 2024-04-13T22:15:35.473 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.473 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.473 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.473 DEBUG:boto:Params: {} 2024-04-13T22:15:35.473 DEBUG:boto:Token: None 2024-04-13T22:15:35.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a7451d50bbe01d-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.477 DEBUG:boto:b'vwnfns-60obj000000000000000000371000falseobj000000000000000000372024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.477 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.477 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.478 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.478 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.478 DEBUG:boto:Method: GET 2024-04-13T22:15:35.478 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.478 DEBUG:boto:Data: 2024-04-13T22:15:35.478 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.478 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.478 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.478 DEBUG:boto:Params: {} 2024-04-13T22:15:35.478 DEBUG:boto:Token: None 2024-04-13T22:15:35.479 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4015bc3fa9316c-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.483 DEBUG:boto:b'vwnfns-60obj000000000000000000371000falseobj000000000000000000372024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.483 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.483 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.483 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.483 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.483 DEBUG:boto:Data: 2024-04-13T22:15:35.483 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.484 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.484 DEBUG:boto:Params: {} 2024-04-13T22:15:35.484 DEBUG:boto:Token: None 2024-04-13T22:15:35.484 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2UDa4L1/wCq6Lul9tIWr3Em8Hyk= 2024-04-13T22:15:35.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2UDa4L1/wCq6Lul9tIWr3Em8Hyk=', 'Content-Length': '0'} 2024-04-13T22:15:35.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fea2e666f723c4a-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.486 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.486 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.486 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.486 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.486 DEBUG:boto:Data: 2024-04-13T22:15:35.486 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.486 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.487 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.487 DEBUG:boto:Params: {} 2024-04-13T22:15:35.487 DEBUG:boto:Token: None 2024-04-13T22:15:35.487 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.487 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2UDa4L1/wCq6Lul9tIWr3Em8Hyk= 2024-04-13T22:15:35.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2UDa4L1/wCq6Lul9tIWr3Em8Hyk=', 'Content-Length': '0'} 2024-04-13T22:15:35.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13659d149af9225-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-13T22:15:35.489 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.489 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.489 DEBUG:boto:Method: GET 2024-04-13T22:15:35.489 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.489 DEBUG:boto:Data: 2024-04-13T22:15:35.489 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.489 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.490 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.490 DEBUG:boto:Params: {} 2024-04-13T22:15:35.490 DEBUG:boto:Token: None 2024-04-13T22:15:35.490 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.490 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8docHMqS6uKqSHXYNvXbesUg/AQ= 2024-04-13T22:15:35.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8docHMqS6uKqSHXYNvXbesUg/AQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e7807dc3ad0971-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.492 DEBUG:boto:path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.492 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.492 DEBUG:boto:Method: GET 2024-04-13T22:15:35.492 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.492 DEBUG:boto:Data: 2024-04-13T22:15:35.492 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.492 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.492 DEBUG:boto:Params: {} 2024-04-13T22:15:35.493 DEBUG:boto:Token: None 2024-04-13T22:15:35.493 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000037 2024-04-13T22:15:35.493 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8docHMqS6uKqSHXYNvXbesUg/AQ= 2024-04-13T22:15:35.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8docHMqS6uKqSHXYNvXbesUg/AQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74b485101c7239d-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.542 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.543 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.543 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.543 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.543 DEBUG:boto:Method: GET 2024-04-13T22:15:35.543 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.543 DEBUG:boto:Data: 2024-04-13T22:15:35.543 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.543 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.543 DEBUG:boto:Params: {} 2024-04-13T22:15:35.544 DEBUG:boto:Token: None 2024-04-13T22:15:35.544 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004797315f8ff97422-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.546 DEBUG:boto:b'vwnfns-60obj000000000000000000371000falseobj000000000000000000372024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.547 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.547 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.547 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.547 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.547 DEBUG:boto:Method: GET 2024-04-13T22:15:35.547 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000037 2024-04-13T22:15:35.547 DEBUG:boto:Data: 2024-04-13T22:15:35.547 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.547 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.548 DEBUG:boto:Params: {} 2024-04-13T22:15:35.548 DEBUG:boto:Token: None 2024-04-13T22:15:35.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf3978ea4f86c18-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.552 DEBUG:boto:b'vwnfns-60obj000000000000000000371000falseobj000000000000000000372024-04-13T22:15:20.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-13T22:15:35.552 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.552 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.552 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.552 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.553 DEBUG:boto:Method: GET 2024-04-13T22:15:35.553 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.553 DEBUG:boto:Data: 2024-04-13T22:15:35.553 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.553 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.553 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.553 DEBUG:boto:Params: {} 2024-04-13T22:15:35.553 DEBUG:boto:Token: None 2024-04-13T22:15:35.553 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.553 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg= 2024-04-13T22:15:35.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg=', 'Content-Length': '0'} 2024-04-13T22:15:35.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e822544aab1d85c-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.598 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.599 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.599 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.599 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.599 DEBUG:boto:Method: GET 2024-04-13T22:15:35.599 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.599 DEBUG:boto:Data: 2024-04-13T22:15:35.599 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.599 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.599 DEBUG:boto:Params: {} 2024-04-13T22:15:35.600 DEBUG:boto:Token: None 2024-04-13T22:15:35.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg= 2024-04-13T22:15:35.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg=', 'Content-Length': '0'} 2024-04-13T22:15:35.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2b4eee0deb68ff-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.602 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.603 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.603 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.603 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.603 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.603 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.603 DEBUG:boto:Data: 2024-04-13T22:15:35.603 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.603 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.603 DEBUG:boto:Params: {} 2024-04-13T22:15:35.604 DEBUG:boto:Token: None 2024-04-13T22:15:35.604 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t37j2B+toiZj2EO4WwWwYsORwbo= 2024-04-13T22:15:35.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t37j2B+toiZj2EO4WwWwYsORwbo=', 'Content-Length': '0'} 2024-04-13T22:15:35.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d2750b2eba93be2-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.606 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.606 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.606 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.606 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.606 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.606 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.606 DEBUG:boto:Data: 2024-04-13T22:15:35.607 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.607 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.607 DEBUG:boto:Params: {} 2024-04-13T22:15:35.607 DEBUG:boto:Token: None 2024-04-13T22:15:35.607 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t37j2B+toiZj2EO4WwWwYsORwbo= 2024-04-13T22:15:35.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t37j2B+toiZj2EO4WwWwYsORwbo=', 'Content-Length': '0'} 2024-04-13T22:15:35.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f39f0602cc13f13-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-13T22:15:35.609 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.609 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.609 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.609 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.609 DEBUG:boto:Method: GET 2024-04-13T22:15:35.609 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.610 DEBUG:boto:Data: 2024-04-13T22:15:35.610 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.610 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.610 DEBUG:boto:Params: {} 2024-04-13T22:15:35.610 DEBUG:boto:Token: None 2024-04-13T22:15:35.610 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.610 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg= 2024-04-13T22:15:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg=', 'Content-Length': '0'} 2024-04-13T22:15:35.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e7524251f68c63-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.612 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.612 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.612 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.612 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.613 DEBUG:boto:Method: GET 2024-04-13T22:15:35.613 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.613 DEBUG:boto:Data: 2024-04-13T22:15:35.613 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.613 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.613 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.613 DEBUG:boto:Params: {} 2024-04-13T22:15:35.613 DEBUG:boto:Token: None 2024-04-13T22:15:35.613 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039?versionId=null 2024-04-13T22:15:35.613 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg= 2024-04-13T22:15:35.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YBGm6MAROKdRctCHE18JRyF2sMg=', 'Content-Length': '0'} 2024-04-13T22:15:35.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0e7e5286d726f6-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.615 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.615 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.615 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.616 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.616 DEBUG:boto:Method: GET 2024-04-13T22:15:35.616 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.616 DEBUG:boto:Data: 2024-04-13T22:15:35.616 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.616 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.616 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.616 DEBUG:boto:Params: {} 2024-04-13T22:15:35.616 DEBUG:boto:Token: None 2024-04-13T22:15:35.616 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f5f0e5e95fb8d2-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.619 DEBUG:boto:b'vwnfns-60obj000000000000000000391000falseobj000000000000000000392024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.619 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.619 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.619 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.620 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.620 DEBUG:boto:Method: GET 2024-04-13T22:15:35.620 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.620 DEBUG:boto:Data: 2024-04-13T22:15:35.620 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.620 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.620 DEBUG:boto:Params: {} 2024-04-13T22:15:35.621 DEBUG:boto:Token: None 2024-04-13T22:15:35.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.621 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004165b9e9758b4206-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.624 DEBUG:boto:b'vwnfns-60obj000000000000000000391000falseobj000000000000000000392024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.625 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.625 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.625 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.625 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.625 DEBUG:boto:Data: 2024-04-13T22:15:35.625 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.625 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.625 DEBUG:boto:Params: {} 2024-04-13T22:15:35.626 DEBUG:boto:Token: None 2024-04-13T22:15:35.626 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.626 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WVRh/SNnUhz+yPXStjBIZaUM9yU= 2024-04-13T22:15:35.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WVRh/SNnUhz+yPXStjBIZaUM9yU=', 'Content-Length': '0'} 2024-04-13T22:15:35.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c474524d4fc08b41-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.628 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.628 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.628 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.628 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.628 DEBUG:boto:Data: 2024-04-13T22:15:35.628 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.628 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.628 DEBUG:boto:Params: {} 2024-04-13T22:15:35.628 DEBUG:boto:Token: None 2024-04-13T22:15:35.629 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WVRh/SNnUhz+yPXStjBIZaUM9yU= 2024-04-13T22:15:35.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WVRh/SNnUhz+yPXStjBIZaUM9yU=', 'Content-Length': '0'} 2024-04-13T22:15:35.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080843711ce682eec-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-13T22:15:35.631 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.631 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.631 DEBUG:boto:Method: GET 2024-04-13T22:15:35.631 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.631 DEBUG:boto:Data: 2024-04-13T22:15:35.631 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.631 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.631 DEBUG:boto:Params: {} 2024-04-13T22:15:35.632 DEBUG:boto:Token: None 2024-04-13T22:15:35.632 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JX6MCETTpBwx7R+H+HMneLpLqRE= 2024-04-13T22:15:35.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JX6MCETTpBwx7R+H+HMneLpLqRE=', 'Content-Length': '0'} 2024-04-13T22:15:35.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221e6aeded831771-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.634 DEBUG:boto:path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.634 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.634 DEBUG:boto:Method: GET 2024-04-13T22:15:35.634 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.634 DEBUG:boto:Data: 2024-04-13T22:15:35.634 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.634 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.635 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.635 DEBUG:boto:Params: {} 2024-04-13T22:15:35.635 DEBUG:boto:Token: None 2024-04-13T22:15:35.635 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000039 2024-04-13T22:15:35.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JX6MCETTpBwx7R+H+HMneLpLqRE= 2024-04-13T22:15:35.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JX6MCETTpBwx7R+H+HMneLpLqRE=', 'Content-Length': '0'} 2024-04-13T22:15:35.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d01aaccbcf9ba0-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.678 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.679 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.679 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.679 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.679 DEBUG:boto:Method: GET 2024-04-13T22:15:35.679 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.679 DEBUG:boto:Data: 2024-04-13T22:15:35.679 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.679 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.680 DEBUG:boto:Params: {} 2024-04-13T22:15:35.680 DEBUG:boto:Token: None 2024-04-13T22:15:35.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb58be3370ff18ed-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.682 DEBUG:boto:b'vwnfns-60obj000000000000000000391000falseobj000000000000000000392024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.683 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.683 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.683 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.683 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.683 DEBUG:boto:Method: GET 2024-04-13T22:15:35.683 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000039 2024-04-13T22:15:35.683 DEBUG:boto:Data: 2024-04-13T22:15:35.683 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.683 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.684 DEBUG:boto:Params: {} 2024-04-13T22:15:35.684 DEBUG:boto:Token: None 2024-04-13T22:15:35.684 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c44626e6a23c08e-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.688 DEBUG:boto:b'vwnfns-60obj000000000000000000391000falseobj000000000000000000392024-04-13T22:15:22.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-13T22:15:35.688 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.688 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.688 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.688 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.688 DEBUG:boto:Method: GET 2024-04-13T22:15:35.689 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.689 DEBUG:boto:Data: 2024-04-13T22:15:35.689 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.689 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.689 DEBUG:boto:Params: {} 2024-04-13T22:15:35.689 DEBUG:boto:Token: None 2024-04-13T22:15:35.689 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0= 2024-04-13T22:15:35.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0=', 'Content-Length': '0'} 2024-04-13T22:15:35.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce533c473beb25a5-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.734 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.735 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.735 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.735 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.735 DEBUG:boto:Method: GET 2024-04-13T22:15:35.735 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.735 DEBUG:boto:Data: 2024-04-13T22:15:35.735 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.735 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.735 DEBUG:boto:Params: {} 2024-04-13T22:15:35.736 DEBUG:boto:Token: None 2024-04-13T22:15:35.736 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0= 2024-04-13T22:15:35.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0=', 'Content-Length': '0'} 2024-04-13T22:15:35.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed766ad23ce81c69-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.738 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.738 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.738 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.738 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.738 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.739 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.739 DEBUG:boto:Data: 2024-04-13T22:15:35.739 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.739 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.739 DEBUG:boto:Params: {} 2024-04-13T22:15:35.739 DEBUG:boto:Token: None 2024-04-13T22:15:35.739 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gRe0H165pmhoSxtqyNYCK8pEiVI= 2024-04-13T22:15:35.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gRe0H165pmhoSxtqyNYCK8pEiVI=', 'Content-Length': '0'} 2024-04-13T22:15:35.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b96122904c17f2-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.741 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.741 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.741 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.742 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.742 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.742 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.742 DEBUG:boto:Data: 2024-04-13T22:15:35.742 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.742 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.742 DEBUG:boto:Params: {} 2024-04-13T22:15:35.742 DEBUG:boto:Token: None 2024-04-13T22:15:35.743 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gRe0H165pmhoSxtqyNYCK8pEiVI= 2024-04-13T22:15:35.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gRe0H165pmhoSxtqyNYCK8pEiVI=', 'Content-Length': '0'} 2024-04-13T22:15:35.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f964f5fd4113a5ba-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-13T22:15:35.744 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.745 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.745 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.745 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.745 DEBUG:boto:Method: GET 2024-04-13T22:15:35.745 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.745 DEBUG:boto:Data: 2024-04-13T22:15:35.745 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.745 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.745 DEBUG:boto:Params: {} 2024-04-13T22:15:35.745 DEBUG:boto:Token: None 2024-04-13T22:15:35.746 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.746 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0= 2024-04-13T22:15:35.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0=', 'Content-Length': '0'} 2024-04-13T22:15:35.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ae01a9dd26cbe6-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.747 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.747 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.748 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.748 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.748 DEBUG:boto:Method: GET 2024-04-13T22:15:35.748 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.748 DEBUG:boto:Data: 2024-04-13T22:15:35.748 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.748 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.748 DEBUG:boto:Params: {} 2024-04-13T22:15:35.748 DEBUG:boto:Token: None 2024-04-13T22:15:35.749 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040?versionId=null 2024-04-13T22:15:35.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0= 2024-04-13T22:15:35.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UuLv0HIA8E8I4R3DMPw/0HoiAc0=', 'Content-Length': '0'} 2024-04-13T22:15:35.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ec8790adc1ea87-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.751 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.751 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.751 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.751 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.751 DEBUG:boto:Method: GET 2024-04-13T22:15:35.751 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.751 DEBUG:boto:Data: 2024-04-13T22:15:35.751 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.752 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.752 DEBUG:boto:Params: {} 2024-04-13T22:15:35.752 DEBUG:boto:Token: None 2024-04-13T22:15:35.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001da91df51d37a98a-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.754 DEBUG:boto:b'vwnfns-60obj000000000000000000401000falseobj000000000000000000402024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.755 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.755 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.755 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.755 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.756 DEBUG:boto:Method: GET 2024-04-13T22:15:35.756 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.756 DEBUG:boto:Data: 2024-04-13T22:15:35.756 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.756 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.756 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.756 DEBUG:boto:Params: {} 2024-04-13T22:15:35.756 DEBUG:boto:Token: None 2024-04-13T22:15:35.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f893e2170774f0c2-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.760 DEBUG:boto:b'vwnfns-60obj000000000000000000401000falseobj000000000000000000402024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.760 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.760 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.760 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.760 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.761 DEBUG:boto:Data: 2024-04-13T22:15:35.761 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.761 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.761 DEBUG:boto:Params: {} 2024-04-13T22:15:35.761 DEBUG:boto:Token: None 2024-04-13T22:15:35.761 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5ODVJ9NJVDH/Yxk7y/Ff+l4x5uo= 2024-04-13T22:15:35.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5ODVJ9NJVDH/Yxk7y/Ff+l4x5uo=', 'Content-Length': '0'} 2024-04-13T22:15:35.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb3326c149c320e-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.763 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.763 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.763 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.763 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.763 DEBUG:boto:Data: 2024-04-13T22:15:35.763 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.764 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.764 DEBUG:boto:Params: {} 2024-04-13T22:15:35.764 DEBUG:boto:Token: None 2024-04-13T22:15:35.764 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5ODVJ9NJVDH/Yxk7y/Ff+l4x5uo= 2024-04-13T22:15:35.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5ODVJ9NJVDH/Yxk7y/Ff+l4x5uo=', 'Content-Length': '0'} 2024-04-13T22:15:35.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087eaf2c484271142-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-13T22:15:35.766 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.766 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.766 DEBUG:boto:Method: GET 2024-04-13T22:15:35.766 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.766 DEBUG:boto:Data: 2024-04-13T22:15:35.766 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.766 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.766 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.766 DEBUG:boto:Params: {} 2024-04-13T22:15:35.767 DEBUG:boto:Token: None 2024-04-13T22:15:35.767 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.767 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w8bqn74OSof9K9UyoINyLSQdt3c= 2024-04-13T22:15:35.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w8bqn74OSof9K9UyoINyLSQdt3c=', 'Content-Length': '0'} 2024-04-13T22:15:35.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac30af482fc7409f-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.810 DEBUG:boto:path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.811 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.811 DEBUG:boto:Method: GET 2024-04-13T22:15:35.811 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.811 DEBUG:boto:Data: 2024-04-13T22:15:35.811 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.811 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.811 DEBUG:boto:Params: {} 2024-04-13T22:15:35.811 DEBUG:boto:Token: None 2024-04-13T22:15:35.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000040 2024-04-13T22:15:35.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w8bqn74OSof9K9UyoINyLSQdt3c= 2024-04-13T22:15:35.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w8bqn74OSof9K9UyoINyLSQdt3c=', 'Content-Length': '0'} 2024-04-13T22:15:35.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436db576a1936383-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.814 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.814 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.814 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.814 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.814 DEBUG:boto:Method: GET 2024-04-13T22:15:35.814 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.814 DEBUG:boto:Data: 2024-04-13T22:15:35.814 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.815 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.815 DEBUG:boto:Params: {} 2024-04-13T22:15:35.815 DEBUG:boto:Token: None 2024-04-13T22:15:35.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a676a1fcd4df49-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.817 DEBUG:boto:b'vwnfns-60obj000000000000000000401000falseobj000000000000000000402024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.818 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.818 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.818 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.818 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.818 DEBUG:boto:Method: GET 2024-04-13T22:15:35.818 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000040 2024-04-13T22:15:35.818 DEBUG:boto:Data: 2024-04-13T22:15:35.819 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.819 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.819 DEBUG:boto:Params: {} 2024-04-13T22:15:35.819 DEBUG:boto:Token: None 2024-04-13T22:15:35.819 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103b3d11015fac36-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.823 DEBUG:boto:b'vwnfns-60obj000000000000000000401000falseobj000000000000000000402024-04-13T22:15:22.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-13T22:15:35.823 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.823 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.823 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.824 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.824 DEBUG:boto:Method: GET 2024-04-13T22:15:35.824 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.824 DEBUG:boto:Data: 2024-04-13T22:15:35.824 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.824 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.824 DEBUG:boto:Params: {} 2024-04-13T22:15:35.824 DEBUG:boto:Token: None 2024-04-13T22:15:35.824 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.825 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY= 2024-04-13T22:15:35.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY=', 'Content-Length': '0'} 2024-04-13T22:15:35.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004816f64096aa52d-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.827 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.827 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.827 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.827 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.827 DEBUG:boto:Method: GET 2024-04-13T22:15:35.828 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.828 DEBUG:boto:Data: 2024-04-13T22:15:35.828 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.828 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.828 DEBUG:boto:Params: {} 2024-04-13T22:15:35.828 DEBUG:boto:Token: None 2024-04-13T22:15:35.828 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY= 2024-04-13T22:15:35.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY=', 'Content-Length': '0'} 2024-04-13T22:15:35.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e2d37e67cbe79f-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.830 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.830 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.830 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.831 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.831 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.831 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.831 DEBUG:boto:Data: 2024-04-13T22:15:35.831 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.831 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.831 DEBUG:boto:Params: {} 2024-04-13T22:15:35.831 DEBUG:boto:Token: None 2024-04-13T22:15:35.831 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.832 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bR0E+mGDrV62b17iuUm+lqRw5Wc= 2024-04-13T22:15:35.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bR0E+mGDrV62b17iuUm+lqRw5Wc=', 'Content-Length': '0'} 2024-04-13T22:15:35.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a5d1c2c64885f8-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.833 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.833 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.833 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.833 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.834 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.834 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.834 DEBUG:boto:Data: 2024-04-13T22:15:35.834 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.834 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.834 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.834 DEBUG:boto:Params: {} 2024-04-13T22:15:35.834 DEBUG:boto:Token: None 2024-04-13T22:15:35.834 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.835 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bR0E+mGDrV62b17iuUm+lqRw5Wc= 2024-04-13T22:15:35.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bR0E+mGDrV62b17iuUm+lqRw5Wc=', 'Content-Length': '0'} 2024-04-13T22:15:35.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095511768a86da268-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-13T22:15:35.836 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.837 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.837 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.837 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.837 DEBUG:boto:Method: GET 2024-04-13T22:15:35.837 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.837 DEBUG:boto:Data: 2024-04-13T22:15:35.837 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.837 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.837 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.837 DEBUG:boto:Params: {} 2024-04-13T22:15:35.837 DEBUG:boto:Token: None 2024-04-13T22:15:35.838 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.838 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY= 2024-04-13T22:15:35.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY=', 'Content-Length': '0'} 2024-04-13T22:15:35.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000defd0b68f1ff7c91-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.840 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.840 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.840 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.840 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.840 DEBUG:boto:Method: GET 2024-04-13T22:15:35.840 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.840 DEBUG:boto:Data: 2024-04-13T22:15:35.840 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.841 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.841 DEBUG:boto:Params: {} 2024-04-13T22:15:35.841 DEBUG:boto:Token: None 2024-04-13T22:15:35.841 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042?versionId=null 2024-04-13T22:15:35.841 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY= 2024-04-13T22:15:35.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IKl4WULAOoV6qRgCXrDtlDUF/cY=', 'Content-Length': '0'} 2024-04-13T22:15:35.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ede42846861e68-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.843 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.843 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.843 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.843 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.843 DEBUG:boto:Method: GET 2024-04-13T22:15:35.843 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.843 DEBUG:boto:Data: 2024-04-13T22:15:35.844 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.844 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.844 DEBUG:boto:Params: {} 2024-04-13T22:15:35.844 DEBUG:boto:Token: None 2024-04-13T22:15:35.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c8e7391265a76d-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.846 DEBUG:boto:b'vwnfns-60obj000000000000000000421000falseobj000000000000000000422024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.847 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.847 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.847 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.847 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.847 DEBUG:boto:Method: GET 2024-04-13T22:15:35.847 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.847 DEBUG:boto:Data: 2024-04-13T22:15:35.847 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.847 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.847 DEBUG:boto:Params: {} 2024-04-13T22:15:35.848 DEBUG:boto:Token: None 2024-04-13T22:15:35.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0616d7157448dc-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.852 DEBUG:boto:b'vwnfns-60obj000000000000000000421000falseobj000000000000000000422024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.852 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.852 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.852 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.852 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.852 DEBUG:boto:Data: 2024-04-13T22:15:35.852 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.853 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.853 DEBUG:boto:Params: {} 2024-04-13T22:15:35.853 DEBUG:boto:Token: None 2024-04-13T22:15:35.853 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5QFXjMSkrVbG6NDyA+8LEKZJKvI= 2024-04-13T22:15:35.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5QFXjMSkrVbG6NDyA+8LEKZJKvI=', 'Content-Length': '0'} 2024-04-13T22:15:35.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d9b2c2496722220-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.855 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.855 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.855 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.855 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.855 DEBUG:boto:Data: 2024-04-13T22:15:35.855 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.856 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.856 DEBUG:boto:Params: {} 2024-04-13T22:15:35.856 DEBUG:boto:Token: None 2024-04-13T22:15:35.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5QFXjMSkrVbG6NDyA+8LEKZJKvI= 2024-04-13T22:15:35.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5QFXjMSkrVbG6NDyA+8LEKZJKvI=', 'Content-Length': '0'} 2024-04-13T22:15:35.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000554e590b4037fdec-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-13T22:15:35.858 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.858 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.858 DEBUG:boto:Method: GET 2024-04-13T22:15:35.858 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.858 DEBUG:boto:Data: 2024-04-13T22:15:35.858 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.858 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.859 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.859 DEBUG:boto:Params: {} 2024-04-13T22:15:35.859 DEBUG:boto:Token: None 2024-04-13T22:15:35.859 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TmsGk37cHLGvp8kg79BMWeTEFCU= 2024-04-13T22:15:35.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TmsGk37cHLGvp8kg79BMWeTEFCU=', 'Content-Length': '0'} 2024-04-13T22:15:35.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006517f9e57dd05cc7-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.902 DEBUG:boto:path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.903 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.903 DEBUG:boto:Method: GET 2024-04-13T22:15:35.903 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.903 DEBUG:boto:Data: 2024-04-13T22:15:35.903 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.903 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.903 DEBUG:boto:Params: {} 2024-04-13T22:15:35.903 DEBUG:boto:Token: None 2024-04-13T22:15:35.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000042 2024-04-13T22:15:35.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TmsGk37cHLGvp8kg79BMWeTEFCU= 2024-04-13T22:15:35.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TmsGk37cHLGvp8kg79BMWeTEFCU=', 'Content-Length': '0'} 2024-04-13T22:15:35.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000064a5ed96008bdd3-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.906 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.906 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.906 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.906 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.906 DEBUG:boto:Method: GET 2024-04-13T22:15:35.906 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.906 DEBUG:boto:Data: 2024-04-13T22:15:35.906 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.907 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.907 DEBUG:boto:Params: {} 2024-04-13T22:15:35.907 DEBUG:boto:Token: None 2024-04-13T22:15:35.907 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ab8a9ded86a989-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.909 DEBUG:boto:b'vwnfns-60obj000000000000000000421000falseobj000000000000000000422024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.910 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.910 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.910 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.910 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.910 DEBUG:boto:Method: GET 2024-04-13T22:15:35.910 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000042 2024-04-13T22:15:35.910 DEBUG:boto:Data: 2024-04-13T22:15:35.911 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.911 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.911 DEBUG:boto:Params: {} 2024-04-13T22:15:35.911 DEBUG:boto:Token: None 2024-04-13T22:15:35.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003848725657740d88-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.916 DEBUG:boto:b'vwnfns-60obj000000000000000000421000falseobj000000000000000000422024-04-13T22:15:20.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-13T22:15:35.916 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.916 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.916 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.916 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.916 DEBUG:boto:Method: GET 2024-04-13T22:15:35.917 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.917 DEBUG:boto:Data: 2024-04-13T22:15:35.917 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.917 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.917 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.917 DEBUG:boto:Params: {} 2024-04-13T22:15:35.917 DEBUG:boto:Token: None 2024-04-13T22:15:35.917 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.917 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y= 2024-04-13T22:15:35.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y=', 'Content-Length': '0'} 2024-04-13T22:15:35.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfdbe49b9bb4bf7-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.920 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.920 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.920 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.920 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.920 DEBUG:boto:Method: GET 2024-04-13T22:15:35.920 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.920 DEBUG:boto:Data: 2024-04-13T22:15:35.921 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.921 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.921 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.921 DEBUG:boto:Params: {} 2024-04-13T22:15:35.921 DEBUG:boto:Token: None 2024-04-13T22:15:35.921 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y= 2024-04-13T22:15:35.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y=', 'Content-Length': '0'} 2024-04-13T22:15:35.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a429095df3aa84bf-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.923 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.923 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.923 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.923 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.923 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.923 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.923 DEBUG:boto:Data: 2024-04-13T22:15:35.924 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.924 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.924 DEBUG:boto:Params: {} 2024-04-13T22:15:35.924 DEBUG:boto:Token: None 2024-04-13T22:15:35.924 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hTlI5PWYq3HJ2UStw9jw8Qx3yY4= 2024-04-13T22:15:35.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hTlI5PWYq3HJ2UStw9jw8Qx3yY4=', 'Content-Length': '0'} 2024-04-13T22:15:35.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8e48d67c693b94-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.926 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.926 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.926 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.926 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.926 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.927 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.927 DEBUG:boto:Data: 2024-04-13T22:15:35.927 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.927 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.927 DEBUG:boto:Params: {} 2024-04-13T22:15:35.927 DEBUG:boto:Token: None 2024-04-13T22:15:35.927 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.927 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hTlI5PWYq3HJ2UStw9jw8Qx3yY4= 2024-04-13T22:15:35.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hTlI5PWYq3HJ2UStw9jw8Qx3yY4=', 'Content-Length': '0'} 2024-04-13T22:15:35.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002356bca3199a97a9-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-13T22:15:35.929 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.929 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.929 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.930 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.930 DEBUG:boto:Method: GET 2024-04-13T22:15:35.930 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.930 DEBUG:boto:Data: 2024-04-13T22:15:35.930 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.930 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.930 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.930 DEBUG:boto:Params: {} 2024-04-13T22:15:35.930 DEBUG:boto:Token: None 2024-04-13T22:15:35.930 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.931 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y= 2024-04-13T22:15:35.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y=', 'Content-Length': '0'} 2024-04-13T22:15:35.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890aa4f78f80ae27-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.932 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.933 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.933 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.933 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.933 DEBUG:boto:Method: GET 2024-04-13T22:15:35.933 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.933 DEBUG:boto:Data: 2024-04-13T22:15:35.933 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.933 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.933 DEBUG:boto:Params: {} 2024-04-13T22:15:35.934 DEBUG:boto:Token: None 2024-04-13T22:15:35.934 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045?versionId=null 2024-04-13T22:15:35.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y= 2024-04-13T22:15:35.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLm871u0GLRcbG3XK2ABegj3/3Y=', 'Content-Length': '0'} 2024-04-13T22:15:35.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e305b0da8928a9a9-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.936 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.936 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.936 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.936 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.936 DEBUG:boto:Method: GET 2024-04-13T22:15:35.936 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.936 DEBUG:boto:Data: 2024-04-13T22:15:35.936 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.937 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.937 DEBUG:boto:Params: {} 2024-04-13T22:15:35.937 DEBUG:boto:Token: None 2024-04-13T22:15:35.937 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059b292d84ec770d-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.939 DEBUG:boto:b'vwnfns-60obj000000000000000000451000falseobj000000000000000000452024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.940 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.940 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.940 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.940 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.940 DEBUG:boto:Method: GET 2024-04-13T22:15:35.940 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.940 DEBUG:boto:Data: 2024-04-13T22:15:35.940 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.941 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.941 DEBUG:boto:Params: {} 2024-04-13T22:15:35.941 DEBUG:boto:Token: None 2024-04-13T22:15:35.941 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.941 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:35.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503aa955b59d0c1a-00661b0407-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.945 DEBUG:boto:b'vwnfns-60obj000000000000000000451000falseobj000000000000000000452024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:35.945 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.945 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.945 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.945 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.946 DEBUG:boto:Data: 2024-04-13T22:15:35.946 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.946 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.946 DEBUG:boto:Params: {} 2024-04-13T22:15:35.946 DEBUG:boto:Token: None 2024-04-13T22:15:35.946 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.946 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q/pi+vI4YIxKQ+03xYNI/H5ORJg= 2024-04-13T22:15:35.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q/pi+vI4YIxKQ+03xYNI/H5ORJg=', 'Content-Length': '0'} 2024-04-13T22:15:35.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a747182bbbedf7d8-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.948 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.948 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.948 DEBUG:boto:Method: HEAD 2024-04-13T22:15:35.948 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.948 DEBUG:boto:Data: 2024-04-13T22:15:35.948 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.949 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.949 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.949 DEBUG:boto:Params: {} 2024-04-13T22:15:35.949 DEBUG:boto:Token: None 2024-04-13T22:15:35.949 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q/pi+vI4YIxKQ+03xYNI/H5ORJg= 2024-04-13T22:15:35.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q/pi+vI4YIxKQ+03xYNI/H5ORJg=', 'Content-Length': '0'} 2024-04-13T22:15:35.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cab6983a08c8238-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-13T22:15:35.951 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.951 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.951 DEBUG:boto:Method: GET 2024-04-13T22:15:35.951 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.951 DEBUG:boto:Data: 2024-04-13T22:15:35.951 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.952 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.952 DEBUG:boto:Params: {} 2024-04-13T22:15:35.952 DEBUG:boto:Token: None 2024-04-13T22:15:35.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oxhWUFvOaroqo5Qwv1nDUfHcFtM= 2024-04-13T22:15:35.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oxhWUFvOaroqo5Qwv1nDUfHcFtM=', 'Content-Length': '0'} 2024-04-13T22:15:35.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff746355e67d9106-00661b0407-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.994 DEBUG:boto:path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.995 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.995 DEBUG:boto:Method: GET 2024-04-13T22:15:35.995 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.995 DEBUG:boto:Data: 2024-04-13T22:15:35.995 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:35.995 DEBUG:boto:Port: 8000 2024-04-13T22:15:35.995 DEBUG:boto:Params: {} 2024-04-13T22:15:35.995 DEBUG:boto:Token: None 2024-04-13T22:15:35.996 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/obj00000000000000000045 2024-04-13T22:15:35.996 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oxhWUFvOaroqo5Qwv1nDUfHcFtM= 2024-04-13T22:15:35.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oxhWUFvOaroqo5Qwv1nDUfHcFtM=', 'Content-Length': '0'} 2024-04-13T22:15:35.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ed0255e8b81bce-00661b0407-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:35.998 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:35.998 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:35.998 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.998 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.998 DEBUG:boto:Method: GET 2024-04-13T22:15:35.998 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:35.998 DEBUG:boto:Data: 2024-04-13T22:15:35.998 DEBUG:boto:Headers: {} 2024-04-13T22:15:35.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:35.999 DEBUG:boto:Port: 8001 2024-04-13T22:15:35.999 DEBUG:boto:Params: {} 2024-04-13T22:15:35.999 DEBUG:boto:Token: None 2024-04-13T22:15:35.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:35 GMT /vwnfns-60/ 2024-04-13T22:15:35.999 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ= 2024-04-13T22:15:35.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mLA0dlSTqe6tox5QAQwbS27WWdQ=', 'Content-Length': '0'} 2024-04-13T22:15:36.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c7a6c9140fcea3-00661b0407-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.002 DEBUG:boto:b'vwnfns-60obj000000000000000000451000falseobj000000000000000000452024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.002 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.002 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.002 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:36.002 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:36.002 DEBUG:boto:Method: GET 2024-04-13T22:15:36.003 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000045 2024-04-13T22:15:36.003 DEBUG:boto:Data: 2024-04-13T22:15:36.003 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.003 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.003 DEBUG:boto:Params: {} 2024-04-13T22:15:36.003 DEBUG:boto:Token: None 2024-04-13T22:15:36.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479e81aad02d0ad7-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.007 DEBUG:boto:b'vwnfns-60obj000000000000000000451000falseobj000000000000000000452024-04-13T22:15:22.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-13T22:15:36.008 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.008 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.008 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.008 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.008 DEBUG:boto:Method: GET 2024-04-13T22:15:36.008 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.008 DEBUG:boto:Data: 2024-04-13T22:15:36.008 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.008 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.008 DEBUG:boto:Params: {} 2024-04-13T22:15:36.009 DEBUG:boto:Token: None 2024-04-13T22:15:36.009 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts= 2024-04-13T22:15:36.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts=', 'Content-Length': '0'} 2024-04-13T22:15:36.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973370cd879a2a89-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.011 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.011 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.011 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.011 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.011 DEBUG:boto:Method: GET 2024-04-13T22:15:36.011 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.011 DEBUG:boto:Data: 2024-04-13T22:15:36.011 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.012 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.012 DEBUG:boto:Params: {} 2024-04-13T22:15:36.012 DEBUG:boto:Token: None 2024-04-13T22:15:36.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts= 2024-04-13T22:15:36.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts=', 'Content-Length': '0'} 2024-04-13T22:15:36.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0de7505f632a858-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.014 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.014 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.014 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.014 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.015 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.015 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.015 DEBUG:boto:Data: 2024-04-13T22:15:36.015 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.015 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.015 DEBUG:boto:Params: {} 2024-04-13T22:15:36.015 DEBUG:boto:Token: None 2024-04-13T22:15:36.015 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cl3jrhpPFMkslb/1M8fXxLhlrC0= 2024-04-13T22:15:36.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cl3jrhpPFMkslb/1M8fXxLhlrC0=', 'Content-Length': '0'} 2024-04-13T22:15:36.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1347f5f965d730f-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.017 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.017 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.017 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.018 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.018 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.018 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.018 DEBUG:boto:Data: 2024-04-13T22:15:36.018 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.018 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.018 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.018 DEBUG:boto:Params: {} 2024-04-13T22:15:36.018 DEBUG:boto:Token: None 2024-04-13T22:15:36.018 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.019 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cl3jrhpPFMkslb/1M8fXxLhlrC0= 2024-04-13T22:15:36.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cl3jrhpPFMkslb/1M8fXxLhlrC0=', 'Content-Length': '0'} 2024-04-13T22:15:36.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35ad49b13102f16-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-13T22:15:36.020 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.020 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.021 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.021 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.021 DEBUG:boto:Method: GET 2024-04-13T22:15:36.021 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.021 DEBUG:boto:Data: 2024-04-13T22:15:36.021 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.021 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.021 DEBUG:boto:Params: {} 2024-04-13T22:15:36.021 DEBUG:boto:Token: None 2024-04-13T22:15:36.022 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.022 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts= 2024-04-13T22:15:36.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts=', 'Content-Length': '0'} 2024-04-13T22:15:36.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b692ac8fd4bf956-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.023 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.024 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.024 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.024 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.024 DEBUG:boto:Method: GET 2024-04-13T22:15:36.024 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.024 DEBUG:boto:Data: 2024-04-13T22:15:36.024 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.024 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.024 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.024 DEBUG:boto:Params: {} 2024-04-13T22:15:36.024 DEBUG:boto:Token: None 2024-04-13T22:15:36.025 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046?versionId=null 2024-04-13T22:15:36.025 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts= 2024-04-13T22:15:36.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A9zCkG8kCmy7z5nr1utFqbP7bts=', 'Content-Length': '0'} 2024-04-13T22:15:36.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000503b881f27de8eda-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.026 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.027 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.027 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.027 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.027 DEBUG:boto:Method: GET 2024-04-13T22:15:36.027 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.027 DEBUG:boto:Data: 2024-04-13T22:15:36.027 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.027 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.027 DEBUG:boto:Params: {} 2024-04-13T22:15:36.027 DEBUG:boto:Token: None 2024-04-13T22:15:36.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000635b61937b5a5d4b-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.030 DEBUG:boto:b'vwnfns-60obj000000000000000000461000falseobj000000000000000000462024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.031 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.031 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.031 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.031 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.031 DEBUG:boto:Method: GET 2024-04-13T22:15:36.031 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.031 DEBUG:boto:Data: 2024-04-13T22:15:36.031 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.031 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.031 DEBUG:boto:Params: {} 2024-04-13T22:15:36.032 DEBUG:boto:Token: None 2024-04-13T22:15:36.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004756a8d3f772990d-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.036 DEBUG:boto:b'vwnfns-60obj000000000000000000461000falseobj000000000000000000462024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.036 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.036 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.036 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.036 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.036 DEBUG:boto:Data: 2024-04-13T22:15:36.036 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.037 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.037 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.037 DEBUG:boto:Params: {} 2024-04-13T22:15:36.037 DEBUG:boto:Token: None 2024-04-13T22:15:36.037 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.037 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vnhkgaBDe+LrhiFieo9nmpZEsuk= 2024-04-13T22:15:36.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vnhkgaBDe+LrhiFieo9nmpZEsuk=', 'Content-Length': '0'} 2024-04-13T22:15:36.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006312939ea4ea752e-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.039 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.039 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.039 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.039 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.039 DEBUG:boto:Data: 2024-04-13T22:15:36.039 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.039 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.040 DEBUG:boto:Params: {} 2024-04-13T22:15:36.040 DEBUG:boto:Token: None 2024-04-13T22:15:36.040 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vnhkgaBDe+LrhiFieo9nmpZEsuk= 2024-04-13T22:15:36.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vnhkgaBDe+LrhiFieo9nmpZEsuk=', 'Content-Length': '0'} 2024-04-13T22:15:36.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a33323a25995ec-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-13T22:15:36.042 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.042 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.042 DEBUG:boto:Method: GET 2024-04-13T22:15:36.042 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.042 DEBUG:boto:Data: 2024-04-13T22:15:36.042 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.042 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.043 DEBUG:boto:Params: {} 2024-04-13T22:15:36.043 DEBUG:boto:Token: None 2024-04-13T22:15:36.043 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gYxFa34TtXMy0VN23Kba2R5Ufqw= 2024-04-13T22:15:36.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gYxFa34TtXMy0VN23Kba2R5Ufqw=', 'Content-Length': '0'} 2024-04-13T22:15:36.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829c9885e352b89a-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.086 DEBUG:boto:path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.086 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.087 DEBUG:boto:Method: GET 2024-04-13T22:15:36.087 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.087 DEBUG:boto:Data: 2024-04-13T22:15:36.087 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.087 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.087 DEBUG:boto:Params: {} 2024-04-13T22:15:36.087 DEBUG:boto:Token: None 2024-04-13T22:15:36.087 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000046 2024-04-13T22:15:36.087 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gYxFa34TtXMy0VN23Kba2R5Ufqw= 2024-04-13T22:15:36.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gYxFa34TtXMy0VN23Kba2R5Ufqw=', 'Content-Length': '0'} 2024-04-13T22:15:36.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005261d7cb61b6de6c-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.089 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.090 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.090 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.090 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.090 DEBUG:boto:Method: GET 2024-04-13T22:15:36.090 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.090 DEBUG:boto:Data: 2024-04-13T22:15:36.090 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.090 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.090 DEBUG:boto:Params: {} 2024-04-13T22:15:36.091 DEBUG:boto:Token: None 2024-04-13T22:15:36.091 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b86ca08ce5915a1e-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.093 DEBUG:boto:b'vwnfns-60obj000000000000000000461000falseobj000000000000000000462024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.094 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.094 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.094 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.094 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.094 DEBUG:boto:Method: GET 2024-04-13T22:15:36.094 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000046 2024-04-13T22:15:36.094 DEBUG:boto:Data: 2024-04-13T22:15:36.094 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.094 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.094 DEBUG:boto:Params: {} 2024-04-13T22:15:36.095 DEBUG:boto:Token: None 2024-04-13T22:15:36.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007915ee403943a497-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.098 DEBUG:boto:b'vwnfns-60obj000000000000000000461000falseobj000000000000000000462024-04-13T22:15:19.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-13T22:15:36.099 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.099 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.099 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.099 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.099 DEBUG:boto:Method: GET 2024-04-13T22:15:36.099 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.099 DEBUG:boto:Data: 2024-04-13T22:15:36.099 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.100 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.100 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.100 DEBUG:boto:Params: {} 2024-04-13T22:15:36.100 DEBUG:boto:Token: None 2024-04-13T22:15:36.100 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc= 2024-04-13T22:15:36.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc=', 'Content-Length': '0'} 2024-04-13T22:15:36.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff9e80a7ffac492-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.102 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.103 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.103 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.103 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.103 DEBUG:boto:Method: GET 2024-04-13T22:15:36.103 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.103 DEBUG:boto:Data: 2024-04-13T22:15:36.103 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.103 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.103 DEBUG:boto:Params: {} 2024-04-13T22:15:36.104 DEBUG:boto:Token: None 2024-04-13T22:15:36.104 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.104 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc= 2024-04-13T22:15:36.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc=', 'Content-Length': '0'} 2024-04-13T22:15:36.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c322f5714bb71e4f-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.106 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.106 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.106 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.106 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.106 DEBUG:boto:Data: 2024-04-13T22:15:36.106 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.107 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.107 DEBUG:boto:Params: {} 2024-04-13T22:15:36.107 DEBUG:boto:Token: None 2024-04-13T22:15:36.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FFL4j/G53P+CyaHjfrg8LZ4FAwM= 2024-04-13T22:15:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FFL4j/G53P+CyaHjfrg8LZ4FAwM=', 'Content-Length': '0'} 2024-04-13T22:15:36.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d9778de70a42200-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.109 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.109 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.109 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.109 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.109 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.109 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.109 DEBUG:boto:Data: 2024-04-13T22:15:36.110 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.110 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.110 DEBUG:boto:Params: {} 2024-04-13T22:15:36.110 DEBUG:boto:Token: None 2024-04-13T22:15:36.110 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FFL4j/G53P+CyaHjfrg8LZ4FAwM= 2024-04-13T22:15:36.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FFL4j/G53P+CyaHjfrg8LZ4FAwM=', 'Content-Length': '0'} 2024-04-13T22:15:36.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c73d3a37f01a7d-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-13T22:15:36.112 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.112 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.112 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.112 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.112 DEBUG:boto:Method: GET 2024-04-13T22:15:36.112 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.113 DEBUG:boto:Data: 2024-04-13T22:15:36.113 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.113 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.113 DEBUG:boto:Params: {} 2024-04-13T22:15:36.113 DEBUG:boto:Token: None 2024-04-13T22:15:36.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc= 2024-04-13T22:15:36.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc=', 'Content-Length': '0'} 2024-04-13T22:15:36.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c44c16a5ae8233-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.116 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.116 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.116 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.116 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.116 DEBUG:boto:Method: GET 2024-04-13T22:15:36.116 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.116 DEBUG:boto:Data: 2024-04-13T22:15:36.116 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.116 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.116 DEBUG:boto:Params: {} 2024-04-13T22:15:36.117 DEBUG:boto:Token: None 2024-04-13T22:15:36.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049?versionId=null 2024-04-13T22:15:36.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc= 2024-04-13T22:15:36.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QTjoLPsiFafqPyx7FNjDo+mDbgc=', 'Content-Length': '0'} 2024-04-13T22:15:36.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092647c7abcc14e42-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.119 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.119 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.119 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.119 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.119 DEBUG:boto:Method: GET 2024-04-13T22:15:36.119 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.119 DEBUG:boto:Data: 2024-04-13T22:15:36.119 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.120 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.120 DEBUG:boto:Params: {} 2024-04-13T22:15:36.120 DEBUG:boto:Token: None 2024-04-13T22:15:36.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1466c4b07c915d4-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.166 DEBUG:boto:b'vwnfns-60obj000000000000000000491000falseobj000000000000000000492024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.167 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.167 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.167 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.167 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.167 DEBUG:boto:Method: GET 2024-04-13T22:15:36.167 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.168 DEBUG:boto:Data: 2024-04-13T22:15:36.168 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.168 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.168 DEBUG:boto:Params: {} 2024-04-13T22:15:36.168 DEBUG:boto:Token: None 2024-04-13T22:15:36.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011200238d46422b1-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.172 DEBUG:boto:b'vwnfns-60obj000000000000000000491000falseobj000000000000000000492024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.172 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.173 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.173 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.173 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.173 DEBUG:boto:Data: 2024-04-13T22:15:36.173 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.173 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.173 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.173 DEBUG:boto:Params: {} 2024-04-13T22:15:36.173 DEBUG:boto:Token: None 2024-04-13T22:15:36.174 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.174 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8HyA+hkp4DhFDhDIkf26POcfySQ= 2024-04-13T22:15:36.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8HyA+hkp4DhFDhDIkf26POcfySQ=', 'Content-Length': '0'} 2024-04-13T22:15:36.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae721a6f508572db-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.176 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.176 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.176 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.176 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.176 DEBUG:boto:Data: 2024-04-13T22:15:36.176 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.176 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.176 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.176 DEBUG:boto:Params: {} 2024-04-13T22:15:36.176 DEBUG:boto:Token: None 2024-04-13T22:15:36.177 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.177 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8HyA+hkp4DhFDhDIkf26POcfySQ= 2024-04-13T22:15:36.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8HyA+hkp4DhFDhDIkf26POcfySQ=', 'Content-Length': '0'} 2024-04-13T22:15:36.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d297b7e382f7fac3-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-13T22:15:36.178 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.179 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.179 DEBUG:boto:Method: GET 2024-04-13T22:15:36.179 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.179 DEBUG:boto:Data: 2024-04-13T22:15:36.179 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.179 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.179 DEBUG:boto:Params: {} 2024-04-13T22:15:36.179 DEBUG:boto:Token: None 2024-04-13T22:15:36.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.180 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SXaa8opJCl0povkSBi5We/+KPdg= 2024-04-13T22:15:36.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SXaa8opJCl0povkSBi5We/+KPdg=', 'Content-Length': '0'} 2024-04-13T22:15:36.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005480904837002bbb-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.181 DEBUG:boto:path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.181 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.182 DEBUG:boto:Method: GET 2024-04-13T22:15:36.182 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.182 DEBUG:boto:Data: 2024-04-13T22:15:36.182 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.182 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.182 DEBUG:boto:Params: {} 2024-04-13T22:15:36.182 DEBUG:boto:Token: None 2024-04-13T22:15:36.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000049 2024-04-13T22:15:36.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SXaa8opJCl0povkSBi5We/+KPdg= 2024-04-13T22:15:36.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SXaa8opJCl0povkSBi5We/+KPdg=', 'Content-Length': '0'} 2024-04-13T22:15:36.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e08208c37015c82-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.185 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.185 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.185 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.185 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.185 DEBUG:boto:Method: GET 2024-04-13T22:15:36.185 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.185 DEBUG:boto:Data: 2024-04-13T22:15:36.185 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.185 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.186 DEBUG:boto:Params: {} 2024-04-13T22:15:36.186 DEBUG:boto:Token: None 2024-04-13T22:15:36.186 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af49874b4b7f7f50-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.188 DEBUG:boto:b'vwnfns-60obj000000000000000000491000falseobj000000000000000000492024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.189 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.189 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.189 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.189 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.189 DEBUG:boto:Method: GET 2024-04-13T22:15:36.189 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000049 2024-04-13T22:15:36.189 DEBUG:boto:Data: 2024-04-13T22:15:36.189 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.189 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.189 DEBUG:boto:Params: {} 2024-04-13T22:15:36.190 DEBUG:boto:Token: None 2024-04-13T22:15:36.190 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004349bff4b08ce5ca-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.194 DEBUG:boto:b'vwnfns-60obj000000000000000000491000falseobj000000000000000000492024-04-13T22:15:19.930Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-13T22:15:36.194 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.194 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.195 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.195 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.195 DEBUG:boto:Method: GET 2024-04-13T22:15:36.195 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.195 DEBUG:boto:Data: 2024-04-13T22:15:36.195 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.195 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.195 DEBUG:boto:Params: {} 2024-04-13T22:15:36.195 DEBUG:boto:Token: None 2024-04-13T22:15:36.197 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.197 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I= 2024-04-13T22:15:36.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I=', 'Content-Length': '0'} 2024-04-13T22:15:36.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000415e31d898bd7bba-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.199 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.199 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.200 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.200 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.200 DEBUG:boto:Method: GET 2024-04-13T22:15:36.200 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.200 DEBUG:boto:Data: 2024-04-13T22:15:36.200 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.200 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.200 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.200 DEBUG:boto:Params: {} 2024-04-13T22:15:36.201 DEBUG:boto:Token: None 2024-04-13T22:15:36.201 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.201 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I= 2024-04-13T22:15:36.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I=', 'Content-Length': '0'} 2024-04-13T22:15:36.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072178ce507e9a677-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.202 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.203 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.203 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.203 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.203 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.203 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.203 DEBUG:boto:Data: 2024-04-13T22:15:36.203 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.203 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.203 DEBUG:boto:Params: {} 2024-04-13T22:15:36.204 DEBUG:boto:Token: None 2024-04-13T22:15:36.204 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7obGeD8FbxjNWMoM28ycK6KjTWc= 2024-04-13T22:15:36.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7obGeD8FbxjNWMoM28ycK6KjTWc=', 'Content-Length': '0'} 2024-04-13T22:15:36.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8980f26cbb872df-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.205 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.206 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.206 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.206 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.206 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.206 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.206 DEBUG:boto:Data: 2024-04-13T22:15:36.206 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.206 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.206 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.206 DEBUG:boto:Params: {} 2024-04-13T22:15:36.207 DEBUG:boto:Token: None 2024-04-13T22:15:36.207 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7obGeD8FbxjNWMoM28ycK6KjTWc= 2024-04-13T22:15:36.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7obGeD8FbxjNWMoM28ycK6KjTWc=', 'Content-Length': '0'} 2024-04-13T22:15:36.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d8be67b218b39e-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-13T22:15:36.209 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.209 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.209 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.209 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.209 DEBUG:boto:Method: GET 2024-04-13T22:15:36.209 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.209 DEBUG:boto:Data: 2024-04-13T22:15:36.209 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.209 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.209 DEBUG:boto:Params: {} 2024-04-13T22:15:36.210 DEBUG:boto:Token: None 2024-04-13T22:15:36.210 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I= 2024-04-13T22:15:36.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I=', 'Content-Length': '0'} 2024-04-13T22:15:36.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095064b35cc50c403-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.212 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.212 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.212 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.212 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.212 DEBUG:boto:Method: GET 2024-04-13T22:15:36.212 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.212 DEBUG:boto:Data: 2024-04-13T22:15:36.212 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.212 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.212 DEBUG:boto:Params: {} 2024-04-13T22:15:36.213 DEBUG:boto:Token: None 2024-04-13T22:15:36.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050?versionId=null 2024-04-13T22:15:36.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I= 2024-04-13T22:15:36.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BlGNe20NF+0LXoUIJMI0DhIr02I=', 'Content-Length': '0'} 2024-04-13T22:15:36.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd32065749b76b0c-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.258 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.259 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.259 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.259 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.259 DEBUG:boto:Method: GET 2024-04-13T22:15:36.259 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.259 DEBUG:boto:Data: 2024-04-13T22:15:36.259 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.259 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.260 DEBUG:boto:Params: {} 2024-04-13T22:15:36.260 DEBUG:boto:Token: None 2024-04-13T22:15:36.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081a1eec0d5fdd5a7-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.262 DEBUG:boto:b'vwnfns-60obj000000000000000000501000falseobj000000000000000000502024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.263 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.263 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.263 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.263 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.263 DEBUG:boto:Method: GET 2024-04-13T22:15:36.263 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.263 DEBUG:boto:Data: 2024-04-13T22:15:36.263 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.264 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.264 DEBUG:boto:Params: {} 2024-04-13T22:15:36.264 DEBUG:boto:Token: None 2024-04-13T22:15:36.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000268895a2880e134b-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.268 DEBUG:boto:b'vwnfns-60obj000000000000000000501000falseobj000000000000000000502024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.268 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.268 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.268 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.269 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.269 DEBUG:boto:Data: 2024-04-13T22:15:36.269 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.269 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.269 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.269 DEBUG:boto:Params: {} 2024-04-13T22:15:36.269 DEBUG:boto:Token: None 2024-04-13T22:15:36.269 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+B4iNzT9CRN7dkAZHEqEIzpG4KE= 2024-04-13T22:15:36.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+B4iNzT9CRN7dkAZHEqEIzpG4KE=', 'Content-Length': '0'} 2024-04-13T22:15:36.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003004ca6d54f940b4-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.271 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.271 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.271 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.272 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.272 DEBUG:boto:Data: 2024-04-13T22:15:36.272 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.272 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.272 DEBUG:boto:Params: {} 2024-04-13T22:15:36.272 DEBUG:boto:Token: None 2024-04-13T22:15:36.272 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+B4iNzT9CRN7dkAZHEqEIzpG4KE= 2024-04-13T22:15:36.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+B4iNzT9CRN7dkAZHEqEIzpG4KE=', 'Content-Length': '0'} 2024-04-13T22:15:36.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9f85333d495d90-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-13T22:15:36.275 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.275 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.275 DEBUG:boto:Method: GET 2024-04-13T22:15:36.275 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.275 DEBUG:boto:Data: 2024-04-13T22:15:36.275 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.275 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.275 DEBUG:boto:Params: {} 2024-04-13T22:15:36.276 DEBUG:boto:Token: None 2024-04-13T22:15:36.276 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:03yuPCn9+HcRMiqaIhxm9iPUjn8= 2024-04-13T22:15:36.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:03yuPCn9+HcRMiqaIhxm9iPUjn8=', 'Content-Length': '0'} 2024-04-13T22:15:36.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c6d03bdc07a1da-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.318 DEBUG:boto:path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.319 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.319 DEBUG:boto:Method: GET 2024-04-13T22:15:36.319 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.319 DEBUG:boto:Data: 2024-04-13T22:15:36.319 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.319 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.319 DEBUG:boto:Params: {} 2024-04-13T22:15:36.319 DEBUG:boto:Token: None 2024-04-13T22:15:36.319 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000050 2024-04-13T22:15:36.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:03yuPCn9+HcRMiqaIhxm9iPUjn8= 2024-04-13T22:15:36.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:03yuPCn9+HcRMiqaIhxm9iPUjn8=', 'Content-Length': '0'} 2024-04-13T22:15:36.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000381ade2177701707-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.322 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.322 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.322 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.322 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.322 DEBUG:boto:Method: GET 2024-04-13T22:15:36.322 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.322 DEBUG:boto:Data: 2024-04-13T22:15:36.322 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.323 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.323 DEBUG:boto:Params: {} 2024-04-13T22:15:36.323 DEBUG:boto:Token: None 2024-04-13T22:15:36.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098ef9ac5d17877a6-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.325 DEBUG:boto:b'vwnfns-60obj000000000000000000501000falseobj000000000000000000502024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.326 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.326 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.326 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.326 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.326 DEBUG:boto:Method: GET 2024-04-13T22:15:36.326 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000050 2024-04-13T22:15:36.326 DEBUG:boto:Data: 2024-04-13T22:15:36.327 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.327 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.327 DEBUG:boto:Params: {} 2024-04-13T22:15:36.327 DEBUG:boto:Token: None 2024-04-13T22:15:36.327 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df78ecd1fa27606a-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.331 DEBUG:boto:b'vwnfns-60obj000000000000000000501000falseobj000000000000000000502024-04-13T22:15:20.916Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-13T22:15:36.331 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.331 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.331 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.331 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.331 DEBUG:boto:Method: GET 2024-04-13T22:15:36.332 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.332 DEBUG:boto:Data: 2024-04-13T22:15:36.332 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.332 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.332 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.332 DEBUG:boto:Params: {} 2024-04-13T22:15:36.332 DEBUG:boto:Token: None 2024-04-13T22:15:36.333 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.333 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w= 2024-04-13T22:15:36.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w=', 'Content-Length': '0'} 2024-04-13T22:15:36.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005af804d4b62bc87a-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.335 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.335 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.335 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.335 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.335 DEBUG:boto:Method: GET 2024-04-13T22:15:36.335 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.335 DEBUG:boto:Data: 2024-04-13T22:15:36.335 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.336 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.336 DEBUG:boto:Params: {} 2024-04-13T22:15:36.336 DEBUG:boto:Token: None 2024-04-13T22:15:36.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w= 2024-04-13T22:15:36.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w=', 'Content-Length': '0'} 2024-04-13T22:15:36.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f52468ec7e3b2773-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.378 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.379 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.379 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.379 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.379 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.379 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.379 DEBUG:boto:Data: 2024-04-13T22:15:36.379 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.379 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.379 DEBUG:boto:Params: {} 2024-04-13T22:15:36.379 DEBUG:boto:Token: None 2024-04-13T22:15:36.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e///z2Oyl8J3CU/704efo0Ylf0c= 2024-04-13T22:15:36.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e///z2Oyl8J3CU/704efo0Ylf0c=', 'Content-Length': '0'} 2024-04-13T22:15:36.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0de9ee0abaa24b7-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.382 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.382 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.382 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.382 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.382 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.382 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.382 DEBUG:boto:Data: 2024-04-13T22:15:36.382 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.382 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.383 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.383 DEBUG:boto:Params: {} 2024-04-13T22:15:36.383 DEBUG:boto:Token: None 2024-04-13T22:15:36.383 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.383 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e///z2Oyl8J3CU/704efo0Ylf0c= 2024-04-13T22:15:36.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e///z2Oyl8J3CU/704efo0Ylf0c=', 'Content-Length': '0'} 2024-04-13T22:15:36.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249a1c97712e7322-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-13T22:15:36.385 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.385 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.386 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.386 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.386 DEBUG:boto:Method: GET 2024-04-13T22:15:36.386 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.386 DEBUG:boto:Data: 2024-04-13T22:15:36.386 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.386 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.386 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.386 DEBUG:boto:Params: {} 2024-04-13T22:15:36.386 DEBUG:boto:Token: None 2024-04-13T22:15:36.387 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w= 2024-04-13T22:15:36.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w=', 'Content-Length': '0'} 2024-04-13T22:15:36.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af039ec5714db64c-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.389 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.389 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.389 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.389 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.389 DEBUG:boto:Method: GET 2024-04-13T22:15:36.389 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.389 DEBUG:boto:Data: 2024-04-13T22:15:36.389 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.389 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.389 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.389 DEBUG:boto:Params: {} 2024-04-13T22:15:36.390 DEBUG:boto:Token: None 2024-04-13T22:15:36.390 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053?versionId=null 2024-04-13T22:15:36.390 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w= 2024-04-13T22:15:36.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5/dy2W3xKye4Gs/dOcuiz0+3U4w=', 'Content-Length': '0'} 2024-04-13T22:15:36.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000168dc3638ff6be0b-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.392 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.392 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.392 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.393 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.393 DEBUG:boto:Method: GET 2024-04-13T22:15:36.393 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.393 DEBUG:boto:Data: 2024-04-13T22:15:36.393 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.393 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.393 DEBUG:boto:Params: {} 2024-04-13T22:15:36.393 DEBUG:boto:Token: None 2024-04-13T22:15:36.393 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.394 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c28ae0f74ef8a9-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.396 DEBUG:boto:b'vwnfns-60obj000000000000000000531000falseobj000000000000000000532024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.396 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.396 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.396 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.396 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.397 DEBUG:boto:Method: GET 2024-04-13T22:15:36.397 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.397 DEBUG:boto:Data: 2024-04-13T22:15:36.397 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.397 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.397 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.397 DEBUG:boto:Params: {} 2024-04-13T22:15:36.397 DEBUG:boto:Token: None 2024-04-13T22:15:36.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.398 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672d0540dcb53837-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.402 DEBUG:boto:b'vwnfns-60obj000000000000000000531000falseobj000000000000000000532024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.402 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.402 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.403 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.403 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.403 DEBUG:boto:Data: 2024-04-13T22:15:36.405 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.406 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.406 DEBUG:boto:Params: {} 2024-04-13T22:15:36.406 DEBUG:boto:Token: None 2024-04-13T22:15:36.406 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AXBGSK2pVMu4hg7ZqcGjfSTTUHA= 2024-04-13T22:15:36.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AXBGSK2pVMu4hg7ZqcGjfSTTUHA=', 'Content-Length': '0'} 2024-04-13T22:15:36.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a0d9dab7fb6cd6-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.408 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.408 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.408 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.408 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.409 DEBUG:boto:Data: 2024-04-13T22:15:36.409 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.409 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.409 DEBUG:boto:Params: {} 2024-04-13T22:15:36.409 DEBUG:boto:Token: None 2024-04-13T22:15:36.409 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.409 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AXBGSK2pVMu4hg7ZqcGjfSTTUHA= 2024-04-13T22:15:36.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AXBGSK2pVMu4hg7ZqcGjfSTTUHA=', 'Content-Length': '0'} 2024-04-13T22:15:36.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13aba63285e42c7-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-13T22:15:36.411 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.411 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.411 DEBUG:boto:Method: GET 2024-04-13T22:15:36.411 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.411 DEBUG:boto:Data: 2024-04-13T22:15:36.412 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.412 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.412 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.412 DEBUG:boto:Params: {} 2024-04-13T22:15:36.412 DEBUG:boto:Token: None 2024-04-13T22:15:36.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+XHns7XZ2a17ZY90vJFL1N1rWO8= 2024-04-13T22:15:36.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+XHns7XZ2a17ZY90vJFL1N1rWO8=', 'Content-Length': '0'} 2024-04-13T22:15:36.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa95b9c30d5a4e89-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.414 DEBUG:boto:path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.414 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.414 DEBUG:boto:Method: GET 2024-04-13T22:15:36.414 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.414 DEBUG:boto:Data: 2024-04-13T22:15:36.415 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.415 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.415 DEBUG:boto:Params: {} 2024-04-13T22:15:36.415 DEBUG:boto:Token: None 2024-04-13T22:15:36.415 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000053 2024-04-13T22:15:36.415 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+XHns7XZ2a17ZY90vJFL1N1rWO8= 2024-04-13T22:15:36.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+XHns7XZ2a17ZY90vJFL1N1rWO8=', 'Content-Length': '0'} 2024-04-13T22:15:36.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8393dabeb5a167-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.458 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.459 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.459 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.459 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.459 DEBUG:boto:Method: GET 2024-04-13T22:15:36.459 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.459 DEBUG:boto:Data: 2024-04-13T22:15:36.459 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.459 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.459 DEBUG:boto:Params: {} 2024-04-13T22:15:36.460 DEBUG:boto:Token: None 2024-04-13T22:15:36.460 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004592dbfd43075c17-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.462 DEBUG:boto:b'vwnfns-60obj000000000000000000531000falseobj000000000000000000532024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.463 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.463 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.463 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.463 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.463 DEBUG:boto:Method: GET 2024-04-13T22:15:36.463 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000053 2024-04-13T22:15:36.463 DEBUG:boto:Data: 2024-04-13T22:15:36.463 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.464 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.464 DEBUG:boto:Params: {} 2024-04-13T22:15:36.464 DEBUG:boto:Token: None 2024-04-13T22:15:36.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b8f4d63cc08f603-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.468 DEBUG:boto:b'vwnfns-60obj000000000000000000531000falseobj000000000000000000532024-04-13T22:15:22.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-13T22:15:36.468 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.468 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.468 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.468 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.469 DEBUG:boto:Method: GET 2024-04-13T22:15:36.469 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.469 DEBUG:boto:Data: 2024-04-13T22:15:36.469 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.469 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.469 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.469 DEBUG:boto:Params: {} 2024-04-13T22:15:36.469 DEBUG:boto:Token: None 2024-04-13T22:15:36.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y= 2024-04-13T22:15:36.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y=', 'Content-Length': '0'} 2024-04-13T22:15:36.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004797533b21569015-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.472 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.472 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.472 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.472 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.472 DEBUG:boto:Method: GET 2024-04-13T22:15:36.472 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.472 DEBUG:boto:Data: 2024-04-13T22:15:36.472 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.472 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.473 DEBUG:boto:Params: {} 2024-04-13T22:15:36.473 DEBUG:boto:Token: None 2024-04-13T22:15:36.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y= 2024-04-13T22:15:36.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y=', 'Content-Length': '0'} 2024-04-13T22:15:36.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3395c06e8af982c-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.475 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.475 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.475 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.475 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.475 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.475 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.475 DEBUG:boto:Data: 2024-04-13T22:15:36.476 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.476 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.476 DEBUG:boto:Params: {} 2024-04-13T22:15:36.476 DEBUG:boto:Token: None 2024-04-13T22:15:36.476 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V9WdrA1cJQoo+gHJKPHGpTxajok= 2024-04-13T22:15:36.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V9WdrA1cJQoo+gHJKPHGpTxajok=', 'Content-Length': '0'} 2024-04-13T22:15:36.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f005297005b75af-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.478 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.478 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.478 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.478 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.478 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.478 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.478 DEBUG:boto:Data: 2024-04-13T22:15:36.479 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.479 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.479 DEBUG:boto:Params: {} 2024-04-13T22:15:36.479 DEBUG:boto:Token: None 2024-04-13T22:15:36.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V9WdrA1cJQoo+gHJKPHGpTxajok= 2024-04-13T22:15:36.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V9WdrA1cJQoo+gHJKPHGpTxajok=', 'Content-Length': '0'} 2024-04-13T22:15:36.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c883c8e7695ee0-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-13T22:15:36.481 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.481 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.481 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.481 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.481 DEBUG:boto:Method: GET 2024-04-13T22:15:36.481 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.482 DEBUG:boto:Data: 2024-04-13T22:15:36.482 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.482 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.482 DEBUG:boto:Params: {} 2024-04-13T22:15:36.482 DEBUG:boto:Token: None 2024-04-13T22:15:36.482 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.482 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y= 2024-04-13T22:15:36.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y=', 'Content-Length': '0'} 2024-04-13T22:15:36.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a239bde6724647-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.484 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.484 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.484 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.484 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.484 DEBUG:boto:Method: GET 2024-04-13T22:15:36.485 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.485 DEBUG:boto:Data: 2024-04-13T22:15:36.485 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.485 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.485 DEBUG:boto:Params: {} 2024-04-13T22:15:36.485 DEBUG:boto:Token: None 2024-04-13T22:15:36.485 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055?versionId=null 2024-04-13T22:15:36.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y= 2024-04-13T22:15:36.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ThSKbAmN3ybppP+kRG2fXb7qx6Y=', 'Content-Length': '0'} 2024-04-13T22:15:36.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f56035272928bea-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.534 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.535 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.535 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.535 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.535 DEBUG:boto:Method: GET 2024-04-13T22:15:36.535 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.535 DEBUG:boto:Data: 2024-04-13T22:15:36.535 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.535 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.535 DEBUG:boto:Params: {} 2024-04-13T22:15:36.536 DEBUG:boto:Token: None 2024-04-13T22:15:36.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000209961a7e9536e6-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.538 DEBUG:boto:b'vwnfns-60obj000000000000000000551000falseobj000000000000000000552024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.539 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.539 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.539 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.539 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.539 DEBUG:boto:Method: GET 2024-04-13T22:15:36.539 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.539 DEBUG:boto:Data: 2024-04-13T22:15:36.539 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.539 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.540 DEBUG:boto:Params: {} 2024-04-13T22:15:36.540 DEBUG:boto:Token: None 2024-04-13T22:15:36.540 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e3fed47f4e7d615-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.544 DEBUG:boto:b'vwnfns-60obj000000000000000000551000falseobj000000000000000000552024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.544 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.544 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.544 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.544 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.544 DEBUG:boto:Data: 2024-04-13T22:15:36.545 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.545 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.545 DEBUG:boto:Params: {} 2024-04-13T22:15:36.545 DEBUG:boto:Token: None 2024-04-13T22:15:36.545 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zLfsjj7PUcIhkeX1NTFze6nVD3Y= 2024-04-13T22:15:36.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zLfsjj7PUcIhkeX1NTFze6nVD3Y=', 'Content-Length': '0'} 2024-04-13T22:15:36.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d59e049f9c219bd-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.547 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.547 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.547 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.547 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.547 DEBUG:boto:Data: 2024-04-13T22:15:36.547 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.548 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.548 DEBUG:boto:Params: {} 2024-04-13T22:15:36.548 DEBUG:boto:Token: None 2024-04-13T22:15:36.548 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zLfsjj7PUcIhkeX1NTFze6nVD3Y= 2024-04-13T22:15:36.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zLfsjj7PUcIhkeX1NTFze6nVD3Y=', 'Content-Length': '0'} 2024-04-13T22:15:36.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006000fb69c0f153b6-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-13T22:15:36.550 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.550 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.551 DEBUG:boto:Method: GET 2024-04-13T22:15:36.551 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.551 DEBUG:boto:Data: 2024-04-13T22:15:36.551 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.551 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.551 DEBUG:boto:Params: {} 2024-04-13T22:15:36.551 DEBUG:boto:Token: None 2024-04-13T22:15:36.551 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0NVbFP3xtzB/Vpj6AkvOWWkyxhM= 2024-04-13T22:15:36.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0NVbFP3xtzB/Vpj6AkvOWWkyxhM=', 'Content-Length': '0'} 2024-04-13T22:15:36.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6cd829fe1431f7b-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.594 DEBUG:boto:path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.595 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.595 DEBUG:boto:Method: GET 2024-04-13T22:15:36.595 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.595 DEBUG:boto:Data: 2024-04-13T22:15:36.595 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.595 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.596 DEBUG:boto:Params: {} 2024-04-13T22:15:36.596 DEBUG:boto:Token: None 2024-04-13T22:15:36.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000055 2024-04-13T22:15:36.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0NVbFP3xtzB/Vpj6AkvOWWkyxhM= 2024-04-13T22:15:36.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0NVbFP3xtzB/Vpj6AkvOWWkyxhM=', 'Content-Length': '0'} 2024-04-13T22:15:36.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087116f42f6fbc874-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.598 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.598 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.598 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.598 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.599 DEBUG:boto:Method: GET 2024-04-13T22:15:36.599 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.599 DEBUG:boto:Data: 2024-04-13T22:15:36.599 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.599 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.599 DEBUG:boto:Params: {} 2024-04-13T22:15:36.599 DEBUG:boto:Token: None 2024-04-13T22:15:36.599 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af654b3d88f0286-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.602 DEBUG:boto:b'vwnfns-60obj000000000000000000551000falseobj000000000000000000552024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.602 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.602 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.602 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.602 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.602 DEBUG:boto:Method: GET 2024-04-13T22:15:36.603 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000055 2024-04-13T22:15:36.603 DEBUG:boto:Data: 2024-04-13T22:15:36.603 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.603 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.603 DEBUG:boto:Params: {} 2024-04-13T22:15:36.603 DEBUG:boto:Token: None 2024-04-13T22:15:36.603 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.603 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000894fa747f3a28b33-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.610 DEBUG:boto:b'vwnfns-60obj000000000000000000551000falseobj000000000000000000552024-04-13T22:15:19.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-13T22:15:36.610 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.610 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.611 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.611 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.611 DEBUG:boto:Method: GET 2024-04-13T22:15:36.611 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.611 DEBUG:boto:Data: 2024-04-13T22:15:36.611 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.611 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.611 DEBUG:boto:Params: {} 2024-04-13T22:15:36.611 DEBUG:boto:Token: None 2024-04-13T22:15:36.612 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8= 2024-04-13T22:15:36.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8=', 'Content-Length': '0'} 2024-04-13T22:15:36.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7a7f132a707778-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.614 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.614 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.614 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.614 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.614 DEBUG:boto:Method: GET 2024-04-13T22:15:36.614 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.614 DEBUG:boto:Data: 2024-04-13T22:15:36.614 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.614 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.615 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.615 DEBUG:boto:Params: {} 2024-04-13T22:15:36.615 DEBUG:boto:Token: None 2024-04-13T22:15:36.615 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8= 2024-04-13T22:15:36.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8=', 'Content-Length': '0'} 2024-04-13T22:15:36.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342598654fdab6a0-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.658 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.659 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.659 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.659 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.659 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.659 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.659 DEBUG:boto:Data: 2024-04-13T22:15:36.659 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.659 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.659 DEBUG:boto:Params: {} 2024-04-13T22:15:36.660 DEBUG:boto:Token: None 2024-04-13T22:15:36.660 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:swEGV0/y2a26+ALaQHpmyvhVTM0= 2024-04-13T22:15:36.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:swEGV0/y2a26+ALaQHpmyvhVTM0=', 'Content-Length': '0'} 2024-04-13T22:15:36.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000262a21b8176f91a1-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.662 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.662 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.663 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.663 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.663 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.663 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.663 DEBUG:boto:Data: 2024-04-13T22:15:36.663 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.663 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.663 DEBUG:boto:Params: {} 2024-04-13T22:15:36.664 DEBUG:boto:Token: None 2024-04-13T22:15:36.664 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:swEGV0/y2a26+ALaQHpmyvhVTM0= 2024-04-13T22:15:36.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:swEGV0/y2a26+ALaQHpmyvhVTM0=', 'Content-Length': '0'} 2024-04-13T22:15:36.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5d654a3bd476cc-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-13T22:15:36.666 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.666 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.666 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.667 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.667 DEBUG:boto:Method: GET 2024-04-13T22:15:36.667 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.667 DEBUG:boto:Data: 2024-04-13T22:15:36.667 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.667 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.667 DEBUG:boto:Params: {} 2024-04-13T22:15:36.667 DEBUG:boto:Token: None 2024-04-13T22:15:36.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.668 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8= 2024-04-13T22:15:36.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8=', 'Content-Length': '0'} 2024-04-13T22:15:36.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2c300d6b5f28c0-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.669 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.669 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.669 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.670 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.670 DEBUG:boto:Method: GET 2024-04-13T22:15:36.670 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.670 DEBUG:boto:Data: 2024-04-13T22:15:36.670 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.670 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.670 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.670 DEBUG:boto:Params: {} 2024-04-13T22:15:36.670 DEBUG:boto:Token: None 2024-04-13T22:15:36.670 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056?versionId=null 2024-04-13T22:15:36.671 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8= 2024-04-13T22:15:36.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bwf6OvTaZ4NjtMtXdx2vfGxrLd8=', 'Content-Length': '0'} 2024-04-13T22:15:36.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000626a581e5dd67fdc-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.672 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.673 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.673 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.673 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.673 DEBUG:boto:Method: GET 2024-04-13T22:15:36.673 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.673 DEBUG:boto:Data: 2024-04-13T22:15:36.673 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.673 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.673 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.673 DEBUG:boto:Params: {} 2024-04-13T22:15:36.674 DEBUG:boto:Token: None 2024-04-13T22:15:36.674 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.674 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001edbfd637e418a90-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.676 DEBUG:boto:b'vwnfns-60obj000000000000000000561000falseobj000000000000000000562024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.676 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.677 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.677 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.677 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.677 DEBUG:boto:Method: GET 2024-04-13T22:15:36.677 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.677 DEBUG:boto:Data: 2024-04-13T22:15:36.677 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.677 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.677 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.677 DEBUG:boto:Params: {} 2024-04-13T22:15:36.678 DEBUG:boto:Token: None 2024-04-13T22:15:36.678 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.678 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097128daea9dfe817-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.681 DEBUG:boto:b'vwnfns-60obj000000000000000000561000falseobj000000000000000000562024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.682 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.682 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.682 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.682 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.682 DEBUG:boto:Data: 2024-04-13T22:15:36.682 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.682 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.682 DEBUG:boto:Params: {} 2024-04-13T22:15:36.683 DEBUG:boto:Token: None 2024-04-13T22:15:36.683 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.683 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:USORzzGQ+CGYwX3u0BsRNiATjZ4= 2024-04-13T22:15:36.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:USORzzGQ+CGYwX3u0BsRNiATjZ4=', 'Content-Length': '0'} 2024-04-13T22:15:36.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f2597a87db5945-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.684 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.685 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.685 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.685 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.685 DEBUG:boto:Data: 2024-04-13T22:15:36.685 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.685 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.685 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.685 DEBUG:boto:Params: {} 2024-04-13T22:15:36.685 DEBUG:boto:Token: None 2024-04-13T22:15:36.686 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.686 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:USORzzGQ+CGYwX3u0BsRNiATjZ4= 2024-04-13T22:15:36.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:USORzzGQ+CGYwX3u0BsRNiATjZ4=', 'Content-Length': '0'} 2024-04-13T22:15:36.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0d52dbc341fbff-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-13T22:15:36.688 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.688 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.688 DEBUG:boto:Method: GET 2024-04-13T22:15:36.688 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.688 DEBUG:boto:Data: 2024-04-13T22:15:36.688 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.688 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.688 DEBUG:boto:Params: {} 2024-04-13T22:15:36.689 DEBUG:boto:Token: None 2024-04-13T22:15:36.689 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dxeugUphU+RAE9wZS0u6uoRfMmk= 2024-04-13T22:15:36.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dxeugUphU+RAE9wZS0u6uoRfMmk=', 'Content-Length': '0'} 2024-04-13T22:15:36.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000084dc35931714ce4-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.690 DEBUG:boto:path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.691 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.691 DEBUG:boto:Method: GET 2024-04-13T22:15:36.691 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.691 DEBUG:boto:Data: 2024-04-13T22:15:36.691 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.691 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.691 DEBUG:boto:Params: {} 2024-04-13T22:15:36.691 DEBUG:boto:Token: None 2024-04-13T22:15:36.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000056 2024-04-13T22:15:36.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dxeugUphU+RAE9wZS0u6uoRfMmk= 2024-04-13T22:15:36.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dxeugUphU+RAE9wZS0u6uoRfMmk=', 'Content-Length': '0'} 2024-04-13T22:15:36.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffb2222611039fc-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.694 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.694 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.694 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.694 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.694 DEBUG:boto:Method: GET 2024-04-13T22:15:36.694 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.694 DEBUG:boto:Data: 2024-04-13T22:15:36.694 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.694 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.695 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.695 DEBUG:boto:Params: {} 2024-04-13T22:15:36.695 DEBUG:boto:Token: None 2024-04-13T22:15:36.695 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.695 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ecd27774280a4d2-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.697 DEBUG:boto:b'vwnfns-60obj000000000000000000561000falseobj000000000000000000562024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.698 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.698 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.698 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.698 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.698 DEBUG:boto:Method: GET 2024-04-13T22:15:36.698 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000056 2024-04-13T22:15:36.698 DEBUG:boto:Data: 2024-04-13T22:15:36.698 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.699 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.699 DEBUG:boto:Params: {} 2024-04-13T22:15:36.699 DEBUG:boto:Token: None 2024-04-13T22:15:36.699 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c51d82b7946f0a38-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.703 DEBUG:boto:b'vwnfns-60obj000000000000000000561000falseobj000000000000000000562024-04-13T22:15:20.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-13T22:15:36.703 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.703 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.704 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.704 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.704 DEBUG:boto:Method: GET 2024-04-13T22:15:36.704 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.704 DEBUG:boto:Data: 2024-04-13T22:15:36.704 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.704 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.704 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.704 DEBUG:boto:Params: {} 2024-04-13T22:15:36.704 DEBUG:boto:Token: None 2024-04-13T22:15:36.704 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg= 2024-04-13T22:15:36.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg=', 'Content-Length': '0'} 2024-04-13T22:15:36.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c5936a4d8cdad7-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.707 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.707 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.707 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.707 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.707 DEBUG:boto:Method: GET 2024-04-13T22:15:36.707 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.707 DEBUG:boto:Data: 2024-04-13T22:15:36.707 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.707 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.708 DEBUG:boto:Params: {} 2024-04-13T22:15:36.708 DEBUG:boto:Token: None 2024-04-13T22:15:36.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg= 2024-04-13T22:15:36.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg=', 'Content-Length': '0'} 2024-04-13T22:15:36.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ff4e0574bbb521-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.710 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.710 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.710 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.710 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.710 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.710 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.710 DEBUG:boto:Data: 2024-04-13T22:15:36.710 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.711 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.711 DEBUG:boto:Params: {} 2024-04-13T22:15:36.711 DEBUG:boto:Token: None 2024-04-13T22:15:36.711 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pNkPfhUVjmClqhja2rwqgWo6DQM= 2024-04-13T22:15:36.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pNkPfhUVjmClqhja2rwqgWo6DQM=', 'Content-Length': '0'} 2024-04-13T22:15:36.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e4981fbbef6df3-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.713 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.713 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.713 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.713 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.713 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.713 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.713 DEBUG:boto:Data: 2024-04-13T22:15:36.714 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.714 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.714 DEBUG:boto:Params: {} 2024-04-13T22:15:36.714 DEBUG:boto:Token: None 2024-04-13T22:15:36.714 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pNkPfhUVjmClqhja2rwqgWo6DQM= 2024-04-13T22:15:36.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pNkPfhUVjmClqhja2rwqgWo6DQM=', 'Content-Length': '0'} 2024-04-13T22:15:36.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f24971c9a25e57-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-13T22:15:36.716 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.716 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.717 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.717 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.717 DEBUG:boto:Method: GET 2024-04-13T22:15:36.717 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.717 DEBUG:boto:Data: 2024-04-13T22:15:36.717 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.717 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.717 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.717 DEBUG:boto:Params: {} 2024-04-13T22:15:36.717 DEBUG:boto:Token: None 2024-04-13T22:15:36.718 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg= 2024-04-13T22:15:36.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg=', 'Content-Length': '0'} 2024-04-13T22:15:36.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d84614ec610f45b-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.766 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.767 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.767 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.767 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.767 DEBUG:boto:Method: GET 2024-04-13T22:15:36.767 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.767 DEBUG:boto:Data: 2024-04-13T22:15:36.767 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.767 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.767 DEBUG:boto:Params: {} 2024-04-13T22:15:36.768 DEBUG:boto:Token: None 2024-04-13T22:15:36.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057?versionId=null 2024-04-13T22:15:36.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg= 2024-04-13T22:15:36.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q+0Z1/pTyubiPrmGlOHLkl2G3Xg=', 'Content-Length': '0'} 2024-04-13T22:15:36.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3818cc90e804201-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.770 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.770 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.770 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.770 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.770 DEBUG:boto:Method: GET 2024-04-13T22:15:36.771 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.771 DEBUG:boto:Data: 2024-04-13T22:15:36.771 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.771 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.771 DEBUG:boto:Params: {} 2024-04-13T22:15:36.771 DEBUG:boto:Token: None 2024-04-13T22:15:36.771 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.771 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009590a66640f8f216-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.774 DEBUG:boto:b'vwnfns-60obj000000000000000000571000falseobj000000000000000000572024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.774 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.774 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.774 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.774 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.774 DEBUG:boto:Method: GET 2024-04-13T22:15:36.774 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.775 DEBUG:boto:Data: 2024-04-13T22:15:36.775 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.775 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.775 DEBUG:boto:Params: {} 2024-04-13T22:15:36.775 DEBUG:boto:Token: None 2024-04-13T22:15:36.775 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.775 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d22f3872ede39d-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.779 DEBUG:boto:b'vwnfns-60obj000000000000000000571000falseobj000000000000000000572024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.779 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.779 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.779 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.779 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.780 DEBUG:boto:Data: 2024-04-13T22:15:36.780 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.780 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.780 DEBUG:boto:Params: {} 2024-04-13T22:15:36.780 DEBUG:boto:Token: None 2024-04-13T22:15:36.780 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rgUPusDuUJ2DayNSLlM5cPKT0VY= 2024-04-13T22:15:36.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rgUPusDuUJ2DayNSLlM5cPKT0VY=', 'Content-Length': '0'} 2024-04-13T22:15:36.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c35e7921d15faf8-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.782 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.782 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.782 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.782 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.783 DEBUG:boto:Data: 2024-04-13T22:15:36.783 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.783 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.783 DEBUG:boto:Params: {} 2024-04-13T22:15:36.783 DEBUG:boto:Token: None 2024-04-13T22:15:36.783 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rgUPusDuUJ2DayNSLlM5cPKT0VY= 2024-04-13T22:15:36.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rgUPusDuUJ2DayNSLlM5cPKT0VY=', 'Content-Length': '0'} 2024-04-13T22:15:36.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1fe39748b0cb5d-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-13T22:15:36.785 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.786 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.786 DEBUG:boto:Method: GET 2024-04-13T22:15:36.786 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.786 DEBUG:boto:Data: 2024-04-13T22:15:36.786 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.786 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.786 DEBUG:boto:Params: {} 2024-04-13T22:15:36.786 DEBUG:boto:Token: None 2024-04-13T22:15:36.786 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JGdp4eWTNzkn8S0AdayutqhOtg4= 2024-04-13T22:15:36.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JGdp4eWTNzkn8S0AdayutqhOtg4=', 'Content-Length': '0'} 2024-04-13T22:15:36.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d4cf08c177e731-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.790 DEBUG:boto:path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.790 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.790 DEBUG:boto:Method: GET 2024-04-13T22:15:36.790 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.790 DEBUG:boto:Data: 2024-04-13T22:15:36.790 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.791 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.791 DEBUG:boto:Params: {} 2024-04-13T22:15:36.791 DEBUG:boto:Token: None 2024-04-13T22:15:36.791 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000057 2024-04-13T22:15:36.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JGdp4eWTNzkn8S0AdayutqhOtg4= 2024-04-13T22:15:36.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JGdp4eWTNzkn8S0AdayutqhOtg4=', 'Content-Length': '0'} 2024-04-13T22:15:36.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d53cfc15d8a722a-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.834 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.834 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.835 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.835 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.835 DEBUG:boto:Method: GET 2024-04-13T22:15:36.835 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.835 DEBUG:boto:Data: 2024-04-13T22:15:36.835 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.835 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.835 DEBUG:boto:Params: {} 2024-04-13T22:15:36.835 DEBUG:boto:Token: None 2024-04-13T22:15:36.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cae400aed244de8a-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.838 DEBUG:boto:b'vwnfns-60obj000000000000000000571000falseobj000000000000000000572024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.839 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.839 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.839 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.839 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.839 DEBUG:boto:Method: GET 2024-04-13T22:15:36.839 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000057 2024-04-13T22:15:36.840 DEBUG:boto:Data: 2024-04-13T22:15:36.840 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.840 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.840 DEBUG:boto:Params: {} 2024-04-13T22:15:36.840 DEBUG:boto:Token: None 2024-04-13T22:15:36.840 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000098a0ef46791458-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.844 DEBUG:boto:b'vwnfns-60obj000000000000000000571000falseobj000000000000000000572024-04-13T22:15:22.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-13T22:15:36.844 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.844 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.845 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.845 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.845 DEBUG:boto:Method: GET 2024-04-13T22:15:36.845 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.845 DEBUG:boto:Data: 2024-04-13T22:15:36.845 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.845 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.845 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.845 DEBUG:boto:Params: {} 2024-04-13T22:15:36.845 DEBUG:boto:Token: None 2024-04-13T22:15:36.845 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.846 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q= 2024-04-13T22:15:36.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q=', 'Content-Length': '0'} 2024-04-13T22:15:36.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4854814426bc7b9-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.848 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.848 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.848 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.848 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.848 DEBUG:boto:Method: GET 2024-04-13T22:15:36.848 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.849 DEBUG:boto:Data: 2024-04-13T22:15:36.849 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.849 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.849 DEBUG:boto:Params: {} 2024-04-13T22:15:36.849 DEBUG:boto:Token: None 2024-04-13T22:15:36.849 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q= 2024-04-13T22:15:36.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q=', 'Content-Length': '0'} 2024-04-13T22:15:36.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008550bce55a021d7b-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.851 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.851 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.851 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.851 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.852 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.852 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.852 DEBUG:boto:Data: 2024-04-13T22:15:36.852 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.852 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.852 DEBUG:boto:Params: {} 2024-04-13T22:15:36.852 DEBUG:boto:Token: None 2024-04-13T22:15:36.852 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ptbvPSHDtqSfKO7LPiJpgSjKKVA= 2024-04-13T22:15:36.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ptbvPSHDtqSfKO7LPiJpgSjKKVA=', 'Content-Length': '0'} 2024-04-13T22:15:36.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c4f0b6c61caaef-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.854 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.854 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.854 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.855 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.855 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.855 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.855 DEBUG:boto:Data: 2024-04-13T22:15:36.855 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.855 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.855 DEBUG:boto:Params: {} 2024-04-13T22:15:36.855 DEBUG:boto:Token: None 2024-04-13T22:15:36.855 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ptbvPSHDtqSfKO7LPiJpgSjKKVA= 2024-04-13T22:15:36.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ptbvPSHDtqSfKO7LPiJpgSjKKVA=', 'Content-Length': '0'} 2024-04-13T22:15:36.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000294786a708950ee9-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-13T22:15:36.858 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.858 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.858 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.858 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.858 DEBUG:boto:Method: GET 2024-04-13T22:15:36.858 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.858 DEBUG:boto:Data: 2024-04-13T22:15:36.858 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.858 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.859 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.859 DEBUG:boto:Params: {} 2024-04-13T22:15:36.859 DEBUG:boto:Token: None 2024-04-13T22:15:36.859 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q= 2024-04-13T22:15:36.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q=', 'Content-Length': '0'} 2024-04-13T22:15:36.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56d61d6aab2b88f-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.861 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.861 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.861 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.861 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.861 DEBUG:boto:Method: GET 2024-04-13T22:15:36.861 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.862 DEBUG:boto:Data: 2024-04-13T22:15:36.862 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.862 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.862 DEBUG:boto:Params: {} 2024-04-13T22:15:36.862 DEBUG:boto:Token: None 2024-04-13T22:15:36.863 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058?versionId=null 2024-04-13T22:15:36.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q= 2024-04-13T22:15:36.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t3kbZ6MobfvXQWJ2Ea5HL0xFu9Q=', 'Content-Length': '0'} 2024-04-13T22:15:36.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163e3e6047e9c1dc-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.906 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.907 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.907 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.907 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.907 DEBUG:boto:Method: GET 2024-04-13T22:15:36.907 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.907 DEBUG:boto:Data: 2024-04-13T22:15:36.907 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.907 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.907 DEBUG:boto:Params: {} 2024-04-13T22:15:36.908 DEBUG:boto:Token: None 2024-04-13T22:15:36.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bab9603cd5fd460-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.910 DEBUG:boto:b'vwnfns-60obj000000000000000000581000falseobj000000000000000000582024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.911 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.911 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.911 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.911 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.911 DEBUG:boto:Method: GET 2024-04-13T22:15:36.911 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.911 DEBUG:boto:Data: 2024-04-13T22:15:36.912 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.912 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.912 DEBUG:boto:Params: {} 2024-04-13T22:15:36.912 DEBUG:boto:Token: None 2024-04-13T22:15:36.912 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e17f817b6e2c1ed-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.916 DEBUG:boto:b'vwnfns-60obj000000000000000000581000falseobj000000000000000000582024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.916 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.917 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.917 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.917 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.917 DEBUG:boto:Data: 2024-04-13T22:15:36.917 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.917 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.917 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.917 DEBUG:boto:Params: {} 2024-04-13T22:15:36.917 DEBUG:boto:Token: None 2024-04-13T22:15:36.917 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.918 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7NwCL3Ke+tXG9T6h/ooTC5W3HA= 2024-04-13T22:15:36.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7NwCL3Ke+tXG9T6h/ooTC5W3HA=', 'Content-Length': '0'} 2024-04-13T22:15:36.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf25138b39e2164-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.919 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.919 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.919 DEBUG:boto:Method: HEAD 2024-04-13T22:15:36.920 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.920 DEBUG:boto:Data: 2024-04-13T22:15:36.920 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.920 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.920 DEBUG:boto:Params: {} 2024-04-13T22:15:36.920 DEBUG:boto:Token: None 2024-04-13T22:15:36.920 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d7NwCL3Ke+tXG9T6h/ooTC5W3HA= 2024-04-13T22:15:36.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d7NwCL3Ke+tXG9T6h/ooTC5W3HA=', 'Content-Length': '0'} 2024-04-13T22:15:36.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023847cdf90331ed5-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-13T22:15:36.922 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.922 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.922 DEBUG:boto:Method: GET 2024-04-13T22:15:36.923 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.923 DEBUG:boto:Data: 2024-04-13T22:15:36.923 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.923 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.923 DEBUG:boto:Params: {} 2024-04-13T22:15:36.923 DEBUG:boto:Token: None 2024-04-13T22:15:36.923 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gmaX8JkCk2dBnqQImbZ6hK9d8Ec= 2024-04-13T22:15:36.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gmaX8JkCk2dBnqQImbZ6hK9d8Ec=', 'Content-Length': '0'} 2024-04-13T22:15:36.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b42b4bea3c770d4-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.966 DEBUG:boto:path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.966 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.967 DEBUG:boto:Method: GET 2024-04-13T22:15:36.967 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.967 DEBUG:boto:Data: 2024-04-13T22:15:36.967 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.967 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.967 DEBUG:boto:Params: {} 2024-04-13T22:15:36.967 DEBUG:boto:Token: None 2024-04-13T22:15:36.967 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000058 2024-04-13T22:15:36.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gmaX8JkCk2dBnqQImbZ6hK9d8Ec= 2024-04-13T22:15:36.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gmaX8JkCk2dBnqQImbZ6hK9d8Ec=', 'Content-Length': '0'} 2024-04-13T22:15:36.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018b2f08334cd2b7f-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.969 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.970 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.970 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.970 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.970 DEBUG:boto:Method: GET 2024-04-13T22:15:36.970 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.970 DEBUG:boto:Data: 2024-04-13T22:15:36.970 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.970 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.970 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.970 DEBUG:boto:Params: {} 2024-04-13T22:15:36.971 DEBUG:boto:Token: None 2024-04-13T22:15:36.971 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511981237e75d88b-00661b0408-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.975 DEBUG:boto:b'vwnfns-60obj000000000000000000581000falseobj000000000000000000582024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.975 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:36.975 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:36.975 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.976 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.976 DEBUG:boto:Method: GET 2024-04-13T22:15:36.976 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000058 2024-04-13T22:15:36.976 DEBUG:boto:Data: 2024-04-13T22:15:36.976 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.976 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.976 DEBUG:boto:Params: {} 2024-04-13T22:15:36.976 DEBUG:boto:Token: None 2024-04-13T22:15:36.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/ 2024-04-13T22:15:36.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA= 2024-04-13T22:15:36.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbgo2epsRjy2Sy7wCLSYDKDYMOA=', 'Content-Length': '0'} 2024-04-13T22:15:36.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9681db5df4f83fa-00661b0408-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.980 DEBUG:boto:b'vwnfns-60obj000000000000000000581000falseobj000000000000000000582024-04-13T22:15:19.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:36.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-13T22:15:36.981 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:36.981 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:36.981 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.981 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.981 DEBUG:boto:Method: GET 2024-04-13T22:15:36.981 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.982 DEBUG:boto:Data: 2024-04-13T22:15:36.982 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.982 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:36.982 DEBUG:boto:Port: 8001 2024-04-13T22:15:36.982 DEBUG:boto:Params: {} 2024-04-13T22:15:36.982 DEBUG:boto:Token: None 2024-04-13T22:15:36.982 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.982 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wMbEoA0ZJa/FO7g8vPaMWeeKSLk= 2024-04-13T22:15:36.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wMbEoA0ZJa/FO7g8vPaMWeeKSLk=', 'Content-Length': '0'} 2024-04-13T22:15:36.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b654a28603ad904-00661b0408-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:36.985 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:36.985 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:36.985 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.985 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.985 DEBUG:boto:Method: GET 2024-04-13T22:15:36.985 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.985 DEBUG:boto:Data: 2024-04-13T22:15:36.985 DEBUG:boto:Headers: {} 2024-04-13T22:15:36.985 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:36.985 DEBUG:boto:Port: 8000 2024-04-13T22:15:36.986 DEBUG:boto:Params: {} 2024-04-13T22:15:36.986 DEBUG:boto:Token: None 2024-04-13T22:15:36.986 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:36 GMT /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:36.986 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wMbEoA0ZJa/FO7g8vPaMWeeKSLk= 2024-04-13T22:15:36.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wMbEoA0ZJa/FO7g8vPaMWeeKSLk=', 'Content-Length': '0'} 2024-04-13T22:15:36.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa4e8e448db1151-00661b0408-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.031 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.031 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.031 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.031 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.031 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.031 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.032 DEBUG:boto:Data: 2024-04-13T22:15:37.032 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.032 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.032 DEBUG:boto:Params: {} 2024-04-13T22:15:37.032 DEBUG:boto:Token: None 2024-04-13T22:15:37.032 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cSLElSGi0hEfrt5P/KTzQafmLo0= 2024-04-13T22:15:37.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cSLElSGi0hEfrt5P/KTzQafmLo0=', 'Content-Length': '0'} 2024-04-13T22:15:37.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d27a4219672e553-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.034 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.035 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.035 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.035 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.035 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.035 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.035 DEBUG:boto:Data: 2024-04-13T22:15:37.035 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.035 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.035 DEBUG:boto:Params: {} 2024-04-13T22:15:37.036 DEBUG:boto:Token: None 2024-04-13T22:15:37.036 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cSLElSGi0hEfrt5P/KTzQafmLo0= 2024-04-13T22:15:37.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cSLElSGi0hEfrt5P/KTzQafmLo0=', 'Content-Length': '0'} 2024-04-13T22:15:37.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004fd744c23c6c8bc-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-13T22:15:37.038 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.038 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.038 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.038 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.038 DEBUG:boto:Method: GET 2024-04-13T22:15:37.038 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.038 DEBUG:boto:Data: 2024-04-13T22:15:37.038 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.039 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.039 DEBUG:boto:Params: {} 2024-04-13T22:15:37.039 DEBUG:boto:Token: None 2024-04-13T22:15:37.039 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4NgCxtpPLlRJObEnxan3a+kNt50= 2024-04-13T22:15:37.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4NgCxtpPLlRJObEnxan3a+kNt50=', 'Content-Length': '0'} 2024-04-13T22:15:37.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f34b5ce1e45ab43-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.041 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.041 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.041 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.041 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.041 DEBUG:boto:Method: GET 2024-04-13T22:15:37.041 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.041 DEBUG:boto:Data: 2024-04-13T22:15:37.042 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.042 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.042 DEBUG:boto:Params: {} 2024-04-13T22:15:37.042 DEBUG:boto:Token: None 2024-04-13T22:15:37.042 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059?versionId=null 2024-04-13T22:15:37.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4NgCxtpPLlRJObEnxan3a+kNt50= 2024-04-13T22:15:37.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4NgCxtpPLlRJObEnxan3a+kNt50=', 'Content-Length': '0'} 2024-04-13T22:15:37.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114cc78e4a4fee16-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.044 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.044 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.044 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.044 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.044 DEBUG:boto:Method: GET 2024-04-13T22:15:37.044 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.045 DEBUG:boto:Data: 2024-04-13T22:15:37.045 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.045 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.045 DEBUG:boto:Params: {} 2024-04-13T22:15:37.045 DEBUG:boto:Token: None 2024-04-13T22:15:37.045 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1e70b5d27a7aa6-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.048 DEBUG:boto:b'vwnfns-60obj000000000000000000591000falseobj000000000000000000592024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.048 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.048 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.048 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.048 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.048 DEBUG:boto:Method: GET 2024-04-13T22:15:37.048 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.049 DEBUG:boto:Data: 2024-04-13T22:15:37.049 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.049 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.049 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.049 DEBUG:boto:Params: {} 2024-04-13T22:15:37.049 DEBUG:boto:Token: None 2024-04-13T22:15:37.049 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aefe65ffc3bdfb0a-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.053 DEBUG:boto:b'vwnfns-60obj000000000000000000591000falseobj000000000000000000592024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.053 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.054 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.054 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.054 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.054 DEBUG:boto:Data: 2024-04-13T22:15:37.054 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.054 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.054 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.054 DEBUG:boto:Params: {} 2024-04-13T22:15:37.054 DEBUG:boto:Token: None 2024-04-13T22:15:37.055 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5B1U4p2kLEVfwLp22O0WiS+66Eg= 2024-04-13T22:15:37.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5B1U4p2kLEVfwLp22O0WiS+66Eg=', 'Content-Length': '0'} 2024-04-13T22:15:37.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009361442fe004a897-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.056 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.056 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.057 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.057 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.057 DEBUG:boto:Data: 2024-04-13T22:15:37.057 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.057 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.057 DEBUG:boto:Params: {} 2024-04-13T22:15:37.057 DEBUG:boto:Token: None 2024-04-13T22:15:37.057 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.057 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5B1U4p2kLEVfwLp22O0WiS+66Eg= 2024-04-13T22:15:37.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5B1U4p2kLEVfwLp22O0WiS+66Eg=', 'Content-Length': '0'} 2024-04-13T22:15:37.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf788bb4085d709-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-13T22:15:37.060 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.060 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.060 DEBUG:boto:Method: GET 2024-04-13T22:15:37.060 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.060 DEBUG:boto:Data: 2024-04-13T22:15:37.060 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.060 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.060 DEBUG:boto:Params: {} 2024-04-13T22:15:37.061 DEBUG:boto:Token: None 2024-04-13T22:15:37.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u9xuKJ3p4ny1g6O2k6XomZRGjHU= 2024-04-13T22:15:37.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u9xuKJ3p4ny1g6O2k6XomZRGjHU=', 'Content-Length': '0'} 2024-04-13T22:15:37.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013957665a4ca7f93-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.106 DEBUG:boto:path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.107 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.107 DEBUG:boto:Method: GET 2024-04-13T22:15:37.107 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.107 DEBUG:boto:Data: 2024-04-13T22:15:37.107 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.107 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.107 DEBUG:boto:Params: {} 2024-04-13T22:15:37.107 DEBUG:boto:Token: None 2024-04-13T22:15:37.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000059 2024-04-13T22:15:37.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u9xuKJ3p4ny1g6O2k6XomZRGjHU= 2024-04-13T22:15:37.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u9xuKJ3p4ny1g6O2k6XomZRGjHU=', 'Content-Length': '0'} 2024-04-13T22:15:37.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087279e4ad7763735-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.110 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.110 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.110 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.110 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.110 DEBUG:boto:Method: GET 2024-04-13T22:15:37.110 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.110 DEBUG:boto:Data: 2024-04-13T22:15:37.110 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.111 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.111 DEBUG:boto:Params: {} 2024-04-13T22:15:37.111 DEBUG:boto:Token: None 2024-04-13T22:15:37.111 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6dda82e8925d5d3-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.113 DEBUG:boto:b'vwnfns-60obj000000000000000000591000falseobj000000000000000000592024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.114 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.114 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.114 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.114 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.114 DEBUG:boto:Method: GET 2024-04-13T22:15:37.114 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000059 2024-04-13T22:15:37.114 DEBUG:boto:Data: 2024-04-13T22:15:37.114 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.115 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.115 DEBUG:boto:Params: {} 2024-04-13T22:15:37.115 DEBUG:boto:Token: None 2024-04-13T22:15:37.115 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.115 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4cfbc10114c59df-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.119 DEBUG:boto:b'vwnfns-60obj000000000000000000591000falseobj000000000000000000592024-04-13T22:15:19.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-13T22:15:37.119 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.120 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.120 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.120 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.120 DEBUG:boto:Method: GET 2024-04-13T22:15:37.120 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.120 DEBUG:boto:Data: 2024-04-13T22:15:37.120 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.120 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.120 DEBUG:boto:Params: {} 2024-04-13T22:15:37.120 DEBUG:boto:Token: None 2024-04-13T22:15:37.121 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0= 2024-04-13T22:15:37.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0=', 'Content-Length': '0'} 2024-04-13T22:15:37.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebe8bc2a49c0036-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.123 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.123 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.123 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.123 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.123 DEBUG:boto:Method: GET 2024-04-13T22:15:37.123 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.123 DEBUG:boto:Data: 2024-04-13T22:15:37.123 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.124 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.124 DEBUG:boto:Params: {} 2024-04-13T22:15:37.124 DEBUG:boto:Token: None 2024-04-13T22:15:37.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0= 2024-04-13T22:15:37.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0=', 'Content-Length': '0'} 2024-04-13T22:15:37.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082ce126542fd51e8-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.166 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.166 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.167 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.167 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.167 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.167 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.167 DEBUG:boto:Data: 2024-04-13T22:15:37.167 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.167 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.167 DEBUG:boto:Params: {} 2024-04-13T22:15:37.167 DEBUG:boto:Token: None 2024-04-13T22:15:37.168 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PS9kNJ5qcN1em7fc0FedaN64v9A= 2024-04-13T22:15:37.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PS9kNJ5qcN1em7fc0FedaN64v9A=', 'Content-Length': '0'} 2024-04-13T22:15:37.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd69fb56bc439fc0-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.169 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.170 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.170 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.170 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.170 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.170 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.170 DEBUG:boto:Data: 2024-04-13T22:15:37.170 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.170 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.170 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.170 DEBUG:boto:Params: {} 2024-04-13T22:15:37.171 DEBUG:boto:Token: None 2024-04-13T22:15:37.171 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PS9kNJ5qcN1em7fc0FedaN64v9A= 2024-04-13T22:15:37.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PS9kNJ5qcN1em7fc0FedaN64v9A=', 'Content-Length': '0'} 2024-04-13T22:15:37.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008254c9298f74b3b9-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-13T22:15:37.173 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.173 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.173 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.173 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.173 DEBUG:boto:Method: GET 2024-04-13T22:15:37.173 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.173 DEBUG:boto:Data: 2024-04-13T22:15:37.173 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.174 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.174 DEBUG:boto:Params: {} 2024-04-13T22:15:37.174 DEBUG:boto:Token: None 2024-04-13T22:15:37.174 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.174 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0= 2024-04-13T22:15:37.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0=', 'Content-Length': '0'} 2024-04-13T22:15:37.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c175a3e31ee1df8b-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.176 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.176 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.176 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.177 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.177 DEBUG:boto:Method: GET 2024-04-13T22:15:37.177 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.177 DEBUG:boto:Data: 2024-04-13T22:15:37.177 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.177 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.177 DEBUG:boto:Params: {} 2024-04-13T22:15:37.177 DEBUG:boto:Token: None 2024-04-13T22:15:37.177 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060?versionId=null 2024-04-13T22:15:37.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0= 2024-04-13T22:15:37.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhUwtJ9MU+uf+Ptls9jlk1d6cT0=', 'Content-Length': '0'} 2024-04-13T22:15:37.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004185029c80ca208a-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.179 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.180 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.180 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.180 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.180 DEBUG:boto:Method: GET 2024-04-13T22:15:37.180 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.180 DEBUG:boto:Data: 2024-04-13T22:15:37.180 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.180 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.180 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.180 DEBUG:boto:Params: {} 2024-04-13T22:15:37.180 DEBUG:boto:Token: None 2024-04-13T22:15:37.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e627ea9af5e7fd-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.183 DEBUG:boto:b'vwnfns-60obj000000000000000000601000falseobj000000000000000000602024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.183 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.184 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.184 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.184 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.184 DEBUG:boto:Method: GET 2024-04-13T22:15:37.184 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.184 DEBUG:boto:Data: 2024-04-13T22:15:37.184 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.184 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.184 DEBUG:boto:Params: {} 2024-04-13T22:15:37.184 DEBUG:boto:Token: None 2024-04-13T22:15:37.185 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa7893241a2a091b-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.188 DEBUG:boto:b'vwnfns-60obj000000000000000000601000falseobj000000000000000000602024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.189 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.189 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.189 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.189 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.189 DEBUG:boto:Data: 2024-04-13T22:15:37.189 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.189 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.189 DEBUG:boto:Params: {} 2024-04-13T22:15:37.190 DEBUG:boto:Token: None 2024-04-13T22:15:37.190 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:56b465igRgh8Q8uPv8m1rerV69Q= 2024-04-13T22:15:37.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:56b465igRgh8Q8uPv8m1rerV69Q=', 'Content-Length': '0'} 2024-04-13T22:15:37.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e696dc1ddc95eb-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.192 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.192 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.192 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.192 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.192 DEBUG:boto:Data: 2024-04-13T22:15:37.192 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.192 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.193 DEBUG:boto:Params: {} 2024-04-13T22:15:37.193 DEBUG:boto:Token: None 2024-04-13T22:15:37.193 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:56b465igRgh8Q8uPv8m1rerV69Q= 2024-04-13T22:15:37.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:56b465igRgh8Q8uPv8m1rerV69Q=', 'Content-Length': '0'} 2024-04-13T22:15:37.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6dcb3564b93e343-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-13T22:15:37.195 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.195 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.195 DEBUG:boto:Method: GET 2024-04-13T22:15:37.195 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.195 DEBUG:boto:Data: 2024-04-13T22:15:37.195 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.196 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.196 DEBUG:boto:Params: {} 2024-04-13T22:15:37.196 DEBUG:boto:Token: None 2024-04-13T22:15:37.196 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gfn8UKkVLroG28IaBNik8uckWS0= 2024-04-13T22:15:37.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gfn8UKkVLroG28IaBNik8uckWS0=', 'Content-Length': '0'} 2024-04-13T22:15:37.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0e7a2bd0b5ec39-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.198 DEBUG:boto:path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.198 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.198 DEBUG:boto:Method: GET 2024-04-13T22:15:37.198 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.198 DEBUG:boto:Data: 2024-04-13T22:15:37.199 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.199 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.199 DEBUG:boto:Params: {} 2024-04-13T22:15:37.199 DEBUG:boto:Token: None 2024-04-13T22:15:37.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000060 2024-04-13T22:15:37.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gfn8UKkVLroG28IaBNik8uckWS0= 2024-04-13T22:15:37.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gfn8UKkVLroG28IaBNik8uckWS0=', 'Content-Length': '0'} 2024-04-13T22:15:37.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d68d7ee46ceed43-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.242 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.243 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.243 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.243 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.243 DEBUG:boto:Method: GET 2024-04-13T22:15:37.244 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.244 DEBUG:boto:Data: 2024-04-13T22:15:37.244 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.244 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.244 DEBUG:boto:Params: {} 2024-04-13T22:15:37.244 DEBUG:boto:Token: None 2024-04-13T22:15:37.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.245 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df28b10c5474ce0-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.247 DEBUG:boto:b'vwnfns-60obj000000000000000000601000falseobj000000000000000000602024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.248 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.248 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.248 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.248 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.248 DEBUG:boto:Method: GET 2024-04-13T22:15:37.248 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000060 2024-04-13T22:15:37.248 DEBUG:boto:Data: 2024-04-13T22:15:37.248 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.249 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.249 DEBUG:boto:Params: {} 2024-04-13T22:15:37.249 DEBUG:boto:Token: None 2024-04-13T22:15:37.249 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.249 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42756dcd4d54aed-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.253 DEBUG:boto:b'vwnfns-60obj000000000000000000601000falseobj000000000000000000602024-04-13T22:15:22.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-13T22:15:37.254 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.254 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.254 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.254 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.254 DEBUG:boto:Method: GET 2024-04-13T22:15:37.254 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.254 DEBUG:boto:Data: 2024-04-13T22:15:37.254 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.254 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.254 DEBUG:boto:Params: {} 2024-04-13T22:15:37.255 DEBUG:boto:Token: None 2024-04-13T22:15:37.255 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA= 2024-04-13T22:15:37.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA=', 'Content-Length': '0'} 2024-04-13T22:15:37.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285f115fd15f90cf-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.298 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.299 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.299 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.299 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.299 DEBUG:boto:Method: GET 2024-04-13T22:15:37.299 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.299 DEBUG:boto:Data: 2024-04-13T22:15:37.299 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.299 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.299 DEBUG:boto:Params: {} 2024-04-13T22:15:37.300 DEBUG:boto:Token: None 2024-04-13T22:15:37.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA= 2024-04-13T22:15:37.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA=', 'Content-Length': '0'} 2024-04-13T22:15:37.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005841742f8161455a-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.302 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.302 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.302 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.303 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.303 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.303 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.303 DEBUG:boto:Data: 2024-04-13T22:15:37.303 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.303 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.303 DEBUG:boto:Params: {} 2024-04-13T22:15:37.303 DEBUG:boto:Token: None 2024-04-13T22:15:37.303 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DBPTjO0u0urBYCyvmH+8viXg+MU= 2024-04-13T22:15:37.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DBPTjO0u0urBYCyvmH+8viXg+MU=', 'Content-Length': '0'} 2024-04-13T22:15:37.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e1bc33ae8f1560-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.305 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.305 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.305 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.306 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.306 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.306 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.306 DEBUG:boto:Data: 2024-04-13T22:15:37.306 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.306 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.306 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.306 DEBUG:boto:Params: {} 2024-04-13T22:15:37.306 DEBUG:boto:Token: None 2024-04-13T22:15:37.306 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DBPTjO0u0urBYCyvmH+8viXg+MU= 2024-04-13T22:15:37.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DBPTjO0u0urBYCyvmH+8viXg+MU=', 'Content-Length': '0'} 2024-04-13T22:15:37.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7244c7583a20a1a-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-13T22:15:37.308 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.309 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.309 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.309 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.309 DEBUG:boto:Method: GET 2024-04-13T22:15:37.309 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.309 DEBUG:boto:Data: 2024-04-13T22:15:37.309 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.309 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.309 DEBUG:boto:Params: {} 2024-04-13T22:15:37.310 DEBUG:boto:Token: None 2024-04-13T22:15:37.310 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA= 2024-04-13T22:15:37.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA=', 'Content-Length': '0'} 2024-04-13T22:15:37.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e392a681d39c217f-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.312 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.312 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.312 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.312 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.312 DEBUG:boto:Method: GET 2024-04-13T22:15:37.312 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.312 DEBUG:boto:Data: 2024-04-13T22:15:37.313 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.313 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.313 DEBUG:boto:Params: {} 2024-04-13T22:15:37.313 DEBUG:boto:Token: None 2024-04-13T22:15:37.313 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061?versionId=null 2024-04-13T22:15:37.313 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA= 2024-04-13T22:15:37.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XRh+gYIRsfTvLbroeW3YdnQ1KBA=', 'Content-Length': '0'} 2024-04-13T22:15:37.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed62e55fb9f9f1bc-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.315 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.315 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.315 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.315 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.315 DEBUG:boto:Method: GET 2024-04-13T22:15:37.316 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.316 DEBUG:boto:Data: 2024-04-13T22:15:37.316 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.316 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.316 DEBUG:boto:Params: {} 2024-04-13T22:15:37.316 DEBUG:boto:Token: None 2024-04-13T22:15:37.316 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e199cc85ea2128a1-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.319 DEBUG:boto:b'vwnfns-60obj000000000000000000611000falseobj000000000000000000612024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.319 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.319 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.319 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.319 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.319 DEBUG:boto:Method: GET 2024-04-13T22:15:37.320 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.320 DEBUG:boto:Data: 2024-04-13T22:15:37.320 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.320 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.320 DEBUG:boto:Params: {} 2024-04-13T22:15:37.320 DEBUG:boto:Token: None 2024-04-13T22:15:37.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae8f67bbd7a7a29-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.324 DEBUG:boto:b'vwnfns-60obj000000000000000000611000falseobj000000000000000000612024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.325 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.325 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.325 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.325 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.325 DEBUG:boto:Data: 2024-04-13T22:15:37.325 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.325 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.325 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.326 DEBUG:boto:Params: {} 2024-04-13T22:15:37.326 DEBUG:boto:Token: None 2024-04-13T22:15:37.326 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p3X1FBTJWQvOSvYOW73HRrPk+ZM= 2024-04-13T22:15:37.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p3X1FBTJWQvOSvYOW73HRrPk+ZM=', 'Content-Length': '0'} 2024-04-13T22:15:37.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab46efa0cf25f4e-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.328 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.328 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.328 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.328 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.328 DEBUG:boto:Data: 2024-04-13T22:15:37.328 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.328 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.328 DEBUG:boto:Params: {} 2024-04-13T22:15:37.329 DEBUG:boto:Token: None 2024-04-13T22:15:37.329 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p3X1FBTJWQvOSvYOW73HRrPk+ZM= 2024-04-13T22:15:37.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p3X1FBTJWQvOSvYOW73HRrPk+ZM=', 'Content-Length': '0'} 2024-04-13T22:15:37.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ad37016d37c43f1-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-13T22:15:37.331 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.331 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.331 DEBUG:boto:Method: GET 2024-04-13T22:15:37.331 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.331 DEBUG:boto:Data: 2024-04-13T22:15:37.331 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.332 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.332 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.332 DEBUG:boto:Params: {} 2024-04-13T22:15:37.332 DEBUG:boto:Token: None 2024-04-13T22:15:37.332 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I8STMRSlG8w5O33+i/NPDjlSIfA= 2024-04-13T22:15:37.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I8STMRSlG8w5O33+i/NPDjlSIfA=', 'Content-Length': '0'} 2024-04-13T22:15:37.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f639816b99dd3a45-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.334 DEBUG:boto:path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.334 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.334 DEBUG:boto:Method: GET 2024-04-13T22:15:37.334 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.334 DEBUG:boto:Data: 2024-04-13T22:15:37.334 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.335 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.335 DEBUG:boto:Params: {} 2024-04-13T22:15:37.335 DEBUG:boto:Token: None 2024-04-13T22:15:37.335 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000061 2024-04-13T22:15:37.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I8STMRSlG8w5O33+i/NPDjlSIfA= 2024-04-13T22:15:37.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I8STMRSlG8w5O33+i/NPDjlSIfA=', 'Content-Length': '0'} 2024-04-13T22:15:37.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe63fed2347d114-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.378 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.379 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.379 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.379 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.379 DEBUG:boto:Method: GET 2024-04-13T22:15:37.379 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.379 DEBUG:boto:Data: 2024-04-13T22:15:37.379 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.379 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.379 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.379 DEBUG:boto:Params: {} 2024-04-13T22:15:37.379 DEBUG:boto:Token: None 2024-04-13T22:15:37.380 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4c5bb54100c1a4-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.382 DEBUG:boto:b'vwnfns-60obj000000000000000000611000falseobj000000000000000000612024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.382 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.382 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.383 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.383 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.383 DEBUG:boto:Method: GET 2024-04-13T22:15:37.383 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000061 2024-04-13T22:15:37.383 DEBUG:boto:Data: 2024-04-13T22:15:37.383 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.383 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.383 DEBUG:boto:Params: {} 2024-04-13T22:15:37.383 DEBUG:boto:Token: None 2024-04-13T22:15:37.383 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80d52f68551ef53-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.387 DEBUG:boto:b'vwnfns-60obj000000000000000000611000falseobj000000000000000000612024-04-13T22:15:21.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-13T22:15:37.387 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.387 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.388 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.388 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.388 DEBUG:boto:Method: GET 2024-04-13T22:15:37.388 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.388 DEBUG:boto:Data: 2024-04-13T22:15:37.388 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.388 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.388 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.388 DEBUG:boto:Params: {} 2024-04-13T22:15:37.388 DEBUG:boto:Token: None 2024-04-13T22:15:37.388 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.389 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM= 2024-04-13T22:15:37.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM=', 'Content-Length': '0'} 2024-04-13T22:15:37.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3463e96247c7b6-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.391 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.391 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.391 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.391 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.391 DEBUG:boto:Method: GET 2024-04-13T22:15:37.391 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.391 DEBUG:boto:Data: 2024-04-13T22:15:37.391 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.391 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.391 DEBUG:boto:Params: {} 2024-04-13T22:15:37.391 DEBUG:boto:Token: None 2024-04-13T22:15:37.392 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM= 2024-04-13T22:15:37.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM=', 'Content-Length': '0'} 2024-04-13T22:15:37.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5fac8e1ef6937c-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.394 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.394 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.394 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.394 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.394 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.394 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.394 DEBUG:boto:Data: 2024-04-13T22:15:37.394 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.395 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.395 DEBUG:boto:Params: {} 2024-04-13T22:15:37.395 DEBUG:boto:Token: None 2024-04-13T22:15:37.395 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.395 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GciU+8ZSMQ2l/g34pCOnM4guNnU= 2024-04-13T22:15:37.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GciU+8ZSMQ2l/g34pCOnM4guNnU=', 'Content-Length': '0'} 2024-04-13T22:15:37.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed4c1dbb8b3b503d-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.397 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.398 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.398 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.398 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.398 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.398 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.398 DEBUG:boto:Data: 2024-04-13T22:15:37.398 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.398 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.398 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.398 DEBUG:boto:Params: {} 2024-04-13T22:15:37.399 DEBUG:boto:Token: None 2024-04-13T22:15:37.399 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.399 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GciU+8ZSMQ2l/g34pCOnM4guNnU= 2024-04-13T22:15:37.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GciU+8ZSMQ2l/g34pCOnM4guNnU=', 'Content-Length': '0'} 2024-04-13T22:15:37.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f3245c1be4f13f-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-13T22:15:37.400 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.401 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.401 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.401 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.401 DEBUG:boto:Method: GET 2024-04-13T22:15:37.401 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.401 DEBUG:boto:Data: 2024-04-13T22:15:37.401 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.401 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.401 DEBUG:boto:Params: {} 2024-04-13T22:15:37.402 DEBUG:boto:Token: None 2024-04-13T22:15:37.402 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.402 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM= 2024-04-13T22:15:37.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM=', 'Content-Length': '0'} 2024-04-13T22:15:37.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba290758cec3d3c-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.446 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.447 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.447 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.447 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.447 DEBUG:boto:Method: GET 2024-04-13T22:15:37.447 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.447 DEBUG:boto:Data: 2024-04-13T22:15:37.447 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.447 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.447 DEBUG:boto:Params: {} 2024-04-13T22:15:37.448 DEBUG:boto:Token: None 2024-04-13T22:15:37.448 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062?versionId=null 2024-04-13T22:15:37.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM= 2024-04-13T22:15:37.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zKGaOq7OnYjPO4DalR0eRXgtNAM=', 'Content-Length': '0'} 2024-04-13T22:15:37.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6b4665860a0251-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.450 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.450 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.450 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.450 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.450 DEBUG:boto:Method: GET 2024-04-13T22:15:37.450 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.451 DEBUG:boto:Data: 2024-04-13T22:15:37.451 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.451 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.451 DEBUG:boto:Params: {} 2024-04-13T22:15:37.451 DEBUG:boto:Token: None 2024-04-13T22:15:37.451 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e75c1ddd0406e21-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.453 DEBUG:boto:b'vwnfns-60obj000000000000000000621000falseobj000000000000000000622024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.454 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.454 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.454 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.454 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.454 DEBUG:boto:Method: GET 2024-04-13T22:15:37.454 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.454 DEBUG:boto:Data: 2024-04-13T22:15:37.455 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.455 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.455 DEBUG:boto:Params: {} 2024-04-13T22:15:37.455 DEBUG:boto:Token: None 2024-04-13T22:15:37.455 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ba91c6628a1f85-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.458 DEBUG:boto:b'vwnfns-60obj000000000000000000621000falseobj000000000000000000622024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.459 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.459 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.459 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.459 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.459 DEBUG:boto:Data: 2024-04-13T22:15:37.459 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.459 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.460 DEBUG:boto:Params: {} 2024-04-13T22:15:37.460 DEBUG:boto:Token: None 2024-04-13T22:15:37.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2j7wvPSHyBRm1g0gPLZBWw5PCT0= 2024-04-13T22:15:37.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2j7wvPSHyBRm1g0gPLZBWw5PCT0=', 'Content-Length': '0'} 2024-04-13T22:15:37.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f238515ad5f1bf3-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.462 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.462 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.462 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.462 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.462 DEBUG:boto:Data: 2024-04-13T22:15:37.462 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.462 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.463 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.463 DEBUG:boto:Params: {} 2024-04-13T22:15:37.463 DEBUG:boto:Token: None 2024-04-13T22:15:37.463 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.463 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2j7wvPSHyBRm1g0gPLZBWw5PCT0= 2024-04-13T22:15:37.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2j7wvPSHyBRm1g0gPLZBWw5PCT0=', 'Content-Length': '0'} 2024-04-13T22:15:37.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069bf75b78a96a241-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-13T22:15:37.465 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.465 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.465 DEBUG:boto:Method: GET 2024-04-13T22:15:37.465 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.465 DEBUG:boto:Data: 2024-04-13T22:15:37.465 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.465 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.465 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.466 DEBUG:boto:Params: {} 2024-04-13T22:15:37.466 DEBUG:boto:Token: None 2024-04-13T22:15:37.466 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.466 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XcMlHn+FxDZTcfUIRVixkbWorm8= 2024-04-13T22:15:37.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XcMlHn+FxDZTcfUIRVixkbWorm8=', 'Content-Length': '0'} 2024-04-13T22:15:37.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365cac8ca316812b-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.468 DEBUG:boto:path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.468 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.468 DEBUG:boto:Method: GET 2024-04-13T22:15:37.468 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.468 DEBUG:boto:Data: 2024-04-13T22:15:37.468 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.468 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.468 DEBUG:boto:Params: {} 2024-04-13T22:15:37.469 DEBUG:boto:Token: None 2024-04-13T22:15:37.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000062 2024-04-13T22:15:37.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XcMlHn+FxDZTcfUIRVixkbWorm8= 2024-04-13T22:15:37.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XcMlHn+FxDZTcfUIRVixkbWorm8=', 'Content-Length': '0'} 2024-04-13T22:15:37.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015cde12ea7bebe4-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.518 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.519 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.519 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.519 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.519 DEBUG:boto:Method: GET 2024-04-13T22:15:37.519 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.519 DEBUG:boto:Data: 2024-04-13T22:15:37.519 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.519 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.519 DEBUG:boto:Params: {} 2024-04-13T22:15:37.519 DEBUG:boto:Token: None 2024-04-13T22:15:37.520 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca83048d954b5bd-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.522 DEBUG:boto:b'vwnfns-60obj000000000000000000621000falseobj000000000000000000622024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.522 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.523 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.523 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.523 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.523 DEBUG:boto:Method: GET 2024-04-13T22:15:37.523 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000062 2024-04-13T22:15:37.524 DEBUG:boto:Data: 2024-04-13T22:15:37.524 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.524 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.524 DEBUG:boto:Params: {} 2024-04-13T22:15:37.524 DEBUG:boto:Token: None 2024-04-13T22:15:37.524 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb601c9c58a64a8-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.528 DEBUG:boto:b'vwnfns-60obj000000000000000000621000falseobj000000000000000000622024-04-13T22:15:22.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-13T22:15:37.529 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.529 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.529 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.529 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.529 DEBUG:boto:Method: GET 2024-04-13T22:15:37.529 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.529 DEBUG:boto:Data: 2024-04-13T22:15:37.529 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.529 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.529 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.530 DEBUG:boto:Params: {} 2024-04-13T22:15:37.530 DEBUG:boto:Token: None 2024-04-13T22:15:37.530 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.530 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk= 2024-04-13T22:15:37.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk=', 'Content-Length': '0'} 2024-04-13T22:15:37.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000548ddf5a0a4efb57-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.532 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.532 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.532 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.532 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.532 DEBUG:boto:Method: GET 2024-04-13T22:15:37.533 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.533 DEBUG:boto:Data: 2024-04-13T22:15:37.533 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.533 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.533 DEBUG:boto:Params: {} 2024-04-13T22:15:37.533 DEBUG:boto:Token: None 2024-04-13T22:15:37.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk= 2024-04-13T22:15:37.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk=', 'Content-Length': '0'} 2024-04-13T22:15:37.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e486ab3e7892381c-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.535 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.536 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.536 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.536 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.536 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.536 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.536 DEBUG:boto:Data: 2024-04-13T22:15:37.536 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.536 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.536 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.536 DEBUG:boto:Params: {} 2024-04-13T22:15:37.536 DEBUG:boto:Token: None 2024-04-13T22:15:37.537 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.537 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LqFGpPAtZThyV20BDWYkAXBtvyo= 2024-04-13T22:15:37.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LqFGpPAtZThyV20BDWYkAXBtvyo=', 'Content-Length': '0'} 2024-04-13T22:15:37.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c696649edcf98d80-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.538 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.538 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.539 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.539 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.539 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.539 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.539 DEBUG:boto:Data: 2024-04-13T22:15:37.539 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.539 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.539 DEBUG:boto:Params: {} 2024-04-13T22:15:37.539 DEBUG:boto:Token: None 2024-04-13T22:15:37.539 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LqFGpPAtZThyV20BDWYkAXBtvyo= 2024-04-13T22:15:37.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LqFGpPAtZThyV20BDWYkAXBtvyo=', 'Content-Length': '0'} 2024-04-13T22:15:37.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000463ad1b466520f4-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-13T22:15:37.541 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.542 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.542 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.542 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.542 DEBUG:boto:Method: GET 2024-04-13T22:15:37.542 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.542 DEBUG:boto:Data: 2024-04-13T22:15:37.542 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.542 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.542 DEBUG:boto:Params: {} 2024-04-13T22:15:37.543 DEBUG:boto:Token: None 2024-04-13T22:15:37.543 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk= 2024-04-13T22:15:37.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk=', 'Content-Length': '0'} 2024-04-13T22:15:37.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba1dbec8053c49e8-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.545 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.545 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.545 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.545 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.545 DEBUG:boto:Method: GET 2024-04-13T22:15:37.545 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.545 DEBUG:boto:Data: 2024-04-13T22:15:37.545 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.546 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.546 DEBUG:boto:Params: {} 2024-04-13T22:15:37.546 DEBUG:boto:Token: None 2024-04-13T22:15:37.546 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065?versionId=null 2024-04-13T22:15:37.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk= 2024-04-13T22:15:37.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BAzDYCS3hQ0PDr2C06pHdFVNhVk=', 'Content-Length': '0'} 2024-04-13T22:15:37.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc62b949395d9e4b-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.590 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.591 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.591 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.591 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.591 DEBUG:boto:Method: GET 2024-04-13T22:15:37.591 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.591 DEBUG:boto:Data: 2024-04-13T22:15:37.591 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.591 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.591 DEBUG:boto:Params: {} 2024-04-13T22:15:37.592 DEBUG:boto:Token: None 2024-04-13T22:15:37.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a87eef34291b7b0d-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.594 DEBUG:boto:b'vwnfns-60obj000000000000000000651000falseobj000000000000000000652024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.595 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.595 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.595 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.595 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.595 DEBUG:boto:Method: GET 2024-04-13T22:15:37.595 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.595 DEBUG:boto:Data: 2024-04-13T22:15:37.595 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.596 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.596 DEBUG:boto:Params: {} 2024-04-13T22:15:37.596 DEBUG:boto:Token: None 2024-04-13T22:15:37.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df8d13cbcff757f-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.600 DEBUG:boto:b'vwnfns-60obj000000000000000000651000falseobj000000000000000000652024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.600 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.600 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.601 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.601 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.601 DEBUG:boto:Data: 2024-04-13T22:15:37.601 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.601 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.601 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.601 DEBUG:boto:Params: {} 2024-04-13T22:15:37.601 DEBUG:boto:Token: None 2024-04-13T22:15:37.601 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hvqNCnREDGjabhhU52ppcsqc9ek= 2024-04-13T22:15:37.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hvqNCnREDGjabhhU52ppcsqc9ek=', 'Content-Length': '0'} 2024-04-13T22:15:37.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029510362e627514b-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.603 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.603 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.603 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.603 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.604 DEBUG:boto:Data: 2024-04-13T22:15:37.604 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.604 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.604 DEBUG:boto:Params: {} 2024-04-13T22:15:37.604 DEBUG:boto:Token: None 2024-04-13T22:15:37.604 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hvqNCnREDGjabhhU52ppcsqc9ek= 2024-04-13T22:15:37.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hvqNCnREDGjabhhU52ppcsqc9ek=', 'Content-Length': '0'} 2024-04-13T22:15:37.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6a2240a437fb52-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-13T22:15:37.606 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.606 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.606 DEBUG:boto:Method: GET 2024-04-13T22:15:37.606 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.607 DEBUG:boto:Data: 2024-04-13T22:15:37.607 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.607 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.607 DEBUG:boto:Params: {} 2024-04-13T22:15:37.607 DEBUG:boto:Token: None 2024-04-13T22:15:37.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rZxlhfa0NPSFgYO9XtMWKPZamqM= 2024-04-13T22:15:37.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rZxlhfa0NPSFgYO9XtMWKPZamqM=', 'Content-Length': '0'} 2024-04-13T22:15:37.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0844541b5b05246-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.650 DEBUG:boto:path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.650 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.651 DEBUG:boto:Method: GET 2024-04-13T22:15:37.651 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.651 DEBUG:boto:Data: 2024-04-13T22:15:37.651 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.651 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.651 DEBUG:boto:Params: {} 2024-04-13T22:15:37.651 DEBUG:boto:Token: None 2024-04-13T22:15:37.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000065 2024-04-13T22:15:37.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rZxlhfa0NPSFgYO9XtMWKPZamqM= 2024-04-13T22:15:37.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rZxlhfa0NPSFgYO9XtMWKPZamqM=', 'Content-Length': '0'} 2024-04-13T22:15:37.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5fc8ff0b8ad6b0f-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.654 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.654 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.654 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.654 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.654 DEBUG:boto:Method: GET 2024-04-13T22:15:37.654 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.654 DEBUG:boto:Data: 2024-04-13T22:15:37.654 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.654 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.654 DEBUG:boto:Params: {} 2024-04-13T22:15:37.655 DEBUG:boto:Token: None 2024-04-13T22:15:37.655 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055aa861b85ec34ea-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.657 DEBUG:boto:b'vwnfns-60obj000000000000000000651000falseobj000000000000000000652024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.657 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.658 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.658 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.658 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.658 DEBUG:boto:Method: GET 2024-04-13T22:15:37.658 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000065 2024-04-13T22:15:37.658 DEBUG:boto:Data: 2024-04-13T22:15:37.658 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.658 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.659 DEBUG:boto:Params: {} 2024-04-13T22:15:37.659 DEBUG:boto:Token: None 2024-04-13T22:15:37.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c964535e0dac923c-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.662 DEBUG:boto:b'vwnfns-60obj000000000000000000651000falseobj000000000000000000652024-04-13T22:15:22.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-13T22:15:37.663 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.663 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.663 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.663 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.663 DEBUG:boto:Method: GET 2024-04-13T22:15:37.663 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.663 DEBUG:boto:Data: 2024-04-13T22:15:37.663 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.664 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.664 DEBUG:boto:Params: {} 2024-04-13T22:15:37.664 DEBUG:boto:Token: None 2024-04-13T22:15:37.664 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8= 2024-04-13T22:15:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8=', 'Content-Length': '0'} 2024-04-13T22:15:37.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff31868b1c2e1eb9-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.666 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.666 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.666 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.666 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.666 DEBUG:boto:Method: GET 2024-04-13T22:15:37.667 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.667 DEBUG:boto:Data: 2024-04-13T22:15:37.667 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.667 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.667 DEBUG:boto:Params: {} 2024-04-13T22:15:37.667 DEBUG:boto:Token: None 2024-04-13T22:15:37.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8= 2024-04-13T22:15:37.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8=', 'Content-Length': '0'} 2024-04-13T22:15:37.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d618a50681f808-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.710 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.711 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.711 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.711 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.711 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.711 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.711 DEBUG:boto:Data: 2024-04-13T22:15:37.711 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.711 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.711 DEBUG:boto:Params: {} 2024-04-13T22:15:37.712 DEBUG:boto:Token: None 2024-04-13T22:15:37.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yQkypqLPcSYaHl+PAlc35Dgp94Q= 2024-04-13T22:15:37.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yQkypqLPcSYaHl+PAlc35Dgp94Q=', 'Content-Length': '0'} 2024-04-13T22:15:37.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d18517bfabbb9fd-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.714 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.714 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.714 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.714 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.714 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.714 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.714 DEBUG:boto:Data: 2024-04-13T22:15:37.714 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.715 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.715 DEBUG:boto:Params: {} 2024-04-13T22:15:37.715 DEBUG:boto:Token: None 2024-04-13T22:15:37.715 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yQkypqLPcSYaHl+PAlc35Dgp94Q= 2024-04-13T22:15:37.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yQkypqLPcSYaHl+PAlc35Dgp94Q=', 'Content-Length': '0'} 2024-04-13T22:15:37.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069615b9c4807036d-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-13T22:15:37.717 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.717 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.717 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.717 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.717 DEBUG:boto:Method: GET 2024-04-13T22:15:37.718 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.718 DEBUG:boto:Data: 2024-04-13T22:15:37.718 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.718 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.718 DEBUG:boto:Params: {} 2024-04-13T22:15:37.718 DEBUG:boto:Token: None 2024-04-13T22:15:37.718 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8= 2024-04-13T22:15:37.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8=', 'Content-Length': '0'} 2024-04-13T22:15:37.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a450ac96c7a59cbc-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.720 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.720 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.720 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.720 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.721 DEBUG:boto:Method: GET 2024-04-13T22:15:37.721 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.721 DEBUG:boto:Data: 2024-04-13T22:15:37.721 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.721 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.721 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.721 DEBUG:boto:Params: {} 2024-04-13T22:15:37.721 DEBUG:boto:Token: None 2024-04-13T22:15:37.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076?versionId=null 2024-04-13T22:15:37.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8= 2024-04-13T22:15:37.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4wEC9cLYVWBmHNOS9C0c09BC7K8=', 'Content-Length': '0'} 2024-04-13T22:15:37.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a580df5d498fded5-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.723 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.723 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.723 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.724 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.724 DEBUG:boto:Method: GET 2024-04-13T22:15:37.724 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.724 DEBUG:boto:Data: 2024-04-13T22:15:37.724 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.724 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.724 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.724 DEBUG:boto:Params: {} 2024-04-13T22:15:37.724 DEBUG:boto:Token: None 2024-04-13T22:15:37.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.725 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000376a396b601757b6-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.727 DEBUG:boto:b'vwnfns-60obj000000000000000000761000falseobj000000000000000000762024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.727 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.727 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.727 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.728 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.728 DEBUG:boto:Method: GET 2024-04-13T22:15:37.728 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.728 DEBUG:boto:Data: 2024-04-13T22:15:37.728 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.728 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.728 DEBUG:boto:Params: {} 2024-04-13T22:15:37.728 DEBUG:boto:Token: None 2024-04-13T22:15:37.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58832c0ab5a362f-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.732 DEBUG:boto:b'vwnfns-60obj000000000000000000761000falseobj000000000000000000762024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.733 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.733 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.733 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.733 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.733 DEBUG:boto:Data: 2024-04-13T22:15:37.733 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.734 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.734 DEBUG:boto:Params: {} 2024-04-13T22:15:37.734 DEBUG:boto:Token: None 2024-04-13T22:15:37.734 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vz4lXqpzP8jglHRzOBqK3VnbwSA= 2024-04-13T22:15:37.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vz4lXqpzP8jglHRzOBqK3VnbwSA=', 'Content-Length': '0'} 2024-04-13T22:15:37.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca53f50672aa8767-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.736 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.736 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.736 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.736 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.736 DEBUG:boto:Data: 2024-04-13T22:15:37.736 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.736 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.736 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.736 DEBUG:boto:Params: {} 2024-04-13T22:15:37.737 DEBUG:boto:Token: None 2024-04-13T22:15:37.737 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vz4lXqpzP8jglHRzOBqK3VnbwSA= 2024-04-13T22:15:37.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vz4lXqpzP8jglHRzOBqK3VnbwSA=', 'Content-Length': '0'} 2024-04-13T22:15:37.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd26f4273c35eb31-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-13T22:15:37.739 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.739 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.739 DEBUG:boto:Method: GET 2024-04-13T22:15:37.739 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.739 DEBUG:boto:Data: 2024-04-13T22:15:37.739 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.739 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.739 DEBUG:boto:Params: {} 2024-04-13T22:15:37.739 DEBUG:boto:Token: None 2024-04-13T22:15:37.739 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.740 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mIZiTYKQirpaGYLALFQQBdHIGTA= 2024-04-13T22:15:37.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mIZiTYKQirpaGYLALFQQBdHIGTA=', 'Content-Length': '0'} 2024-04-13T22:15:37.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eac70345b4516da-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.741 DEBUG:boto:path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.742 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.742 DEBUG:boto:Method: GET 2024-04-13T22:15:37.742 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.742 DEBUG:boto:Data: 2024-04-13T22:15:37.742 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.742 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.742 DEBUG:boto:Params: {} 2024-04-13T22:15:37.742 DEBUG:boto:Token: None 2024-04-13T22:15:37.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000076 2024-04-13T22:15:37.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mIZiTYKQirpaGYLALFQQBdHIGTA= 2024-04-13T22:15:37.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mIZiTYKQirpaGYLALFQQBdHIGTA=', 'Content-Length': '0'} 2024-04-13T22:15:37.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014991b12cd1ce345-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.786 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.786 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.787 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.787 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.787 DEBUG:boto:Method: GET 2024-04-13T22:15:37.787 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.787 DEBUG:boto:Data: 2024-04-13T22:15:37.787 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.787 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.787 DEBUG:boto:Params: {} 2024-04-13T22:15:37.787 DEBUG:boto:Token: None 2024-04-13T22:15:37.787 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5e444cae3648ce-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.790 DEBUG:boto:b'vwnfns-60obj000000000000000000761000falseobj000000000000000000762024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.790 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.790 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.790 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.791 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.791 DEBUG:boto:Method: GET 2024-04-13T22:15:37.791 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000076 2024-04-13T22:15:37.791 DEBUG:boto:Data: 2024-04-13T22:15:37.791 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.791 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.791 DEBUG:boto:Params: {} 2024-04-13T22:15:37.791 DEBUG:boto:Token: None 2024-04-13T22:15:37.791 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5c70576e35747a3-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.795 DEBUG:boto:b'vwnfns-60obj000000000000000000761000falseobj000000000000000000762024-04-13T22:15:17.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-13T22:15:37.795 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.796 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.796 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.796 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.796 DEBUG:boto:Method: GET 2024-04-13T22:15:37.796 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.796 DEBUG:boto:Data: 2024-04-13T22:15:37.796 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.796 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.796 DEBUG:boto:Params: {} 2024-04-13T22:15:37.796 DEBUG:boto:Token: None 2024-04-13T22:15:37.797 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.797 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU= 2024-04-13T22:15:37.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU=', 'Content-Length': '0'} 2024-04-13T22:15:37.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000457f293fa8deb8bc-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.799 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.799 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.799 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.799 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.799 DEBUG:boto:Method: GET 2024-04-13T22:15:37.799 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.799 DEBUG:boto:Data: 2024-04-13T22:15:37.799 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.799 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.799 DEBUG:boto:Params: {} 2024-04-13T22:15:37.800 DEBUG:boto:Token: None 2024-04-13T22:15:37.800 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU= 2024-04-13T22:15:37.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU=', 'Content-Length': '0'} 2024-04-13T22:15:37.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4c357039627e3e-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.802 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.802 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.802 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.802 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.802 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.803 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.803 DEBUG:boto:Data: 2024-04-13T22:15:37.803 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.803 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.803 DEBUG:boto:Params: {} 2024-04-13T22:15:37.803 DEBUG:boto:Token: None 2024-04-13T22:15:37.803 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VeSqkYxdBW2ea8gwpLLlqJkpvJY= 2024-04-13T22:15:37.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VeSqkYxdBW2ea8gwpLLlqJkpvJY=', 'Content-Length': '0'} 2024-04-13T22:15:37.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013277bd73dc96817-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.805 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.805 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.805 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.805 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.806 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.806 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.806 DEBUG:boto:Data: 2024-04-13T22:15:37.806 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.806 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.806 DEBUG:boto:Params: {} 2024-04-13T22:15:37.806 DEBUG:boto:Token: None 2024-04-13T22:15:37.806 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VeSqkYxdBW2ea8gwpLLlqJkpvJY= 2024-04-13T22:15:37.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VeSqkYxdBW2ea8gwpLLlqJkpvJY=', 'Content-Length': '0'} 2024-04-13T22:15:37.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdf18b2c020c8ebd-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-13T22:15:37.808 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.808 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.808 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.808 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.809 DEBUG:boto:Method: GET 2024-04-13T22:15:37.809 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.809 DEBUG:boto:Data: 2024-04-13T22:15:37.809 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.809 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.809 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.809 DEBUG:boto:Params: {} 2024-04-13T22:15:37.809 DEBUG:boto:Token: None 2024-04-13T22:15:37.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU= 2024-04-13T22:15:37.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU=', 'Content-Length': '0'} 2024-04-13T22:15:37.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232016cf12f32c29-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.854 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.854 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.855 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.855 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.855 DEBUG:boto:Method: GET 2024-04-13T22:15:37.855 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.855 DEBUG:boto:Data: 2024-04-13T22:15:37.855 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.855 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.855 DEBUG:boto:Params: {} 2024-04-13T22:15:37.855 DEBUG:boto:Token: None 2024-04-13T22:15:37.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133?versionId=null 2024-04-13T22:15:37.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU= 2024-04-13T22:15:37.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:geq8SEhm0KQvP1vF9EZ0bZTX3VU=', 'Content-Length': '0'} 2024-04-13T22:15:37.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a0c0f9a539d690-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.858 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.858 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.858 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.858 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.858 DEBUG:boto:Method: GET 2024-04-13T22:15:37.858 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.858 DEBUG:boto:Data: 2024-04-13T22:15:37.858 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.858 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.859 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.859 DEBUG:boto:Params: {} 2024-04-13T22:15:37.859 DEBUG:boto:Token: None 2024-04-13T22:15:37.859 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d1dec56da988e51-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.861 DEBUG:boto:b'vwnfns-60obj000000000000000001331000falseobj000000000000000001332024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.862 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.862 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.862 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.862 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.862 DEBUG:boto:Method: GET 2024-04-13T22:15:37.862 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.862 DEBUG:boto:Data: 2024-04-13T22:15:37.862 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.862 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.863 DEBUG:boto:Params: {} 2024-04-13T22:15:37.863 DEBUG:boto:Token: None 2024-04-13T22:15:37.863 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edd4a66910a39b70-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.866 DEBUG:boto:b'vwnfns-60obj000000000000000001331000falseobj000000000000000001332024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.867 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.867 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.867 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.867 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.867 DEBUG:boto:Data: 2024-04-13T22:15:37.867 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.867 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.867 DEBUG:boto:Params: {} 2024-04-13T22:15:37.868 DEBUG:boto:Token: None 2024-04-13T22:15:37.868 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cyifGaodEUXlUACEoHw97K0P9MQ= 2024-04-13T22:15:37.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cyifGaodEUXlUACEoHw97K0P9MQ=', 'Content-Length': '0'} 2024-04-13T22:15:37.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8449b13bf43331-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.869 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.870 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.870 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.870 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.870 DEBUG:boto:Data: 2024-04-13T22:15:37.870 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.870 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.870 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.870 DEBUG:boto:Params: {} 2024-04-13T22:15:37.870 DEBUG:boto:Token: None 2024-04-13T22:15:37.871 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cyifGaodEUXlUACEoHw97K0P9MQ= 2024-04-13T22:15:37.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cyifGaodEUXlUACEoHw97K0P9MQ=', 'Content-Length': '0'} 2024-04-13T22:15:37.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b19b2c49be8e9ce-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-13T22:15:37.873 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.873 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.873 DEBUG:boto:Method: GET 2024-04-13T22:15:37.873 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.873 DEBUG:boto:Data: 2024-04-13T22:15:37.873 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.873 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.873 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.873 DEBUG:boto:Params: {} 2024-04-13T22:15:37.873 DEBUG:boto:Token: None 2024-04-13T22:15:37.874 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.874 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wgOdMJERm46ftd0TMHoMgjCGiCU= 2024-04-13T22:15:37.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wgOdMJERm46ftd0TMHoMgjCGiCU=', 'Content-Length': '0'} 2024-04-13T22:15:37.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170ae668bd0eaea3-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.875 DEBUG:boto:path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.876 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.876 DEBUG:boto:Method: GET 2024-04-13T22:15:37.876 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.876 DEBUG:boto:Data: 2024-04-13T22:15:37.876 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.876 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.876 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.876 DEBUG:boto:Params: {} 2024-04-13T22:15:37.876 DEBUG:boto:Token: None 2024-04-13T22:15:37.876 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000133 2024-04-13T22:15:37.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wgOdMJERm46ftd0TMHoMgjCGiCU= 2024-04-13T22:15:37.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wgOdMJERm46ftd0TMHoMgjCGiCU=', 'Content-Length': '0'} 2024-04-13T22:15:37.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029569eddc655f239-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.922 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.923 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.923 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.923 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.923 DEBUG:boto:Method: GET 2024-04-13T22:15:37.923 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.923 DEBUG:boto:Data: 2024-04-13T22:15:37.923 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.923 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.923 DEBUG:boto:Params: {} 2024-04-13T22:15:37.924 DEBUG:boto:Token: None 2024-04-13T22:15:37.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3882cfb146436e5-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.926 DEBUG:boto:b'vwnfns-60obj000000000000000001331000falseobj000000000000000001332024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.927 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.927 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.927 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.927 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.927 DEBUG:boto:Method: GET 2024-04-13T22:15:37.927 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000133 2024-04-13T22:15:37.927 DEBUG:boto:Data: 2024-04-13T22:15:37.927 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.927 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.927 DEBUG:boto:Params: {} 2024-04-13T22:15:37.928 DEBUG:boto:Token: None 2024-04-13T22:15:37.928 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/ 2024-04-13T22:15:37.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk= 2024-04-13T22:15:37.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I9FIGmIVaNzkoGkH41yPdpGdfXk=', 'Content-Length': '0'} 2024-04-13T22:15:37.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017cb7933ffd6ff16-00661b0409-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.932 DEBUG:boto:b'vwnfns-60obj000000000000000001331000falseobj000000000000000001332024-04-13T22:15:20.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:37.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-13T22:15:37.932 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.932 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.932 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.932 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.933 DEBUG:boto:Method: GET 2024-04-13T22:15:37.933 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.933 DEBUG:boto:Data: 2024-04-13T22:15:37.933 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.933 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.933 DEBUG:boto:Params: {} 2024-04-13T22:15:37.933 DEBUG:boto:Token: None 2024-04-13T22:15:37.933 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o= 2024-04-13T22:15:37.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o=', 'Content-Length': '0'} 2024-04-13T22:15:37.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212240659adb331f-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.935 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.936 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.936 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.936 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.936 DEBUG:boto:Method: GET 2024-04-13T22:15:37.936 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.936 DEBUG:boto:Data: 2024-04-13T22:15:37.936 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.936 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.936 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.936 DEBUG:boto:Params: {} 2024-04-13T22:15:37.937 DEBUG:boto:Token: None 2024-04-13T22:15:37.937 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o= 2024-04-13T22:15:37.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o=', 'Content-Length': '0'} 2024-04-13T22:15:37.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d8afc626967186-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.939 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.939 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.939 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.939 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.939 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.939 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.939 DEBUG:boto:Data: 2024-04-13T22:15:37.939 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.939 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.939 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.940 DEBUG:boto:Params: {} 2024-04-13T22:15:37.940 DEBUG:boto:Token: None 2024-04-13T22:15:37.940 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MoJA4US8E+NTf5Y+dBxPp5wS8mE= 2024-04-13T22:15:37.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MoJA4US8E+NTf5Y+dBxPp5wS8mE=', 'Content-Length': '0'} 2024-04-13T22:15:37.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078bd7837db4b2c47-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.942 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.942 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.942 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.942 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.942 DEBUG:boto:Method: HEAD 2024-04-13T22:15:37.942 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.942 DEBUG:boto:Data: 2024-04-13T22:15:37.942 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.942 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.943 DEBUG:boto:Params: {} 2024-04-13T22:15:37.943 DEBUG:boto:Token: None 2024-04-13T22:15:37.943 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.943 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MoJA4US8E+NTf5Y+dBxPp5wS8mE= 2024-04-13T22:15:37.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MoJA4US8E+NTf5Y+dBxPp5wS8mE=', 'Content-Length': '0'} 2024-04-13T22:15:37.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2e99f45e42f982-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-13T22:15:37.945 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.945 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.945 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.945 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.945 DEBUG:boto:Method: GET 2024-04-13T22:15:37.945 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.945 DEBUG:boto:Data: 2024-04-13T22:15:37.945 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.946 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.946 DEBUG:boto:Params: {} 2024-04-13T22:15:37.946 DEBUG:boto:Token: None 2024-04-13T22:15:37.946 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.946 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o= 2024-04-13T22:15:37.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o=', 'Content-Length': '0'} 2024-04-13T22:15:37.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c6a15dfdfdc4122-00661b0409-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.948 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.948 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:37.948 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.948 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.948 DEBUG:boto:Method: GET 2024-04-13T22:15:37.948 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.948 DEBUG:boto:Data: 2024-04-13T22:15:37.948 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.949 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:37.949 DEBUG:boto:Port: 8000 2024-04-13T22:15:37.949 DEBUG:boto:Params: {} 2024-04-13T22:15:37.949 DEBUG:boto:Token: None 2024-04-13T22:15:37.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:37 GMT /vwnfns-60/obj00000000000000000162?versionId=null 2024-04-13T22:15:37.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o= 2024-04-13T22:15:37.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JtXEQMHGXbYpE8MjzgSoLw2Yp/o=', 'Content-Length': '0'} 2024-04-13T22:15:37.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066e9a3f74e75d1a-00661b0409-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:37.998 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:37.999 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:37.999 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:37.999 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:37.999 DEBUG:boto:Method: GET 2024-04-13T22:15:37.999 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:37.999 DEBUG:boto:Data: 2024-04-13T22:15:37.999 DEBUG:boto:Headers: {} 2024-04-13T22:15:37.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:37.999 DEBUG:boto:Port: 8001 2024-04-13T22:15:37.999 DEBUG:boto:Params: {} 2024-04-13T22:15:38.000 DEBUG:boto:Token: None 2024-04-13T22:15:38.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f35a2a877bfad0e-00661b0409-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.002 DEBUG:boto:b'vwnfns-60obj000000000000000001621000falseobj000000000000000001622024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.003 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.003 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.003 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.003 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.003 DEBUG:boto:Method: GET 2024-04-13T22:15:38.003 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.003 DEBUG:boto:Data: 2024-04-13T22:15:38.003 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.004 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.004 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.004 DEBUG:boto:Params: {} 2024-04-13T22:15:38.004 DEBUG:boto:Token: None 2024-04-13T22:15:38.004 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000432d94873e2f4f86-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.008 DEBUG:boto:b'vwnfns-60obj000000000000000001621000falseobj000000000000000001622024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.008 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.008 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.008 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.008 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.008 DEBUG:boto:Data: 2024-04-13T22:15:38.008 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.008 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.009 DEBUG:boto:Params: {} 2024-04-13T22:15:38.009 DEBUG:boto:Token: None 2024-04-13T22:15:38.009 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yP0kZ3mlJDUnhZk+Q53Ckwk1UH8= 2024-04-13T22:15:38.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yP0kZ3mlJDUnhZk+Q53Ckwk1UH8=', 'Content-Length': '0'} 2024-04-13T22:15:38.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad364500d72ab89-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.011 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.011 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.011 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.011 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.011 DEBUG:boto:Data: 2024-04-13T22:15:38.011 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.011 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.012 DEBUG:boto:Params: {} 2024-04-13T22:15:38.012 DEBUG:boto:Token: None 2024-04-13T22:15:38.012 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yP0kZ3mlJDUnhZk+Q53Ckwk1UH8= 2024-04-13T22:15:38.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yP0kZ3mlJDUnhZk+Q53Ckwk1UH8=', 'Content-Length': '0'} 2024-04-13T22:15:38.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f7cd5ea9bb0ab2-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-13T22:15:38.014 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.014 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.014 DEBUG:boto:Method: GET 2024-04-13T22:15:38.014 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.014 DEBUG:boto:Data: 2024-04-13T22:15:38.014 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.015 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.015 DEBUG:boto:Params: {} 2024-04-13T22:15:38.015 DEBUG:boto:Token: None 2024-04-13T22:15:38.015 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.015 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1anFuu3UTG9SC3Uu4Va0WcTXqaA= 2024-04-13T22:15:38.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1anFuu3UTG9SC3Uu4Va0WcTXqaA=', 'Content-Length': '0'} 2024-04-13T22:15:38.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e433de8301df1714-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.058 DEBUG:boto:path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.058 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.059 DEBUG:boto:Method: GET 2024-04-13T22:15:38.059 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.059 DEBUG:boto:Data: 2024-04-13T22:15:38.059 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.059 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.059 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.059 DEBUG:boto:Params: {} 2024-04-13T22:15:38.059 DEBUG:boto:Token: None 2024-04-13T22:15:38.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000162 2024-04-13T22:15:38.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1anFuu3UTG9SC3Uu4Va0WcTXqaA= 2024-04-13T22:15:38.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1anFuu3UTG9SC3Uu4Va0WcTXqaA=', 'Content-Length': '0'} 2024-04-13T22:15:38.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86200541282724f-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.062 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.062 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.062 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.062 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.062 DEBUG:boto:Method: GET 2024-04-13T22:15:38.062 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.062 DEBUG:boto:Data: 2024-04-13T22:15:38.062 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.062 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.062 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.063 DEBUG:boto:Params: {} 2024-04-13T22:15:38.063 DEBUG:boto:Token: None 2024-04-13T22:15:38.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b662e8b455ae69f2-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.065 DEBUG:boto:b'vwnfns-60obj000000000000000001621000falseobj000000000000000001622024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.066 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.066 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.066 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.066 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.066 DEBUG:boto:Method: GET 2024-04-13T22:15:38.066 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000162 2024-04-13T22:15:38.066 DEBUG:boto:Data: 2024-04-13T22:15:38.066 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.066 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.066 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.067 DEBUG:boto:Params: {} 2024-04-13T22:15:38.067 DEBUG:boto:Token: None 2024-04-13T22:15:38.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000514c42c086352721-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.071 DEBUG:boto:b'vwnfns-60obj000000000000000001621000falseobj000000000000000001622024-04-13T22:15:21.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-13T22:15:38.071 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.071 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.071 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.071 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.071 DEBUG:boto:Method: GET 2024-04-13T22:15:38.072 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.072 DEBUG:boto:Data: 2024-04-13T22:15:38.072 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.072 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.072 DEBUG:boto:Params: {} 2024-04-13T22:15:38.072 DEBUG:boto:Token: None 2024-04-13T22:15:38.073 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.073 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o= 2024-04-13T22:15:38.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o=', 'Content-Length': '0'} 2024-04-13T22:15:38.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d2de8104d21042-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.075 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.075 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.075 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.075 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.075 DEBUG:boto:Method: GET 2024-04-13T22:15:38.075 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.075 DEBUG:boto:Data: 2024-04-13T22:15:38.075 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.075 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.076 DEBUG:boto:Params: {} 2024-04-13T22:15:38.076 DEBUG:boto:Token: None 2024-04-13T22:15:38.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o= 2024-04-13T22:15:38.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o=', 'Content-Length': '0'} 2024-04-13T22:15:38.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc2c97d7d6be31d-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.122 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.123 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.123 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.123 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.123 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.123 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.123 DEBUG:boto:Data: 2024-04-13T22:15:38.123 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.123 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.123 DEBUG:boto:Params: {} 2024-04-13T22:15:38.124 DEBUG:boto:Token: None 2024-04-13T22:15:38.124 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7OfRuwnDZ/IGsNEfVO0D/nLvXw8= 2024-04-13T22:15:38.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7OfRuwnDZ/IGsNEfVO0D/nLvXw8=', 'Content-Length': '0'} 2024-04-13T22:15:38.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2bc499d841ba48d-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.126 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.126 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.126 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.126 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.126 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.126 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.126 DEBUG:boto:Data: 2024-04-13T22:15:38.127 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.127 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.127 DEBUG:boto:Params: {} 2024-04-13T22:15:38.127 DEBUG:boto:Token: None 2024-04-13T22:15:38.127 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.127 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7OfRuwnDZ/IGsNEfVO0D/nLvXw8= 2024-04-13T22:15:38.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7OfRuwnDZ/IGsNEfVO0D/nLvXw8=', 'Content-Length': '0'} 2024-04-13T22:15:38.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce2b545fc4be992-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-13T22:15:38.129 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.129 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.129 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.129 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.129 DEBUG:boto:Method: GET 2024-04-13T22:15:38.130 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.130 DEBUG:boto:Data: 2024-04-13T22:15:38.130 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.130 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.130 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.130 DEBUG:boto:Params: {} 2024-04-13T22:15:38.130 DEBUG:boto:Token: None 2024-04-13T22:15:38.130 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.131 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o= 2024-04-13T22:15:38.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o=', 'Content-Length': '0'} 2024-04-13T22:15:38.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad846456e7a3dc6-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.132 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.133 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.133 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.133 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.133 DEBUG:boto:Method: GET 2024-04-13T22:15:38.133 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.133 DEBUG:boto:Data: 2024-04-13T22:15:38.133 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.133 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.133 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.133 DEBUG:boto:Params: {} 2024-04-13T22:15:38.133 DEBUG:boto:Token: None 2024-04-13T22:15:38.134 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271?versionId=null 2024-04-13T22:15:38.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o= 2024-04-13T22:15:38.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jk7bZbCkLJSffQmwr0oqQ0F4t7o=', 'Content-Length': '0'} 2024-04-13T22:15:38.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879c99da56015da8-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.135 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.136 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.136 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.136 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.136 DEBUG:boto:Method: GET 2024-04-13T22:15:38.136 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.136 DEBUG:boto:Data: 2024-04-13T22:15:38.136 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.136 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.136 DEBUG:boto:Params: {} 2024-04-13T22:15:38.136 DEBUG:boto:Token: None 2024-04-13T22:15:38.137 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c6e72a6bc98f816-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.139 DEBUG:boto:b'vwnfns-60obj000000000000000002711000falseobj000000000000000002712024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.140 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.140 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.140 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.140 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.140 DEBUG:boto:Method: GET 2024-04-13T22:15:38.140 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.140 DEBUG:boto:Data: 2024-04-13T22:15:38.140 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.141 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.141 DEBUG:boto:Params: {} 2024-04-13T22:15:38.141 DEBUG:boto:Token: None 2024-04-13T22:15:38.141 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd77a94d00512df-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.145 DEBUG:boto:b'vwnfns-60obj000000000000000002711000falseobj000000000000000002712024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.145 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.146 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.146 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.146 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.146 DEBUG:boto:Data: 2024-04-13T22:15:38.146 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.146 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.146 DEBUG:boto:Params: {} 2024-04-13T22:15:38.146 DEBUG:boto:Token: None 2024-04-13T22:15:38.146 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6EeMdvpJq3iu4SwRotaoTF7f44o= 2024-04-13T22:15:38.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6EeMdvpJq3iu4SwRotaoTF7f44o=', 'Content-Length': '0'} 2024-04-13T22:15:38.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cccae8260775ef89-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.148 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.148 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.148 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.149 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.149 DEBUG:boto:Data: 2024-04-13T22:15:38.149 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.149 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.149 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.149 DEBUG:boto:Params: {} 2024-04-13T22:15:38.149 DEBUG:boto:Token: None 2024-04-13T22:15:38.149 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6EeMdvpJq3iu4SwRotaoTF7f44o= 2024-04-13T22:15:38.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6EeMdvpJq3iu4SwRotaoTF7f44o=', 'Content-Length': '0'} 2024-04-13T22:15:38.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b6ab34aa6b0c46-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-13T22:15:38.151 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.151 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.151 DEBUG:boto:Method: GET 2024-04-13T22:15:38.151 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.152 DEBUG:boto:Data: 2024-04-13T22:15:38.152 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.152 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.152 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.152 DEBUG:boto:Params: {} 2024-04-13T22:15:38.152 DEBUG:boto:Token: None 2024-04-13T22:15:38.152 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9WBkSZlefiG9m1VmcJEqsFtwueU= 2024-04-13T22:15:38.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9WBkSZlefiG9m1VmcJEqsFtwueU=', 'Content-Length': '0'} 2024-04-13T22:15:38.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045a04031212cef2d-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.154 DEBUG:boto:path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.154 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.154 DEBUG:boto:Method: GET 2024-04-13T22:15:38.154 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.154 DEBUG:boto:Data: 2024-04-13T22:15:38.155 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.155 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.155 DEBUG:boto:Params: {} 2024-04-13T22:15:38.155 DEBUG:boto:Token: None 2024-04-13T22:15:38.155 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000271 2024-04-13T22:15:38.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9WBkSZlefiG9m1VmcJEqsFtwueU= 2024-04-13T22:15:38.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9WBkSZlefiG9m1VmcJEqsFtwueU=', 'Content-Length': '0'} 2024-04-13T22:15:38.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004abd7ca4918f884f-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.198 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.199 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.199 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.199 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.199 DEBUG:boto:Method: GET 2024-04-13T22:15:38.199 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.199 DEBUG:boto:Data: 2024-04-13T22:15:38.199 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.199 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.199 DEBUG:boto:Params: {} 2024-04-13T22:15:38.199 DEBUG:boto:Token: None 2024-04-13T22:15:38.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bdd387a81d9c39c-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.202 DEBUG:boto:b'vwnfns-60obj000000000000000002711000falseobj000000000000000002712024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.202 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.203 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.203 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.203 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.203 DEBUG:boto:Method: GET 2024-04-13T22:15:38.203 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000271 2024-04-13T22:15:38.203 DEBUG:boto:Data: 2024-04-13T22:15:38.203 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.203 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.203 DEBUG:boto:Params: {} 2024-04-13T22:15:38.203 DEBUG:boto:Token: None 2024-04-13T22:15:38.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd8bd9dd5aed28bc-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.208 DEBUG:boto:b'vwnfns-60obj000000000000000002711000falseobj000000000000000002712024-04-13T22:15:21.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-13T22:15:38.209 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.209 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.209 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.209 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.209 DEBUG:boto:Method: GET 2024-04-13T22:15:38.209 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.210 DEBUG:boto:Data: 2024-04-13T22:15:38.210 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.210 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.210 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.210 DEBUG:boto:Params: {} 2024-04-13T22:15:38.210 DEBUG:boto:Token: None 2024-04-13T22:15:38.210 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM= 2024-04-13T22:15:38.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM=', 'Content-Length': '0'} 2024-04-13T22:15:38.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f288d54c20cc1c6-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.212 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.212 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.213 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.213 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.213 DEBUG:boto:Method: GET 2024-04-13T22:15:38.213 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.213 DEBUG:boto:Data: 2024-04-13T22:15:38.213 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.213 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.213 DEBUG:boto:Params: {} 2024-04-13T22:15:38.213 DEBUG:boto:Token: None 2024-04-13T22:15:38.214 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.214 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM= 2024-04-13T22:15:38.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM=', 'Content-Length': '0'} 2024-04-13T22:15:38.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389106d4906b743e-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.216 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.216 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.216 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.216 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.216 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.216 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.216 DEBUG:boto:Data: 2024-04-13T22:15:38.216 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.216 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.216 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.216 DEBUG:boto:Params: {} 2024-04-13T22:15:38.217 DEBUG:boto:Token: None 2024-04-13T22:15:38.217 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:69YRnT4lXb8TKXjuV79RVRkpTGs= 2024-04-13T22:15:38.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:69YRnT4lXb8TKXjuV79RVRkpTGs=', 'Content-Length': '0'} 2024-04-13T22:15:38.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177ed37c30e2313d-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.219 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.219 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.219 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.219 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.219 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.219 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.219 DEBUG:boto:Data: 2024-04-13T22:15:38.219 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.219 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.219 DEBUG:boto:Params: {} 2024-04-13T22:15:38.220 DEBUG:boto:Token: None 2024-04-13T22:15:38.220 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:69YRnT4lXb8TKXjuV79RVRkpTGs= 2024-04-13T22:15:38.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:69YRnT4lXb8TKXjuV79RVRkpTGs=', 'Content-Length': '0'} 2024-04-13T22:15:38.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007017b02985ffe3b7-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-13T22:15:38.222 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.222 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.222 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.222 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.222 DEBUG:boto:Method: GET 2024-04-13T22:15:38.222 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.222 DEBUG:boto:Data: 2024-04-13T22:15:38.222 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.222 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.222 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.222 DEBUG:boto:Params: {} 2024-04-13T22:15:38.223 DEBUG:boto:Token: None 2024-04-13T22:15:38.223 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.223 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM= 2024-04-13T22:15:38.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM=', 'Content-Length': '0'} 2024-04-13T22:15:38.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb16a76d8c3d316d-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.266 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.266 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.267 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.267 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.267 DEBUG:boto:Method: GET 2024-04-13T22:15:38.267 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.267 DEBUG:boto:Data: 2024-04-13T22:15:38.267 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.267 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.267 DEBUG:boto:Params: {} 2024-04-13T22:15:38.267 DEBUG:boto:Token: None 2024-04-13T22:15:38.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317?versionId=null 2024-04-13T22:15:38.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM= 2024-04-13T22:15:38.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ATmgzM1xJqNhJ/B2RXDGJHg9OeM=', 'Content-Length': '0'} 2024-04-13T22:15:38.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707b83c7b824e9c9-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.270 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.270 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.270 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.270 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.270 DEBUG:boto:Method: GET 2024-04-13T22:15:38.270 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.271 DEBUG:boto:Data: 2024-04-13T22:15:38.271 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.271 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.271 DEBUG:boto:Params: {} 2024-04-13T22:15:38.271 DEBUG:boto:Token: None 2024-04-13T22:15:38.271 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a4206f400a7171-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.274 DEBUG:boto:b'vwnfns-60obj000000000000000003171000falseobj000000000000000003172024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.274 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.274 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.274 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.274 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.275 DEBUG:boto:Method: GET 2024-04-13T22:15:38.275 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.275 DEBUG:boto:Data: 2024-04-13T22:15:38.275 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.275 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.275 DEBUG:boto:Params: {} 2024-04-13T22:15:38.275 DEBUG:boto:Token: None 2024-04-13T22:15:38.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ac3830868b7182-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.279 DEBUG:boto:b'vwnfns-60obj000000000000000003171000falseobj000000000000000003172024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.279 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.279 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.279 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.279 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.280 DEBUG:boto:Data: 2024-04-13T22:15:38.280 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.280 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.280 DEBUG:boto:Params: {} 2024-04-13T22:15:38.280 DEBUG:boto:Token: None 2024-04-13T22:15:38.280 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+9Zfm1JyESmNpAo4JkyFbtrAZTU= 2024-04-13T22:15:38.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+9Zfm1JyESmNpAo4JkyFbtrAZTU=', 'Content-Length': '0'} 2024-04-13T22:15:38.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4223c6f96d54cb0-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.282 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.282 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.283 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.283 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.283 DEBUG:boto:Data: 2024-04-13T22:15:38.283 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.283 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.283 DEBUG:boto:Params: {} 2024-04-13T22:15:38.283 DEBUG:boto:Token: None 2024-04-13T22:15:38.283 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+9Zfm1JyESmNpAo4JkyFbtrAZTU= 2024-04-13T22:15:38.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+9Zfm1JyESmNpAo4JkyFbtrAZTU=', 'Content-Length': '0'} 2024-04-13T22:15:38.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000049e97dcfd8e4345-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-13T22:15:38.285 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.285 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.285 DEBUG:boto:Method: GET 2024-04-13T22:15:38.286 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.286 DEBUG:boto:Data: 2024-04-13T22:15:38.286 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.286 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.286 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.286 DEBUG:boto:Params: {} 2024-04-13T22:15:38.286 DEBUG:boto:Token: None 2024-04-13T22:15:38.286 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eJ1ujaIVY3wUhtS34f6bAQN9l/k= 2024-04-13T22:15:38.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eJ1ujaIVY3wUhtS34f6bAQN9l/k=', 'Content-Length': '0'} 2024-04-13T22:15:38.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000455197deca4e0f83-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.288 DEBUG:boto:path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.289 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.289 DEBUG:boto:Method: GET 2024-04-13T22:15:38.289 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.289 DEBUG:boto:Data: 2024-04-13T22:15:38.289 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.289 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.289 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.289 DEBUG:boto:Params: {} 2024-04-13T22:15:38.289 DEBUG:boto:Token: None 2024-04-13T22:15:38.289 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000317 2024-04-13T22:15:38.290 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eJ1ujaIVY3wUhtS34f6bAQN9l/k= 2024-04-13T22:15:38.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eJ1ujaIVY3wUhtS34f6bAQN9l/k=', 'Content-Length': '0'} 2024-04-13T22:15:38.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000018e86fb6245b8cc-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.338 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.339 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.339 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.339 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.339 DEBUG:boto:Method: GET 2024-04-13T22:15:38.339 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.339 DEBUG:boto:Data: 2024-04-13T22:15:38.339 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.339 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.339 DEBUG:boto:Params: {} 2024-04-13T22:15:38.339 DEBUG:boto:Token: None 2024-04-13T22:15:38.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab75f04f6a12e28b-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.342 DEBUG:boto:b'vwnfns-60obj000000000000000003171000falseobj000000000000000003172024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.343 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.343 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.343 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.343 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.343 DEBUG:boto:Method: GET 2024-04-13T22:15:38.343 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000317 2024-04-13T22:15:38.343 DEBUG:boto:Data: 2024-04-13T22:15:38.343 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.343 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.343 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.344 DEBUG:boto:Params: {} 2024-04-13T22:15:38.344 DEBUG:boto:Token: None 2024-04-13T22:15:38.344 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018b88f6f16bf9e17-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.348 DEBUG:boto:b'vwnfns-60obj000000000000000003171000falseobj000000000000000003172024-04-13T22:15:21.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-13T22:15:38.348 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.348 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.348 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.348 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.348 DEBUG:boto:Method: GET 2024-04-13T22:15:38.349 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.349 DEBUG:boto:Data: 2024-04-13T22:15:38.349 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.349 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.349 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.349 DEBUG:boto:Params: {} 2024-04-13T22:15:38.349 DEBUG:boto:Token: None 2024-04-13T22:15:38.349 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.349 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc= 2024-04-13T22:15:38.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc=', 'Content-Length': '0'} 2024-04-13T22:15:38.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de3be5dfe3f58731-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.352 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.352 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.352 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.352 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.352 DEBUG:boto:Method: GET 2024-04-13T22:15:38.352 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.352 DEBUG:boto:Data: 2024-04-13T22:15:38.352 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.352 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.352 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.353 DEBUG:boto:Params: {} 2024-04-13T22:15:38.353 DEBUG:boto:Token: None 2024-04-13T22:15:38.353 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.353 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc= 2024-04-13T22:15:38.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc=', 'Content-Length': '0'} 2024-04-13T22:15:38.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f632874d05ef942-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.355 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.355 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.355 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.355 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.355 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.355 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.355 DEBUG:boto:Data: 2024-04-13T22:15:38.356 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.356 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.356 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.356 DEBUG:boto:Params: {} 2024-04-13T22:15:38.356 DEBUG:boto:Token: None 2024-04-13T22:15:38.356 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AUVRhjEfIPu5Uj3ljegYvfLJPg4= 2024-04-13T22:15:38.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AUVRhjEfIPu5Uj3ljegYvfLJPg4=', 'Content-Length': '0'} 2024-04-13T22:15:38.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004562397fd1d22b1e-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.358 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.358 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.358 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.358 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.358 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.358 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.358 DEBUG:boto:Data: 2024-04-13T22:15:38.359 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.359 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.359 DEBUG:boto:Params: {} 2024-04-13T22:15:38.359 DEBUG:boto:Token: None 2024-04-13T22:15:38.359 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.359 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AUVRhjEfIPu5Uj3ljegYvfLJPg4= 2024-04-13T22:15:38.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AUVRhjEfIPu5Uj3ljegYvfLJPg4=', 'Content-Length': '0'} 2024-04-13T22:15:38.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e34aedc10b56fd0-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-13T22:15:38.361 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.361 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.361 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.361 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.361 DEBUG:boto:Method: GET 2024-04-13T22:15:38.362 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.362 DEBUG:boto:Data: 2024-04-13T22:15:38.362 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.362 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.362 DEBUG:boto:Params: {} 2024-04-13T22:15:38.362 DEBUG:boto:Token: None 2024-04-13T22:15:38.362 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.362 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc= 2024-04-13T22:15:38.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc=', 'Content-Length': '0'} 2024-04-13T22:15:38.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c3c5116070b0b3-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.364 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.364 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.364 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.364 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.365 DEBUG:boto:Method: GET 2024-04-13T22:15:38.365 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.365 DEBUG:boto:Data: 2024-04-13T22:15:38.365 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.365 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.365 DEBUG:boto:Params: {} 2024-04-13T22:15:38.365 DEBUG:boto:Token: None 2024-04-13T22:15:38.365 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423?versionId=null 2024-04-13T22:15:38.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc= 2024-04-13T22:15:38.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nC+oMWqKYj/qjc31pL+QVFwFEGc=', 'Content-Length': '0'} 2024-04-13T22:15:38.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000580a0d4197b1ba55-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.410 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.410 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.411 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.411 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.411 DEBUG:boto:Method: GET 2024-04-13T22:15:38.411 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.411 DEBUG:boto:Data: 2024-04-13T22:15:38.411 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.411 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.411 DEBUG:boto:Params: {} 2024-04-13T22:15:38.411 DEBUG:boto:Token: None 2024-04-13T22:15:38.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050ea464cc5b312da-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.414 DEBUG:boto:b'vwnfns-60obj000000000000000004231000falseobj000000000000000004232024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.415 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.415 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.415 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.415 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.415 DEBUG:boto:Method: GET 2024-04-13T22:15:38.415 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.415 DEBUG:boto:Data: 2024-04-13T22:15:38.415 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.415 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.416 DEBUG:boto:Params: {} 2024-04-13T22:15:38.416 DEBUG:boto:Token: None 2024-04-13T22:15:38.416 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fdc95a2965819a3-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.419 DEBUG:boto:b'vwnfns-60obj000000000000000004231000falseobj000000000000000004232024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.420 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.420 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.420 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.420 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.420 DEBUG:boto:Data: 2024-04-13T22:15:38.420 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.420 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.420 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.420 DEBUG:boto:Params: {} 2024-04-13T22:15:38.421 DEBUG:boto:Token: None 2024-04-13T22:15:38.421 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.421 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F7lwvJe45EHmj22xD6xUglEMOM4= 2024-04-13T22:15:38.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F7lwvJe45EHmj22xD6xUglEMOM4=', 'Content-Length': '0'} 2024-04-13T22:15:38.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a8abdcc8b3acf13-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.423 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.423 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.423 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.423 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.423 DEBUG:boto:Data: 2024-04-13T22:15:38.423 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.423 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.423 DEBUG:boto:Params: {} 2024-04-13T22:15:38.423 DEBUG:boto:Token: None 2024-04-13T22:15:38.424 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F7lwvJe45EHmj22xD6xUglEMOM4= 2024-04-13T22:15:38.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F7lwvJe45EHmj22xD6xUglEMOM4=', 'Content-Length': '0'} 2024-04-13T22:15:38.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a36db035dee304c9-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-13T22:15:38.426 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.426 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.426 DEBUG:boto:Method: GET 2024-04-13T22:15:38.426 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.426 DEBUG:boto:Data: 2024-04-13T22:15:38.426 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.426 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.427 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.427 DEBUG:boto:Params: {} 2024-04-13T22:15:38.427 DEBUG:boto:Token: None 2024-04-13T22:15:38.427 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7fI+J5wSBfqsNFlvyQ8E+vUN77g= 2024-04-13T22:15:38.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7fI+J5wSBfqsNFlvyQ8E+vUN77g=', 'Content-Length': '0'} 2024-04-13T22:15:38.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fdd2bdf03627fda-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.470 DEBUG:boto:path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.471 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.471 DEBUG:boto:Method: GET 2024-04-13T22:15:38.471 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.471 DEBUG:boto:Data: 2024-04-13T22:15:38.471 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.471 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.471 DEBUG:boto:Params: {} 2024-04-13T22:15:38.471 DEBUG:boto:Token: None 2024-04-13T22:15:38.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000423 2024-04-13T22:15:38.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7fI+J5wSBfqsNFlvyQ8E+vUN77g= 2024-04-13T22:15:38.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7fI+J5wSBfqsNFlvyQ8E+vUN77g=', 'Content-Length': '0'} 2024-04-13T22:15:38.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012c11043372192e6-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.474 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.474 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.474 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.474 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.474 DEBUG:boto:Method: GET 2024-04-13T22:15:38.474 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.474 DEBUG:boto:Data: 2024-04-13T22:15:38.475 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.475 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.475 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.475 DEBUG:boto:Params: {} 2024-04-13T22:15:38.475 DEBUG:boto:Token: None 2024-04-13T22:15:38.475 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.475 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b5cff8e7fd4c73-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.477 DEBUG:boto:b'vwnfns-60obj000000000000000004231000falseobj000000000000000004232024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.478 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.478 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.478 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.478 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.478 DEBUG:boto:Method: GET 2024-04-13T22:15:38.478 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000423 2024-04-13T22:15:38.478 DEBUG:boto:Data: 2024-04-13T22:15:38.479 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.479 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.479 DEBUG:boto:Params: {} 2024-04-13T22:15:38.479 DEBUG:boto:Token: None 2024-04-13T22:15:38.479 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca2afee4f0e03d0e-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.483 DEBUG:boto:b'vwnfns-60obj000000000000000004231000falseobj000000000000000004232024-04-13T22:15:21.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-13T22:15:38.483 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.483 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.483 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.484 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.484 DEBUG:boto:Method: GET 2024-04-13T22:15:38.484 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.484 DEBUG:boto:Data: 2024-04-13T22:15:38.484 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.484 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.484 DEBUG:boto:Params: {} 2024-04-13T22:15:38.484 DEBUG:boto:Token: None 2024-04-13T22:15:38.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns= 2024-04-13T22:15:38.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns=', 'Content-Length': '0'} 2024-04-13T22:15:38.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000125582940e89cf3c-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.487 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.487 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.487 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.487 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.487 DEBUG:boto:Method: GET 2024-04-13T22:15:38.487 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.487 DEBUG:boto:Data: 2024-04-13T22:15:38.487 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.487 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.488 DEBUG:boto:Params: {} 2024-04-13T22:15:38.488 DEBUG:boto:Token: None 2024-04-13T22:15:38.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns= 2024-04-13T22:15:38.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns=', 'Content-Length': '0'} 2024-04-13T22:15:38.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a48fe6013505d4-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.530 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.530 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.530 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.531 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.531 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.531 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.531 DEBUG:boto:Data: 2024-04-13T22:15:38.531 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.531 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.531 DEBUG:boto:Params: {} 2024-04-13T22:15:38.531 DEBUG:boto:Token: None 2024-04-13T22:15:38.532 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/O3nq4jX8DPpZ3gltZKmEYQl0Ac= 2024-04-13T22:15:38.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/O3nq4jX8DPpZ3gltZKmEYQl0Ac=', 'Content-Length': '0'} 2024-04-13T22:15:38.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001927dec72c86d79f-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.534 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.534 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.534 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.534 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.534 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.534 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.534 DEBUG:boto:Data: 2024-04-13T22:15:38.534 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.535 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.535 DEBUG:boto:Params: {} 2024-04-13T22:15:38.535 DEBUG:boto:Token: None 2024-04-13T22:15:38.535 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/O3nq4jX8DPpZ3gltZKmEYQl0Ac= 2024-04-13T22:15:38.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/O3nq4jX8DPpZ3gltZKmEYQl0Ac=', 'Content-Length': '0'} 2024-04-13T22:15:38.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a043cab7048181-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-13T22:15:38.537 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.537 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.537 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.537 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.537 DEBUG:boto:Method: GET 2024-04-13T22:15:38.538 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.538 DEBUG:boto:Data: 2024-04-13T22:15:38.538 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.538 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.538 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.538 DEBUG:boto:Params: {} 2024-04-13T22:15:38.538 DEBUG:boto:Token: None 2024-04-13T22:15:38.538 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.538 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns= 2024-04-13T22:15:38.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns=', 'Content-Length': '0'} 2024-04-13T22:15:38.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f78d3116cef597d-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.540 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.540 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.540 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.540 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.540 DEBUG:boto:Method: GET 2024-04-13T22:15:38.541 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.541 DEBUG:boto:Data: 2024-04-13T22:15:38.541 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.541 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.541 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.541 DEBUG:boto:Params: {} 2024-04-13T22:15:38.541 DEBUG:boto:Token: None 2024-04-13T22:15:38.542 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467?versionId=null 2024-04-13T22:15:38.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns= 2024-04-13T22:15:38.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lZRPqyMoska7bqM0NxVXZBPr9Ns=', 'Content-Length': '0'} 2024-04-13T22:15:38.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a44c3c9e825e74-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.543 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.544 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.544 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.544 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.544 DEBUG:boto:Method: GET 2024-04-13T22:15:38.544 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.544 DEBUG:boto:Data: 2024-04-13T22:15:38.544 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.544 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.544 DEBUG:boto:Params: {} 2024-04-13T22:15:38.545 DEBUG:boto:Token: None 2024-04-13T22:15:38.545 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000859bd9e95f918cf0-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.547 DEBUG:boto:b'vwnfns-60obj000000000000000004671000falseobj000000000000000004672024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.548 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.548 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.548 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.548 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.548 DEBUG:boto:Method: GET 2024-04-13T22:15:38.548 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.548 DEBUG:boto:Data: 2024-04-13T22:15:38.548 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.548 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.549 DEBUG:boto:Params: {} 2024-04-13T22:15:38.549 DEBUG:boto:Token: None 2024-04-13T22:15:38.549 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb73db2278d4ce6-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.553 DEBUG:boto:b'vwnfns-60obj000000000000000004671000falseobj000000000000000004672024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.553 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.553 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.553 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.553 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.553 DEBUG:boto:Data: 2024-04-13T22:15:38.553 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.554 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.554 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.554 DEBUG:boto:Params: {} 2024-04-13T22:15:38.554 DEBUG:boto:Token: None 2024-04-13T22:15:38.554 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.554 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m54lvwlVYOL9vEcCjKMMJrHGa5Q= 2024-04-13T22:15:38.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m54lvwlVYOL9vEcCjKMMJrHGa5Q=', 'Content-Length': '0'} 2024-04-13T22:15:38.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8bfa938bb02844b-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.556 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.556 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.556 DEBUG:boto:Method: HEAD 2024-04-13T22:15:38.556 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.556 DEBUG:boto:Data: 2024-04-13T22:15:38.556 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.556 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.557 DEBUG:boto:Params: {} 2024-04-13T22:15:38.557 DEBUG:boto:Token: None 2024-04-13T22:15:38.557 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m54lvwlVYOL9vEcCjKMMJrHGa5Q= 2024-04-13T22:15:38.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m54lvwlVYOL9vEcCjKMMJrHGa5Q=', 'Content-Length': '0'} 2024-04-13T22:15:38.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17b8a53c5825fa5-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-13T22:15:38.559 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.559 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.559 DEBUG:boto:Method: GET 2024-04-13T22:15:38.559 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.559 DEBUG:boto:Data: 2024-04-13T22:15:38.559 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.559 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.559 DEBUG:boto:Params: {} 2024-04-13T22:15:38.560 DEBUG:boto:Token: None 2024-04-13T22:15:38.560 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5ojR4UKWRWI9VYR7VFuOZefrDew= 2024-04-13T22:15:38.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5ojR4UKWRWI9VYR7VFuOZefrDew=', 'Content-Length': '0'} 2024-04-13T22:15:38.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-60:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ed702e2d0ba710-00661b040a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.561 DEBUG:boto:path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.562 DEBUG:boto:auth_path=/vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.562 DEBUG:boto:Method: GET 2024-04-13T22:15:38.562 DEBUG:boto:Path: /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.562 DEBUG:boto:Data: 2024-04-13T22:15:38.562 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.562 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.562 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.562 DEBUG:boto:Params: {} 2024-04-13T22:15:38.562 DEBUG:boto:Token: None 2024-04-13T22:15:38.563 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/obj00000000000000000467 2024-04-13T22:15:38.563 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5ojR4UKWRWI9VYR7VFuOZefrDew= 2024-04-13T22:15:38.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5ojR4UKWRWI9VYR7VFuOZefrDew=', 'Content-Length': '0'} 2024-04-13T22:15:38.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:15:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a2cddad06959ce-00661b040a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.606 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.606 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.607 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.607 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.607 DEBUG:boto:Method: GET 2024-04-13T22:15:38.607 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.607 DEBUG:boto:Data: 2024-04-13T22:15:38.607 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:15:38.607 DEBUG:boto:Port: 8001 2024-04-13T22:15:38.607 DEBUG:boto:Params: {} 2024-04-13T22:15:38.608 DEBUG:boto:Token: None 2024-04-13T22:15:38.608 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005057ec8fb77928fc-00661b040a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.610 DEBUG:boto:b'vwnfns-60obj000000000000000004671000falseobj000000000000000004672024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.611 DEBUG:boto:path=/vwnfns-60/ 2024-04-13T22:15:38.611 DEBUG:boto:auth_path=/vwnfns-60/ 2024-04-13T22:15:38.611 DEBUG:boto:path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.611 DEBUG:boto:auth_path=/vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.611 DEBUG:boto:Method: GET 2024-04-13T22:15:38.611 DEBUG:boto:Path: /vwnfns-60/?prefix=obj00000000000000000467 2024-04-13T22:15:38.611 DEBUG:boto:Data: 2024-04-13T22:15:38.611 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.612 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.612 DEBUG:boto:Params: {} 2024-04-13T22:15:38.612 DEBUG:boto:Token: None 2024-04-13T22:15:38.612 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-60/ 2024-04-13T22:15:38.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI= 2024-04-13T22:15:38.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:loFThe5+w+bBGI50hHv5b/E7qfI=', 'Content-Length': '0'} 2024-04-13T22:15:38.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d888fe2ab1b8308-00661b040a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.616 DEBUG:boto:b'vwnfns-60obj000000000000000004671000falseobj000000000000000004672024-04-13T22:15:21.441Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:15:38.617 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-60 zones={a2, a1} 2024-04-13T22:15:38.617 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2024-04-13T22:15:38.619 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:15:38.619 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:15:38.620 DEBUG:boto:Method: PUT 2024-04-13T22:15:38.620 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:15:38.620 DEBUG:boto:Data: 2024-04-13T22:15:38.620 DEBUG:boto:Headers: {} 2024-04-13T22:15:38.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:15:38.620 DEBUG:boto:Port: 8000 2024-04-13T22:15:38.620 DEBUG:boto:Params: {} 2024-04-13T22:15:38.620 DEBUG:boto:Token: None 2024-04-13T22:15:38.620 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:15:38 GMT /vwnfns-61/ 2024-04-13T22:15:38.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b8TDaYrIEAJo0sd9mbVq0TullnU= 2024-04-13T22:15:38.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:15:38 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b8TDaYrIEAJo0sd9mbVq0TullnU=', 'Content-Length': '0'} 2024-04-13T22:15:38.634 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1bb6b0dc7f7df8b-00661b040a-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:15:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:15:38.634 DEBUG:rgw_multi.tests:created bucket=vwnfns-61 2024-04-13T22:15:38.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:38.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:38.635 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:38.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:38.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:38.712 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:15:38.725 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:15:38.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.725 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046538.631420_800.1", 2024-04-13T22:15:38.725 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:15:38.631420Z" 2024-04-13T22:15:38.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:38.726 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "0.000000" 2024-04-13T22:15:38.727 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:38.727 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:15:38.728 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:38.728 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:15:38.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:38.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:38.728 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:38.776 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:38.776 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:38.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:38.802+0000 7fdd9b4ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:38.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:38.803+0000 7fdd9b4ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:38.818 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.819 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:15:38.820 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:38.821 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:15:38.822 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:15:38.823 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:38.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.825 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:15:38.825 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:15:38.825 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:15:38.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:38.825 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:15:38Z" 2024-04-13T22:15:38.825 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:38.826 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:15:38Z" } 2024-04-13T22:15:38.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:15:38.826 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:38.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:38.827 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:15:43.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:43.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:43.828 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:43.883 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:43.883 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:43.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:43.914+0000 7f309190c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:43.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:43.915+0000 7f309190c780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:43.931 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:15:43.932 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:43.933 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:43.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:43.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:43.936 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:43.937 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:15:43Z" 2024-04-13T22:15:43.938 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:43.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:15:43Z" } 2024-04-13T22:15:43.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:15:43.939 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:43.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:43.939 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:15:48.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:48.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:48.941 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:48.997 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:48.997 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:49.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:49.026+0000 7fa43396f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:49.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:49.027+0000 7fa43396f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:15:49.044 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:15:49.045 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:49.046 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:49.047 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:49.048 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:15:49.049 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:15:49.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:49.050 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:15:49Z" 2024-04-13T22:15:49.050 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:49.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:15:49Z" } 2024-04-13T22:15:49.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:15:49.051 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:49.051 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:49.051 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:15:54.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:54.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:54.053 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:54.109 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:54.109 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:54.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:54.138+0000 7f0faf395780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:54.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:54.139+0000 7f0faf395780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:54.156 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:54.156 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:15:54.156 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:15:54.156 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:54.157 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:15:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:54.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:54.160 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:54.161 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:15:54Z" 2024-04-13T22:15:54.162 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:54.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:15:54Z" } 2024-04-13T22:15:54.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:15:54.164 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:54.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:54.164 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:15:59.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:59.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:15:59.166 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:15:59.221 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:15:59.221 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:15:59.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:59.254+0000 7f8980ff8780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:59.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:15:59.255+0000 7f8980ff8780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:59.273 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:15:59.274 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:59.275 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:15:59.276 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:15:59Z" 2024-04-13T22:15:59.277 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:15:59.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:15:59Z" } 2024-04-13T22:15:59.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:15:59.279 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:59.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:15:59.279 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:16:04.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:04.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:04.280 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:04.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:04.335 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:04.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:04.365+0000 7f3726d08780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:04.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:04.366+0000 7f3726d08780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:04.383 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:04.383 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:04.384 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:04.385 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:04.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:04.388 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:04Z" 2024-04-13T22:16:04.389 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:04.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:04Z" } 2024-04-13T22:16:04.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:04.391 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:04.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:04.391 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:16:09.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:09.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:09.393 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:09.449 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:09.449 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:09.478+0000 7f41cb5d5780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:09.479+0000 7f41cb5d5780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:09.495 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:09.496 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:09.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:09.498 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:09.499 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:09.500 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:09Z" 2024-04-13T22:16:09.501 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:09.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:09Z" } 2024-04-13T22:16:09.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:09.503 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:09.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:09.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:16:14.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:14.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:14.504 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:14.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:14.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:14.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:14.591+0000 7f738a85a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:14.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:14.592+0000 7f738a85a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:14.609 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:14.609 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:14.609 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:14.609 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:14.610 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:14.611 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:14.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:14.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:14.614 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:14Z" 2024-04-13T22:16:14.615 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:14.616 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:14Z" } 2024-04-13T22:16:14.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:14.617 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:14.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:14.617 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:16:19.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:19.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:19.619 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:19.675 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:19.675 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:19.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:19.705+0000 7f5b94072780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:19.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:19.706+0000 7f5b94072780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:19.722 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:19.722 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:19.722 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:19.722 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:19.722 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:19.722 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046494.022684_767.1", 2024-04-13T22:16:19.723 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:14:54.022684Z", 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:19.725 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:19.726 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:19.727 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:19Z" 2024-04-13T22:16:19.728 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:19.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046494.022684_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:14:54.022684Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:19Z" } 2024-04-13T22:16:19.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046494.022684_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:14:54.022684Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:19.729 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:19.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046494.022684_767.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:19.730 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046494.022684_767.1 2024-04-13T22:16:24.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:24.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:24.731 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:24.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:24.791 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:24.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:24.826+0000 7f9d76e09780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:24.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:24.827+0000 7f9d76e09780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:24.842 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046517.246368_771.1", 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:15:17.246368Z", 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:24.844 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:24.845 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:24.846 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:24.847 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:24.848 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:24Z" 2024-04-13T22:16:24.849 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:24.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046517.246368_771.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:15:17.246368Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:24Z" } 2024-04-13T22:16:24.851 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046517.246368_771.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:15:17.246368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:24.851 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:24.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046517.246368_771.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:24.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046517.246368_771.1 2024-04-13T22:16:29.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:29.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:29.853 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:29.908 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:29.909 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:29.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:29.937+0000 7facaf865780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:29.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:29.938+0000 7facaf865780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:29.955 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046517.246368_771.1", 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:15:17.246368Z", 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:29Z" 2024-04-13T22:16:29.961 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:29.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046517.246368_771.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:15:17.246368Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:29Z" } 2024-04-13T22:16:29.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046517.246368_771.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:15:17.246368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:29.963 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:29.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046517.246368_771.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:29.963 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713046538.631420_800.1 target=1_1713046517.246368_771.1 2024-04-13T22:16:34.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:34.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:16:34.965 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:16:35.019 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:35.019 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:35.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:35.048+0000 7f0a23fcf780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:35.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:35.049+0000 7f0a23fcf780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:16:35.066 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:16:35.066 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:16:35.066 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:35.067 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713046538.631420_800.1", 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:15:38.631420Z", 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:35.068 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:35.069 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:16:35.070 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:16:35.071 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:16:35.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:16:35.072 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:16:35Z" 2024-04-13T22:16:35.072 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:16:35.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713046538.631420_800.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:15:38.631420Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:16:35Z" } 2024-04-13T22:16:35.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046538.631420_800.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:15:38.631420Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:16:35.073 DEBUG:rgw_multi.tests:log_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:35.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1713046538.631420_800.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:16:35.073 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:16:35.074 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:16:35.074 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:16:35.074 DEBUG:boto:Method: HEAD 2024-04-13T22:16:35.074 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:16:35.074 DEBUG:boto:Data: 2024-04-13T22:16:35.074 DEBUG:boto:Headers: {} 2024-04-13T22:16:35.074 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.074 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.074 DEBUG:boto:Params: {} 2024-04-13T22:16:35.075 DEBUG:boto:Token: None 2024-04-13T22:16:35.075 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/ 2024-04-13T22:16:35.075 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk= 2024-04-13T22:16:35.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk=', 'Content-Length': '0'} 2024-04-13T22:16:35.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008743166eede19f5f-00661b0443-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.079 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:16:35.079 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:16:35.079 DEBUG:boto:Method: PUT 2024-04-13T22:16:35.079 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:16:35.079 DEBUG:boto:Data: 2024-04-13T22:16:35.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:16:35.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.079 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.079 DEBUG:boto:Params: {} 2024-04-13T22:16:35.079 DEBUG:boto:Token: None 2024-04-13T22:16:35.080 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/obj11 2024-04-13T22:16:35.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txN+rauRzYLaHdUJTt/izuFjoJM= 2024-04-13T22:16:35.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txN+rauRzYLaHdUJTt/izuFjoJM='} 2024-04-13T22:16:35.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025f96b059bdd6a56-00661b0443-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.086 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:16:35.086 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:16:35.086 DEBUG:boto:Method: HEAD 2024-04-13T22:16:35.087 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:16:35.087 DEBUG:boto:Data: 2024-04-13T22:16:35.087 DEBUG:boto:Headers: {} 2024-04-13T22:16:35.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.087 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.087 DEBUG:boto:Params: {} 2024-04-13T22:16:35.087 DEBUG:boto:Token: None 2024-04-13T22:16:35.087 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/ 2024-04-13T22:16:35.087 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk= 2024-04-13T22:16:35.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk=', 'Content-Length': '0'} 2024-04-13T22:16:35.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000478e85251d55bbb0-00661b0443-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.090 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:16:35.090 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:16:35.090 DEBUG:boto:Method: PUT 2024-04-13T22:16:35.090 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:16:35.091 DEBUG:boto:Data: 2024-04-13T22:16:35.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:16:35.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.091 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.091 DEBUG:boto:Params: {} 2024-04-13T22:16:35.091 DEBUG:boto:Token: None 2024-04-13T22:16:35.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/obj22 2024-04-13T22:16:35.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T8FNCb8+cKBY9gbpo3FSykR1CXo= 2024-04-13T22:16:35.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T8FNCb8+cKBY9gbpo3FSykR1CXo='} 2024-04-13T22:16:35.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054d77f624c7b9798-00661b0443-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.134 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:16:35.135 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:16:35.135 DEBUG:boto:Method: HEAD 2024-04-13T22:16:35.135 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:16:35.135 DEBUG:boto:Data: 2024-04-13T22:16:35.135 DEBUG:boto:Headers: {} 2024-04-13T22:16:35.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.135 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.135 DEBUG:boto:Params: {} 2024-04-13T22:16:35.135 DEBUG:boto:Token: None 2024-04-13T22:16:35.136 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/ 2024-04-13T22:16:35.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk= 2024-04-13T22:16:35.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk=', 'Content-Length': '0'} 2024-04-13T22:16:35.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000069f87150c3d1383-00661b0443-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.138 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:16:35.138 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:16:35.138 DEBUG:boto:Method: PUT 2024-04-13T22:16:35.138 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:16:35.138 DEBUG:boto:Data: 2024-04-13T22:16:35.138 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:16:35.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.139 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.139 DEBUG:boto:Params: {} 2024-04-13T22:16:35.139 DEBUG:boto:Token: None 2024-04-13T22:16:35.139 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/obj33 2024-04-13T22:16:35.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tqERIODy47YdZ5M5fLcr1svso5I= 2024-04-13T22:16:35.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tqERIODy47YdZ5M5fLcr1svso5I='} 2024-04-13T22:16:35.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e429a38bc2bd64fe-00661b0443-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.144 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:16:35.144 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:16:35.144 DEBUG:boto:Method: HEAD 2024-04-13T22:16:35.144 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:16:35.144 DEBUG:boto:Data: 2024-04-13T22:16:35.144 DEBUG:boto:Headers: {} 2024-04-13T22:16:35.145 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.145 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.145 DEBUG:boto:Params: {} 2024-04-13T22:16:35.145 DEBUG:boto:Token: None 2024-04-13T22:16:35.145 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/ 2024-04-13T22:16:35.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk= 2024-04-13T22:16:35.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk=', 'Content-Length': '0'} 2024-04-13T22:16:35.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a471fafddc78ba9-00661b0443-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.148 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:16:35.148 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:16:35.148 DEBUG:boto:Method: PUT 2024-04-13T22:16:35.148 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:16:35.148 DEBUG:boto:Data: 2024-04-13T22:16:35.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:16:35.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.148 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.148 DEBUG:boto:Params: {} 2024-04-13T22:16:35.148 DEBUG:boto:Token: None 2024-04-13T22:16:35.149 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/obj44 2024-04-13T22:16:35.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UbXywQETaD4GAthxBgwZ87xi4wY= 2024-04-13T22:16:35.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UbXywQETaD4GAthxBgwZ87xi4wY='} 2024-04-13T22:16:35.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000958eeeb8df7c116d-00661b0443-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.195 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:16:35.195 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:16:35.195 DEBUG:boto:Method: HEAD 2024-04-13T22:16:35.195 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:16:35.195 DEBUG:boto:Data: 2024-04-13T22:16:35.196 DEBUG:boto:Headers: {} 2024-04-13T22:16:35.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.196 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.196 DEBUG:boto:Params: {} 2024-04-13T22:16:35.196 DEBUG:boto:Token: None 2024-04-13T22:16:35.196 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/ 2024-04-13T22:16:35.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk= 2024-04-13T22:16:35.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qKKcT7BW0INVotBwO7k6b5V75wk=', 'Content-Length': '0'} 2024-04-13T22:16:35.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc77d38e4be8e7cb-00661b0443-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.199 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:16:35.199 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:16:35.199 DEBUG:boto:Method: PUT 2024-04-13T22:16:35.199 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:16:35.199 DEBUG:boto:Data: 2024-04-13T22:16:35.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:16:35.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:16:35.199 DEBUG:boto:Port: 8000 2024-04-13T22:16:35.199 DEBUG:boto:Params: {} 2024-04-13T22:16:35.200 DEBUG:boto:Token: None 2024-04-13T22:16:35.200 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:16:35 GMT /vwnfns-61/obj55 2024-04-13T22:16:35.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6l+J8T5d0S0t98cWIyWmU4WSJgI= 2024-04-13T22:16:35.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:16:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6l+J8T5d0S0t98cWIyWmU4WSJgI='} 2024-04-13T22:16:35.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d1590cd515552f9-00661b0443-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:16:35.204 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:16:35.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:16:35.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:16:35.204 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:16:35.259 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:35.259 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:35.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:35.290+0000 7f84a1676780 1 waiting to reach incremental sync.. 2024-04-13T22:16:40.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:40.291+0000 7f84a1676780 1 waiting to reach incremental sync.. 2024-04-13T22:16:45.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:45.291+0000 7f84a1676780 1 waiting to reach incremental sync.. 2024-04-13T22:16:50.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:50.294+0000 7f84a1676780 1 bucket sync caught up with source: 2024-04-13T22:16:50.297 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.116.6, 00000000002.64.6, 00000000002.76.6] 2024-04-13T22:16:50.297 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.116.6, 00000000002.64.6, 00000000002.76.6] 2024-04-13T22:16:50.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:50.294+0000 7f84a1676780 0 bucket checkpoint complete 2024-04-13T22:16:50.313 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:16:50.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:16:50.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:16:50.314 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:16:50.372 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:16:50.372 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:16:50.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:50.405+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:16:55.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:16:55.405+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:00.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:00.406+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:05.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:05.406+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:10.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:10.406+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:15.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:15.407+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:20.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:20.407+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:25.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:25.408+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:30.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:30.409+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:35.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:35.410+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:40.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:40.411+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:45.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:45.410+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:50.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:50.411+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:17:55.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:17:55.411+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:18:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:00.412+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:18:05.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:05.414+0000 7f184354f780 1 waiting to reach incremental sync.. 2024-04-13T22:18:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:10.415+0000 7f184354f780 1 bucket sync caught up with source: 2024-04-13T22:18:10.418 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.613.6, 00000000002.1884.6, 00000000002.1718.6] 2024-04-13T22:18:10.418 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.613.6, 00000000002.1884.6, 00000000002.1718.6] 2024-04-13T22:18:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:10.415+0000 7f184354f780 0 bucket checkpoint complete 2024-04-13T22:18:10.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:18:10.436 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.436 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.436 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.436 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:10.436 DEBUG:boto:Data: 2024-04-13T22:18:10.436 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.437 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.437 DEBUG:boto:Params: {} 2024-04-13T22:18:10.437 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:18:10.437 DEBUG:boto:Token: None 2024-04-13T22:18:10.438 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d+HcZJYVNfPXJd+x4tKgsEvOu3Q= 2024-04-13T22:18:10.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d+HcZJYVNfPXJd+x4tKgsEvOu3Q=', 'Content-Length': '0'} 2024-04-13T22:18:10.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008921a287d9e6a9de-00661b04a2-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.443 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.443 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.443 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.443 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:10.444 DEBUG:boto:Data: 2024-04-13T22:18:10.444 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.444 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.444 DEBUG:boto:Params: {} 2024-04-13T22:18:10.444 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:18:10.444 DEBUG:boto:Token: None 2024-04-13T22:18:10.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d+HcZJYVNfPXJd+x4tKgsEvOu3Q= 2024-04-13T22:18:10.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d+HcZJYVNfPXJd+x4tKgsEvOu3Q=', 'Content-Length': '0'} 2024-04-13T22:18:10.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001acc8015e4a8a57f-00661b04a2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.449 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:18:10.449 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.450 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.450 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.450 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.450 DEBUG:boto:Method: GET 2024-04-13T22:18:10.450 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.450 DEBUG:boto:Data: 2024-04-13T22:18:10.450 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.450 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.450 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.450 DEBUG:boto:Params: {} 2024-04-13T22:18:10.451 DEBUG:boto:Token: None 2024-04-13T22:18:10.451 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/?versions 2024-04-13T22:18:10.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E= 2024-04-13T22:18:10.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E=', 'Content-Length': '0'} 2024-04-13T22:18:10.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007147659447a016dd-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.453 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:18:10.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:18:10.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:18:10.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:18:10.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:18:10.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:18:10.455 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.455 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.455 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.455 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.455 DEBUG:boto:Method: GET 2024-04-13T22:18:10.455 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.456 DEBUG:boto:Data: 2024-04-13T22:18:10.456 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.456 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.456 DEBUG:boto:Params: {} 2024-04-13T22:18:10.456 DEBUG:boto:Token: None 2024-04-13T22:18:10.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/?versions 2024-04-13T22:18:10.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E= 2024-04-13T22:18:10.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E=', 'Content-Length': '0'} 2024-04-13T22:18:10.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e3f31be54bf025-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.459 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:18:10.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:18:10.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:18:10.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:18:10.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:18:10.460 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.460 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.460 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.460 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.460 DEBUG:boto:Method: GET 2024-04-13T22:18:10.460 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.460 DEBUG:boto:Data: 2024-04-13T22:18:10.461 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.461 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.461 DEBUG:boto:Params: {} 2024-04-13T22:18:10.461 DEBUG:boto:Token: None 2024-04-13T22:18:10.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/?versions 2024-04-13T22:18:10.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E= 2024-04-13T22:18:10.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E=', 'Content-Length': '0'} 2024-04-13T22:18:10.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9dae4cffc68bc1-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.463 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.464 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.464 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.464 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.464 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.464 DEBUG:boto:Method: GET 2024-04-13T22:18:10.464 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:18:10.465 DEBUG:boto:Data: 2024-04-13T22:18:10.465 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.465 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.465 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.465 DEBUG:boto:Params: {} 2024-04-13T22:18:10.465 DEBUG:boto:Token: None 2024-04-13T22:18:10.465 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/?versions 2024-04-13T22:18:10.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E= 2024-04-13T22:18:10.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LKyvITAEctlt0MKcVUAY2U7M67E=', 'Content-Length': '0'} 2024-04-13T22:18:10.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf538579abb270b-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.468 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:18:10.468 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.468 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.468 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.468 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.469 DEBUG:boto:Method: GET 2024-04-13T22:18:10.469 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.469 DEBUG:boto:Data: 2024-04-13T22:18:10.469 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.469 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.469 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.469 DEBUG:boto:Params: {} 2024-04-13T22:18:10.469 DEBUG:boto:Token: None 2024-04-13T22:18:10.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE= 2024-04-13T22:18:10.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE=', 'Content-Length': '0'} 2024-04-13T22:18:10.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de7b91298b253f2-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.471 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.472 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.472 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.472 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.472 DEBUG:boto:Method: GET 2024-04-13T22:18:10.472 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.472 DEBUG:boto:Data: 2024-04-13T22:18:10.472 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.472 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.472 DEBUG:boto:Params: {} 2024-04-13T22:18:10.473 DEBUG:boto:Token: None 2024-04-13T22:18:10.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE= 2024-04-13T22:18:10.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE=', 'Content-Length': '0'} 2024-04-13T22:18:10.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1182e3ef0ab1296-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.475 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.475 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.475 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.475 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.475 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.476 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.476 DEBUG:boto:Data: 2024-04-13T22:18:10.476 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.476 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.476 DEBUG:boto:Params: {} 2024-04-13T22:18:10.476 DEBUG:boto:Token: None 2024-04-13T22:18:10.476 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eX6HRzbK0u3EZczGGKU+QLtMYgM= 2024-04-13T22:18:10.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eX6HRzbK0u3EZczGGKU+QLtMYgM=', 'Content-Length': '0'} 2024-04-13T22:18:10.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a99506dc479f724-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.478 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.478 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.478 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.479 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.479 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.479 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.479 DEBUG:boto:Data: 2024-04-13T22:18:10.479 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.479 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.479 DEBUG:boto:Params: {} 2024-04-13T22:18:10.479 DEBUG:boto:Token: None 2024-04-13T22:18:10.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eX6HRzbK0u3EZczGGKU+QLtMYgM= 2024-04-13T22:18:10.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eX6HRzbK0u3EZczGGKU+QLtMYgM=', 'Content-Length': '0'} 2024-04-13T22:18:10.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f59730c35bcc40c-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:18:10.482 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.482 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.482 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.482 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.482 DEBUG:boto:Method: GET 2024-04-13T22:18:10.482 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.482 DEBUG:boto:Data: 2024-04-13T22:18:10.482 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.483 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.483 DEBUG:boto:Params: {} 2024-04-13T22:18:10.483 DEBUG:boto:Token: None 2024-04-13T22:18:10.483 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE= 2024-04-13T22:18:10.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE=', 'Content-Length': '0'} 2024-04-13T22:18:10.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bdd8c55ad12345-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.526 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.527 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.527 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.527 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.527 DEBUG:boto:Method: GET 2024-04-13T22:18:10.527 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.527 DEBUG:boto:Data: 2024-04-13T22:18:10.527 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.527 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.527 DEBUG:boto:Params: {} 2024-04-13T22:18:10.528 DEBUG:boto:Token: None 2024-04-13T22:18:10.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:18:10.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE= 2024-04-13T22:18:10.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T0pye/MkQQbRhqE8YWSs6dn3RAE=', 'Content-Length': '0'} 2024-04-13T22:18:10.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4092e0e7d3c25ac-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.530 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.530 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.530 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.531 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.531 DEBUG:boto:Method: GET 2024-04-13T22:18:10.531 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.531 DEBUG:boto:Data: 2024-04-13T22:18:10.531 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.531 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.531 DEBUG:boto:Params: {} 2024-04-13T22:18:10.531 DEBUG:boto:Token: None 2024-04-13T22:18:10.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ab3a70ec6a86d1-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.534 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.535 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.535 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.535 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.535 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.535 DEBUG:boto:Method: GET 2024-04-13T22:18:10.535 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.535 DEBUG:boto:Data: 2024-04-13T22:18:10.535 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.535 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.536 DEBUG:boto:Params: {} 2024-04-13T22:18:10.536 DEBUG:boto:Token: None 2024-04-13T22:18:10.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814a5327680b4e6e-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.538 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.539 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.539 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.539 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.539 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:18:10.539 DEBUG:boto:Data: 2024-04-13T22:18:10.539 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.539 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.539 DEBUG:boto:Params: {} 2024-04-13T22:18:10.540 DEBUG:boto:Token: None 2024-04-13T22:18:10.540 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11 2024-04-13T22:18:10.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:740YQ/UVLJBMglJxgGu0nTNDrd4= 2024-04-13T22:18:10.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:740YQ/UVLJBMglJxgGu0nTNDrd4=', 'Content-Length': '0'} 2024-04-13T22:18:10.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9b30296c815d6e9-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.542 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.542 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.542 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.542 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:18:10.542 DEBUG:boto:Data: 2024-04-13T22:18:10.542 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.542 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.542 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.542 DEBUG:boto:Params: {} 2024-04-13T22:18:10.543 DEBUG:boto:Token: None 2024-04-13T22:18:10.543 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11 2024-04-13T22:18:10.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:740YQ/UVLJBMglJxgGu0nTNDrd4= 2024-04-13T22:18:10.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:740YQ/UVLJBMglJxgGu0nTNDrd4=', 'Content-Length': '0'} 2024-04-13T22:18:10.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5dc7517a1ab1d4-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:18:10.545 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.545 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.545 DEBUG:boto:Method: GET 2024-04-13T22:18:10.545 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:18:10.545 DEBUG:boto:Data: 2024-04-13T22:18:10.545 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.546 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.546 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.546 DEBUG:boto:Params: {} 2024-04-13T22:18:10.546 DEBUG:boto:Token: None 2024-04-13T22:18:10.546 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11 2024-04-13T22:18:10.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ja3razka8QRYgjeCzaCEoy6nZrY= 2024-04-13T22:18:10.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ja3razka8QRYgjeCzaCEoy6nZrY=', 'Content-Length': '0'} 2024-04-13T22:18:10.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004adb6e62ecebe4ff-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.548 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:18:10.548 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:18:10.548 DEBUG:boto:Method: GET 2024-04-13T22:18:10.548 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:18:10.549 DEBUG:boto:Data: 2024-04-13T22:18:10.549 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.549 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.549 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.549 DEBUG:boto:Params: {} 2024-04-13T22:18:10.549 DEBUG:boto:Token: None 2024-04-13T22:18:10.549 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj11 2024-04-13T22:18:10.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ja3razka8QRYgjeCzaCEoy6nZrY= 2024-04-13T22:18:10.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ja3razka8QRYgjeCzaCEoy6nZrY=', 'Content-Length': '0'} 2024-04-13T22:18:10.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce2ab26ef1a9c50c-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.551 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.551 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.552 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.552 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.552 DEBUG:boto:Method: GET 2024-04-13T22:18:10.552 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.552 DEBUG:boto:Data: 2024-04-13T22:18:10.552 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.552 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.552 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.552 DEBUG:boto:Params: {} 2024-04-13T22:18:10.552 DEBUG:boto:Token: None 2024-04-13T22:18:10.553 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.553 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe593156aa0b9f2e-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.555 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.556 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.556 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.556 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.556 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.556 DEBUG:boto:Method: GET 2024-04-13T22:18:10.556 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:18:10.556 DEBUG:boto:Data: 2024-04-13T22:18:10.556 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.557 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.557 DEBUG:boto:Params: {} 2024-04-13T22:18:10.557 DEBUG:boto:Token: None 2024-04-13T22:18:10.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005599bc80e9e7deb0-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.559 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:18:10.560 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.560 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.560 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.560 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.560 DEBUG:boto:Method: GET 2024-04-13T22:18:10.560 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.561 DEBUG:boto:Data: 2024-04-13T22:18:10.561 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.561 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.561 DEBUG:boto:Params: {} 2024-04-13T22:18:10.561 DEBUG:boto:Token: None 2024-04-13T22:18:10.561 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.561 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U= 2024-04-13T22:18:10.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U=', 'Content-Length': '0'} 2024-04-13T22:18:10.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007021e518c209bf7d-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.563 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.563 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.564 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.564 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.564 DEBUG:boto:Method: GET 2024-04-13T22:18:10.564 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.564 DEBUG:boto:Data: 2024-04-13T22:18:10.564 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.564 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.564 DEBUG:boto:Params: {} 2024-04-13T22:18:10.564 DEBUG:boto:Token: None 2024-04-13T22:18:10.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U= 2024-04-13T22:18:10.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U=', 'Content-Length': '0'} 2024-04-13T22:18:10.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ebe3a38dafd030-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.610 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.611 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.611 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.611 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.611 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.611 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.611 DEBUG:boto:Data: 2024-04-13T22:18:10.611 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.611 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.611 DEBUG:boto:Params: {} 2024-04-13T22:18:10.612 DEBUG:boto:Token: None 2024-04-13T22:18:10.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xNCaeJqrdbVOw3ga2cPzRvLpXRs= 2024-04-13T22:18:10.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xNCaeJqrdbVOw3ga2cPzRvLpXRs=', 'Content-Length': '0'} 2024-04-13T22:18:10.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d190684ef921fb48-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.614 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.614 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.614 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.614 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.614 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.614 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.615 DEBUG:boto:Data: 2024-04-13T22:18:10.615 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.615 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.615 DEBUG:boto:Params: {} 2024-04-13T22:18:10.615 DEBUG:boto:Token: None 2024-04-13T22:18:10.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xNCaeJqrdbVOw3ga2cPzRvLpXRs= 2024-04-13T22:18:10.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xNCaeJqrdbVOw3ga2cPzRvLpXRs=', 'Content-Length': '0'} 2024-04-13T22:18:10.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429ac0cdf1662ed0-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:18:10.618 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.618 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.618 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.618 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.618 DEBUG:boto:Method: GET 2024-04-13T22:18:10.618 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.618 DEBUG:boto:Data: 2024-04-13T22:18:10.618 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.618 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.618 DEBUG:boto:Params: {} 2024-04-13T22:18:10.619 DEBUG:boto:Token: None 2024-04-13T22:18:10.619 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U= 2024-04-13T22:18:10.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U=', 'Content-Length': '0'} 2024-04-13T22:18:10.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe874d92d008988-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.621 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.621 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.621 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.621 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.621 DEBUG:boto:Method: GET 2024-04-13T22:18:10.621 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.621 DEBUG:boto:Data: 2024-04-13T22:18:10.621 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.622 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.622 DEBUG:boto:Params: {} 2024-04-13T22:18:10.622 DEBUG:boto:Token: None 2024-04-13T22:18:10.622 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:18:10.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U= 2024-04-13T22:18:10.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8KlbGGrUcjdi5Oibjuqh2NMem+U=', 'Content-Length': '0'} 2024-04-13T22:18:10.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f140de21ef035c-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.624 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.624 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.624 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.624 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.625 DEBUG:boto:Method: GET 2024-04-13T22:18:10.625 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.625 DEBUG:boto:Data: 2024-04-13T22:18:10.625 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.625 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.625 DEBUG:boto:Params: {} 2024-04-13T22:18:10.625 DEBUG:boto:Token: None 2024-04-13T22:18:10.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d5b24db6035bc2e-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.628 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.628 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.628 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.629 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.629 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.629 DEBUG:boto:Method: GET 2024-04-13T22:18:10.629 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.629 DEBUG:boto:Data: 2024-04-13T22:18:10.629 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.629 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.629 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.629 DEBUG:boto:Params: {} 2024-04-13T22:18:10.630 DEBUG:boto:Token: None 2024-04-13T22:18:10.630 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f709178d10007f-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.632 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.632 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.633 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.633 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.633 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:18:10.633 DEBUG:boto:Data: 2024-04-13T22:18:10.633 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.633 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.633 DEBUG:boto:Params: {} 2024-04-13T22:18:10.633 DEBUG:boto:Token: None 2024-04-13T22:18:10.634 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22 2024-04-13T22:18:10.634 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0xXc87OqlCWFdDCt+M7uXaaka1M= 2024-04-13T22:18:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0xXc87OqlCWFdDCt+M7uXaaka1M=', 'Content-Length': '0'} 2024-04-13T22:18:10.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b53a69ad4a3b4e-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.635 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.635 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.636 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.636 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:18:10.636 DEBUG:boto:Data: 2024-04-13T22:18:10.636 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.636 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.636 DEBUG:boto:Params: {} 2024-04-13T22:18:10.636 DEBUG:boto:Token: None 2024-04-13T22:18:10.636 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22 2024-04-13T22:18:10.637 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0xXc87OqlCWFdDCt+M7uXaaka1M= 2024-04-13T22:18:10.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0xXc87OqlCWFdDCt+M7uXaaka1M=', 'Content-Length': '0'} 2024-04-13T22:18:10.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f810d5a0d232f420-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:18:10.639 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.639 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.639 DEBUG:boto:Method: GET 2024-04-13T22:18:10.639 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:18:10.639 DEBUG:boto:Data: 2024-04-13T22:18:10.639 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.639 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.639 DEBUG:boto:Params: {} 2024-04-13T22:18:10.640 DEBUG:boto:Token: None 2024-04-13T22:18:10.640 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22 2024-04-13T22:18:10.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T25ZtdurWOmHvV0uR0T3sXTpOW8= 2024-04-13T22:18:10.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T25ZtdurWOmHvV0uR0T3sXTpOW8=', 'Content-Length': '0'} 2024-04-13T22:18:10.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7da65a60728b178-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.682 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:18:10.682 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:18:10.683 DEBUG:boto:Method: GET 2024-04-13T22:18:10.683 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:18:10.683 DEBUG:boto:Data: 2024-04-13T22:18:10.683 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.683 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.683 DEBUG:boto:Params: {} 2024-04-13T22:18:10.683 DEBUG:boto:Token: None 2024-04-13T22:18:10.684 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj22 2024-04-13T22:18:10.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T25ZtdurWOmHvV0uR0T3sXTpOW8= 2024-04-13T22:18:10.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T25ZtdurWOmHvV0uR0T3sXTpOW8=', 'Content-Length': '0'} 2024-04-13T22:18:10.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8807216cf862e36-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.686 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.686 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.686 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.686 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.687 DEBUG:boto:Method: GET 2024-04-13T22:18:10.687 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.687 DEBUG:boto:Data: 2024-04-13T22:18:10.687 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.687 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.687 DEBUG:boto:Params: {} 2024-04-13T22:18:10.687 DEBUG:boto:Token: None 2024-04-13T22:18:10.687 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a22a6e0516930ff9-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.690 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.690 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.690 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.690 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.691 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.691 DEBUG:boto:Method: GET 2024-04-13T22:18:10.691 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:18:10.691 DEBUG:boto:Data: 2024-04-13T22:18:10.691 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.691 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.691 DEBUG:boto:Params: {} 2024-04-13T22:18:10.691 DEBUG:boto:Token: None 2024-04-13T22:18:10.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162eeb706c0af86c-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.694 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:18:10.695 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.695 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.695 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.695 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.695 DEBUG:boto:Method: GET 2024-04-13T22:18:10.695 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.695 DEBUG:boto:Data: 2024-04-13T22:18:10.695 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.695 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.695 DEBUG:boto:Params: {} 2024-04-13T22:18:10.696 DEBUG:boto:Token: None 2024-04-13T22:18:10.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4= 2024-04-13T22:18:10.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4=', 'Content-Length': '0'} 2024-04-13T22:18:10.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f25b75f2dbcefa5-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.698 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.698 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.698 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.698 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.698 DEBUG:boto:Method: GET 2024-04-13T22:18:10.698 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.699 DEBUG:boto:Data: 2024-04-13T22:18:10.699 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.699 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.699 DEBUG:boto:Params: {} 2024-04-13T22:18:10.699 DEBUG:boto:Token: None 2024-04-13T22:18:10.699 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4= 2024-04-13T22:18:10.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4=', 'Content-Length': '0'} 2024-04-13T22:18:10.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d571d760f4494b0-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.701 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.701 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.702 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.702 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.702 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.702 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.702 DEBUG:boto:Data: 2024-04-13T22:18:10.702 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.702 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.702 DEBUG:boto:Params: {} 2024-04-13T22:18:10.702 DEBUG:boto:Token: None 2024-04-13T22:18:10.703 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.703 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3dh+q8whhPg8vn9BVBIfAKKqFgs= 2024-04-13T22:18:10.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3dh+q8whhPg8vn9BVBIfAKKqFgs=', 'Content-Length': '0'} 2024-04-13T22:18:10.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002933502eaadda67f-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.704 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.705 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.705 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.705 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.705 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.705 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.705 DEBUG:boto:Data: 2024-04-13T22:18:10.705 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.705 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.705 DEBUG:boto:Params: {} 2024-04-13T22:18:10.706 DEBUG:boto:Token: None 2024-04-13T22:18:10.706 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3dh+q8whhPg8vn9BVBIfAKKqFgs= 2024-04-13T22:18:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3dh+q8whhPg8vn9BVBIfAKKqFgs=', 'Content-Length': '0'} 2024-04-13T22:18:10.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd74326db96c095-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:18:10.708 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.708 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.708 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.708 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.708 DEBUG:boto:Method: GET 2024-04-13T22:18:10.709 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.709 DEBUG:boto:Data: 2024-04-13T22:18:10.709 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.709 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.709 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.709 DEBUG:boto:Params: {} 2024-04-13T22:18:10.709 DEBUG:boto:Token: None 2024-04-13T22:18:10.709 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.709 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4= 2024-04-13T22:18:10.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4=', 'Content-Length': '0'} 2024-04-13T22:18:10.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477258e0aa89e246-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.711 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.711 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.711 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.711 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.712 DEBUG:boto:Method: GET 2024-04-13T22:18:10.712 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.712 DEBUG:boto:Data: 2024-04-13T22:18:10.712 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.712 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.712 DEBUG:boto:Params: {} 2024-04-13T22:18:10.712 DEBUG:boto:Token: None 2024-04-13T22:18:10.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:18:10.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4= 2024-04-13T22:18:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZ+qDmarUouY+4pYBqwx7VT+kI4=', 'Content-Length': '0'} 2024-04-13T22:18:10.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e219d7ce977a3c-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.714 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.715 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.715 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.715 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.715 DEBUG:boto:Method: GET 2024-04-13T22:18:10.715 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.715 DEBUG:boto:Data: 2024-04-13T22:18:10.715 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.715 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.715 DEBUG:boto:Params: {} 2024-04-13T22:18:10.715 DEBUG:boto:Token: None 2024-04-13T22:18:10.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3fa7e9bb55c6f1-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.718 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.718 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.718 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.719 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.719 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.719 DEBUG:boto:Method: GET 2024-04-13T22:18:10.719 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.719 DEBUG:boto:Data: 2024-04-13T22:18:10.719 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.719 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.719 DEBUG:boto:Params: {} 2024-04-13T22:18:10.719 DEBUG:boto:Token: None 2024-04-13T22:18:10.720 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040328dc76d2b18f7-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.722 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.722 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.722 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.722 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.723 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:18:10.723 DEBUG:boto:Data: 2024-04-13T22:18:10.723 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.723 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.723 DEBUG:boto:Params: {} 2024-04-13T22:18:10.723 DEBUG:boto:Token: None 2024-04-13T22:18:10.723 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33 2024-04-13T22:18:10.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J8q0wOxP8+FqKxlis6nBhlcdYrk= 2024-04-13T22:18:10.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J8q0wOxP8+FqKxlis6nBhlcdYrk=', 'Content-Length': '0'} 2024-04-13T22:18:10.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be46e05f4553dfe-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.725 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.725 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.725 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.725 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:18:10.725 DEBUG:boto:Data: 2024-04-13T22:18:10.726 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.726 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.726 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.726 DEBUG:boto:Params: {} 2024-04-13T22:18:10.726 DEBUG:boto:Token: None 2024-04-13T22:18:10.726 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33 2024-04-13T22:18:10.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J8q0wOxP8+FqKxlis6nBhlcdYrk= 2024-04-13T22:18:10.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J8q0wOxP8+FqKxlis6nBhlcdYrk=', 'Content-Length': '0'} 2024-04-13T22:18:10.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e98388bfd07d45-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:18:10.728 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.728 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.728 DEBUG:boto:Method: GET 2024-04-13T22:18:10.729 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:18:10.729 DEBUG:boto:Data: 2024-04-13T22:18:10.729 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.729 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.729 DEBUG:boto:Params: {} 2024-04-13T22:18:10.729 DEBUG:boto:Token: None 2024-04-13T22:18:10.729 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33 2024-04-13T22:18:10.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CnYu95OWvVniUkITE5bASNBUW48= 2024-04-13T22:18:10.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CnYu95OWvVniUkITE5bASNBUW48=', 'Content-Length': '0'} 2024-04-13T22:18:10.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068539ca77156b22c-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.731 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:18:10.731 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:18:10.732 DEBUG:boto:Method: GET 2024-04-13T22:18:10.732 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:18:10.732 DEBUG:boto:Data: 2024-04-13T22:18:10.732 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.732 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.732 DEBUG:boto:Params: {} 2024-04-13T22:18:10.732 DEBUG:boto:Token: None 2024-04-13T22:18:10.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj33 2024-04-13T22:18:10.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CnYu95OWvVniUkITE5bASNBUW48= 2024-04-13T22:18:10.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CnYu95OWvVniUkITE5bASNBUW48=', 'Content-Length': '0'} 2024-04-13T22:18:10.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051034a0c19d8e529-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.734 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.735 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.735 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.735 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.735 DEBUG:boto:Method: GET 2024-04-13T22:18:10.735 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.735 DEBUG:boto:Data: 2024-04-13T22:18:10.735 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.735 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.735 DEBUG:boto:Params: {} 2024-04-13T22:18:10.736 DEBUG:boto:Token: None 2024-04-13T22:18:10.736 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db6a38bac7d619a-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.778 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.779 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.779 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.779 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.779 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.779 DEBUG:boto:Method: GET 2024-04-13T22:18:10.779 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:18:10.780 DEBUG:boto:Data: 2024-04-13T22:18:10.780 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.780 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.780 DEBUG:boto:Params: {} 2024-04-13T22:18:10.780 DEBUG:boto:Token: None 2024-04-13T22:18:10.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf9c451cfa8d36d3-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.782 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:18:10.783 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.783 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.783 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.783 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.783 DEBUG:boto:Method: GET 2024-04-13T22:18:10.783 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.784 DEBUG:boto:Data: 2024-04-13T22:18:10.784 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.784 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.784 DEBUG:boto:Params: {} 2024-04-13T22:18:10.784 DEBUG:boto:Token: None 2024-04-13T22:18:10.784 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw= 2024-04-13T22:18:10.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw=', 'Content-Length': '0'} 2024-04-13T22:18:10.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051ec94383e3d1c76-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.786 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.786 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.786 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.787 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.787 DEBUG:boto:Method: GET 2024-04-13T22:18:10.787 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.787 DEBUG:boto:Data: 2024-04-13T22:18:10.787 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.787 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.787 DEBUG:boto:Params: {} 2024-04-13T22:18:10.787 DEBUG:boto:Token: None 2024-04-13T22:18:10.787 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw= 2024-04-13T22:18:10.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw=', 'Content-Length': '0'} 2024-04-13T22:18:10.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e71e532d55e5fe-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.789 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.789 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.789 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.790 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.790 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.790 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.790 DEBUG:boto:Data: 2024-04-13T22:18:10.790 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.790 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.790 DEBUG:boto:Params: {} 2024-04-13T22:18:10.790 DEBUG:boto:Token: None 2024-04-13T22:18:10.790 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5O5FyDGgQDV4W50NK9uKx8U2ZSs= 2024-04-13T22:18:10.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5O5FyDGgQDV4W50NK9uKx8U2ZSs=', 'Content-Length': '0'} 2024-04-13T22:18:10.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6890bc424b0e46-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.792 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.792 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.792 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.793 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.793 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.793 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.793 DEBUG:boto:Data: 2024-04-13T22:18:10.793 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.793 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.793 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.793 DEBUG:boto:Params: {} 2024-04-13T22:18:10.793 DEBUG:boto:Token: None 2024-04-13T22:18:10.793 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.793 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5O5FyDGgQDV4W50NK9uKx8U2ZSs= 2024-04-13T22:18:10.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5O5FyDGgQDV4W50NK9uKx8U2ZSs=', 'Content-Length': '0'} 2024-04-13T22:18:10.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ceb19992a2417b-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:18:10.796 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.796 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.796 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.796 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.796 DEBUG:boto:Method: GET 2024-04-13T22:18:10.796 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.796 DEBUG:boto:Data: 2024-04-13T22:18:10.796 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.797 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.797 DEBUG:boto:Params: {} 2024-04-13T22:18:10.797 DEBUG:boto:Token: None 2024-04-13T22:18:10.797 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.797 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw= 2024-04-13T22:18:10.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw=', 'Content-Length': '0'} 2024-04-13T22:18:10.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3ed7fadcb3b574-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.799 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.799 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.800 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.800 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.800 DEBUG:boto:Method: GET 2024-04-13T22:18:10.800 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.800 DEBUG:boto:Data: 2024-04-13T22:18:10.800 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.800 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.800 DEBUG:boto:Params: {} 2024-04-13T22:18:10.800 DEBUG:boto:Token: None 2024-04-13T22:18:10.800 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:18:10.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw= 2024-04-13T22:18:10.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8hW76qw0Sd8d8x+N8f07r/K7drw=', 'Content-Length': '0'} 2024-04-13T22:18:10.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004766c5323a9ee649-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.846 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.847 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.847 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.847 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.847 DEBUG:boto:Method: GET 2024-04-13T22:18:10.847 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.847 DEBUG:boto:Data: 2024-04-13T22:18:10.847 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.847 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.847 DEBUG:boto:Params: {} 2024-04-13T22:18:10.848 DEBUG:boto:Token: None 2024-04-13T22:18:10.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46083fd8774ac94-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.850 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.852 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.852 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.852 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.852 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.852 DEBUG:boto:Method: GET 2024-04-13T22:18:10.852 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.852 DEBUG:boto:Data: 2024-04-13T22:18:10.852 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.853 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.853 DEBUG:boto:Params: {} 2024-04-13T22:18:10.853 DEBUG:boto:Token: None 2024-04-13T22:18:10.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c228c5b8fd06793-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.855 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.856 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.856 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.856 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.856 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:18:10.856 DEBUG:boto:Data: 2024-04-13T22:18:10.856 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.857 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.857 DEBUG:boto:Params: {} 2024-04-13T22:18:10.857 DEBUG:boto:Token: None 2024-04-13T22:18:10.857 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44 2024-04-13T22:18:10.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:owBH0OCYRONsupZd3t6xKGJUd7o= 2024-04-13T22:18:10.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:owBH0OCYRONsupZd3t6xKGJUd7o=', 'Content-Length': '0'} 2024-04-13T22:18:10.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d5974532264f8d-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.859 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.859 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.859 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.859 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:18:10.860 DEBUG:boto:Data: 2024-04-13T22:18:10.860 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.860 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.860 DEBUG:boto:Params: {} 2024-04-13T22:18:10.860 DEBUG:boto:Token: None 2024-04-13T22:18:10.860 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44 2024-04-13T22:18:10.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:owBH0OCYRONsupZd3t6xKGJUd7o= 2024-04-13T22:18:10.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:owBH0OCYRONsupZd3t6xKGJUd7o=', 'Content-Length': '0'} 2024-04-13T22:18:10.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e5ec4aaa6218ab-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:18:10.863 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.863 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.863 DEBUG:boto:Method: GET 2024-04-13T22:18:10.863 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:18:10.863 DEBUG:boto:Data: 2024-04-13T22:18:10.863 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.863 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.863 DEBUG:boto:Params: {} 2024-04-13T22:18:10.863 DEBUG:boto:Token: None 2024-04-13T22:18:10.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44 2024-04-13T22:18:10.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A/rl/flhEbXpzG/+3sZE7WW3Ops= 2024-04-13T22:18:10.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A/rl/flhEbXpzG/+3sZE7WW3Ops=', 'Content-Length': '0'} 2024-04-13T22:18:10.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e5795b196fffe7-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.906 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:18:10.906 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:18:10.907 DEBUG:boto:Method: GET 2024-04-13T22:18:10.907 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:18:10.907 DEBUG:boto:Data: 2024-04-13T22:18:10.907 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.907 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.907 DEBUG:boto:Params: {} 2024-04-13T22:18:10.907 DEBUG:boto:Token: None 2024-04-13T22:18:10.907 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj44 2024-04-13T22:18:10.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A/rl/flhEbXpzG/+3sZE7WW3Ops= 2024-04-13T22:18:10.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A/rl/flhEbXpzG/+3sZE7WW3Ops=', 'Content-Length': '0'} 2024-04-13T22:18:10.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9e152148861884-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.910 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.910 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.910 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.910 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.910 DEBUG:boto:Method: GET 2024-04-13T22:18:10.910 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.911 DEBUG:boto:Data: 2024-04-13T22:18:10.911 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.911 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.911 DEBUG:boto:Params: {} 2024-04-13T22:18:10.911 DEBUG:boto:Token: None 2024-04-13T22:18:10.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3bc09e9cda2b4b-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.913 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.914 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.914 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.914 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.914 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.914 DEBUG:boto:Method: GET 2024-04-13T22:18:10.914 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:18:10.915 DEBUG:boto:Data: 2024-04-13T22:18:10.915 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.915 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.915 DEBUG:boto:Params: {} 2024-04-13T22:18:10.915 DEBUG:boto:Token: None 2024-04-13T22:18:10.915 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.915 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048ecdad5487391f7-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.918 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:18:10.918 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.918 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.918 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.918 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.918 DEBUG:boto:Method: GET 2024-04-13T22:18:10.919 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.919 DEBUG:boto:Data: 2024-04-13T22:18:10.919 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.919 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.919 DEBUG:boto:Params: {} 2024-04-13T22:18:10.919 DEBUG:boto:Token: None 2024-04-13T22:18:10.919 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI= 2024-04-13T22:18:10.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI=', 'Content-Length': '0'} 2024-04-13T22:18:10.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000796b4e6f02293ddf-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.921 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.921 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.921 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.921 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.922 DEBUG:boto:Method: GET 2024-04-13T22:18:10.922 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.922 DEBUG:boto:Data: 2024-04-13T22:18:10.922 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.922 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.922 DEBUG:boto:Params: {} 2024-04-13T22:18:10.922 DEBUG:boto:Token: None 2024-04-13T22:18:10.922 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI= 2024-04-13T22:18:10.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI=', 'Content-Length': '0'} 2024-04-13T22:18:10.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301aa7200b3a63e3-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.924 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.925 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.925 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.925 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.925 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.925 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.925 DEBUG:boto:Data: 2024-04-13T22:18:10.925 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.925 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.925 DEBUG:boto:Params: {} 2024-04-13T22:18:10.925 DEBUG:boto:Token: None 2024-04-13T22:18:10.925 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xtXdBnb8iNXwoyuLCxsNqVzuWD0= 2024-04-13T22:18:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xtXdBnb8iNXwoyuLCxsNqVzuWD0=', 'Content-Length': '0'} 2024-04-13T22:18:10.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc1a6ee26676333-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.927 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.927 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.927 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.928 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.928 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.928 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.928 DEBUG:boto:Data: 2024-04-13T22:18:10.928 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.928 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.928 DEBUG:boto:Params: {} 2024-04-13T22:18:10.928 DEBUG:boto:Token: None 2024-04-13T22:18:10.928 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xtXdBnb8iNXwoyuLCxsNqVzuWD0= 2024-04-13T22:18:10.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xtXdBnb8iNXwoyuLCxsNqVzuWD0=', 'Content-Length': '0'} 2024-04-13T22:18:10.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3006ae1f4d456b8-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:18:10.930 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.931 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.931 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.931 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.931 DEBUG:boto:Method: GET 2024-04-13T22:18:10.931 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.931 DEBUG:boto:Data: 2024-04-13T22:18:10.931 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.931 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.931 DEBUG:boto:Params: {} 2024-04-13T22:18:10.931 DEBUG:boto:Token: None 2024-04-13T22:18:10.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI= 2024-04-13T22:18:10.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI=', 'Content-Length': '0'} 2024-04-13T22:18:10.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0a963523ef9d1d-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.933 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.934 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.934 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.934 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.934 DEBUG:boto:Method: GET 2024-04-13T22:18:10.934 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.934 DEBUG:boto:Data: 2024-04-13T22:18:10.934 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.934 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.934 DEBUG:boto:Params: {} 2024-04-13T22:18:10.934 DEBUG:boto:Token: None 2024-04-13T22:18:10.935 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:18:10.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI= 2024-04-13T22:18:10.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Snb4Ls0rYMObQA5emQltXfd4nuI=', 'Content-Length': '0'} 2024-04-13T22:18:10.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad68e114cb818e22-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.937 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.937 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.937 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.937 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.937 DEBUG:boto:Method: GET 2024-04-13T22:18:10.937 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.937 DEBUG:boto:Data: 2024-04-13T22:18:10.937 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.937 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.937 DEBUG:boto:Params: {} 2024-04-13T22:18:10.938 DEBUG:boto:Token: None 2024-04-13T22:18:10.938 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fc4e4672fcbc120-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.941 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.941 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.941 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.941 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.941 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.941 DEBUG:boto:Method: GET 2024-04-13T22:18:10.941 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.942 DEBUG:boto:Data: 2024-04-13T22:18:10.942 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.942 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.942 DEBUG:boto:Params: {} 2024-04-13T22:18:10.942 DEBUG:boto:Token: None 2024-04-13T22:18:10.942 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/ 2024-04-13T22:18:10.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg= 2024-04-13T22:18:10.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTTQTGXZOvVW7WOD4BH8QpL/+wg=', 'Content-Length': '0'} 2024-04-13T22:18:10.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eef74060196d8300-00661b04a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.944 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:10.945 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.945 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.945 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.945 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:18:10.945 DEBUG:boto:Data: 2024-04-13T22:18:10.945 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.945 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.945 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.945 DEBUG:boto:Params: {} 2024-04-13T22:18:10.945 DEBUG:boto:Token: None 2024-04-13T22:18:10.946 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55 2024-04-13T22:18:10.946 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKa1VPVwG7f0Le6XvHrPf7ocKrc= 2024-04-13T22:18:10.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKa1VPVwG7f0Le6XvHrPf7ocKrc=', 'Content-Length': '0'} 2024-04-13T22:18:10.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7243805b939aae-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.947 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.948 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.948 DEBUG:boto:Method: HEAD 2024-04-13T22:18:10.948 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:18:10.948 DEBUG:boto:Data: 2024-04-13T22:18:10.948 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.948 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.948 DEBUG:boto:Params: {} 2024-04-13T22:18:10.948 DEBUG:boto:Token: None 2024-04-13T22:18:10.948 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55 2024-04-13T22:18:10.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKa1VPVwG7f0Le6XvHrPf7ocKrc= 2024-04-13T22:18:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKa1VPVwG7f0Le6XvHrPf7ocKrc=', 'Content-Length': '0'} 2024-04-13T22:18:10.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a859046e37024baa-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:18:10.951 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.951 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.951 DEBUG:boto:Method: GET 2024-04-13T22:18:10.951 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:18:10.951 DEBUG:boto:Data: 2024-04-13T22:18:10.951 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.951 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.951 DEBUG:boto:Params: {} 2024-04-13T22:18:10.951 DEBUG:boto:Token: None 2024-04-13T22:18:10.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55 2024-04-13T22:18:10.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ck16IFpDlx9vXSY3TMrxYJVfOuk= 2024-04-13T22:18:10.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ck16IFpDlx9vXSY3TMrxYJVfOuk=', 'Content-Length': '0'} 2024-04-13T22:18:10.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f77864582446d3a-00661b04a2-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.953 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:18:10.954 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:18:10.954 DEBUG:boto:Method: GET 2024-04-13T22:18:10.954 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:18:10.954 DEBUG:boto:Data: 2024-04-13T22:18:10.954 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.954 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:10.954 DEBUG:boto:Port: 8000 2024-04-13T22:18:10.954 DEBUG:boto:Params: {} 2024-04-13T22:18:10.954 DEBUG:boto:Token: None 2024-04-13T22:18:10.955 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:10 GMT /vwnfns-61/obj55 2024-04-13T22:18:10.955 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ck16IFpDlx9vXSY3TMrxYJVfOuk= 2024-04-13T22:18:10.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ck16IFpDlx9vXSY3TMrxYJVfOuk=', 'Content-Length': '0'} 2024-04-13T22:18:10.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c355ecdb6f1929a-00661b04a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:10.998 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:10.999 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:10.999 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.999 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.999 DEBUG:boto:Method: GET 2024-04-13T22:18:10.999 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:18:10.999 DEBUG:boto:Data: 2024-04-13T22:18:10.999 DEBUG:boto:Headers: {} 2024-04-13T22:18:10.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:18:10.999 DEBUG:boto:Port: 8001 2024-04-13T22:18:10.999 DEBUG:boto:Params: {} 2024-04-13T22:18:11.000 DEBUG:boto:Token: None 2024-04-13T22:18:11.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QOejbV55Qrcec7Ai1u15ob7UazU= 2024-04-13T22:18:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QOejbV55Qrcec7Ai1u15ob7UazU=', 'Content-Length': '0'} 2024-04-13T22:18:11.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ce840a4544ea2c-00661b04a2-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.002 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:11.003 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:11.003 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:11.003 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:11.003 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:18:11.003 DEBUG:boto:Method: GET 2024-04-13T22:18:11.003 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:18:11.003 DEBUG:boto:Data: 2024-04-13T22:18:11.003 DEBUG:boto:Headers: {} 2024-04-13T22:18:11.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.003 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.004 DEBUG:boto:Params: {} 2024-04-13T22:18:11.004 DEBUG:boto:Token: None 2024-04-13T22:18:11.004 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QOejbV55Qrcec7Ai1u15ob7UazU= 2024-04-13T22:18:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QOejbV55Qrcec7Ai1u15ob7UazU=', 'Content-Length': '0'} 2024-04-13T22:18:11.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5561b497ae3b02-00661b04a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.006 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:18:11.007 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:18:11.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:18:11.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:18:11.007 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:18:11.060 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:18:11.060 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:18:11.102 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:18:11.102 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-61 2024-04-13T22:18:11.106 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 5 2024-04-13T22:18:11.333 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:11.333 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:11.334 DEBUG:boto:Method: HEAD 2024-04-13T22:18:11.334 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:11.334 DEBUG:boto:Data: 2024-04-13T22:18:11.334 DEBUG:boto:Headers: {} 2024-04-13T22:18:11.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.334 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.334 DEBUG:boto:Params: {} 2024-04-13T22:18:11.334 DEBUG:boto:Token: None 2024-04-13T22:18:11.334 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o= 2024-04-13T22:18:11.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o=', 'Content-Length': '0'} 2024-04-13T22:18:11.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003ac06b51728d4bc-00661b04a3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.338 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:18:11.338 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:18:11.338 DEBUG:boto:Method: PUT 2024-04-13T22:18:11.338 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:18:11.338 DEBUG:boto:Data: 2024-04-13T22:18:11.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:18:11.338 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.338 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.339 DEBUG:boto:Params: {} 2024-04-13T22:18:11.339 DEBUG:boto:Token: None 2024-04-13T22:18:11.339 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/obj17 2024-04-13T22:18:11.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0GrQMBGGIrlahiia1tFL+uJR85w= 2024-04-13T22:18:11.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0GrQMBGGIrlahiia1tFL+uJR85w='} 2024-04-13T22:18:11.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030a0bc72cee6076b-00661b04a3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.345 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:11.345 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:11.345 DEBUG:boto:Method: HEAD 2024-04-13T22:18:11.345 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:11.345 DEBUG:boto:Data: 2024-04-13T22:18:11.345 DEBUG:boto:Headers: {} 2024-04-13T22:18:11.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.345 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.345 DEBUG:boto:Params: {} 2024-04-13T22:18:11.345 DEBUG:boto:Token: None 2024-04-13T22:18:11.346 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.346 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o= 2024-04-13T22:18:11.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o=', 'Content-Length': '0'} 2024-04-13T22:18:11.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098c2da305dfc4bc1-00661b04a3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.350 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:18:11.350 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:18:11.350 DEBUG:boto:Method: PUT 2024-04-13T22:18:11.350 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:18:11.350 DEBUG:boto:Data: 2024-04-13T22:18:11.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:18:11.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.350 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.350 DEBUG:boto:Params: {} 2024-04-13T22:18:11.351 DEBUG:boto:Token: None 2024-04-13T22:18:11.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/obj34 2024-04-13T22:18:11.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ag9ladTuX/qgXGAptQwIrUjgXWk= 2024-04-13T22:18:11.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ag9ladTuX/qgXGAptQwIrUjgXWk='} 2024-04-13T22:18:11.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000757ccad4d1df6153-00661b04a3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.357 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:11.357 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:11.357 DEBUG:boto:Method: HEAD 2024-04-13T22:18:11.357 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:11.357 DEBUG:boto:Data: 2024-04-13T22:18:11.357 DEBUG:boto:Headers: {} 2024-04-13T22:18:11.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.357 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.357 DEBUG:boto:Params: {} 2024-04-13T22:18:11.358 DEBUG:boto:Token: None 2024-04-13T22:18:11.358 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o= 2024-04-13T22:18:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o=', 'Content-Length': '0'} 2024-04-13T22:18:11.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f74c18456c8264f-00661b04a3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.363 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:18:11.363 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:18:11.363 DEBUG:boto:Method: PUT 2024-04-13T22:18:11.363 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:18:11.363 DEBUG:boto:Data: 2024-04-13T22:18:11.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:18:11.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.363 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.363 DEBUG:boto:Params: {} 2024-04-13T22:18:11.364 DEBUG:boto:Token: None 2024-04-13T22:18:11.364 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/obj51 2024-04-13T22:18:11.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4/oCZRsVSfTPIKlyX49XwexiJAo= 2024-04-13T22:18:11.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4/oCZRsVSfTPIKlyX49XwexiJAo='} 2024-04-13T22:18:11.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007be89e445a3c94a4-00661b04a3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.411 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:11.411 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:11.411 DEBUG:boto:Method: HEAD 2024-04-13T22:18:11.411 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:11.411 DEBUG:boto:Data: 2024-04-13T22:18:11.412 DEBUG:boto:Headers: {} 2024-04-13T22:18:11.412 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.412 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.412 DEBUG:boto:Params: {} 2024-04-13T22:18:11.412 DEBUG:boto:Token: None 2024-04-13T22:18:11.412 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o= 2024-04-13T22:18:11.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o=', 'Content-Length': '0'} 2024-04-13T22:18:11.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f0e4b654ab71d58-00661b04a3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.415 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:18:11.416 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:18:11.416 DEBUG:boto:Method: PUT 2024-04-13T22:18:11.416 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:18:11.416 DEBUG:boto:Data: 2024-04-13T22:18:11.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:18:11.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.416 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.416 DEBUG:boto:Params: {} 2024-04-13T22:18:11.416 DEBUG:boto:Token: None 2024-04-13T22:18:11.416 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/obj68 2024-04-13T22:18:11.417 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NHelDgA+tjgHDpOZQrO+ZKxw9gk= 2024-04-13T22:18:11.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NHelDgA+tjgHDpOZQrO+ZKxw9gk='} 2024-04-13T22:18:11.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d13ba5d5811d9752-00661b04a3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.422 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:18:11.422 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:18:11.422 DEBUG:boto:Method: HEAD 2024-04-13T22:18:11.422 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:18:11.422 DEBUG:boto:Data: 2024-04-13T22:18:11.423 DEBUG:boto:Headers: {} 2024-04-13T22:18:11.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.423 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.423 DEBUG:boto:Params: {} 2024-04-13T22:18:11.423 DEBUG:boto:Token: None 2024-04-13T22:18:11.423 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/ 2024-04-13T22:18:11.423 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o= 2024-04-13T22:18:11.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKfgdJrBxZhTuoKtmiCAQ26dC+o=', 'Content-Length': '0'} 2024-04-13T22:18:11.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e2e15c26e768afb-00661b04a3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.426 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:18:11.426 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:18:11.426 DEBUG:boto:Method: PUT 2024-04-13T22:18:11.426 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:18:11.426 DEBUG:boto:Data: 2024-04-13T22:18:11.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:18:11.426 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:18:11.426 DEBUG:boto:Port: 8000 2024-04-13T22:18:11.426 DEBUG:boto:Params: {} 2024-04-13T22:18:11.426 DEBUG:boto:Token: None 2024-04-13T22:18:11.427 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:18:11 GMT /vwnfns-61/obj85 2024-04-13T22:18:11.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:61Uj8ZGW5MFo0Bh5fpUwX23w8Zc= 2024-04-13T22:18:11.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:18:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:61Uj8ZGW5MFo0Bh5fpUwX23w8Zc='} 2024-04-13T22:18:11.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7b1b7050a09400e-00661b04a3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:18:11.471 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:18:11.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:18:11.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:18:11.472 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:18:11.528 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:18:11.528 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:18:11.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:11.559+0000 7fe180115780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T22:18:16.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:16.560+0000 7fe180115780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T22:18:21.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:21.561+0000 7fe180115780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T22:18:26.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:26.561+0000 7fe180115780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T22:18:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:31.563+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:18:31.567 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-13T22:18:31.567 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:18:36.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:36.564+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:18:36.568 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-13T22:18:36.568 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:18:41.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:41.566+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:18:41.570 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-13T22:18:41.570 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:18:46.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:46.568+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:18:46.571 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-13T22:18:46.571 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:18:51.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:51.570+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:18:51.573 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, , , , ] 2024-04-13T22:18:51.573 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:18:56.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:18:56.571+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:18:56.574 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, , , , ] 2024-04-13T22:18:56.574 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:01.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:01.572+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:01.576 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, , , , ] 2024-04-13T22:19:01.576 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:06.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:06.573+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:06.577 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, , , , ] 2024-04-13T22:19:06.577 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:11.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:11.575+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:11.579 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, , , , ] 2024-04-13T22:19:11.579 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:16.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:16.576+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:16.580 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, , , , ] 2024-04-13T22:19:16.580 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:21.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:21.578+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:21.582 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:21.582 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:26.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:26.580+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:26.583 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:26.583 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:31.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:31.581+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:31.585 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:31.585 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:36.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:36.583+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:36.586 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:36.586 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:41.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:41.584+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:41.588 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:41.588 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:46.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:46.585+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:46.589 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:46.590 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:51.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:51.587+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:51.591 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:51.591 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:56.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:19:56.588+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:19:56.592 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:19:56.592 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:01.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:01.590+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:01.593 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:01.594 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:06.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:06.592+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:06.595 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:06.595 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:11.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:11.593+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:11.597 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:11.597 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:16.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:16.595+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:16.598 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:16.598 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:21.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:21.595+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:21.600 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:21.600 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:26.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:26.597+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:26.601 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:26.601 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:31.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:31.598+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:31.603 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:31.603 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:36.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:36.600+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:36.604 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:36.604 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:41.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:41.602+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:41.606 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:41.606 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:46.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:46.604+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:46.607 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:46.607 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:51.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:51.605+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:51.608 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:51.609 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:56.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:20:56.605+0000 7fe180115780 1 waiting for incremental sync to catch up: 2024-04-13T22:20:56.610 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , , , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:20:56.610 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:21:01.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:01.607+0000 7fe180115780 1 bucket sync caught up with source: 2024-04-13T22:21:01.611 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:21:01.611 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000001.86.6, , , , , , 00000000001.62.6, , 00000000001.44.6, , , , 00000000001.47.6, 00000000001.68.6, , , ] 2024-04-13T22:21:01.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:01.607+0000 7fe180115780 0 bucket checkpoint complete 2024-04-13T22:21:01.630 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:21:01.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:21:01.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:21:01.631 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:21:01.692 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:21:01.693 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:21:01.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:01.727+0000 7fa21c37c780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:01.731 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.2040.6, 00000000001.2283.6, , , 00000000001.1138.6, , , 00000000001.613.6, 00000000003.1891.6, 00000000002.1718.6] 2024-04-13T22:21:01.731 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.2040.6, 00000000001.2283.6, , , 00000000001.1138.6, , , 00000000001.613.6, 00000000003.1891.6, 00000000003.1730.6] 2024-04-13T22:21:06.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:06.728+0000 7fa21c37c780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:06.732 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.2040.6, 00000000001.2283.6, , , 00000000001.1138.6, , , 00000000001.613.6, 00000000003.1891.6, 00000000002.1718.6] 2024-04-13T22:21:06.732 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.2040.6, 00000000001.2283.6, , , 00000000001.1138.6, , , 00000000001.613.6, 00000000003.1891.6, 00000000003.1730.6] 2024-04-13T22:21:11.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:11.729+0000 7fa21c37c780 1 bucket sync caught up with source: 2024-04-13T22:21:11.734 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.2040.6, 00000000001.2283.6, , , 00000000001.1138.6, , , 00000000001.613.6, 00000000003.1891.6, 00000000003.1730.6] 2024-04-13T22:21:11.734 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000001.2040.6, 00000000001.2283.6, , , 00000000001.1138.6, , , 00000000001.613.6, 00000000003.1891.6, 00000000003.1730.6] 2024-04-13T22:21:11.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:11.729+0000 7fa21c37c780 0 bucket checkpoint complete 2024-04-13T22:21:11.751 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:21:11.751 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.751 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.751 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.751 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:11.752 DEBUG:boto:Data: 2024-04-13T22:21:11.752 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.752 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.752 DEBUG:boto:Params: {} 2024-04-13T22:21:11.752 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:21:11.753 DEBUG:boto:Token: None 2024-04-13T22:21:11.753 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/ 2024-04-13T22:21:11.753 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9MuPTbs3FPEeF6MJ7lEWPvKepoQ= 2024-04-13T22:21:11.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9MuPTbs3FPEeF6MJ7lEWPvKepoQ=', 'Content-Length': '0'} 2024-04-13T22:21:11.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002902e769d7428e5b-00661b0557-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.758 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.758 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.759 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.759 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:11.759 DEBUG:boto:Data: 2024-04-13T22:21:11.759 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.759 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.759 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.759 DEBUG:boto:Params: {} 2024-04-13T22:21:11.759 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:21:11.760 DEBUG:boto:Token: None 2024-04-13T22:21:11.760 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/ 2024-04-13T22:21:11.760 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9MuPTbs3FPEeF6MJ7lEWPvKepoQ= 2024-04-13T22:21:11.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9MuPTbs3FPEeF6MJ7lEWPvKepoQ=', 'Content-Length': '0'} 2024-04-13T22:21:11.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086c96e7409c19f54-00661b0557-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.764 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:21:11.765 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.765 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.765 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.765 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.765 DEBUG:boto:Method: GET 2024-04-13T22:21:11.765 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.765 DEBUG:boto:Data: 2024-04-13T22:21:11.765 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.765 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.766 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.766 DEBUG:boto:Params: {} 2024-04-13T22:21:11.766 DEBUG:boto:Token: None 2024-04-13T22:21:11.766 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/?versions 2024-04-13T22:21:11.766 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI= 2024-04-13T22:21:11.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI=', 'Content-Length': '0'} 2024-04-13T22:21:11.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c8372a521ab4a3e-00661b0557-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.769 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:21:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:21:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:21:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:21:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:21:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:21:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:21:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:21:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:21:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:21:11.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:21:11.771 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.771 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.771 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.771 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.772 DEBUG:boto:Method: GET 2024-04-13T22:21:11.772 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.772 DEBUG:boto:Data: 2024-04-13T22:21:11.772 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.772 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.772 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.772 DEBUG:boto:Params: {} 2024-04-13T22:21:11.772 DEBUG:boto:Token: None 2024-04-13T22:21:11.772 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/?versions 2024-04-13T22:21:11.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI= 2024-04-13T22:21:11.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI=', 'Content-Length': '0'} 2024-04-13T22:21:11.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c439443b867b8576-00661b0557-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.775 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:21:11.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:21:11.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:21:11.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:21:11.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:21:11.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:21:11.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:21:11.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:21:11.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:21:11.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:21:11.777 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.777 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.777 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.777 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.778 DEBUG:boto:Method: GET 2024-04-13T22:21:11.778 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.778 DEBUG:boto:Data: 2024-04-13T22:21:11.778 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.778 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.778 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.778 DEBUG:boto:Params: {} 2024-04-13T22:21:11.778 DEBUG:boto:Token: None 2024-04-13T22:21:11.779 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/?versions 2024-04-13T22:21:11.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI= 2024-04-13T22:21:11.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI=', 'Content-Length': '0'} 2024-04-13T22:21:11.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688ea385b6ea4894-00661b0557-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.782 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.782 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.783 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.783 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.783 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.783 DEBUG:boto:Method: GET 2024-04-13T22:21:11.783 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:21:11.783 DEBUG:boto:Data: 2024-04-13T22:21:11.784 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.784 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.784 DEBUG:boto:Params: {} 2024-04-13T22:21:11.784 DEBUG:boto:Token: None 2024-04-13T22:21:11.784 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/?versions 2024-04-13T22:21:11.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI= 2024-04-13T22:21:11.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d3PWAoX3Pj433vjg86JDbKb9kFI=', 'Content-Length': '0'} 2024-04-13T22:21:11.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc9f59a590f3b13f-00661b0557-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.788 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:21:11.789 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.790 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.790 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.790 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.790 DEBUG:boto:Method: GET 2024-04-13T22:21:11.791 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.791 DEBUG:boto:Data: 2024-04-13T22:21:11.791 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.791 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.792 DEBUG:boto:Params: {} 2024-04-13T22:21:11.792 DEBUG:boto:Token: None 2024-04-13T22:21:11.792 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o= 2024-04-13T22:21:11.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o=', 'Content-Length': '0'} 2024-04-13T22:21:11.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029bde999bf6fc0fb-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.795 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.796 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.796 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.796 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.796 DEBUG:boto:Method: GET 2024-04-13T22:21:11.796 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.797 DEBUG:boto:Data: 2024-04-13T22:21:11.797 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.797 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.797 DEBUG:boto:Params: {} 2024-04-13T22:21:11.798 DEBUG:boto:Token: None 2024-04-13T22:21:11.798 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o= 2024-04-13T22:21:11.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o=', 'Content-Length': '0'} 2024-04-13T22:21:11.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6bffa459093b7d3-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.842 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.843 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.843 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.843 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.844 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.844 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.844 DEBUG:boto:Data: 2024-04-13T22:21:11.844 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.845 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.845 DEBUG:boto:Params: {} 2024-04-13T22:21:11.845 DEBUG:boto:Token: None 2024-04-13T22:21:11.845 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.846 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6o375L6DFcjzw0kONRxfFsgdw38= 2024-04-13T22:21:11.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6o375L6DFcjzw0kONRxfFsgdw38=', 'Content-Length': '0'} 2024-04-13T22:21:11.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011be05f85dfc372e-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.848 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.848 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.848 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.848 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.848 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.848 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.848 DEBUG:boto:Data: 2024-04-13T22:21:11.848 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.848 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.849 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.849 DEBUG:boto:Params: {} 2024-04-13T22:21:11.849 DEBUG:boto:Token: None 2024-04-13T22:21:11.849 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6o375L6DFcjzw0kONRxfFsgdw38= 2024-04-13T22:21:11.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6o375L6DFcjzw0kONRxfFsgdw38=', 'Content-Length': '0'} 2024-04-13T22:21:11.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757479b78df5d643-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:21:11.851 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.852 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.852 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.852 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.852 DEBUG:boto:Method: GET 2024-04-13T22:21:11.852 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.852 DEBUG:boto:Data: 2024-04-13T22:21:11.852 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.852 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.852 DEBUG:boto:Params: {} 2024-04-13T22:21:11.852 DEBUG:boto:Token: None 2024-04-13T22:21:11.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o= 2024-04-13T22:21:11.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o=', 'Content-Length': '0'} 2024-04-13T22:21:11.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002060391799082bcc-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.855 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.855 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.855 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.855 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.855 DEBUG:boto:Method: GET 2024-04-13T22:21:11.855 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.856 DEBUG:boto:Data: 2024-04-13T22:21:11.856 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.856 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.856 DEBUG:boto:Params: {} 2024-04-13T22:21:11.856 DEBUG:boto:Token: None 2024-04-13T22:21:11.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:21:11.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o= 2024-04-13T22:21:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:inmv1tUIiuinYeF7IGFFJFkyM0o=', 'Content-Length': '0'} 2024-04-13T22:21:11.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0bf228265ec8a4d-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.859 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.859 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.859 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.859 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.859 DEBUG:boto:Method: GET 2024-04-13T22:21:11.859 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.859 DEBUG:boto:Data: 2024-04-13T22:21:11.859 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.859 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.859 DEBUG:boto:Params: {} 2024-04-13T22:21:11.860 DEBUG:boto:Token: None 2024-04-13T22:21:11.860 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/ 2024-04-13T22:21:11.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI= 2024-04-13T22:21:11.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI=', 'Content-Length': '0'} 2024-04-13T22:21:11.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128a7975382e9122-00661b0557-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.863 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.863 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.863 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.863 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.863 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.863 DEBUG:boto:Method: GET 2024-04-13T22:21:11.863 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.864 DEBUG:boto:Data: 2024-04-13T22:21:11.864 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.864 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.864 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.864 DEBUG:boto:Params: {} 2024-04-13T22:21:11.864 DEBUG:boto:Token: None 2024-04-13T22:21:11.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/ 2024-04-13T22:21:11.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI= 2024-04-13T22:21:11.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI=', 'Content-Length': '0'} 2024-04-13T22:21:11.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000098cd3d3d31ac114-00661b0557-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.910 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.911 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.911 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.911 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.911 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:21:11.911 DEBUG:boto:Data: 2024-04-13T22:21:11.912 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.912 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.912 DEBUG:boto:Params: {} 2024-04-13T22:21:11.912 DEBUG:boto:Token: None 2024-04-13T22:21:11.913 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11 2024-04-13T22:21:11.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6bOqKK8UJcY69CtRQZyM4qYHuxE= 2024-04-13T22:21:11.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6bOqKK8UJcY69CtRQZyM4qYHuxE=', 'Content-Length': '0'} 2024-04-13T22:21:11.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ba1760adc1dbbf-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.915 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.915 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.915 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.915 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:21:11.915 DEBUG:boto:Data: 2024-04-13T22:21:11.915 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.915 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.916 DEBUG:boto:Params: {} 2024-04-13T22:21:11.916 DEBUG:boto:Token: None 2024-04-13T22:21:11.916 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11 2024-04-13T22:21:11.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6bOqKK8UJcY69CtRQZyM4qYHuxE= 2024-04-13T22:21:11.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6bOqKK8UJcY69CtRQZyM4qYHuxE=', 'Content-Length': '0'} 2024-04-13T22:21:11.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bdf77c288f4d00-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:21:11.918 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.918 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.918 DEBUG:boto:Method: GET 2024-04-13T22:21:11.918 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:21:11.919 DEBUG:boto:Data: 2024-04-13T22:21:11.919 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.919 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.919 DEBUG:boto:Params: {} 2024-04-13T22:21:11.919 DEBUG:boto:Token: None 2024-04-13T22:21:11.919 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11 2024-04-13T22:21:11.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a2LtTNoZzqZCtkqYMhTo39U/qRE= 2024-04-13T22:21:11.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a2LtTNoZzqZCtkqYMhTo39U/qRE=', 'Content-Length': '0'} 2024-04-13T22:21:11.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eab1edb4e418ea5-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.922 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:21:11.922 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:21:11.922 DEBUG:boto:Method: GET 2024-04-13T22:21:11.922 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:21:11.922 DEBUG:boto:Data: 2024-04-13T22:21:11.922 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.922 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.922 DEBUG:boto:Params: {} 2024-04-13T22:21:11.923 DEBUG:boto:Token: None 2024-04-13T22:21:11.923 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj11 2024-04-13T22:21:11.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:a2LtTNoZzqZCtkqYMhTo39U/qRE= 2024-04-13T22:21:11.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:a2LtTNoZzqZCtkqYMhTo39U/qRE=', 'Content-Length': '0'} 2024-04-13T22:21:11.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c08d5c600298da-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.926 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.926 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.926 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.926 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.926 DEBUG:boto:Method: GET 2024-04-13T22:21:11.926 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.926 DEBUG:boto:Data: 2024-04-13T22:21:11.927 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.927 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.927 DEBUG:boto:Params: {} 2024-04-13T22:21:11.927 DEBUG:boto:Token: None 2024-04-13T22:21:11.927 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/ 2024-04-13T22:21:11.927 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI= 2024-04-13T22:21:11.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI=', 'Content-Length': '0'} 2024-04-13T22:21:11.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f6635fbc12dabd4-00661b0557-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.930 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.931 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:11.931 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:11.931 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.931 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.931 DEBUG:boto:Method: GET 2024-04-13T22:21:11.931 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:21:11.931 DEBUG:boto:Data: 2024-04-13T22:21:11.931 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.932 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.932 DEBUG:boto:Params: {} 2024-04-13T22:21:11.932 DEBUG:boto:Token: None 2024-04-13T22:21:11.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/ 2024-04-13T22:21:11.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI= 2024-04-13T22:21:11.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PbStNJRu7qxKgQBv8t+2yai+VDI=', 'Content-Length': '0'} 2024-04-13T22:21:11.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6da9506159dec6-00661b0557-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.935 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:11.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:21:11.936 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:11.936 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:11.936 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.936 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.936 DEBUG:boto:Method: GET 2024-04-13T22:21:11.936 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.937 DEBUG:boto:Data: 2024-04-13T22:21:11.937 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.937 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.937 DEBUG:boto:Params: {} 2024-04-13T22:21:11.937 DEBUG:boto:Token: None 2024-04-13T22:21:11.937 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WjQh7Tquy9Jmls+1bTUTVwx970M= 2024-04-13T22:21:11.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WjQh7Tquy9Jmls+1bTUTVwx970M=', 'Content-Length': '0'} 2024-04-13T22:21:11.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080160a7e4dbfd158-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.986 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:11.987 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:11.987 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.987 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.987 DEBUG:boto:Method: GET 2024-04-13T22:21:11.987 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.987 DEBUG:boto:Data: 2024-04-13T22:21:11.987 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.988 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.988 DEBUG:boto:Params: {} 2024-04-13T22:21:11.988 DEBUG:boto:Token: None 2024-04-13T22:21:11.988 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WjQh7Tquy9Jmls+1bTUTVwx970M= 2024-04-13T22:21:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WjQh7Tquy9Jmls+1bTUTVwx970M=', 'Content-Length': '0'} 2024-04-13T22:21:11.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061fc4607eac3c16a-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.991 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:11.991 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:11.991 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.991 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.991 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.991 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.991 DEBUG:boto:Data: 2024-04-13T22:21:11.991 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.991 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.992 DEBUG:boto:Params: {} 2024-04-13T22:21:11.992 DEBUG:boto:Token: None 2024-04-13T22:21:11.992 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+3BUmuIzAjJ2aclRfwl35W6LzTY= 2024-04-13T22:21:11.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+3BUmuIzAjJ2aclRfwl35W6LzTY=', 'Content-Length': '0'} 2024-04-13T22:21:11.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a0eb9c4715292dc-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.994 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:11.994 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:11.994 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.994 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.994 DEBUG:boto:Method: HEAD 2024-04-13T22:21:11.995 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.995 DEBUG:boto:Data: 2024-04-13T22:21:11.995 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:11.995 DEBUG:boto:Port: 8000 2024-04-13T22:21:11.995 DEBUG:boto:Params: {} 2024-04-13T22:21:11.995 DEBUG:boto:Token: None 2024-04-13T22:21:11.995 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+3BUmuIzAjJ2aclRfwl35W6LzTY= 2024-04-13T22:21:11.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+3BUmuIzAjJ2aclRfwl35W6LzTY=', 'Content-Length': '0'} 2024-04-13T22:21:11.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f97f3132b4f523cc-00661b0557-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:11.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:21:11.998 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:11.998 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:11.998 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.998 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.998 DEBUG:boto:Method: GET 2024-04-13T22:21:11.998 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.998 DEBUG:boto:Data: 2024-04-13T22:21:11.998 DEBUG:boto:Headers: {} 2024-04-13T22:21:11.998 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:11.998 DEBUG:boto:Port: 8001 2024-04-13T22:21:11.999 DEBUG:boto:Params: {} 2024-04-13T22:21:11.999 DEBUG:boto:Token: None 2024-04-13T22:21:11.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:11 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:11.999 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WjQh7Tquy9Jmls+1bTUTVwx970M= 2024-04-13T22:21:11.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:11 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WjQh7Tquy9Jmls+1bTUTVwx970M=', 'Content-Length': '0'} 2024-04-13T22:21:12.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024701477e2dfb9b6-00661b0557-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.001 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:12.001 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:12.001 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:12.001 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:21:12.001 DEBUG:boto:Method: GET 2024-04-13T22:21:12.001 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:12.002 DEBUG:boto:Data: 2024-04-13T22:21:12.002 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.002 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.002 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.002 DEBUG:boto:Params: {} 2024-04-13T22:21:12.002 DEBUG:boto:Token: None 2024-04-13T22:21:12.002 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:21:12.002 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aEr/WOBA7cC8T5XmkbCYwax2YXs= 2024-04-13T22:21:12.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aEr/WOBA7cC8T5XmkbCYwax2YXs=', 'Content-Length': '0'} 2024-04-13T22:21:12.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71a342c164905d8-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.004 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.005 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.005 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.005 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.005 DEBUG:boto:Method: GET 2024-04-13T22:21:12.005 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.005 DEBUG:boto:Data: 2024-04-13T22:21:12.005 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.005 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.005 DEBUG:boto:Params: {} 2024-04-13T22:21:12.006 DEBUG:boto:Token: None 2024-04-13T22:21:12.006 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e5f5b3b73422f47-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.008 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.010 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.010 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.010 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.010 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.011 DEBUG:boto:Method: GET 2024-04-13T22:21:12.011 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.011 DEBUG:boto:Data: 2024-04-13T22:21:12.011 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.012 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.012 DEBUG:boto:Params: {} 2024-04-13T22:21:12.012 DEBUG:boto:Token: None 2024-04-13T22:21:12.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc51a01b348a2f66-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.015 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.015 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:12.015 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:12.016 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.016 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:21:12.016 DEBUG:boto:Data: 2024-04-13T22:21:12.016 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.016 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.016 DEBUG:boto:Params: {} 2024-04-13T22:21:12.016 DEBUG:boto:Token: None 2024-04-13T22:21:12.016 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj17 2024-04-13T22:21:12.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:afBaJcLOb/8Us8NZmmDeOYtGse0= 2024-04-13T22:21:12.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:afBaJcLOb/8Us8NZmmDeOYtGse0=', 'Content-Length': '0'} 2024-04-13T22:21:12.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33d33c0c12fcfe5-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.018 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:12.019 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:12.019 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.019 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:21:12.019 DEBUG:boto:Data: 2024-04-13T22:21:12.019 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.019 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.019 DEBUG:boto:Params: {} 2024-04-13T22:21:12.019 DEBUG:boto:Token: None 2024-04-13T22:21:12.019 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj17 2024-04-13T22:21:12.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:afBaJcLOb/8Us8NZmmDeOYtGse0= 2024-04-13T22:21:12.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:afBaJcLOb/8Us8NZmmDeOYtGse0=', 'Content-Length': '0'} 2024-04-13T22:21:12.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2b03c1855ed17b-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:21:12.022 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:12.022 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:12.022 DEBUG:boto:Method: GET 2024-04-13T22:21:12.022 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:21:12.022 DEBUG:boto:Data: 2024-04-13T22:21:12.022 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.023 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.023 DEBUG:boto:Params: {} 2024-04-13T22:21:12.023 DEBUG:boto:Token: None 2024-04-13T22:21:12.023 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj17 2024-04-13T22:21:12.023 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TkKY8lZiuMGaokFp0pZEefpbUnA= 2024-04-13T22:21:12.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TkKY8lZiuMGaokFp0pZEefpbUnA=', 'Content-Length': '0'} 2024-04-13T22:21:12.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000655e9dfd3833d25a-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.025 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:21:12.026 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:21:12.026 DEBUG:boto:Method: GET 2024-04-13T22:21:12.026 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:21:12.026 DEBUG:boto:Data: 2024-04-13T22:21:12.026 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.026 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.026 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.026 DEBUG:boto:Params: {} 2024-04-13T22:21:12.026 DEBUG:boto:Token: None 2024-04-13T22:21:12.026 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj17 2024-04-13T22:21:12.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TkKY8lZiuMGaokFp0pZEefpbUnA= 2024-04-13T22:21:12.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TkKY8lZiuMGaokFp0pZEefpbUnA=', 'Content-Length': '0'} 2024-04-13T22:21:12.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936873380f2958ee-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.070 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.071 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.071 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.071 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.071 DEBUG:boto:Method: GET 2024-04-13T22:21:12.071 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.071 DEBUG:boto:Data: 2024-04-13T22:21:12.071 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.071 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.072 DEBUG:boto:Params: {} 2024-04-13T22:21:12.072 DEBUG:boto:Token: None 2024-04-13T22:21:12.072 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007902282c953026ba-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.075 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.075 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.076 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.076 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.076 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.076 DEBUG:boto:Method: GET 2024-04-13T22:21:12.076 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:21:12.076 DEBUG:boto:Data: 2024-04-13T22:21:12.076 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.076 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.076 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.076 DEBUG:boto:Params: {} 2024-04-13T22:21:12.076 DEBUG:boto:Token: None 2024-04-13T22:21:12.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb163862534f983-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.080 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:21:12.080 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.080 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.080 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.080 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.081 DEBUG:boto:Method: GET 2024-04-13T22:21:12.081 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.081 DEBUG:boto:Data: 2024-04-13T22:21:12.081 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.081 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.081 DEBUG:boto:Params: {} 2024-04-13T22:21:12.081 DEBUG:boto:Token: None 2024-04-13T22:21:12.081 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk= 2024-04-13T22:21:12.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk=', 'Content-Length': '0'} 2024-04-13T22:21:12.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b236b3dd81dd1a84-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.084 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.084 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.084 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.084 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.084 DEBUG:boto:Method: GET 2024-04-13T22:21:12.084 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.084 DEBUG:boto:Data: 2024-04-13T22:21:12.084 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.084 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.084 DEBUG:boto:Params: {} 2024-04-13T22:21:12.085 DEBUG:boto:Token: None 2024-04-13T22:21:12.085 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk= 2024-04-13T22:21:12.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk=', 'Content-Length': '0'} 2024-04-13T22:21:12.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058208ce1a181c052-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.087 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.087 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.087 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.088 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.088 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.088 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.088 DEBUG:boto:Data: 2024-04-13T22:21:12.088 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.088 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.088 DEBUG:boto:Params: {} 2024-04-13T22:21:12.088 DEBUG:boto:Token: None 2024-04-13T22:21:12.088 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4BW5fB9alI0fucFFwLdPTCsc5eY= 2024-04-13T22:21:12.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4BW5fB9alI0fucFFwLdPTCsc5eY=', 'Content-Length': '0'} 2024-04-13T22:21:12.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68c4665ec954382-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.090 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.090 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.091 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.091 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.091 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.091 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.091 DEBUG:boto:Data: 2024-04-13T22:21:12.091 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.091 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.091 DEBUG:boto:Params: {} 2024-04-13T22:21:12.091 DEBUG:boto:Token: None 2024-04-13T22:21:12.091 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4BW5fB9alI0fucFFwLdPTCsc5eY= 2024-04-13T22:21:12.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4BW5fB9alI0fucFFwLdPTCsc5eY=', 'Content-Length': '0'} 2024-04-13T22:21:12.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e95a2820973f14-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:21:12.093 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.094 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.094 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.094 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.094 DEBUG:boto:Method: GET 2024-04-13T22:21:12.094 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.094 DEBUG:boto:Data: 2024-04-13T22:21:12.094 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.094 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.094 DEBUG:boto:Params: {} 2024-04-13T22:21:12.095 DEBUG:boto:Token: None 2024-04-13T22:21:12.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk= 2024-04-13T22:21:12.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk=', 'Content-Length': '0'} 2024-04-13T22:21:12.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fea0eff4f5539323-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.097 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.097 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.097 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.097 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.097 DEBUG:boto:Method: GET 2024-04-13T22:21:12.097 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.097 DEBUG:boto:Data: 2024-04-13T22:21:12.097 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.097 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.097 DEBUG:boto:Params: {} 2024-04-13T22:21:12.098 DEBUG:boto:Token: None 2024-04-13T22:21:12.098 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:21:12.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk= 2024-04-13T22:21:12.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AuQPa3BTieAlHRvocONOf5gTmYk=', 'Content-Length': '0'} 2024-04-13T22:21:12.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060426e90d1c9923f-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.143 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.143 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.143 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.143 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.143 DEBUG:boto:Method: GET 2024-04-13T22:21:12.143 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.143 DEBUG:boto:Data: 2024-04-13T22:21:12.143 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.143 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.144 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.144 DEBUG:boto:Params: {} 2024-04-13T22:21:12.144 DEBUG:boto:Token: None 2024-04-13T22:21:12.144 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079d936c6dfd36fc0-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.147 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.147 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.147 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.147 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.147 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.148 DEBUG:boto:Method: GET 2024-04-13T22:21:12.148 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.148 DEBUG:boto:Data: 2024-04-13T22:21:12.148 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.148 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.148 DEBUG:boto:Params: {} 2024-04-13T22:21:12.148 DEBUG:boto:Token: None 2024-04-13T22:21:12.148 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029c75a0c96b51b43-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.151 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.152 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.152 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.152 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.152 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:21:12.152 DEBUG:boto:Data: 2024-04-13T22:21:12.152 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.152 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.152 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.152 DEBUG:boto:Params: {} 2024-04-13T22:21:12.152 DEBUG:boto:Token: None 2024-04-13T22:21:12.153 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22 2024-04-13T22:21:12.153 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VwmOKl/oLOC4aSdaRskqYcLVoJE= 2024-04-13T22:21:12.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VwmOKl/oLOC4aSdaRskqYcLVoJE=', 'Content-Length': '0'} 2024-04-13T22:21:12.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab813e87b65e26c6-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.155 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.155 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.155 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.155 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:21:12.155 DEBUG:boto:Data: 2024-04-13T22:21:12.155 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.155 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.155 DEBUG:boto:Params: {} 2024-04-13T22:21:12.156 DEBUG:boto:Token: None 2024-04-13T22:21:12.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22 2024-04-13T22:21:12.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VwmOKl/oLOC4aSdaRskqYcLVoJE= 2024-04-13T22:21:12.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VwmOKl/oLOC4aSdaRskqYcLVoJE=', 'Content-Length': '0'} 2024-04-13T22:21:12.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277feabde0197f16-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:21:12.158 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.158 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.158 DEBUG:boto:Method: GET 2024-04-13T22:21:12.158 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:21:12.158 DEBUG:boto:Data: 2024-04-13T22:21:12.158 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.158 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.159 DEBUG:boto:Params: {} 2024-04-13T22:21:12.159 DEBUG:boto:Token: None 2024-04-13T22:21:12.159 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22 2024-04-13T22:21:12.159 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VJaFXrjPLNRH0lKKeJPQWjO/ObY= 2024-04-13T22:21:12.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VJaFXrjPLNRH0lKKeJPQWjO/ObY=', 'Content-Length': '0'} 2024-04-13T22:21:12.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5d4af866569c1d-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.202 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:21:12.203 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:21:12.203 DEBUG:boto:Method: GET 2024-04-13T22:21:12.203 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:21:12.203 DEBUG:boto:Data: 2024-04-13T22:21:12.203 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.203 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.203 DEBUG:boto:Params: {} 2024-04-13T22:21:12.203 DEBUG:boto:Token: None 2024-04-13T22:21:12.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj22 2024-04-13T22:21:12.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VJaFXrjPLNRH0lKKeJPQWjO/ObY= 2024-04-13T22:21:12.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VJaFXrjPLNRH0lKKeJPQWjO/ObY=', 'Content-Length': '0'} 2024-04-13T22:21:12.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbea02c05110bf35-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.206 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.206 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.207 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.207 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.207 DEBUG:boto:Method: GET 2024-04-13T22:21:12.207 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.207 DEBUG:boto:Data: 2024-04-13T22:21:12.207 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.208 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.208 DEBUG:boto:Params: {} 2024-04-13T22:21:12.208 DEBUG:boto:Token: None 2024-04-13T22:21:12.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06d0dfb2db02d4c-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.211 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.212 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.212 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.212 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.212 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.212 DEBUG:boto:Method: GET 2024-04-13T22:21:12.212 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:21:12.212 DEBUG:boto:Data: 2024-04-13T22:21:12.212 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.212 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.213 DEBUG:boto:Params: {} 2024-04-13T22:21:12.213 DEBUG:boto:Token: None 2024-04-13T22:21:12.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2625659c29f761f-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.216 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:21:12.216 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.217 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.217 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.217 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.217 DEBUG:boto:Method: GET 2024-04-13T22:21:12.217 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.217 DEBUG:boto:Data: 2024-04-13T22:21:12.217 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.217 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.217 DEBUG:boto:Params: {} 2024-04-13T22:21:12.218 DEBUG:boto:Token: None 2024-04-13T22:21:12.218 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8= 2024-04-13T22:21:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8=', 'Content-Length': '0'} 2024-04-13T22:21:12.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed92433384ae8d35-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.220 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.220 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.220 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.221 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.221 DEBUG:boto:Method: GET 2024-04-13T22:21:12.221 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.221 DEBUG:boto:Data: 2024-04-13T22:21:12.221 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.221 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.221 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.221 DEBUG:boto:Params: {} 2024-04-13T22:21:12.221 DEBUG:boto:Token: None 2024-04-13T22:21:12.221 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.222 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8= 2024-04-13T22:21:12.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8=', 'Content-Length': '0'} 2024-04-13T22:21:12.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ec81c610c6817b-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.266 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.267 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.267 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.267 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.267 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.267 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.267 DEBUG:boto:Data: 2024-04-13T22:21:12.267 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.267 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.267 DEBUG:boto:Params: {} 2024-04-13T22:21:12.268 DEBUG:boto:Token: None 2024-04-13T22:21:12.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RE1llTG2BJqrwWTYybVNreRBMNQ= 2024-04-13T22:21:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RE1llTG2BJqrwWTYybVNreRBMNQ=', 'Content-Length': '0'} 2024-04-13T22:21:12.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682c68c25673412c-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.270 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.270 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.270 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.271 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.271 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.271 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.271 DEBUG:boto:Data: 2024-04-13T22:21:12.271 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.271 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.271 DEBUG:boto:Params: {} 2024-04-13T22:21:12.271 DEBUG:boto:Token: None 2024-04-13T22:21:12.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RE1llTG2BJqrwWTYybVNreRBMNQ= 2024-04-13T22:21:12.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RE1llTG2BJqrwWTYybVNreRBMNQ=', 'Content-Length': '0'} 2024-04-13T22:21:12.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f494410c92d327-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:21:12.274 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.274 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.274 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.274 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.274 DEBUG:boto:Method: GET 2024-04-13T22:21:12.275 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.275 DEBUG:boto:Data: 2024-04-13T22:21:12.275 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.275 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.275 DEBUG:boto:Params: {} 2024-04-13T22:21:12.275 DEBUG:boto:Token: None 2024-04-13T22:21:12.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8= 2024-04-13T22:21:12.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8=', 'Content-Length': '0'} 2024-04-13T22:21:12.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9eeee2da330be0-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.277 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.278 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.278 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.278 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.278 DEBUG:boto:Method: GET 2024-04-13T22:21:12.278 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.278 DEBUG:boto:Data: 2024-04-13T22:21:12.278 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.278 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.278 DEBUG:boto:Params: {} 2024-04-13T22:21:12.278 DEBUG:boto:Token: None 2024-04-13T22:21:12.279 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:21:12.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8= 2024-04-13T22:21:12.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zHvf4pfm/zcPm0iJVCCD+hiiUM8=', 'Content-Length': '0'} 2024-04-13T22:21:12.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5aacacc4928c0ae-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.281 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.281 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.281 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.281 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.281 DEBUG:boto:Method: GET 2024-04-13T22:21:12.281 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.282 DEBUG:boto:Data: 2024-04-13T22:21:12.282 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.282 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.282 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.282 DEBUG:boto:Params: {} 2024-04-13T22:21:12.282 DEBUG:boto:Token: None 2024-04-13T22:21:12.282 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.282 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7342802a58fe2fa-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.285 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.285 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.285 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.285 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.286 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.286 DEBUG:boto:Method: GET 2024-04-13T22:21:12.286 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.286 DEBUG:boto:Data: 2024-04-13T22:21:12.286 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.286 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.286 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.286 DEBUG:boto:Params: {} 2024-04-13T22:21:12.286 DEBUG:boto:Token: None 2024-04-13T22:21:12.286 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.287 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066877637f4a88599-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.289 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.290 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.290 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.290 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.290 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:21:12.290 DEBUG:boto:Data: 2024-04-13T22:21:12.290 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.291 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.291 DEBUG:boto:Params: {} 2024-04-13T22:21:12.291 DEBUG:boto:Token: None 2024-04-13T22:21:12.291 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33 2024-04-13T22:21:12.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CTKgkBjAqkGjC/2n6ilSQ9TUeQM= 2024-04-13T22:21:12.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CTKgkBjAqkGjC/2n6ilSQ9TUeQM=', 'Content-Length': '0'} 2024-04-13T22:21:12.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c25a65c5f038d36e-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.293 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.293 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.293 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.293 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:21:12.293 DEBUG:boto:Data: 2024-04-13T22:21:12.293 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.294 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.294 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.294 DEBUG:boto:Params: {} 2024-04-13T22:21:12.294 DEBUG:boto:Token: None 2024-04-13T22:21:12.294 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33 2024-04-13T22:21:12.294 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CTKgkBjAqkGjC/2n6ilSQ9TUeQM= 2024-04-13T22:21:12.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CTKgkBjAqkGjC/2n6ilSQ9TUeQM=', 'Content-Length': '0'} 2024-04-13T22:21:12.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c4ffcd2dda811f-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:21:12.297 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.297 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.297 DEBUG:boto:Method: GET 2024-04-13T22:21:12.297 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:21:12.297 DEBUG:boto:Data: 2024-04-13T22:21:12.297 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.297 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.297 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.297 DEBUG:boto:Params: {} 2024-04-13T22:21:12.298 DEBUG:boto:Token: None 2024-04-13T22:21:12.298 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33 2024-04-13T22:21:12.298 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sEK1LEvkberogY+tXIfKpRa2Ei8= 2024-04-13T22:21:12.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sEK1LEvkberogY+tXIfKpRa2Ei8=', 'Content-Length': '0'} 2024-04-13T22:21:12.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a57fb9a034837f-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.300 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:21:12.300 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:21:12.300 DEBUG:boto:Method: GET 2024-04-13T22:21:12.300 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:21:12.300 DEBUG:boto:Data: 2024-04-13T22:21:12.301 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.301 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.301 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.301 DEBUG:boto:Params: {} 2024-04-13T22:21:12.301 DEBUG:boto:Token: None 2024-04-13T22:21:12.301 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj33 2024-04-13T22:21:12.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sEK1LEvkberogY+tXIfKpRa2Ei8= 2024-04-13T22:21:12.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sEK1LEvkberogY+tXIfKpRa2Ei8=', 'Content-Length': '0'} 2024-04-13T22:21:12.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6ec2141b570267-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.303 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.304 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.304 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.304 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.304 DEBUG:boto:Method: GET 2024-04-13T22:21:12.304 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.304 DEBUG:boto:Data: 2024-04-13T22:21:12.304 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.304 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.304 DEBUG:boto:Params: {} 2024-04-13T22:21:12.304 DEBUG:boto:Token: None 2024-04-13T22:21:12.305 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099341789711c92ef-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.307 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.308 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.308 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.308 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.308 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.308 DEBUG:boto:Method: GET 2024-04-13T22:21:12.308 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:21:12.308 DEBUG:boto:Data: 2024-04-13T22:21:12.309 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.309 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.309 DEBUG:boto:Params: {} 2024-04-13T22:21:12.309 DEBUG:boto:Token: None 2024-04-13T22:21:12.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.309 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b635aac021ef9ee-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.313 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:21:12.314 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.314 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.314 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.314 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.314 DEBUG:boto:Method: GET 2024-04-13T22:21:12.314 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.314 DEBUG:boto:Data: 2024-04-13T22:21:12.314 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.315 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.315 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.315 DEBUG:boto:Params: {} 2024-04-13T22:21:12.315 DEBUG:boto:Token: None 2024-04-13T22:21:12.315 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo= 2024-04-13T22:21:12.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo=', 'Content-Length': '0'} 2024-04-13T22:21:12.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009186840366725453-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.358 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.359 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.359 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.359 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.359 DEBUG:boto:Method: GET 2024-04-13T22:21:12.359 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.359 DEBUG:boto:Data: 2024-04-13T22:21:12.359 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.359 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.359 DEBUG:boto:Params: {} 2024-04-13T22:21:12.360 DEBUG:boto:Token: None 2024-04-13T22:21:12.360 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo= 2024-04-13T22:21:12.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo=', 'Content-Length': '0'} 2024-04-13T22:21:12.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a39b515dc58b1288-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.362 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.362 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.362 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.362 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.363 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.363 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.363 DEBUG:boto:Data: 2024-04-13T22:21:12.363 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.363 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.363 DEBUG:boto:Params: {} 2024-04-13T22:21:12.363 DEBUG:boto:Token: None 2024-04-13T22:21:12.363 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FL8cMzIsR6Z/qAUubV1EJBPKtqY= 2024-04-13T22:21:12.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FL8cMzIsR6Z/qAUubV1EJBPKtqY=', 'Content-Length': '0'} 2024-04-13T22:21:12.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfd68e603efcbe8-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.366 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.366 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.366 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.366 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.366 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.366 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.366 DEBUG:boto:Data: 2024-04-13T22:21:12.366 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.366 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.367 DEBUG:boto:Params: {} 2024-04-13T22:21:12.367 DEBUG:boto:Token: None 2024-04-13T22:21:12.367 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FL8cMzIsR6Z/qAUubV1EJBPKtqY= 2024-04-13T22:21:12.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FL8cMzIsR6Z/qAUubV1EJBPKtqY=', 'Content-Length': '0'} 2024-04-13T22:21:12.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c342efcfb21bcc5-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:21:12.369 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.369 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.369 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.370 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.370 DEBUG:boto:Method: GET 2024-04-13T22:21:12.370 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.370 DEBUG:boto:Data: 2024-04-13T22:21:12.370 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.370 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.370 DEBUG:boto:Params: {} 2024-04-13T22:21:12.370 DEBUG:boto:Token: None 2024-04-13T22:21:12.370 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo= 2024-04-13T22:21:12.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo=', 'Content-Length': '0'} 2024-04-13T22:21:12.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a91c1d30452c5f5-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.373 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.373 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.373 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.373 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.373 DEBUG:boto:Method: GET 2024-04-13T22:21:12.373 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.373 DEBUG:boto:Data: 2024-04-13T22:21:12.373 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.373 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.374 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.374 DEBUG:boto:Params: {} 2024-04-13T22:21:12.374 DEBUG:boto:Token: None 2024-04-13T22:21:12.374 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:21:12.374 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo= 2024-04-13T22:21:12.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zOslC+g9dFhQi/dvQjGeldDaXAo=', 'Content-Length': '0'} 2024-04-13T22:21:12.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f796f161cf2afe1-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.376 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.377 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.377 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.377 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.377 DEBUG:boto:Method: GET 2024-04-13T22:21:12.377 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.377 DEBUG:boto:Data: 2024-04-13T22:21:12.377 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.377 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.377 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.377 DEBUG:boto:Params: {} 2024-04-13T22:21:12.377 DEBUG:boto:Token: None 2024-04-13T22:21:12.378 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.378 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000072d3bfbb8c55235-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.380 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.381 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.381 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.381 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.381 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.381 DEBUG:boto:Method: GET 2024-04-13T22:21:12.381 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.381 DEBUG:boto:Data: 2024-04-13T22:21:12.381 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.381 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.381 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.382 DEBUG:boto:Params: {} 2024-04-13T22:21:12.382 DEBUG:boto:Token: None 2024-04-13T22:21:12.382 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.382 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688863d8028617e7-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.386 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.386 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.386 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.386 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.386 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:21:12.386 DEBUG:boto:Data: 2024-04-13T22:21:12.386 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.387 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.387 DEBUG:boto:Params: {} 2024-04-13T22:21:12.387 DEBUG:boto:Token: None 2024-04-13T22:21:12.387 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34 2024-04-13T22:21:12.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P0y7XdPf0EjFfzXA99GSsjA4fnc= 2024-04-13T22:21:12.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P0y7XdPf0EjFfzXA99GSsjA4fnc=', 'Content-Length': '0'} 2024-04-13T22:21:12.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ea79690feb31ed-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.390 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.390 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.390 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.391 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:21:12.391 DEBUG:boto:Data: 2024-04-13T22:21:12.391 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.391 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.391 DEBUG:boto:Params: {} 2024-04-13T22:21:12.391 DEBUG:boto:Token: None 2024-04-13T22:21:12.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34 2024-04-13T22:21:12.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P0y7XdPf0EjFfzXA99GSsjA4fnc= 2024-04-13T22:21:12.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P0y7XdPf0EjFfzXA99GSsjA4fnc=', 'Content-Length': '0'} 2024-04-13T22:21:12.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3abd32b5cf4f9e-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:21:12.393 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.394 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.394 DEBUG:boto:Method: GET 2024-04-13T22:21:12.394 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:21:12.394 DEBUG:boto:Data: 2024-04-13T22:21:12.394 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.394 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.394 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.394 DEBUG:boto:Params: {} 2024-04-13T22:21:12.394 DEBUG:boto:Token: None 2024-04-13T22:21:12.394 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34 2024-04-13T22:21:12.395 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+39xP/Uwtlip6q2AADkEfI5/DQY= 2024-04-13T22:21:12.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+39xP/Uwtlip6q2AADkEfI5/DQY=', 'Content-Length': '0'} 2024-04-13T22:21:12.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000634848eeb038ce85-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.396 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:21:12.397 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:21:12.397 DEBUG:boto:Method: GET 2024-04-13T22:21:12.397 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:21:12.397 DEBUG:boto:Data: 2024-04-13T22:21:12.397 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.397 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.397 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.397 DEBUG:boto:Params: {} 2024-04-13T22:21:12.397 DEBUG:boto:Token: None 2024-04-13T22:21:12.397 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj34 2024-04-13T22:21:12.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+39xP/Uwtlip6q2AADkEfI5/DQY= 2024-04-13T22:21:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+39xP/Uwtlip6q2AADkEfI5/DQY=', 'Content-Length': '0'} 2024-04-13T22:21:12.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3063e3951b093e-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.399 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.399 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.400 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.400 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.400 DEBUG:boto:Method: GET 2024-04-13T22:21:12.400 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.400 DEBUG:boto:Data: 2024-04-13T22:21:12.400 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.400 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.401 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.401 DEBUG:boto:Params: {} 2024-04-13T22:21:12.401 DEBUG:boto:Token: None 2024-04-13T22:21:12.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a109d216938eff-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.403 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.404 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.404 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.404 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.404 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.404 DEBUG:boto:Method: GET 2024-04-13T22:21:12.404 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:21:12.404 DEBUG:boto:Data: 2024-04-13T22:21:12.404 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.405 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.405 DEBUG:boto:Params: {} 2024-04-13T22:21:12.405 DEBUG:boto:Token: None 2024-04-13T22:21:12.405 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875b861376131285-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.450 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:21:12.451 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.451 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.451 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.451 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.452 DEBUG:boto:Method: GET 2024-04-13T22:21:12.452 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.452 DEBUG:boto:Data: 2024-04-13T22:21:12.452 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.452 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.452 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.453 DEBUG:boto:Params: {} 2024-04-13T22:21:12.453 DEBUG:boto:Token: None 2024-04-13T22:21:12.453 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.453 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs= 2024-04-13T22:21:12.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs=', 'Content-Length': '0'} 2024-04-13T22:21:12.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cdeeb862b22425-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.455 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.456 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.456 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.456 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.456 DEBUG:boto:Method: GET 2024-04-13T22:21:12.456 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.456 DEBUG:boto:Data: 2024-04-13T22:21:12.456 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.456 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.457 DEBUG:boto:Params: {} 2024-04-13T22:21:12.457 DEBUG:boto:Token: None 2024-04-13T22:21:12.457 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.457 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs= 2024-04-13T22:21:12.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs=', 'Content-Length': '0'} 2024-04-13T22:21:12.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388c1fee0cd8c218-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.459 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.459 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.459 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.459 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.459 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.460 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.460 DEBUG:boto:Data: 2024-04-13T22:21:12.460 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.460 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.460 DEBUG:boto:Params: {} 2024-04-13T22:21:12.460 DEBUG:boto:Token: None 2024-04-13T22:21:12.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KQDRDjpWK2ibKzJzDacS46Yny6I= 2024-04-13T22:21:12.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KQDRDjpWK2ibKzJzDacS46Yny6I=', 'Content-Length': '0'} 2024-04-13T22:21:12.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009100329083f0ef8b-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.463 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.463 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.463 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.463 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.463 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.463 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.464 DEBUG:boto:Data: 2024-04-13T22:21:12.464 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.464 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.464 DEBUG:boto:Params: {} 2024-04-13T22:21:12.464 DEBUG:boto:Token: None 2024-04-13T22:21:12.464 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KQDRDjpWK2ibKzJzDacS46Yny6I= 2024-04-13T22:21:12.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KQDRDjpWK2ibKzJzDacS46Yny6I=', 'Content-Length': '0'} 2024-04-13T22:21:12.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3f91e9b20b4a44-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:21:12.467 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.467 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.467 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.467 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.467 DEBUG:boto:Method: GET 2024-04-13T22:21:12.467 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.467 DEBUG:boto:Data: 2024-04-13T22:21:12.467 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.467 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.467 DEBUG:boto:Params: {} 2024-04-13T22:21:12.468 DEBUG:boto:Token: None 2024-04-13T22:21:12.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs= 2024-04-13T22:21:12.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs=', 'Content-Length': '0'} 2024-04-13T22:21:12.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b8ef2d5fdd38cf-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.471 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.471 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.471 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.471 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.471 DEBUG:boto:Method: GET 2024-04-13T22:21:12.471 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.471 DEBUG:boto:Data: 2024-04-13T22:21:12.471 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.471 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.471 DEBUG:boto:Params: {} 2024-04-13T22:21:12.472 DEBUG:boto:Token: None 2024-04-13T22:21:12.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:21:12.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs= 2024-04-13T22:21:12.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cvk5P1sfsigMT6EpGHVXjxX/EBs=', 'Content-Length': '0'} 2024-04-13T22:21:12.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde32c7aa3709c85-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.514 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.515 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.515 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.515 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.515 DEBUG:boto:Method: GET 2024-04-13T22:21:12.515 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.515 DEBUG:boto:Data: 2024-04-13T22:21:12.515 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.515 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.515 DEBUG:boto:Params: {} 2024-04-13T22:21:12.518 DEBUG:boto:Token: None 2024-04-13T22:21:12.518 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ef019900e7d1920-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.521 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.522 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.522 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.522 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.522 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.522 DEBUG:boto:Method: GET 2024-04-13T22:21:12.522 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.522 DEBUG:boto:Data: 2024-04-13T22:21:12.522 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.522 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.523 DEBUG:boto:Params: {} 2024-04-13T22:21:12.523 DEBUG:boto:Token: None 2024-04-13T22:21:12.523 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d37440911ca8464c-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.526 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.526 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.526 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.526 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.526 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:21:12.526 DEBUG:boto:Data: 2024-04-13T22:21:12.527 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.527 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.527 DEBUG:boto:Params: {} 2024-04-13T22:21:12.527 DEBUG:boto:Token: None 2024-04-13T22:21:12.527 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44 2024-04-13T22:21:12.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/GPnb1gaxw4QAywCLLq/cengc9o= 2024-04-13T22:21:12.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/GPnb1gaxw4QAywCLLq/cengc9o=', 'Content-Length': '0'} 2024-04-13T22:21:12.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9ddec0469efe65a-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.529 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.529 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.529 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.529 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:21:12.529 DEBUG:boto:Data: 2024-04-13T22:21:12.530 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.530 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.530 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.530 DEBUG:boto:Params: {} 2024-04-13T22:21:12.530 DEBUG:boto:Token: None 2024-04-13T22:21:12.530 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44 2024-04-13T22:21:12.530 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/GPnb1gaxw4QAywCLLq/cengc9o= 2024-04-13T22:21:12.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/GPnb1gaxw4QAywCLLq/cengc9o=', 'Content-Length': '0'} 2024-04-13T22:21:12.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f52634ba6a0a7b01-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:21:12.532 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.532 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.532 DEBUG:boto:Method: GET 2024-04-13T22:21:12.533 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:21:12.533 DEBUG:boto:Data: 2024-04-13T22:21:12.533 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.533 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.533 DEBUG:boto:Params: {} 2024-04-13T22:21:12.533 DEBUG:boto:Token: None 2024-04-13T22:21:12.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44 2024-04-13T22:21:12.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3YFyEuTJicKiR+n1OfP+n8nG2nk= 2024-04-13T22:21:12.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3YFyEuTJicKiR+n1OfP+n8nG2nk=', 'Content-Length': '0'} 2024-04-13T22:21:12.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170de4c446150799-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.535 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:21:12.535 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:21:12.535 DEBUG:boto:Method: GET 2024-04-13T22:21:12.536 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:21:12.536 DEBUG:boto:Data: 2024-04-13T22:21:12.536 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.536 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.536 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.536 DEBUG:boto:Params: {} 2024-04-13T22:21:12.536 DEBUG:boto:Token: None 2024-04-13T22:21:12.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj44 2024-04-13T22:21:12.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3YFyEuTJicKiR+n1OfP+n8nG2nk= 2024-04-13T22:21:12.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3YFyEuTJicKiR+n1OfP+n8nG2nk=', 'Content-Length': '0'} 2024-04-13T22:21:12.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025750446b503f998-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.538 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.538 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.539 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.539 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.539 DEBUG:boto:Method: GET 2024-04-13T22:21:12.539 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.539 DEBUG:boto:Data: 2024-04-13T22:21:12.539 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.539 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.539 DEBUG:boto:Params: {} 2024-04-13T22:21:12.539 DEBUG:boto:Token: None 2024-04-13T22:21:12.540 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009237134472ca00d-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.582 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.583 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.583 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.583 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.583 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.583 DEBUG:boto:Method: GET 2024-04-13T22:21:12.583 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:21:12.584 DEBUG:boto:Data: 2024-04-13T22:21:12.584 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.584 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.584 DEBUG:boto:Params: {} 2024-04-13T22:21:12.584 DEBUG:boto:Token: None 2024-04-13T22:21:12.584 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb7cf2e30277292-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.587 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:21:12.588 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.588 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.588 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.588 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.588 DEBUG:boto:Method: GET 2024-04-13T22:21:12.589 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.589 DEBUG:boto:Data: 2024-04-13T22:21:12.589 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.589 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.589 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.589 DEBUG:boto:Params: {} 2024-04-13T22:21:12.589 DEBUG:boto:Token: None 2024-04-13T22:21:12.589 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI= 2024-04-13T22:21:12.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI=', 'Content-Length': '0'} 2024-04-13T22:21:12.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825008f01fb7b695-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.592 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.592 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.592 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.592 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.592 DEBUG:boto:Method: GET 2024-04-13T22:21:12.592 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.592 DEBUG:boto:Data: 2024-04-13T22:21:12.592 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.593 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.593 DEBUG:boto:Params: {} 2024-04-13T22:21:12.593 DEBUG:boto:Token: None 2024-04-13T22:21:12.593 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.593 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI= 2024-04-13T22:21:12.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI=', 'Content-Length': '0'} 2024-04-13T22:21:12.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031fbb0e43e46c61e-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.595 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.595 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.595 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.595 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.595 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.596 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.596 DEBUG:boto:Data: 2024-04-13T22:21:12.596 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.596 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.596 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.596 DEBUG:boto:Params: {} 2024-04-13T22:21:12.596 DEBUG:boto:Token: None 2024-04-13T22:21:12.596 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuimWlYbQpOil97fYtOCY0HAKSs= 2024-04-13T22:21:12.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuimWlYbQpOil97fYtOCY0HAKSs=', 'Content-Length': '0'} 2024-04-13T22:21:12.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1d0b44512a124f-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.598 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.598 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.598 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.598 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.598 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.599 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.599 DEBUG:boto:Data: 2024-04-13T22:21:12.599 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.599 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.599 DEBUG:boto:Params: {} 2024-04-13T22:21:12.599 DEBUG:boto:Token: None 2024-04-13T22:21:12.599 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.599 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nuimWlYbQpOil97fYtOCY0HAKSs= 2024-04-13T22:21:12.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuimWlYbQpOil97fYtOCY0HAKSs=', 'Content-Length': '0'} 2024-04-13T22:21:12.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015257c204b5e715-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:21:12.601 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.601 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.602 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.602 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.602 DEBUG:boto:Method: GET 2024-04-13T22:21:12.602 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.602 DEBUG:boto:Data: 2024-04-13T22:21:12.602 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.602 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.602 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.602 DEBUG:boto:Params: {} 2024-04-13T22:21:12.602 DEBUG:boto:Token: None 2024-04-13T22:21:12.603 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.603 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI= 2024-04-13T22:21:12.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI=', 'Content-Length': '0'} 2024-04-13T22:21:12.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff9f08d645bca1a-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.605 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.605 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.605 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.605 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.605 DEBUG:boto:Method: GET 2024-04-13T22:21:12.605 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.605 DEBUG:boto:Data: 2024-04-13T22:21:12.605 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.605 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.605 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.605 DEBUG:boto:Params: {} 2024-04-13T22:21:12.606 DEBUG:boto:Token: None 2024-04-13T22:21:12.606 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:21:12.606 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI= 2024-04-13T22:21:12.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v+xRIhCVN0EYLATAatImZ//DziI=', 'Content-Length': '0'} 2024-04-13T22:21:12.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000669e4320a61d4ae-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.650 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.651 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.651 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.651 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.651 DEBUG:boto:Method: GET 2024-04-13T22:21:12.651 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.651 DEBUG:boto:Data: 2024-04-13T22:21:12.651 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.651 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.651 DEBUG:boto:Params: {} 2024-04-13T22:21:12.652 DEBUG:boto:Token: None 2024-04-13T22:21:12.652 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb94107cfd64f96-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.654 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.655 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.655 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.655 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.655 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.655 DEBUG:boto:Method: GET 2024-04-13T22:21:12.655 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.655 DEBUG:boto:Data: 2024-04-13T22:21:12.655 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.656 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.656 DEBUG:boto:Params: {} 2024-04-13T22:21:12.656 DEBUG:boto:Token: None 2024-04-13T22:21:12.656 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f1a8742f0fb936-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.659 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.659 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.659 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.659 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.659 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:21:12.660 DEBUG:boto:Data: 2024-04-13T22:21:12.660 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.660 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.660 DEBUG:boto:Params: {} 2024-04-13T22:21:12.660 DEBUG:boto:Token: None 2024-04-13T22:21:12.660 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51 2024-04-13T22:21:12.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fl1O+4D0jrB9HbJ+e+vUlp22Yk0= 2024-04-13T22:21:12.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fl1O+4D0jrB9HbJ+e+vUlp22Yk0=', 'Content-Length': '0'} 2024-04-13T22:21:12.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a68b5e9d75c0270-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.662 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.662 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.662 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.663 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:21:12.663 DEBUG:boto:Data: 2024-04-13T22:21:12.663 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.663 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.663 DEBUG:boto:Params: {} 2024-04-13T22:21:12.663 DEBUG:boto:Token: None 2024-04-13T22:21:12.663 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51 2024-04-13T22:21:12.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fl1O+4D0jrB9HbJ+e+vUlp22Yk0= 2024-04-13T22:21:12.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fl1O+4D0jrB9HbJ+e+vUlp22Yk0=', 'Content-Length': '0'} 2024-04-13T22:21:12.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e838b85226e133a-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:21:12.665 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.666 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.666 DEBUG:boto:Method: GET 2024-04-13T22:21:12.666 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:21:12.666 DEBUG:boto:Data: 2024-04-13T22:21:12.666 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.666 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.666 DEBUG:boto:Params: {} 2024-04-13T22:21:12.666 DEBUG:boto:Token: None 2024-04-13T22:21:12.666 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51 2024-04-13T22:21:12.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:myG+FxrHJYyEiPhMr3dZPdKWj8w= 2024-04-13T22:21:12.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:myG+FxrHJYyEiPhMr3dZPdKWj8w=', 'Content-Length': '0'} 2024-04-13T22:21:12.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde83dc1bcbff6a8-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.669 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:21:12.669 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:21:12.669 DEBUG:boto:Method: GET 2024-04-13T22:21:12.669 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:21:12.669 DEBUG:boto:Data: 2024-04-13T22:21:12.669 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.669 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.669 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.670 DEBUG:boto:Params: {} 2024-04-13T22:21:12.670 DEBUG:boto:Token: None 2024-04-13T22:21:12.670 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj51 2024-04-13T22:21:12.670 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:myG+FxrHJYyEiPhMr3dZPdKWj8w= 2024-04-13T22:21:12.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:myG+FxrHJYyEiPhMr3dZPdKWj8w=', 'Content-Length': '0'} 2024-04-13T22:21:12.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea214614b4829c1-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.672 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.672 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.672 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.672 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.672 DEBUG:boto:Method: GET 2024-04-13T22:21:12.673 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.673 DEBUG:boto:Data: 2024-04-13T22:21:12.673 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.673 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.673 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.673 DEBUG:boto:Params: {} 2024-04-13T22:21:12.673 DEBUG:boto:Token: None 2024-04-13T22:21:12.673 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.673 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000751f38d0551ade26-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.676 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.676 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.676 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.676 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.676 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.676 DEBUG:boto:Method: GET 2024-04-13T22:21:12.677 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:21:12.677 DEBUG:boto:Data: 2024-04-13T22:21:12.677 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.677 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.677 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.677 DEBUG:boto:Params: {} 2024-04-13T22:21:12.677 DEBUG:boto:Token: None 2024-04-13T22:21:12.677 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009365f5c5c83fbff8-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.680 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:21:12.680 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.680 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.681 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.681 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.681 DEBUG:boto:Method: GET 2024-04-13T22:21:12.681 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.681 DEBUG:boto:Data: 2024-04-13T22:21:12.681 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.681 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.681 DEBUG:boto:Params: {} 2024-04-13T22:21:12.681 DEBUG:boto:Token: None 2024-04-13T22:21:12.682 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0= 2024-04-13T22:21:12.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0=', 'Content-Length': '0'} 2024-04-13T22:21:12.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3ee5c645c568a3-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.730 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.731 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.731 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.731 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.731 DEBUG:boto:Method: GET 2024-04-13T22:21:12.731 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.731 DEBUG:boto:Data: 2024-04-13T22:21:12.731 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.731 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.731 DEBUG:boto:Params: {} 2024-04-13T22:21:12.732 DEBUG:boto:Token: None 2024-04-13T22:21:12.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0= 2024-04-13T22:21:12.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0=', 'Content-Length': '0'} 2024-04-13T22:21:12.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009936fed66e4fd634-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.734 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.734 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.734 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.734 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.735 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.735 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.735 DEBUG:boto:Data: 2024-04-13T22:21:12.735 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.735 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.735 DEBUG:boto:Params: {} 2024-04-13T22:21:12.735 DEBUG:boto:Token: None 2024-04-13T22:21:12.735 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bEPSrTL+sWPuhYYscUs+QJvowYI= 2024-04-13T22:21:12.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bEPSrTL+sWPuhYYscUs+QJvowYI=', 'Content-Length': '0'} 2024-04-13T22:21:12.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050583fff6243975d-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.738 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.738 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.738 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.738 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.738 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.738 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.738 DEBUG:boto:Data: 2024-04-13T22:21:12.738 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.738 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.738 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.739 DEBUG:boto:Params: {} 2024-04-13T22:21:12.739 DEBUG:boto:Token: None 2024-04-13T22:21:12.739 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bEPSrTL+sWPuhYYscUs+QJvowYI= 2024-04-13T22:21:12.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bEPSrTL+sWPuhYYscUs+QJvowYI=', 'Content-Length': '0'} 2024-04-13T22:21:12.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f82cc448a56bceb-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:21:12.741 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.741 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.741 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.741 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.742 DEBUG:boto:Method: GET 2024-04-13T22:21:12.742 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.742 DEBUG:boto:Data: 2024-04-13T22:21:12.742 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.742 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.742 DEBUG:boto:Params: {} 2024-04-13T22:21:12.742 DEBUG:boto:Token: None 2024-04-13T22:21:12.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0= 2024-04-13T22:21:12.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0=', 'Content-Length': '0'} 2024-04-13T22:21:12.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006478998ec171fdfd-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.745 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.745 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.746 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.746 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.746 DEBUG:boto:Method: GET 2024-04-13T22:21:12.746 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.746 DEBUG:boto:Data: 2024-04-13T22:21:12.746 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.746 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.746 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.746 DEBUG:boto:Params: {} 2024-04-13T22:21:12.746 DEBUG:boto:Token: None 2024-04-13T22:21:12.747 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:21:12.747 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0= 2024-04-13T22:21:12.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1sZbKmBbU1wchlgK5YWX5N9zD0=', 'Content-Length': '0'} 2024-04-13T22:21:12.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6f9e72b45b1553-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.749 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.749 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.749 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.749 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.749 DEBUG:boto:Method: GET 2024-04-13T22:21:12.749 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.749 DEBUG:boto:Data: 2024-04-13T22:21:12.750 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.750 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.750 DEBUG:boto:Params: {} 2024-04-13T22:21:12.750 DEBUG:boto:Token: None 2024-04-13T22:21:12.750 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.750 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2553eb70bbf8031-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.753 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.753 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.753 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.753 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.753 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.754 DEBUG:boto:Method: GET 2024-04-13T22:21:12.754 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.754 DEBUG:boto:Data: 2024-04-13T22:21:12.754 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.754 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.754 DEBUG:boto:Params: {} 2024-04-13T22:21:12.754 DEBUG:boto:Token: None 2024-04-13T22:21:12.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad95a98ac307a0c6-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.757 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.757 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.758 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.758 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.758 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:21:12.758 DEBUG:boto:Data: 2024-04-13T22:21:12.758 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.758 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.758 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.758 DEBUG:boto:Params: {} 2024-04-13T22:21:12.758 DEBUG:boto:Token: None 2024-04-13T22:21:12.759 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55 2024-04-13T22:21:12.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kraQ1DN7s6Q+pADsYznryLQHEUc= 2024-04-13T22:21:12.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kraQ1DN7s6Q+pADsYznryLQHEUc=', 'Content-Length': '0'} 2024-04-13T22:21:12.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b324ad840d677919-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.761 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.761 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.761 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.761 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:21:12.761 DEBUG:boto:Data: 2024-04-13T22:21:12.761 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.761 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.761 DEBUG:boto:Params: {} 2024-04-13T22:21:12.761 DEBUG:boto:Token: None 2024-04-13T22:21:12.762 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55 2024-04-13T22:21:12.762 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kraQ1DN7s6Q+pADsYznryLQHEUc= 2024-04-13T22:21:12.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kraQ1DN7s6Q+pADsYznryLQHEUc=', 'Content-Length': '0'} 2024-04-13T22:21:12.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2e4eb5adac5dbd-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:21:12.764 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.764 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.764 DEBUG:boto:Method: GET 2024-04-13T22:21:12.764 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:21:12.764 DEBUG:boto:Data: 2024-04-13T22:21:12.764 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.765 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.765 DEBUG:boto:Params: {} 2024-04-13T22:21:12.765 DEBUG:boto:Token: None 2024-04-13T22:21:12.765 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55 2024-04-13T22:21:12.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7uxE3m9qnV6XS8+h0tTSLBCLeCc= 2024-04-13T22:21:12.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7uxE3m9qnV6XS8+h0tTSLBCLeCc=', 'Content-Length': '0'} 2024-04-13T22:21:12.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63104eeffb52647-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.767 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:21:12.767 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:21:12.767 DEBUG:boto:Method: GET 2024-04-13T22:21:12.767 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:21:12.767 DEBUG:boto:Data: 2024-04-13T22:21:12.768 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.768 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.768 DEBUG:boto:Params: {} 2024-04-13T22:21:12.768 DEBUG:boto:Token: None 2024-04-13T22:21:12.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj55 2024-04-13T22:21:12.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7uxE3m9qnV6XS8+h0tTSLBCLeCc= 2024-04-13T22:21:12.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7uxE3m9qnV6XS8+h0tTSLBCLeCc=', 'Content-Length': '0'} 2024-04-13T22:21:12.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b83fd76fedb8745-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.770 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.770 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.770 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.770 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.771 DEBUG:boto:Method: GET 2024-04-13T22:21:12.771 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.771 DEBUG:boto:Data: 2024-04-13T22:21:12.771 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.771 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.771 DEBUG:boto:Params: {} 2024-04-13T22:21:12.771 DEBUG:boto:Token: None 2024-04-13T22:21:12.771 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.771 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002027a7a59640ca9e-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.774 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.774 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.774 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.774 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.774 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.774 DEBUG:boto:Method: GET 2024-04-13T22:21:12.775 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:21:12.775 DEBUG:boto:Data: 2024-04-13T22:21:12.775 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.775 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.775 DEBUG:boto:Params: {} 2024-04-13T22:21:12.775 DEBUG:boto:Token: None 2024-04-13T22:21:12.775 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.775 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c0b94ebc8bc782-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.778 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:21:12.778 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.779 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.779 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.779 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.779 DEBUG:boto:Method: GET 2024-04-13T22:21:12.779 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.779 DEBUG:boto:Data: 2024-04-13T22:21:12.779 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.779 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.779 DEBUG:boto:Params: {} 2024-04-13T22:21:12.779 DEBUG:boto:Token: None 2024-04-13T22:21:12.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds= 2024-04-13T22:21:12.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds=', 'Content-Length': '0'} 2024-04-13T22:21:12.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a45abbb3f05f9e4-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.782 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.782 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.782 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.782 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.782 DEBUG:boto:Method: GET 2024-04-13T22:21:12.782 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.782 DEBUG:boto:Data: 2024-04-13T22:21:12.783 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.783 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.783 DEBUG:boto:Params: {} 2024-04-13T22:21:12.783 DEBUG:boto:Token: None 2024-04-13T22:21:12.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds= 2024-04-13T22:21:12.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds=', 'Content-Length': '0'} 2024-04-13T22:21:12.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a717d5bb7f5b3f6-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.785 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.785 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.785 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.786 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.786 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.786 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.786 DEBUG:boto:Data: 2024-04-13T22:21:12.786 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.786 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.786 DEBUG:boto:Params: {} 2024-04-13T22:21:12.786 DEBUG:boto:Token: None 2024-04-13T22:21:12.787 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.787 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xj65wtwmY2wKqcodF7uCRZCvCPM= 2024-04-13T22:21:12.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xj65wtwmY2wKqcodF7uCRZCvCPM=', 'Content-Length': '0'} 2024-04-13T22:21:12.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037024bb61368c054-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.789 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.789 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.789 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.789 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.789 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.789 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.789 DEBUG:boto:Data: 2024-04-13T22:21:12.790 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.790 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.790 DEBUG:boto:Params: {} 2024-04-13T22:21:12.790 DEBUG:boto:Token: None 2024-04-13T22:21:12.790 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.790 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xj65wtwmY2wKqcodF7uCRZCvCPM= 2024-04-13T22:21:12.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xj65wtwmY2wKqcodF7uCRZCvCPM=', 'Content-Length': '0'} 2024-04-13T22:21:12.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1543c65fd6861c-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:21:12.792 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.792 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.792 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.792 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.792 DEBUG:boto:Method: GET 2024-04-13T22:21:12.793 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.793 DEBUG:boto:Data: 2024-04-13T22:21:12.793 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.793 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.793 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.793 DEBUG:boto:Params: {} 2024-04-13T22:21:12.793 DEBUG:boto:Token: None 2024-04-13T22:21:12.793 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.793 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds= 2024-04-13T22:21:12.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds=', 'Content-Length': '0'} 2024-04-13T22:21:12.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a44d6acf1c143a9-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.795 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.795 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.795 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.796 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.796 DEBUG:boto:Method: GET 2024-04-13T22:21:12.796 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.796 DEBUG:boto:Data: 2024-04-13T22:21:12.796 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.796 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.796 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.796 DEBUG:boto:Params: {} 2024-04-13T22:21:12.796 DEBUG:boto:Token: None 2024-04-13T22:21:12.796 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:21:12.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds= 2024-04-13T22:21:12.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n9nuUa0l7VXZzeqJ/z5oCRGehds=', 'Content-Length': '0'} 2024-04-13T22:21:12.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc5ff430cdbc49d-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.798 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.798 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.799 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.799 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.799 DEBUG:boto:Method: GET 2024-04-13T22:21:12.799 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.799 DEBUG:boto:Data: 2024-04-13T22:21:12.799 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.799 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.799 DEBUG:boto:Params: {} 2024-04-13T22:21:12.799 DEBUG:boto:Token: None 2024-04-13T22:21:12.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009db3f0c8aa3eadf-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.842 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.843 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.843 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.843 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.843 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.843 DEBUG:boto:Method: GET 2024-04-13T22:21:12.843 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.843 DEBUG:boto:Data: 2024-04-13T22:21:12.843 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.843 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.844 DEBUG:boto:Params: {} 2024-04-13T22:21:12.844 DEBUG:boto:Token: None 2024-04-13T22:21:12.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678bbbecef904bf6-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.846 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.847 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.847 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.847 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.847 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:21:12.847 DEBUG:boto:Data: 2024-04-13T22:21:12.847 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.847 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.847 DEBUG:boto:Params: {} 2024-04-13T22:21:12.848 DEBUG:boto:Token: None 2024-04-13T22:21:12.848 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68 2024-04-13T22:21:12.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LjLA3+3G7RhOZF/y91tv5HXz8fE= 2024-04-13T22:21:12.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LjLA3+3G7RhOZF/y91tv5HXz8fE=', 'Content-Length': '0'} 2024-04-13T22:21:12.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38b4e97ef7fe96a-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.850 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.850 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.850 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.850 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:21:12.850 DEBUG:boto:Data: 2024-04-13T22:21:12.850 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.850 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.850 DEBUG:boto:Params: {} 2024-04-13T22:21:12.851 DEBUG:boto:Token: None 2024-04-13T22:21:12.851 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68 2024-04-13T22:21:12.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LjLA3+3G7RhOZF/y91tv5HXz8fE= 2024-04-13T22:21:12.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LjLA3+3G7RhOZF/y91tv5HXz8fE=', 'Content-Length': '0'} 2024-04-13T22:21:12.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3919e06cbf8bcf0-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:21:12.853 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.853 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.853 DEBUG:boto:Method: GET 2024-04-13T22:21:12.853 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:21:12.853 DEBUG:boto:Data: 2024-04-13T22:21:12.853 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.853 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.854 DEBUG:boto:Params: {} 2024-04-13T22:21:12.854 DEBUG:boto:Token: None 2024-04-13T22:21:12.854 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68 2024-04-13T22:21:12.854 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6mKCqMlqd7TpxLZH0Wvh0NMBsMU= 2024-04-13T22:21:12.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6mKCqMlqd7TpxLZH0Wvh0NMBsMU=', 'Content-Length': '0'} 2024-04-13T22:21:12.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686f8cd945537517-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.856 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:21:12.856 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:21:12.856 DEBUG:boto:Method: GET 2024-04-13T22:21:12.856 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:21:12.856 DEBUG:boto:Data: 2024-04-13T22:21:12.856 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.856 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.857 DEBUG:boto:Params: {} 2024-04-13T22:21:12.857 DEBUG:boto:Token: None 2024-04-13T22:21:12.857 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj68 2024-04-13T22:21:12.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6mKCqMlqd7TpxLZH0Wvh0NMBsMU= 2024-04-13T22:21:12.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6mKCqMlqd7TpxLZH0Wvh0NMBsMU=', 'Content-Length': '0'} 2024-04-13T22:21:12.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ddaa79ea5311bd-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.902 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.903 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.903 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.903 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.903 DEBUG:boto:Method: GET 2024-04-13T22:21:12.903 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.903 DEBUG:boto:Data: 2024-04-13T22:21:12.903 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.903 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.903 DEBUG:boto:Params: {} 2024-04-13T22:21:12.904 DEBUG:boto:Token: None 2024-04-13T22:21:12.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c4615452952896-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.906 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.907 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.907 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.907 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.907 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.907 DEBUG:boto:Method: GET 2024-04-13T22:21:12.907 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:21:12.907 DEBUG:boto:Data: 2024-04-13T22:21:12.907 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.908 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.908 DEBUG:boto:Params: {} 2024-04-13T22:21:12.908 DEBUG:boto:Token: None 2024-04-13T22:21:12.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b372c7df9b8aa2b-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.911 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:21:12.911 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.911 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.912 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.912 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.912 DEBUG:boto:Method: GET 2024-04-13T22:21:12.912 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.912 DEBUG:boto:Data: 2024-04-13T22:21:12.912 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.912 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.912 DEBUG:boto:Params: {} 2024-04-13T22:21:12.913 DEBUG:boto:Token: None 2024-04-13T22:21:12.913 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84= 2024-04-13T22:21:12.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84=', 'Content-Length': '0'} 2024-04-13T22:21:12.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010481476974ee01a-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.915 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.915 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.915 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.915 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.915 DEBUG:boto:Method: GET 2024-04-13T22:21:12.915 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.916 DEBUG:boto:Data: 2024-04-13T22:21:12.916 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.916 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.916 DEBUG:boto:Params: {} 2024-04-13T22:21:12.916 DEBUG:boto:Token: None 2024-04-13T22:21:12.916 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84= 2024-04-13T22:21:12.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84=', 'Content-Length': '0'} 2024-04-13T22:21:12.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ada38524de897ab-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.918 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.918 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.918 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.919 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.919 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.919 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.919 DEBUG:boto:Data: 2024-04-13T22:21:12.919 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.919 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.919 DEBUG:boto:Params: {} 2024-04-13T22:21:12.919 DEBUG:boto:Token: None 2024-04-13T22:21:12.919 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:52rddSdlZ3KWaGPhfsqhAbSpbxE= 2024-04-13T22:21:12.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:52rddSdlZ3KWaGPhfsqhAbSpbxE=', 'Content-Length': '0'} 2024-04-13T22:21:12.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690fadc064736f27-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.921 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.921 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.922 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.922 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.922 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.922 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.922 DEBUG:boto:Data: 2024-04-13T22:21:12.922 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.925 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.925 DEBUG:boto:Params: {} 2024-04-13T22:21:12.925 DEBUG:boto:Token: None 2024-04-13T22:21:12.925 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:52rddSdlZ3KWaGPhfsqhAbSpbxE= 2024-04-13T22:21:12.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:52rddSdlZ3KWaGPhfsqhAbSpbxE=', 'Content-Length': '0'} 2024-04-13T22:21:12.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7416679905b287e-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:21:12.928 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.928 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.928 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.929 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.929 DEBUG:boto:Method: GET 2024-04-13T22:21:12.929 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.929 DEBUG:boto:Data: 2024-04-13T22:21:12.929 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.929 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.929 DEBUG:boto:Params: {} 2024-04-13T22:21:12.929 DEBUG:boto:Token: None 2024-04-13T22:21:12.929 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84= 2024-04-13T22:21:12.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84=', 'Content-Length': '0'} 2024-04-13T22:21:12.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035305ea4206a0142-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.978 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.979 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.979 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.979 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.979 DEBUG:boto:Method: GET 2024-04-13T22:21:12.979 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.979 DEBUG:boto:Data: 2024-04-13T22:21:12.979 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.979 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.979 DEBUG:boto:Params: {} 2024-04-13T22:21:12.979 DEBUG:boto:Token: None 2024-04-13T22:21:12.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:21:12.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84= 2024-04-13T22:21:12.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Yx6Ny9KRCB02Wu8mJr/h8C/6T84=', 'Content-Length': '0'} 2024-04-13T22:21:12.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccaf6afcee77c693-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.982 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.982 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.982 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:12.982 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:12.982 DEBUG:boto:Method: GET 2024-04-13T22:21:12.983 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:21:12.983 DEBUG:boto:Data: 2024-04-13T22:21:12.983 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.983 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.983 DEBUG:boto:Params: {} 2024-04-13T22:21:12.983 DEBUG:boto:Token: None 2024-04-13T22:21:12.983 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.983 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008183d964e85d94f0-00661b0558-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.986 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.986 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:12.986 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:12.986 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:12.986 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:12.986 DEBUG:boto:Method: GET 2024-04-13T22:21:12.987 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:21:12.987 DEBUG:boto:Data: 2024-04-13T22:21:12.987 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.987 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.987 DEBUG:boto:Params: {} 2024-04-13T22:21:12.987 DEBUG:boto:Token: None 2024-04-13T22:21:12.987 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/ 2024-04-13T22:21:12.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg= 2024-04-13T22:21:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zuuVZ48x0EVWzn6zT5G4nqe+yqg=', 'Content-Length': '0'} 2024-04-13T22:21:12.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad64686f8079fdc6-00661b0558-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.990 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:12.990 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.990 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.990 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.991 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:21:12.991 DEBUG:boto:Data: 2024-04-13T22:21:12.991 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.991 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.991 DEBUG:boto:Params: {} 2024-04-13T22:21:12.991 DEBUG:boto:Token: None 2024-04-13T22:21:12.991 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85 2024-04-13T22:21:12.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LeAuw8RgJzu2VDgobNtfvsyEUWM= 2024-04-13T22:21:12.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LeAuw8RgJzu2VDgobNtfvsyEUWM=', 'Content-Length': '0'} 2024-04-13T22:21:12.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060311e6a70c4b38c-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.993 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.993 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.993 DEBUG:boto:Method: HEAD 2024-04-13T22:21:12.993 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:21:12.993 DEBUG:boto:Data: 2024-04-13T22:21:12.993 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.993 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.993 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.993 DEBUG:boto:Params: {} 2024-04-13T22:21:12.994 DEBUG:boto:Token: None 2024-04-13T22:21:12.994 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85 2024-04-13T22:21:12.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LeAuw8RgJzu2VDgobNtfvsyEUWM= 2024-04-13T22:21:12.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LeAuw8RgJzu2VDgobNtfvsyEUWM=', 'Content-Length': '0'} 2024-04-13T22:21:12.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046919df2e9ec5039-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:21:12.996 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.996 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.996 DEBUG:boto:Method: GET 2024-04-13T22:21:12.996 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:21:12.996 DEBUG:boto:Data: 2024-04-13T22:21:12.996 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.996 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:12.997 DEBUG:boto:Port: 8001 2024-04-13T22:21:12.997 DEBUG:boto:Params: {} 2024-04-13T22:21:12.997 DEBUG:boto:Token: None 2024-04-13T22:21:12.997 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:12 GMT /vwnfns-61/obj85 2024-04-13T22:21:12.997 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IbEGs6vVrz1EGzh7kWBOPxj5Gz0= 2024-04-13T22:21:12.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:12 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IbEGs6vVrz1EGzh7kWBOPxj5Gz0=', 'Content-Length': '0'} 2024-04-13T22:21:12.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd63962e74a6a884-00661b0558-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:12.999 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:21:12.999 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:21:12.999 DEBUG:boto:Method: GET 2024-04-13T22:21:12.999 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:21:12.999 DEBUG:boto:Data: 2024-04-13T22:21:12.999 DEBUG:boto:Headers: {} 2024-04-13T22:21:12.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:12.999 DEBUG:boto:Port: 8000 2024-04-13T22:21:12.999 DEBUG:boto:Params: {} 2024-04-13T22:21:13.000 DEBUG:boto:Token: None 2024-04-13T22:21:13.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/obj85 2024-04-13T22:21:13.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fff79yIv9bMDN3ETgtm1TgcWre4= 2024-04-13T22:21:13.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fff79yIv9bMDN3ETgtm1TgcWre4=', 'Content-Length': '0'} 2024-04-13T22:21:13.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c97e76bdf5dac8-00661b0558-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:21:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.002 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.002 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.002 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:13.002 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:13.002 DEBUG:boto:Method: GET 2024-04-13T22:21:13.002 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:21:13.002 DEBUG:boto:Data: 2024-04-13T22:21:13.003 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:21:13.003 DEBUG:boto:Port: 8001 2024-04-13T22:21:13.003 DEBUG:boto:Params: {} 2024-04-13T22:21:13.003 DEBUG:boto:Token: None 2024-04-13T22:21:13.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s3sVmTmwucP6rhXaJVyPGiUrV/8= 2024-04-13T22:21:13.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s3sVmTmwucP6rhXaJVyPGiUrV/8=', 'Content-Length': '0'} 2024-04-13T22:21:13.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b56e4d4fe375106a-00661b0559-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.005 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:13.006 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.006 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.006 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:13.006 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:21:13.006 DEBUG:boto:Method: GET 2024-04-13T22:21:13.006 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:21:13.006 DEBUG:boto:Data: 2024-04-13T22:21:13.006 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.006 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.007 DEBUG:boto:Params: {} 2024-04-13T22:21:13.007 DEBUG:boto:Token: None 2024-04-13T22:21:13.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s3sVmTmwucP6rhXaJVyPGiUrV/8= 2024-04-13T22:21:13.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s3sVmTmwucP6rhXaJVyPGiUrV/8=', 'Content-Length': '0'} 2024-04-13T22:21:13.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f02551c6863367-00661b0559-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.010 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:21:13.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:21:13.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:21:13.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:21:13.011 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:21:13.062 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:21:13.062 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:21:13.104 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:21:13.105 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-61 2024-04-13T22:21:13.110 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 10 2024-04-13T22:21:13.155 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.155 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.155 DEBUG:boto:Method: HEAD 2024-04-13T22:21:13.155 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:13.155 DEBUG:boto:Data: 2024-04-13T22:21:13.155 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.156 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.156 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.156 DEBUG:boto:Params: {} 2024-04-13T22:21:13.156 DEBUG:boto:Token: None 2024-04-13T22:21:13.156 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw= 2024-04-13T22:21:13.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw=', 'Content-Length': '0'} 2024-04-13T22:21:13.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049612616ec941983-00661b0559-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.160 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:21:13.160 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:21:13.160 DEBUG:boto:Method: PUT 2024-04-13T22:21:13.160 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:21:13.160 DEBUG:boto:Data: 2024-04-13T22:21:13.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:21:13.160 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.160 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.160 DEBUG:boto:Params: {} 2024-04-13T22:21:13.160 DEBUG:boto:Token: None 2024-04-13T22:21:13.161 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/obj19 2024-04-13T22:21:13.161 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cWoBtQI0xVSdvBIBWmFSbaYf2SY= 2024-04-13T22:21:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cWoBtQI0xVSdvBIBWmFSbaYf2SY='} 2024-04-13T22:21:13.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b9d6ccf6a2ff00b-00661b0559-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.207 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.207 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.207 DEBUG:boto:Method: HEAD 2024-04-13T22:21:13.207 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:13.207 DEBUG:boto:Data: 2024-04-13T22:21:13.207 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.207 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.208 DEBUG:boto:Params: {} 2024-04-13T22:21:13.208 DEBUG:boto:Token: None 2024-04-13T22:21:13.208 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw= 2024-04-13T22:21:13.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw=', 'Content-Length': '0'} 2024-04-13T22:21:13.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000478923b8daed0ccd-00661b0559-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.211 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:21:13.211 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:21:13.211 DEBUG:boto:Method: PUT 2024-04-13T22:21:13.211 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:21:13.211 DEBUG:boto:Data: 2024-04-13T22:21:13.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:21:13.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.212 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.212 DEBUG:boto:Params: {} 2024-04-13T22:21:13.212 DEBUG:boto:Token: None 2024-04-13T22:21:13.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/obj38 2024-04-13T22:21:13.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rJqB/Q3yyDxHONEh+QUB4XwNPRw= 2024-04-13T22:21:13.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rJqB/Q3yyDxHONEh+QUB4XwNPRw='} 2024-04-13T22:21:13.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093f0effb0e7532d8-00661b0559-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.216 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.216 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.216 DEBUG:boto:Method: HEAD 2024-04-13T22:21:13.216 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:13.216 DEBUG:boto:Data: 2024-04-13T22:21:13.216 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.217 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.217 DEBUG:boto:Params: {} 2024-04-13T22:21:13.217 DEBUG:boto:Token: None 2024-04-13T22:21:13.217 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw= 2024-04-13T22:21:13.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw=', 'Content-Length': '0'} 2024-04-13T22:21:13.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001659753abcd96ca1-00661b0559-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.220 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:21:13.220 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:21:13.220 DEBUG:boto:Method: PUT 2024-04-13T22:21:13.220 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:21:13.220 DEBUG:boto:Data: 2024-04-13T22:21:13.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:21:13.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.220 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.220 DEBUG:boto:Params: {} 2024-04-13T22:21:13.220 DEBUG:boto:Token: None 2024-04-13T22:21:13.221 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/obj57 2024-04-13T22:21:13.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rMsQtncs5vntRc+Gh4vJ9wotZvo= 2024-04-13T22:21:13.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rMsQtncs5vntRc+Gh4vJ9wotZvo='} 2024-04-13T22:21:13.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abb3a8b711472735-00661b0559-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.267 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.267 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.267 DEBUG:boto:Method: HEAD 2024-04-13T22:21:13.267 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:13.267 DEBUG:boto:Data: 2024-04-13T22:21:13.267 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.267 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.267 DEBUG:boto:Params: {} 2024-04-13T22:21:13.268 DEBUG:boto:Token: None 2024-04-13T22:21:13.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw= 2024-04-13T22:21:13.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw=', 'Content-Length': '0'} 2024-04-13T22:21:13.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c04020f26368b1a3-00661b0559-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.271 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:21:13.271 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:21:13.271 DEBUG:boto:Method: PUT 2024-04-13T22:21:13.271 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:21:13.271 DEBUG:boto:Data: 2024-04-13T22:21:13.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:21:13.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.271 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.271 DEBUG:boto:Params: {} 2024-04-13T22:21:13.272 DEBUG:boto:Token: None 2024-04-13T22:21:13.272 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/obj76 2024-04-13T22:21:13.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z1AekEmUrnZmSun0lH5IowRogrw= 2024-04-13T22:21:13.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z1AekEmUrnZmSun0lH5IowRogrw='} 2024-04-13T22:21:13.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025b458ae2f835644-00661b0559-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.277 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:21:13.277 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:21:13.277 DEBUG:boto:Method: HEAD 2024-04-13T22:21:13.277 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:21:13.277 DEBUG:boto:Data: 2024-04-13T22:21:13.278 DEBUG:boto:Headers: {} 2024-04-13T22:21:13.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.278 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.278 DEBUG:boto:Params: {} 2024-04-13T22:21:13.278 DEBUG:boto:Token: None 2024-04-13T22:21:13.278 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/ 2024-04-13T22:21:13.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw= 2024-04-13T22:21:13.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3KEQeKQ6k0Ap3DICrCtiPCh41Tw=', 'Content-Length': '0'} 2024-04-13T22:21:13.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b67bad2a0976ec2-00661b0559-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.281 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:21:13.281 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:21:13.281 DEBUG:boto:Method: PUT 2024-04-13T22:21:13.282 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:21:13.282 DEBUG:boto:Data: 2024-04-13T22:21:13.282 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:21:13.282 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:21:13.282 DEBUG:boto:Port: 8000 2024-04-13T22:21:13.282 DEBUG:boto:Params: {} 2024-04-13T22:21:13.282 DEBUG:boto:Token: None 2024-04-13T22:21:13.282 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:21:13 GMT /vwnfns-61/obj95 2024-04-13T22:21:13.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/X/23oJMX2nc8FI9PG4SnC9ahUo= 2024-04-13T22:21:13.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:21:13 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/X/23oJMX2nc8FI9PG4SnC9ahUo='} 2024-04-13T22:21:13.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7371c7e8986b75e-00661b0559-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:21:13.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:21:13.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:21:13.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:21:13.327 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:21:13.385 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:21:13.386 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:21:13.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:13.418+0000 7fcd93b4f780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T22:21:18.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:18.420+0000 7fcd93b4f780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T22:21:23.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:23.420+0000 7fcd93b4f780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T22:21:28.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:28.422+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:28.425 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-13T22:21:28.425 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:33.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:33.423+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:33.427 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-13T22:21:33.427 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:38.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:38.423+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:38.427 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:38.427 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:43.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:43.424+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:43.429 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:43.429 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:48.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:48.426+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:48.431 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:48.431 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:53.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:53.428+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:53.432 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:53.432 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:58.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:21:58.430+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:21:58.433 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:21:58.434 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:03.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:03.432+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:03.435 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:03.435 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:08.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:08.433+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:08.436 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:08.437 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:13.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:13.434+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:13.438 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:13.438 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:18.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:18.435+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:18.440 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:18.440 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:23.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:23.437+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:23.441 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:23.442 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:28.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:28.439+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:28.443 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:28.443 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:33.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:33.440+0000 7fcd93b4f780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:33.445 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:33.445 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:38.443+0000 7fcd93b4f780 1 bucket sync caught up with source: 2024-04-13T22:22:38.446 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:38.446 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.94.6, 00000000004.68.6, , , , , , , , , , , ] 2024-04-13T22:22:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:38.443+0000 7fcd93b4f780 0 bucket checkpoint complete 2024-04-13T22:22:38.461 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:22:38.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:22:38.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:22:38.462 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:22:38.529 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:22:38.529 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:22:38.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:38.563+0000 7f10fe023780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:38.566 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, , 00000000001.1138.6, , , 00000000002.624.6, 00000000004.1897.6, 00000000003.1730.6] 2024-04-13T22:22:38.566 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000001.1202.6, 00000000001.1138.6, , , 00000000002.624.6, 00000000004.1897.6, 00000000003.1730.6] 2024-04-13T22:22:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:43.564+0000 7f10fe023780 1 waiting for incremental sync to catch up: 2024-04-13T22:22:43.568 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, , 00000000001.1138.6, , , 00000000002.624.6, 00000000004.1897.6, 00000000003.1730.6] 2024-04-13T22:22:43.568 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000001.1202.6, 00000000001.1138.6, , , 00000000002.624.6, 00000000004.1897.6, 00000000003.1730.6] 2024-04-13T22:22:48.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:48.565+0000 7f10fe023780 1 bucket sync caught up with source: 2024-04-13T22:22:48.569 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000001.1202.6, 00000000001.1138.6, , , 00000000002.624.6, 00000000004.1897.6, 00000000003.1730.6] 2024-04-13T22:22:48.569 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000001.1202.6, 00000000001.1138.6, , , 00000000002.624.6, 00000000004.1897.6, 00000000003.1730.6] 2024-04-13T22:22:48.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:48.565+0000 7f10fe023780 0 bucket checkpoint complete 2024-04-13T22:22:48.585 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:22:48.585 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.585 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.586 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.586 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:48.586 DEBUG:boto:Data: 2024-04-13T22:22:48.586 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.586 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.586 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.586 DEBUG:boto:Params: {} 2024-04-13T22:22:48.587 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:22:48.587 DEBUG:boto:Token: None 2024-04-13T22:22:48.587 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sif5GuQZYo7VuWt+P9MBlNY/+6w= 2024-04-13T22:22:48.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sif5GuQZYo7VuWt+P9MBlNY/+6w=', 'Content-Length': '0'} 2024-04-13T22:22:48.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e9b40ca2b096f9d-00661b05b8-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.592 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.592 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.593 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.593 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:48.593 DEBUG:boto:Data: 2024-04-13T22:22:48.593 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.593 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.593 DEBUG:boto:Params: {} 2024-04-13T22:22:48.593 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:22:48.593 DEBUG:boto:Token: None 2024-04-13T22:22:48.594 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Sif5GuQZYo7VuWt+P9MBlNY/+6w= 2024-04-13T22:22:48.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sif5GuQZYo7VuWt+P9MBlNY/+6w=', 'Content-Length': '0'} 2024-04-13T22:22:48.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ca22d8fe4328cb4-00661b05b8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.598 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:22:48.598 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.598 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.598 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.599 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.599 DEBUG:boto:Method: GET 2024-04-13T22:22:48.599 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.599 DEBUG:boto:Data: 2024-04-13T22:22:48.599 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.599 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.599 DEBUG:boto:Params: {} 2024-04-13T22:22:48.599 DEBUG:boto:Token: None 2024-04-13T22:22:48.599 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/?versions 2024-04-13T22:22:48.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ= 2024-04-13T22:22:48.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3e7272bea06a76-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.602 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:22:48.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:22:48.605 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:22:48.605 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.605 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.605 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.606 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.606 DEBUG:boto:Method: GET 2024-04-13T22:22:48.606 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.606 DEBUG:boto:Data: 2024-04-13T22:22:48.606 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.606 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.606 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.606 DEBUG:boto:Params: {} 2024-04-13T22:22:48.606 DEBUG:boto:Token: None 2024-04-13T22:22:48.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/?versions 2024-04-13T22:22:48.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ= 2024-04-13T22:22:48.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4cc62b19998c692-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.609 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:22:48.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:22:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:22:48.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:22:48.612 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.612 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.612 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.612 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.612 DEBUG:boto:Method: GET 2024-04-13T22:22:48.612 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.612 DEBUG:boto:Data: 2024-04-13T22:22:48.612 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.612 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.612 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.613 DEBUG:boto:Params: {} 2024-04-13T22:22:48.613 DEBUG:boto:Token: None 2024-04-13T22:22:48.613 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/?versions 2024-04-13T22:22:48.613 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ= 2024-04-13T22:22:48.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005d69f368383b43-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.615 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.616 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.616 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.616 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.616 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.616 DEBUG:boto:Method: GET 2024-04-13T22:22:48.617 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:22:48.617 DEBUG:boto:Data: 2024-04-13T22:22:48.617 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.617 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.617 DEBUG:boto:Params: {} 2024-04-13T22:22:48.617 DEBUG:boto:Token: None 2024-04-13T22:22:48.617 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/?versions 2024-04-13T22:22:48.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ= 2024-04-13T22:22:48.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rZ976erW2wZSyMA7R8PuI36OhAQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005779b423105d5b70-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.620 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:22:48.621 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.621 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.622 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.622 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.622 DEBUG:boto:Method: GET 2024-04-13T22:22:48.622 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.622 DEBUG:boto:Data: 2024-04-13T22:22:48.622 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.622 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.622 DEBUG:boto:Params: {} 2024-04-13T22:22:48.622 DEBUG:boto:Token: None 2024-04-13T22:22:48.623 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc= 2024-04-13T22:22:48.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc=', 'Content-Length': '0'} 2024-04-13T22:22:48.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001a086af9d53008c-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.625 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.625 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.625 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.625 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.625 DEBUG:boto:Method: GET 2024-04-13T22:22:48.625 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.625 DEBUG:boto:Data: 2024-04-13T22:22:48.625 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.625 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.625 DEBUG:boto:Params: {} 2024-04-13T22:22:48.626 DEBUG:boto:Token: None 2024-04-13T22:22:48.626 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.626 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc= 2024-04-13T22:22:48.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc=', 'Content-Length': '0'} 2024-04-13T22:22:48.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57dcaddd52f0586-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.628 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.628 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.628 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.628 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.628 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.629 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.629 DEBUG:boto:Data: 2024-04-13T22:22:48.629 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.629 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.629 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.629 DEBUG:boto:Params: {} 2024-04-13T22:22:48.629 DEBUG:boto:Token: None 2024-04-13T22:22:48.629 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aMR0IYqkXfcxJ3OQ0IEjZhjI2Og= 2024-04-13T22:22:48.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aMR0IYqkXfcxJ3OQ0IEjZhjI2Og=', 'Content-Length': '0'} 2024-04-13T22:22:48.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562fd157e1f3ea70-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.631 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.631 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.631 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.632 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.632 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.632 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.632 DEBUG:boto:Data: 2024-04-13T22:22:48.632 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.632 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.632 DEBUG:boto:Params: {} 2024-04-13T22:22:48.632 DEBUG:boto:Token: None 2024-04-13T22:22:48.632 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aMR0IYqkXfcxJ3OQ0IEjZhjI2Og= 2024-04-13T22:22:48.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aMR0IYqkXfcxJ3OQ0IEjZhjI2Og=', 'Content-Length': '0'} 2024-04-13T22:22:48.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb934234f3760618-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:22:48.634 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.635 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.635 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.635 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.635 DEBUG:boto:Method: GET 2024-04-13T22:22:48.635 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.635 DEBUG:boto:Data: 2024-04-13T22:22:48.635 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.635 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.635 DEBUG:boto:Params: {} 2024-04-13T22:22:48.635 DEBUG:boto:Token: None 2024-04-13T22:22:48.636 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc= 2024-04-13T22:22:48.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc=', 'Content-Length': '0'} 2024-04-13T22:22:48.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3f6623f199a8a8-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.638 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.638 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.638 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.638 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.638 DEBUG:boto:Method: GET 2024-04-13T22:22:48.638 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.638 DEBUG:boto:Data: 2024-04-13T22:22:48.638 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.639 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.639 DEBUG:boto:Params: {} 2024-04-13T22:22:48.639 DEBUG:boto:Token: None 2024-04-13T22:22:48.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:22:48.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc= 2024-04-13T22:22:48.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8XSlSAyltykA8SJm8fu/2HmCzgc=', 'Content-Length': '0'} 2024-04-13T22:22:48.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a655755bcfc476c0-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.641 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.641 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.641 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.641 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.641 DEBUG:boto:Method: GET 2024-04-13T22:22:48.641 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.641 DEBUG:boto:Data: 2024-04-13T22:22:48.641 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.642 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.642 DEBUG:boto:Params: {} 2024-04-13T22:22:48.642 DEBUG:boto:Token: None 2024-04-13T22:22:48.642 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e3af7c9c9da5bd3-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.645 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.645 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.645 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.645 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.645 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.645 DEBUG:boto:Method: GET 2024-04-13T22:22:48.645 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.646 DEBUG:boto:Data: 2024-04-13T22:22:48.646 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.646 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.646 DEBUG:boto:Params: {} 2024-04-13T22:22:48.646 DEBUG:boto:Token: None 2024-04-13T22:22:48.646 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.646 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272f2c9ef778ee73-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.650 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.650 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.650 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.650 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.650 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:22:48.650 DEBUG:boto:Data: 2024-04-13T22:22:48.650 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.651 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.651 DEBUG:boto:Params: {} 2024-04-13T22:22:48.651 DEBUG:boto:Token: None 2024-04-13T22:22:48.651 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11 2024-04-13T22:22:48.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KV3cIUPIlaKXHNvfHJfLWojh6lA= 2024-04-13T22:22:48.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KV3cIUPIlaKXHNvfHJfLWojh6lA=', 'Content-Length': '0'} 2024-04-13T22:22:48.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44f27295feef8c2-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.653 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.653 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.653 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.653 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:22:48.654 DEBUG:boto:Data: 2024-04-13T22:22:48.654 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.654 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.654 DEBUG:boto:Params: {} 2024-04-13T22:22:48.654 DEBUG:boto:Token: None 2024-04-13T22:22:48.654 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11 2024-04-13T22:22:48.654 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KV3cIUPIlaKXHNvfHJfLWojh6lA= 2024-04-13T22:22:48.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KV3cIUPIlaKXHNvfHJfLWojh6lA=', 'Content-Length': '0'} 2024-04-13T22:22:48.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005250620ded6add13-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:22:48.656 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.656 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.656 DEBUG:boto:Method: GET 2024-04-13T22:22:48.656 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:22:48.656 DEBUG:boto:Data: 2024-04-13T22:22:48.657 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.657 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.657 DEBUG:boto:Params: {} 2024-04-13T22:22:48.657 DEBUG:boto:Token: None 2024-04-13T22:22:48.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11 2024-04-13T22:22:48.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sAfpni5tKt6ohHrP6RWxQv6rODE= 2024-04-13T22:22:48.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sAfpni5tKt6ohHrP6RWxQv6rODE=', 'Content-Length': '0'} 2024-04-13T22:22:48.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256ccdfffede35af-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.659 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:22:48.659 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:22:48.659 DEBUG:boto:Method: GET 2024-04-13T22:22:48.659 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:22:48.659 DEBUG:boto:Data: 2024-04-13T22:22:48.659 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.660 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.660 DEBUG:boto:Params: {} 2024-04-13T22:22:48.660 DEBUG:boto:Token: None 2024-04-13T22:22:48.660 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj11 2024-04-13T22:22:48.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sAfpni5tKt6ohHrP6RWxQv6rODE= 2024-04-13T22:22:48.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sAfpni5tKt6ohHrP6RWxQv6rODE=', 'Content-Length': '0'} 2024-04-13T22:22:48.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028d9b460363bf0f2-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.662 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.662 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.662 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.662 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.662 DEBUG:boto:Method: GET 2024-04-13T22:22:48.662 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.662 DEBUG:boto:Data: 2024-04-13T22:22:48.662 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.663 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.663 DEBUG:boto:Params: {} 2024-04-13T22:22:48.663 DEBUG:boto:Token: None 2024-04-13T22:22:48.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f291bc18b16adb1-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.707 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.707 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.708 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.708 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.708 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.708 DEBUG:boto:Method: GET 2024-04-13T22:22:48.708 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:22:48.708 DEBUG:boto:Data: 2024-04-13T22:22:48.708 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.708 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.708 DEBUG:boto:Params: {} 2024-04-13T22:22:48.709 DEBUG:boto:Token: None 2024-04-13T22:22:48.709 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.709 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de0b5b1fff0f886-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.712 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:22:48.712 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.712 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.712 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.713 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.713 DEBUG:boto:Method: GET 2024-04-13T22:22:48.713 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.713 DEBUG:boto:Data: 2024-04-13T22:22:48.713 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.713 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.713 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.713 DEBUG:boto:Params: {} 2024-04-13T22:22:48.713 DEBUG:boto:Token: None 2024-04-13T22:22:48.713 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c= 2024-04-13T22:22:48.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c=', 'Content-Length': '0'} 2024-04-13T22:22:48.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c575feeaf5914bf6-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.715 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.716 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.716 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.716 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.716 DEBUG:boto:Method: GET 2024-04-13T22:22:48.716 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.716 DEBUG:boto:Data: 2024-04-13T22:22:48.716 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.716 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.716 DEBUG:boto:Params: {} 2024-04-13T22:22:48.716 DEBUG:boto:Token: None 2024-04-13T22:22:48.717 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c= 2024-04-13T22:22:48.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c=', 'Content-Length': '0'} 2024-04-13T22:22:48.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f4a4a10af95200-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.719 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.719 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.719 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.719 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.719 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.719 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.719 DEBUG:boto:Data: 2024-04-13T22:22:48.719 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.719 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.720 DEBUG:boto:Params: {} 2024-04-13T22:22:48.720 DEBUG:boto:Token: None 2024-04-13T22:22:48.720 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kJPl8fLDzV1qOS2Xl2pKMPgJ6xw= 2024-04-13T22:22:48.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kJPl8fLDzV1qOS2Xl2pKMPgJ6xw=', 'Content-Length': '0'} 2024-04-13T22:22:48.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba3446129322448-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.722 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.722 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.722 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.722 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.722 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.722 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.722 DEBUG:boto:Data: 2024-04-13T22:22:48.722 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.722 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.723 DEBUG:boto:Params: {} 2024-04-13T22:22:48.723 DEBUG:boto:Token: None 2024-04-13T22:22:48.723 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kJPl8fLDzV1qOS2Xl2pKMPgJ6xw= 2024-04-13T22:22:48.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kJPl8fLDzV1qOS2Xl2pKMPgJ6xw=', 'Content-Length': '0'} 2024-04-13T22:22:48.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78387e1da90b6c1-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:22:48.725 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.725 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.725 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.725 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.725 DEBUG:boto:Method: GET 2024-04-13T22:22:48.725 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.725 DEBUG:boto:Data: 2024-04-13T22:22:48.725 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.726 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.726 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.726 DEBUG:boto:Params: {} 2024-04-13T22:22:48.726 DEBUG:boto:Token: None 2024-04-13T22:22:48.726 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c= 2024-04-13T22:22:48.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c=', 'Content-Length': '0'} 2024-04-13T22:22:48.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd4858a1fc642d9-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.728 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.728 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.728 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.728 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.728 DEBUG:boto:Method: GET 2024-04-13T22:22:48.729 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.729 DEBUG:boto:Data: 2024-04-13T22:22:48.729 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.729 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.729 DEBUG:boto:Params: {} 2024-04-13T22:22:48.729 DEBUG:boto:Token: None 2024-04-13T22:22:48.729 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:22:48.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c= 2024-04-13T22:22:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+I6yN6pEqOFCkIYBeR5dlh6j/5c=', 'Content-Length': '0'} 2024-04-13T22:22:48.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d122e7f15399d3-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.774 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.775 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.775 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.775 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.775 DEBUG:boto:Method: GET 2024-04-13T22:22:48.775 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.775 DEBUG:boto:Data: 2024-04-13T22:22:48.775 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.775 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.775 DEBUG:boto:Params: {} 2024-04-13T22:22:48.776 DEBUG:boto:Token: None 2024-04-13T22:22:48.776 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007180c1d8326b4d1a-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.778 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.779 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.779 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.779 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.779 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.779 DEBUG:boto:Method: GET 2024-04-13T22:22:48.779 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.779 DEBUG:boto:Data: 2024-04-13T22:22:48.779 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.780 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.780 DEBUG:boto:Params: {} 2024-04-13T22:22:48.780 DEBUG:boto:Token: None 2024-04-13T22:22:48.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ba9331db4118f1b-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.783 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.783 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.784 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.784 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.784 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:22:48.784 DEBUG:boto:Data: 2024-04-13T22:22:48.784 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.784 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.784 DEBUG:boto:Params: {} 2024-04-13T22:22:48.784 DEBUG:boto:Token: None 2024-04-13T22:22:48.785 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17 2024-04-13T22:22:48.785 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YYv3KLwRnpZNzSO4zBBdjgl8iQ0= 2024-04-13T22:22:48.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YYv3KLwRnpZNzSO4zBBdjgl8iQ0=', 'Content-Length': '0'} 2024-04-13T22:22:48.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be00d61359d20ed-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.786 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.786 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.787 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.787 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:22:48.787 DEBUG:boto:Data: 2024-04-13T22:22:48.787 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.787 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.787 DEBUG:boto:Params: {} 2024-04-13T22:22:48.787 DEBUG:boto:Token: None 2024-04-13T22:22:48.787 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17 2024-04-13T22:22:48.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YYv3KLwRnpZNzSO4zBBdjgl8iQ0= 2024-04-13T22:22:48.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YYv3KLwRnpZNzSO4zBBdjgl8iQ0=', 'Content-Length': '0'} 2024-04-13T22:22:48.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a6d5539564b4ba-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:22:48.789 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.790 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.790 DEBUG:boto:Method: GET 2024-04-13T22:22:48.790 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:22:48.790 DEBUG:boto:Data: 2024-04-13T22:22:48.790 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.790 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.790 DEBUG:boto:Params: {} 2024-04-13T22:22:48.790 DEBUG:boto:Token: None 2024-04-13T22:22:48.791 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17 2024-04-13T22:22:48.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hUfBgmlt20NDsMBptYTqq39Akxc= 2024-04-13T22:22:48.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hUfBgmlt20NDsMBptYTqq39Akxc=', 'Content-Length': '0'} 2024-04-13T22:22:48.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d7cb474b67b2b0-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.793 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:22:48.793 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:22:48.793 DEBUG:boto:Method: GET 2024-04-13T22:22:48.793 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:22:48.793 DEBUG:boto:Data: 2024-04-13T22:22:48.793 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.794 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.794 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.794 DEBUG:boto:Params: {} 2024-04-13T22:22:48.794 DEBUG:boto:Token: None 2024-04-13T22:22:48.794 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj17 2024-04-13T22:22:48.794 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hUfBgmlt20NDsMBptYTqq39Akxc= 2024-04-13T22:22:48.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hUfBgmlt20NDsMBptYTqq39Akxc=', 'Content-Length': '0'} 2024-04-13T22:22:48.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e29d57320533512-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.796 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.796 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.797 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.797 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.797 DEBUG:boto:Method: GET 2024-04-13T22:22:48.797 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.797 DEBUG:boto:Data: 2024-04-13T22:22:48.797 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.797 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.797 DEBUG:boto:Params: {} 2024-04-13T22:22:48.797 DEBUG:boto:Token: None 2024-04-13T22:22:48.798 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14451bfb39762df-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.800 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.801 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.801 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.801 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.801 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.801 DEBUG:boto:Method: GET 2024-04-13T22:22:48.801 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:22:48.802 DEBUG:boto:Data: 2024-04-13T22:22:48.802 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.802 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.802 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.802 DEBUG:boto:Params: {} 2024-04-13T22:22:48.802 DEBUG:boto:Token: None 2024-04-13T22:22:48.802 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.802 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ead14920f8f5b60-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.805 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:22:48.806 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.806 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.806 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.806 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.807 DEBUG:boto:Method: GET 2024-04-13T22:22:48.807 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.807 DEBUG:boto:Data: 2024-04-13T22:22:48.807 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.807 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.807 DEBUG:boto:Params: {} 2024-04-13T22:22:48.808 DEBUG:boto:Token: None 2024-04-13T22:22:48.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic= 2024-04-13T22:22:48.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic=', 'Content-Length': '0'} 2024-04-13T22:22:48.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fee2daa9c391d6d-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.810 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.810 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.811 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.811 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.811 DEBUG:boto:Method: GET 2024-04-13T22:22:48.811 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.811 DEBUG:boto:Data: 2024-04-13T22:22:48.811 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.811 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.811 DEBUG:boto:Params: {} 2024-04-13T22:22:48.811 DEBUG:boto:Token: None 2024-04-13T22:22:48.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic= 2024-04-13T22:22:48.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic=', 'Content-Length': '0'} 2024-04-13T22:22:48.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f89810e7aca1bb0-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.814 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.814 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.814 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.814 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.814 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.814 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.815 DEBUG:boto:Data: 2024-04-13T22:22:48.815 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.815 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.815 DEBUG:boto:Params: {} 2024-04-13T22:22:48.815 DEBUG:boto:Token: None 2024-04-13T22:22:48.815 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c9dyotVNogxngcvupMWIb3HhpTQ= 2024-04-13T22:22:48.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c9dyotVNogxngcvupMWIb3HhpTQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a419cf82a4ae5ea-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.817 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.817 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.817 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.818 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.818 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.818 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.818 DEBUG:boto:Data: 2024-04-13T22:22:48.818 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.818 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.818 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.818 DEBUG:boto:Params: {} 2024-04-13T22:22:48.818 DEBUG:boto:Token: None 2024-04-13T22:22:48.819 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c9dyotVNogxngcvupMWIb3HhpTQ= 2024-04-13T22:22:48.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c9dyotVNogxngcvupMWIb3HhpTQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ae9bb6f42a5e02-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:22:48.821 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.821 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.821 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.821 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.821 DEBUG:boto:Method: GET 2024-04-13T22:22:48.822 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.822 DEBUG:boto:Data: 2024-04-13T22:22:48.822 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.822 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.822 DEBUG:boto:Params: {} 2024-04-13T22:22:48.822 DEBUG:boto:Token: None 2024-04-13T22:22:48.822 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.822 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic= 2024-04-13T22:22:48.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic=', 'Content-Length': '0'} 2024-04-13T22:22:48.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a92bb79a45bb74d-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.824 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.824 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.825 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.825 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.825 DEBUG:boto:Method: GET 2024-04-13T22:22:48.825 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.825 DEBUG:boto:Data: 2024-04-13T22:22:48.825 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.825 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.825 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.825 DEBUG:boto:Params: {} 2024-04-13T22:22:48.826 DEBUG:boto:Token: None 2024-04-13T22:22:48.826 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:22:48.826 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic= 2024-04-13T22:22:48.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NIdqI2krqwIMm5cdCodV+Mx+0ic=', 'Content-Length': '0'} 2024-04-13T22:22:48.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bff13441e9a0adb-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.828 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.828 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.828 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.828 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.828 DEBUG:boto:Method: GET 2024-04-13T22:22:48.829 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.829 DEBUG:boto:Data: 2024-04-13T22:22:48.829 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.829 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.829 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.829 DEBUG:boto:Params: {} 2024-04-13T22:22:48.829 DEBUG:boto:Token: None 2024-04-13T22:22:48.829 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.829 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9526409bb04d42c-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.832 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.832 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.833 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.833 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.833 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.833 DEBUG:boto:Method: GET 2024-04-13T22:22:48.833 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.833 DEBUG:boto:Data: 2024-04-13T22:22:48.833 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.833 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.833 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.833 DEBUG:boto:Params: {} 2024-04-13T22:22:48.833 DEBUG:boto:Token: None 2024-04-13T22:22:48.834 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.834 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eee8152b51ef2844-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.837 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.837 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.837 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.837 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.837 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:22:48.837 DEBUG:boto:Data: 2024-04-13T22:22:48.838 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.838 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.838 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.838 DEBUG:boto:Params: {} 2024-04-13T22:22:48.838 DEBUG:boto:Token: None 2024-04-13T22:22:48.838 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19 2024-04-13T22:22:48.838 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yiAtTOQYKAHBdwec4LB28ZoHsKc= 2024-04-13T22:22:48.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yiAtTOQYKAHBdwec4LB28ZoHsKc=', 'Content-Length': '0'} 2024-04-13T22:22:48.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064623ea6ca76fed6-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.840 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.840 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.840 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.840 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:22:48.841 DEBUG:boto:Data: 2024-04-13T22:22:48.841 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.841 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.841 DEBUG:boto:Params: {} 2024-04-13T22:22:48.841 DEBUG:boto:Token: None 2024-04-13T22:22:48.841 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19 2024-04-13T22:22:48.841 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yiAtTOQYKAHBdwec4LB28ZoHsKc= 2024-04-13T22:22:48.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yiAtTOQYKAHBdwec4LB28ZoHsKc=', 'Content-Length': '0'} 2024-04-13T22:22:48.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3a5d45309ca7f1-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:22:48.843 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.843 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.844 DEBUG:boto:Method: GET 2024-04-13T22:22:48.844 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:22:48.844 DEBUG:boto:Data: 2024-04-13T22:22:48.844 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.844 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.844 DEBUG:boto:Params: {} 2024-04-13T22:22:48.844 DEBUG:boto:Token: None 2024-04-13T22:22:48.844 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19 2024-04-13T22:22:48.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGkM7o2TZoLAS5nYdtxfPJE90f8= 2024-04-13T22:22:48.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGkM7o2TZoLAS5nYdtxfPJE90f8=', 'Content-Length': '0'} 2024-04-13T22:22:48.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe65805d7a24f2b-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.890 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:22:48.891 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:22:48.891 DEBUG:boto:Method: GET 2024-04-13T22:22:48.891 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:22:48.891 DEBUG:boto:Data: 2024-04-13T22:22:48.891 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.891 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.891 DEBUG:boto:Params: {} 2024-04-13T22:22:48.891 DEBUG:boto:Token: None 2024-04-13T22:22:48.891 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj19 2024-04-13T22:22:48.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dGkM7o2TZoLAS5nYdtxfPJE90f8= 2024-04-13T22:22:48.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dGkM7o2TZoLAS5nYdtxfPJE90f8=', 'Content-Length': '0'} 2024-04-13T22:22:48.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000719d8f0577bce4df-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.893 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.894 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.894 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.894 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.894 DEBUG:boto:Method: GET 2024-04-13T22:22:48.894 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.894 DEBUG:boto:Data: 2024-04-13T22:22:48.894 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.894 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.894 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.894 DEBUG:boto:Params: {} 2024-04-13T22:22:48.895 DEBUG:boto:Token: None 2024-04-13T22:22:48.895 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.895 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000838e484fd623e7f6-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.898 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.899 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.899 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.899 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.899 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.899 DEBUG:boto:Method: GET 2024-04-13T22:22:48.899 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:22:48.899 DEBUG:boto:Data: 2024-04-13T22:22:48.899 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.899 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.899 DEBUG:boto:Params: {} 2024-04-13T22:22:48.900 DEBUG:boto:Token: None 2024-04-13T22:22:48.900 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a9ed46778d7800-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.903 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:22:48.903 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.903 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.904 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.904 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.904 DEBUG:boto:Method: GET 2024-04-13T22:22:48.904 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.904 DEBUG:boto:Data: 2024-04-13T22:22:48.904 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.904 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.904 DEBUG:boto:Params: {} 2024-04-13T22:22:48.904 DEBUG:boto:Token: None 2024-04-13T22:22:48.905 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs= 2024-04-13T22:22:48.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs=', 'Content-Length': '0'} 2024-04-13T22:22:48.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d913e2bbecd913b-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.906 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.907 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.907 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.907 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.907 DEBUG:boto:Method: GET 2024-04-13T22:22:48.907 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.907 DEBUG:boto:Data: 2024-04-13T22:22:48.907 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.907 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.907 DEBUG:boto:Params: {} 2024-04-13T22:22:48.908 DEBUG:boto:Token: None 2024-04-13T22:22:48.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs= 2024-04-13T22:22:48.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs=', 'Content-Length': '0'} 2024-04-13T22:22:48.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088cb2ca58295f032-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.954 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.955 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.955 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.955 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.955 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.955 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.955 DEBUG:boto:Data: 2024-04-13T22:22:48.955 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.956 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.956 DEBUG:boto:Params: {} 2024-04-13T22:22:48.956 DEBUG:boto:Token: None 2024-04-13T22:22:48.956 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gy5udpxmKU5YGuqU7i8FIkhMPBQ= 2024-04-13T22:22:48.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gy5udpxmKU5YGuqU7i8FIkhMPBQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a886007e48ff5878-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.958 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.958 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.958 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.958 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.958 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.959 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.959 DEBUG:boto:Data: 2024-04-13T22:22:48.959 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.959 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.959 DEBUG:boto:Params: {} 2024-04-13T22:22:48.959 DEBUG:boto:Token: None 2024-04-13T22:22:48.959 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.959 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gy5udpxmKU5YGuqU7i8FIkhMPBQ= 2024-04-13T22:22:48.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gy5udpxmKU5YGuqU7i8FIkhMPBQ=', 'Content-Length': '0'} 2024-04-13T22:22:48.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eeca566e90cdc53-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:22:48.961 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.961 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.962 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.962 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.962 DEBUG:boto:Method: GET 2024-04-13T22:22:48.962 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.962 DEBUG:boto:Data: 2024-04-13T22:22:48.962 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.962 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.962 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.962 DEBUG:boto:Params: {} 2024-04-13T22:22:48.963 DEBUG:boto:Token: None 2024-04-13T22:22:48.963 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.963 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs= 2024-04-13T22:22:48.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs=', 'Content-Length': '0'} 2024-04-13T22:22:48.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c0a07b31d11154-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.965 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.965 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.965 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.965 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.965 DEBUG:boto:Method: GET 2024-04-13T22:22:48.965 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.965 DEBUG:boto:Data: 2024-04-13T22:22:48.965 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.966 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.966 DEBUG:boto:Params: {} 2024-04-13T22:22:48.966 DEBUG:boto:Token: None 2024-04-13T22:22:48.966 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:22:48.966 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs= 2024-04-13T22:22:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StV45Jc3BzqHlVEtl6qeD1+DVfs=', 'Content-Length': '0'} 2024-04-13T22:22:48.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baa9486b9500be3a-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.968 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.968 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.968 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.968 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.969 DEBUG:boto:Method: GET 2024-04-13T22:22:48.969 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.969 DEBUG:boto:Data: 2024-04-13T22:22:48.969 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.970 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.970 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.970 DEBUG:boto:Params: {} 2024-04-13T22:22:48.970 DEBUG:boto:Token: None 2024-04-13T22:22:48.970 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.970 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a22298b892294254-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.972 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.973 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.973 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.973 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.973 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.973 DEBUG:boto:Method: GET 2024-04-13T22:22:48.973 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.973 DEBUG:boto:Data: 2024-04-13T22:22:48.974 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.974 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.974 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.974 DEBUG:boto:Params: {} 2024-04-13T22:22:48.974 DEBUG:boto:Token: None 2024-04-13T22:22:48.974 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.974 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001968b17c6b165f33-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.977 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.977 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.977 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.978 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.978 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:22:48.978 DEBUG:boto:Data: 2024-04-13T22:22:48.978 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.978 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.978 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.978 DEBUG:boto:Params: {} 2024-04-13T22:22:48.978 DEBUG:boto:Token: None 2024-04-13T22:22:48.978 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22 2024-04-13T22:22:48.979 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLSAQwAQxrC358mUeBp10quCCs4= 2024-04-13T22:22:48.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLSAQwAQxrC358mUeBp10quCCs4=', 'Content-Length': '0'} 2024-04-13T22:22:48.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e52962986b84d7d-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.980 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.980 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.980 DEBUG:boto:Method: HEAD 2024-04-13T22:22:48.981 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:22:48.981 DEBUG:boto:Data: 2024-04-13T22:22:48.981 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.981 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.981 DEBUG:boto:Params: {} 2024-04-13T22:22:48.981 DEBUG:boto:Token: None 2024-04-13T22:22:48.981 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22 2024-04-13T22:22:48.981 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLSAQwAQxrC358mUeBp10quCCs4= 2024-04-13T22:22:48.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLSAQwAQxrC358mUeBp10quCCs4=', 'Content-Length': '0'} 2024-04-13T22:22:48.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000513201218f09d210-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:22:48.983 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.984 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.984 DEBUG:boto:Method: GET 2024-04-13T22:22:48.984 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:22:48.984 DEBUG:boto:Data: 2024-04-13T22:22:48.984 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.984 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.984 DEBUG:boto:Params: {} 2024-04-13T22:22:48.984 DEBUG:boto:Token: None 2024-04-13T22:22:48.984 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22 2024-04-13T22:22:48.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nkebtHsUBYVVIewmrpNfiT9gRmI= 2024-04-13T22:22:48.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nkebtHsUBYVVIewmrpNfiT9gRmI=', 'Content-Length': '0'} 2024-04-13T22:22:48.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b990fae0d44c7392-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.986 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:22:48.986 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:22:48.987 DEBUG:boto:Method: GET 2024-04-13T22:22:48.987 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:22:48.987 DEBUG:boto:Data: 2024-04-13T22:22:48.987 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.987 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.987 DEBUG:boto:Params: {} 2024-04-13T22:22:48.987 DEBUG:boto:Token: None 2024-04-13T22:22:48.987 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj22 2024-04-13T22:22:48.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nkebtHsUBYVVIewmrpNfiT9gRmI= 2024-04-13T22:22:48.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nkebtHsUBYVVIewmrpNfiT9gRmI=', 'Content-Length': '0'} 2024-04-13T22:22:48.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b39a89cdffb236-00661b05b8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.989 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.990 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.990 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.990 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.990 DEBUG:boto:Method: GET 2024-04-13T22:22:48.990 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.990 DEBUG:boto:Data: 2024-04-13T22:22:48.990 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.990 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.990 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.990 DEBUG:boto:Params: {} 2024-04-13T22:22:48.991 DEBUG:boto:Token: None 2024-04-13T22:22:48.991 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caa34e4da449ab3e-00661b05b8-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.993 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.994 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:48.994 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:48.994 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.994 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.994 DEBUG:boto:Method: GET 2024-04-13T22:22:48.994 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:22:48.994 DEBUG:boto:Data: 2024-04-13T22:22:48.994 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:48.994 DEBUG:boto:Port: 8000 2024-04-13T22:22:48.995 DEBUG:boto:Params: {} 2024-04-13T22:22:48.995 DEBUG:boto:Token: None 2024-04-13T22:22:48.995 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/ 2024-04-13T22:22:48.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM= 2024-04-13T22:22:48.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NhxH+5VVW6/iJYQv657H07xEOhM=', 'Content-Length': '0'} 2024-04-13T22:22:48.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8ec625a6aab8888-00661b05b8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:48.997 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:48.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:22:48.998 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:48.998 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:48.998 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:48.998 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:48.998 DEBUG:boto:Method: GET 2024-04-13T22:22:48.999 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:48.999 DEBUG:boto:Data: 2024-04-13T22:22:48.999 DEBUG:boto:Headers: {} 2024-04-13T22:22:48.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:48.999 DEBUG:boto:Port: 8001 2024-04-13T22:22:48.999 DEBUG:boto:Params: {} 2024-04-13T22:22:48.999 DEBUG:boto:Token: None 2024-04-13T22:22:48.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:48 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:48.999 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXscfvxb5a8urkEuL4qq6pp4na0= 2024-04-13T22:22:48.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXscfvxb5a8urkEuL4qq6pp4na0=', 'Content-Length': '0'} 2024-04-13T22:22:49.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c606826c93b3b0-00661b05b8-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.001 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.001 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.001 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.001 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.002 DEBUG:boto:Method: GET 2024-04-13T22:22:49.002 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.002 DEBUG:boto:Data: 2024-04-13T22:22:49.002 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.002 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.002 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.002 DEBUG:boto:Params: {} 2024-04-13T22:22:49.002 DEBUG:boto:Token: None 2024-04-13T22:22:49.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gGlcLspQCPsOaKvIvQxlYP8vOmY= 2024-04-13T22:22:49.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gGlcLspQCPsOaKvIvQxlYP8vOmY=', 'Content-Length': '0'} 2024-04-13T22:22:49.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e3241d1d237da1a-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.004 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.005 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.005 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.005 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.005 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.005 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.005 DEBUG:boto:Data: 2024-04-13T22:22:49.005 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.005 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.005 DEBUG:boto:Params: {} 2024-04-13T22:22:49.006 DEBUG:boto:Token: None 2024-04-13T22:22:49.006 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mAe/eDNbRnW0G6XwdtVHsN3bqVM= 2024-04-13T22:22:49.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mAe/eDNbRnW0G6XwdtVHsN3bqVM=', 'Content-Length': '0'} 2024-04-13T22:22:49.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000284c2e9821ddec91-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.008 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.008 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.008 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.008 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.008 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.008 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.009 DEBUG:boto:Data: 2024-04-13T22:22:49.009 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.009 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.009 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.009 DEBUG:boto:Params: {} 2024-04-13T22:22:49.009 DEBUG:boto:Token: None 2024-04-13T22:22:49.009 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mAe/eDNbRnW0G6XwdtVHsN3bqVM= 2024-04-13T22:22:49.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mAe/eDNbRnW0G6XwdtVHsN3bqVM=', 'Content-Length': '0'} 2024-04-13T22:22:49.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000048b26cc8515acd-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:22:49.011 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.011 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.011 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.011 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.012 DEBUG:boto:Method: GET 2024-04-13T22:22:49.012 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.012 DEBUG:boto:Data: 2024-04-13T22:22:49.012 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.012 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.012 DEBUG:boto:Params: {} 2024-04-13T22:22:49.012 DEBUG:boto:Token: None 2024-04-13T22:22:49.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gGlcLspQCPsOaKvIvQxlYP8vOmY= 2024-04-13T22:22:49.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gGlcLspQCPsOaKvIvQxlYP8vOmY=', 'Content-Length': '0'} 2024-04-13T22:22:49.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b854a7109bad3d-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.014 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.014 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.014 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.015 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.015 DEBUG:boto:Method: GET 2024-04-13T22:22:49.015 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.015 DEBUG:boto:Data: 2024-04-13T22:22:49.015 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.015 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.015 DEBUG:boto:Params: {} 2024-04-13T22:22:49.015 DEBUG:boto:Token: None 2024-04-13T22:22:49.015 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:22:49.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gGlcLspQCPsOaKvIvQxlYP8vOmY= 2024-04-13T22:22:49.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gGlcLspQCPsOaKvIvQxlYP8vOmY=', 'Content-Length': '0'} 2024-04-13T22:22:49.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a5d8e40bf0c77d-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.058 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.059 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.059 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.059 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.059 DEBUG:boto:Method: GET 2024-04-13T22:22:49.059 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.059 DEBUG:boto:Data: 2024-04-13T22:22:49.059 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.059 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.059 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.059 DEBUG:boto:Params: {} 2024-04-13T22:22:49.060 DEBUG:boto:Token: None 2024-04-13T22:22:49.060 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.060 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2d08b39a791005-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.062 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.063 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.063 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.063 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.063 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.063 DEBUG:boto:Method: GET 2024-04-13T22:22:49.063 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.063 DEBUG:boto:Data: 2024-04-13T22:22:49.063 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.064 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.064 DEBUG:boto:Params: {} 2024-04-13T22:22:49.064 DEBUG:boto:Token: None 2024-04-13T22:22:49.064 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.064 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de8a1486fc77b41-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.067 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.067 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.067 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.068 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.068 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:22:49.068 DEBUG:boto:Data: 2024-04-13T22:22:49.068 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.068 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.068 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.068 DEBUG:boto:Params: {} 2024-04-13T22:22:49.068 DEBUG:boto:Token: None 2024-04-13T22:22:49.068 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33 2024-04-13T22:22:49.068 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q1zNbqel6MQGdvWCoJ6OOs0JFzE= 2024-04-13T22:22:49.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q1zNbqel6MQGdvWCoJ6OOs0JFzE=', 'Content-Length': '0'} 2024-04-13T22:22:49.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f55cc4b165f925-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.070 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.070 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.071 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.071 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:22:49.071 DEBUG:boto:Data: 2024-04-13T22:22:49.071 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.071 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.071 DEBUG:boto:Params: {} 2024-04-13T22:22:49.071 DEBUG:boto:Token: None 2024-04-13T22:22:49.071 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33 2024-04-13T22:22:49.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q1zNbqel6MQGdvWCoJ6OOs0JFzE= 2024-04-13T22:22:49.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q1zNbqel6MQGdvWCoJ6OOs0JFzE=', 'Content-Length': '0'} 2024-04-13T22:22:49.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e539f38aec0cac-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:22:49.073 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.073 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.073 DEBUG:boto:Method: GET 2024-04-13T22:22:49.073 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:22:49.074 DEBUG:boto:Data: 2024-04-13T22:22:49.074 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.074 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.074 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.074 DEBUG:boto:Params: {} 2024-04-13T22:22:49.074 DEBUG:boto:Token: None 2024-04-13T22:22:49.074 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33 2024-04-13T22:22:49.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YNakkCFn0azJlUH7t9vMyFgJ6xc= 2024-04-13T22:22:49.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YNakkCFn0azJlUH7t9vMyFgJ6xc=', 'Content-Length': '0'} 2024-04-13T22:22:49.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df9d27d40edd3a5-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.076 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:22:49.077 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:22:49.077 DEBUG:boto:Method: GET 2024-04-13T22:22:49.077 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:22:49.077 DEBUG:boto:Data: 2024-04-13T22:22:49.077 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.077 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.077 DEBUG:boto:Params: {} 2024-04-13T22:22:49.077 DEBUG:boto:Token: None 2024-04-13T22:22:49.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj33 2024-04-13T22:22:49.078 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YNakkCFn0azJlUH7t9vMyFgJ6xc= 2024-04-13T22:22:49.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YNakkCFn0azJlUH7t9vMyFgJ6xc=', 'Content-Length': '0'} 2024-04-13T22:22:49.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f048f672cb37ded9-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.079 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.080 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.080 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.080 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.080 DEBUG:boto:Method: GET 2024-04-13T22:22:49.080 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.080 DEBUG:boto:Data: 2024-04-13T22:22:49.080 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.080 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.080 DEBUG:boto:Params: {} 2024-04-13T22:22:49.080 DEBUG:boto:Token: None 2024-04-13T22:22:49.081 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.081 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682ebf2a0957733f-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.083 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.083 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.083 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.083 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.083 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.084 DEBUG:boto:Method: GET 2024-04-13T22:22:49.084 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:22:49.084 DEBUG:boto:Data: 2024-04-13T22:22:49.084 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.084 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.084 DEBUG:boto:Params: {} 2024-04-13T22:22:49.084 DEBUG:boto:Token: None 2024-04-13T22:22:49.084 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e14a120406ca296-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.087 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:22:49.087 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.088 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.088 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.088 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.088 DEBUG:boto:Method: GET 2024-04-13T22:22:49.088 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.088 DEBUG:boto:Data: 2024-04-13T22:22:49.088 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.088 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.088 DEBUG:boto:Params: {} 2024-04-13T22:22:49.088 DEBUG:boto:Token: None 2024-04-13T22:22:49.089 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0= 2024-04-13T22:22:49.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0=', 'Content-Length': '0'} 2024-04-13T22:22:49.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca5dab586de74c6-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.134 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.135 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.135 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.135 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.135 DEBUG:boto:Method: GET 2024-04-13T22:22:49.135 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.135 DEBUG:boto:Data: 2024-04-13T22:22:49.135 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.135 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.136 DEBUG:boto:Params: {} 2024-04-13T22:22:49.136 DEBUG:boto:Token: None 2024-04-13T22:22:49.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0= 2024-04-13T22:22:49.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0=', 'Content-Length': '0'} 2024-04-13T22:22:49.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000660e456acda74135-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.138 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.139 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.139 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.139 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.139 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.139 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.139 DEBUG:boto:Data: 2024-04-13T22:22:49.139 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.139 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.139 DEBUG:boto:Params: {} 2024-04-13T22:22:49.139 DEBUG:boto:Token: None 2024-04-13T22:22:49.140 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TW3Qx9bZFyYas98BPYJJUdHt9WE= 2024-04-13T22:22:49.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TW3Qx9bZFyYas98BPYJJUdHt9WE=', 'Content-Length': '0'} 2024-04-13T22:22:49.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096056c6e24100738-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.142 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.142 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.142 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.142 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.142 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.143 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.143 DEBUG:boto:Data: 2024-04-13T22:22:49.143 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.143 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.143 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.143 DEBUG:boto:Params: {} 2024-04-13T22:22:49.143 DEBUG:boto:Token: None 2024-04-13T22:22:49.143 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.143 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TW3Qx9bZFyYas98BPYJJUdHt9WE= 2024-04-13T22:22:49.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TW3Qx9bZFyYas98BPYJJUdHt9WE=', 'Content-Length': '0'} 2024-04-13T22:22:49.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000669be7ddf0b951aa-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:22:49.145 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.145 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.146 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.146 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.146 DEBUG:boto:Method: GET 2024-04-13T22:22:49.146 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.146 DEBUG:boto:Data: 2024-04-13T22:22:49.146 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.146 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.146 DEBUG:boto:Params: {} 2024-04-13T22:22:49.147 DEBUG:boto:Token: None 2024-04-13T22:22:49.147 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0= 2024-04-13T22:22:49.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0=', 'Content-Length': '0'} 2024-04-13T22:22:49.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16e83f9577d95cf-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.149 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.150 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.150 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.150 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.150 DEBUG:boto:Method: GET 2024-04-13T22:22:49.150 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.150 DEBUG:boto:Data: 2024-04-13T22:22:49.150 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.150 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.150 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.150 DEBUG:boto:Params: {} 2024-04-13T22:22:49.151 DEBUG:boto:Token: None 2024-04-13T22:22:49.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:22:49.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0= 2024-04-13T22:22:49.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n+FwRu+p4ecdItWjhDWbUKGCaa0=', 'Content-Length': '0'} 2024-04-13T22:22:49.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcfff3025b59f966-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.198 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.199 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.199 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.199 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.199 DEBUG:boto:Method: GET 2024-04-13T22:22:49.199 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.199 DEBUG:boto:Data: 2024-04-13T22:22:49.199 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.199 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.199 DEBUG:boto:Params: {} 2024-04-13T22:22:49.200 DEBUG:boto:Token: None 2024-04-13T22:22:49.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009008ef7eb5255559-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.202 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.203 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.203 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.203 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.203 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.203 DEBUG:boto:Method: GET 2024-04-13T22:22:49.203 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.203 DEBUG:boto:Data: 2024-04-13T22:22:49.204 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.204 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.204 DEBUG:boto:Params: {} 2024-04-13T22:22:49.204 DEBUG:boto:Token: None 2024-04-13T22:22:49.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d2aaa8f43c30f8-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.207 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.208 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.208 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.208 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.208 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:22:49.208 DEBUG:boto:Data: 2024-04-13T22:22:49.208 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.208 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.208 DEBUG:boto:Params: {} 2024-04-13T22:22:49.209 DEBUG:boto:Token: None 2024-04-13T22:22:49.209 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34 2024-04-13T22:22:49.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JxPkZwk5mBPmLW9jqS1Jaf4k5o= 2024-04-13T22:22:49.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JxPkZwk5mBPmLW9jqS1Jaf4k5o=', 'Content-Length': '0'} 2024-04-13T22:22:49.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089822d4d5cd2197e-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.211 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.211 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.211 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.211 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:22:49.211 DEBUG:boto:Data: 2024-04-13T22:22:49.212 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.212 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.212 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.212 DEBUG:boto:Params: {} 2024-04-13T22:22:49.212 DEBUG:boto:Token: None 2024-04-13T22:22:49.212 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34 2024-04-13T22:22:49.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JxPkZwk5mBPmLW9jqS1Jaf4k5o= 2024-04-13T22:22:49.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JxPkZwk5mBPmLW9jqS1Jaf4k5o=', 'Content-Length': '0'} 2024-04-13T22:22:49.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4e2c560d4a7cdc-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:22:49.214 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.214 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.214 DEBUG:boto:Method: GET 2024-04-13T22:22:49.214 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:22:49.214 DEBUG:boto:Data: 2024-04-13T22:22:49.215 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.215 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.215 DEBUG:boto:Params: {} 2024-04-13T22:22:49.215 DEBUG:boto:Token: None 2024-04-13T22:22:49.215 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34 2024-04-13T22:22:49.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tGmaq8MXvN9c1NY6VDdY23L+inI= 2024-04-13T22:22:49.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tGmaq8MXvN9c1NY6VDdY23L+inI=', 'Content-Length': '0'} 2024-04-13T22:22:49.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006360714093d66c52-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.262 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:22:49.263 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:22:49.263 DEBUG:boto:Method: GET 2024-04-13T22:22:49.263 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:22:49.263 DEBUG:boto:Data: 2024-04-13T22:22:49.263 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.263 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.263 DEBUG:boto:Params: {} 2024-04-13T22:22:49.263 DEBUG:boto:Token: None 2024-04-13T22:22:49.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj34 2024-04-13T22:22:49.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tGmaq8MXvN9c1NY6VDdY23L+inI= 2024-04-13T22:22:49.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tGmaq8MXvN9c1NY6VDdY23L+inI=', 'Content-Length': '0'} 2024-04-13T22:22:49.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fbb0c1203406e0d-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.266 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.266 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.266 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.266 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.266 DEBUG:boto:Method: GET 2024-04-13T22:22:49.266 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.266 DEBUG:boto:Data: 2024-04-13T22:22:49.266 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.266 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.267 DEBUG:boto:Params: {} 2024-04-13T22:22:49.267 DEBUG:boto:Token: None 2024-04-13T22:22:49.267 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa19e80b8d3ec87f-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.269 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.270 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.270 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.270 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.270 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.270 DEBUG:boto:Method: GET 2024-04-13T22:22:49.270 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:22:49.270 DEBUG:boto:Data: 2024-04-13T22:22:49.271 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.271 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.271 DEBUG:boto:Params: {} 2024-04-13T22:22:49.271 DEBUG:boto:Token: None 2024-04-13T22:22:49.271 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006094929ef1fb289f-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.274 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:22:49.274 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.274 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.275 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.275 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.275 DEBUG:boto:Method: GET 2024-04-13T22:22:49.275 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.275 DEBUG:boto:Data: 2024-04-13T22:22:49.275 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.275 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.275 DEBUG:boto:Params: {} 2024-04-13T22:22:49.275 DEBUG:boto:Token: None 2024-04-13T22:22:49.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g= 2024-04-13T22:22:49.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g=', 'Content-Length': '0'} 2024-04-13T22:22:49.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0fd25b1f37a624f-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.278 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.278 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.278 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.278 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.278 DEBUG:boto:Method: GET 2024-04-13T22:22:49.278 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.278 DEBUG:boto:Data: 2024-04-13T22:22:49.278 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.278 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.279 DEBUG:boto:Params: {} 2024-04-13T22:22:49.279 DEBUG:boto:Token: None 2024-04-13T22:22:49.279 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g= 2024-04-13T22:22:49.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g=', 'Content-Length': '0'} 2024-04-13T22:22:49.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009035400c6f2d4893-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.281 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.281 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.281 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.281 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.281 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.281 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.281 DEBUG:boto:Data: 2024-04-13T22:22:49.281 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.281 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.282 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.282 DEBUG:boto:Params: {} 2024-04-13T22:22:49.282 DEBUG:boto:Token: None 2024-04-13T22:22:49.282 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.282 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g/lgsVLfHGBVkSRA/pbYuleiRpE= 2024-04-13T22:22:49.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g/lgsVLfHGBVkSRA/pbYuleiRpE=', 'Content-Length': '0'} 2024-04-13T22:22:49.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004958666e6ff97945-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.284 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.284 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.284 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.284 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.284 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.285 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.285 DEBUG:boto:Data: 2024-04-13T22:22:49.285 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.285 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.285 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.285 DEBUG:boto:Params: {} 2024-04-13T22:22:49.285 DEBUG:boto:Token: None 2024-04-13T22:22:49.285 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.285 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g/lgsVLfHGBVkSRA/pbYuleiRpE= 2024-04-13T22:22:49.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g/lgsVLfHGBVkSRA/pbYuleiRpE=', 'Content-Length': '0'} 2024-04-13T22:22:49.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001944698ccfca0553-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:22:49.287 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.287 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.287 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.288 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.288 DEBUG:boto:Method: GET 2024-04-13T22:22:49.288 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.288 DEBUG:boto:Data: 2024-04-13T22:22:49.288 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.288 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.288 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.288 DEBUG:boto:Params: {} 2024-04-13T22:22:49.288 DEBUG:boto:Token: None 2024-04-13T22:22:49.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.289 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g= 2024-04-13T22:22:49.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g=', 'Content-Length': '0'} 2024-04-13T22:22:49.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d1cfae75eeba2f-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.290 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.291 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.291 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.291 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.291 DEBUG:boto:Method: GET 2024-04-13T22:22:49.291 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.291 DEBUG:boto:Data: 2024-04-13T22:22:49.291 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.291 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.291 DEBUG:boto:Params: {} 2024-04-13T22:22:49.292 DEBUG:boto:Token: None 2024-04-13T22:22:49.292 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:22:49.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g= 2024-04-13T22:22:49.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZzU5Lf4x0QcSWTAyXMmS2fblP9g=', 'Content-Length': '0'} 2024-04-13T22:22:49.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e09c89910a5edd-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.334 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.335 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.335 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.335 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.335 DEBUG:boto:Method: GET 2024-04-13T22:22:49.335 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.335 DEBUG:boto:Data: 2024-04-13T22:22:49.335 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.336 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.336 DEBUG:boto:Params: {} 2024-04-13T22:22:49.336 DEBUG:boto:Token: None 2024-04-13T22:22:49.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab8b6be0ee840f30-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.339 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.340 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.340 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.340 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.340 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.341 DEBUG:boto:Method: GET 2024-04-13T22:22:49.341 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.341 DEBUG:boto:Data: 2024-04-13T22:22:49.341 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.341 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.341 DEBUG:boto:Params: {} 2024-04-13T22:22:49.341 DEBUG:boto:Token: None 2024-04-13T22:22:49.342 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b68ea605c7fe35-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.344 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.345 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.345 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.345 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.345 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:22:49.345 DEBUG:boto:Data: 2024-04-13T22:22:49.345 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.346 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.346 DEBUG:boto:Params: {} 2024-04-13T22:22:49.346 DEBUG:boto:Token: None 2024-04-13T22:22:49.346 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38 2024-04-13T22:22:49.346 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k0GCeByQx+bgAaTBT5uJXF8hnyU= 2024-04-13T22:22:49.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k0GCeByQx+bgAaTBT5uJXF8hnyU=', 'Content-Length': '0'} 2024-04-13T22:22:49.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f8e27b73da08d5-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.348 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.348 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.348 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.348 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:22:49.349 DEBUG:boto:Data: 2024-04-13T22:22:49.349 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.349 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.349 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.349 DEBUG:boto:Params: {} 2024-04-13T22:22:49.349 DEBUG:boto:Token: None 2024-04-13T22:22:49.349 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38 2024-04-13T22:22:49.349 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k0GCeByQx+bgAaTBT5uJXF8hnyU= 2024-04-13T22:22:49.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k0GCeByQx+bgAaTBT5uJXF8hnyU=', 'Content-Length': '0'} 2024-04-13T22:22:49.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520b218a6962ec49-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:22:49.351 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.352 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.352 DEBUG:boto:Method: GET 2024-04-13T22:22:49.352 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:22:49.352 DEBUG:boto:Data: 2024-04-13T22:22:49.352 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.352 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.352 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.352 DEBUG:boto:Params: {} 2024-04-13T22:22:49.352 DEBUG:boto:Token: None 2024-04-13T22:22:49.353 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38 2024-04-13T22:22:49.353 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ps32hrUrS+onYIFZ5bAT2PGOuTg= 2024-04-13T22:22:49.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ps32hrUrS+onYIFZ5bAT2PGOuTg=', 'Content-Length': '0'} 2024-04-13T22:22:49.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95515181c677478-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.355 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:22:49.355 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:22:49.355 DEBUG:boto:Method: GET 2024-04-13T22:22:49.355 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:22:49.355 DEBUG:boto:Data: 2024-04-13T22:22:49.356 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.356 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.356 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.356 DEBUG:boto:Params: {} 2024-04-13T22:22:49.356 DEBUG:boto:Token: None 2024-04-13T22:22:49.356 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj38 2024-04-13T22:22:49.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ps32hrUrS+onYIFZ5bAT2PGOuTg= 2024-04-13T22:22:49.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ps32hrUrS+onYIFZ5bAT2PGOuTg=', 'Content-Length': '0'} 2024-04-13T22:22:49.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86849edcbfd4534-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.358 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.358 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.359 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.359 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.359 DEBUG:boto:Method: GET 2024-04-13T22:22:49.359 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.359 DEBUG:boto:Data: 2024-04-13T22:22:49.359 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.359 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.359 DEBUG:boto:Params: {} 2024-04-13T22:22:49.360 DEBUG:boto:Token: None 2024-04-13T22:22:49.360 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc6ff5f9833f3938-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.402 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.403 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.403 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.403 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.403 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.403 DEBUG:boto:Method: GET 2024-04-13T22:22:49.404 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:22:49.404 DEBUG:boto:Data: 2024-04-13T22:22:49.404 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.404 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.404 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.404 DEBUG:boto:Params: {} 2024-04-13T22:22:49.404 DEBUG:boto:Token: None 2024-04-13T22:22:49.404 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009afc5f3671aed3c5-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.408 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:22:49.408 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.408 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.408 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.409 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.409 DEBUG:boto:Method: GET 2024-04-13T22:22:49.409 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.409 DEBUG:boto:Data: 2024-04-13T22:22:49.409 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.409 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.409 DEBUG:boto:Params: {} 2024-04-13T22:22:49.409 DEBUG:boto:Token: None 2024-04-13T22:22:49.410 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do= 2024-04-13T22:22:49.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do=', 'Content-Length': '0'} 2024-04-13T22:22:49.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d374ba9cdb3fb1c5-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.412 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.412 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.413 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.413 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.413 DEBUG:boto:Method: GET 2024-04-13T22:22:49.413 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.413 DEBUG:boto:Data: 2024-04-13T22:22:49.413 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.413 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.413 DEBUG:boto:Params: {} 2024-04-13T22:22:49.414 DEBUG:boto:Token: None 2024-04-13T22:22:49.414 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.414 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do= 2024-04-13T22:22:49.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do=', 'Content-Length': '0'} 2024-04-13T22:22:49.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f1aed20253b297-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.458 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.459 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.459 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.459 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.459 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.459 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.459 DEBUG:boto:Data: 2024-04-13T22:22:49.459 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.459 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.460 DEBUG:boto:Params: {} 2024-04-13T22:22:49.460 DEBUG:boto:Token: None 2024-04-13T22:22:49.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MvAfAIGd8Y758w0CiJUb+SdOh54= 2024-04-13T22:22:49.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MvAfAIGd8Y758w0CiJUb+SdOh54=', 'Content-Length': '0'} 2024-04-13T22:22:49.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd65e993e6149ab-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.463 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.463 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.463 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.463 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.464 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.464 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.464 DEBUG:boto:Data: 2024-04-13T22:22:49.464 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.464 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.464 DEBUG:boto:Params: {} 2024-04-13T22:22:49.464 DEBUG:boto:Token: None 2024-04-13T22:22:49.464 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MvAfAIGd8Y758w0CiJUb+SdOh54= 2024-04-13T22:22:49.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MvAfAIGd8Y758w0CiJUb+SdOh54=', 'Content-Length': '0'} 2024-04-13T22:22:49.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000298f0da03757154-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:22:49.467 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.467 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.467 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.467 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.467 DEBUG:boto:Method: GET 2024-04-13T22:22:49.467 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.467 DEBUG:boto:Data: 2024-04-13T22:22:49.467 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.468 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.468 DEBUG:boto:Params: {} 2024-04-13T22:22:49.468 DEBUG:boto:Token: None 2024-04-13T22:22:49.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do= 2024-04-13T22:22:49.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do=', 'Content-Length': '0'} 2024-04-13T22:22:49.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b86afb808da23eff-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.470 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.470 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.470 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.470 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.470 DEBUG:boto:Method: GET 2024-04-13T22:22:49.471 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.471 DEBUG:boto:Data: 2024-04-13T22:22:49.471 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.471 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.471 DEBUG:boto:Params: {} 2024-04-13T22:22:49.471 DEBUG:boto:Token: None 2024-04-13T22:22:49.471 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:22:49.471 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do= 2024-04-13T22:22:49.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:udP4LOmuZ9QmDQFk6V2A3i688do=', 'Content-Length': '0'} 2024-04-13T22:22:49.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7db08f4db14683-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.473 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.474 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.474 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.474 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.474 DEBUG:boto:Method: GET 2024-04-13T22:22:49.474 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.474 DEBUG:boto:Data: 2024-04-13T22:22:49.474 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.474 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.474 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.474 DEBUG:boto:Params: {} 2024-04-13T22:22:49.475 DEBUG:boto:Token: None 2024-04-13T22:22:49.475 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.475 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e5bc5b8bceb4f1-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.477 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.477 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.478 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.478 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.478 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.478 DEBUG:boto:Method: GET 2024-04-13T22:22:49.478 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.478 DEBUG:boto:Data: 2024-04-13T22:22:49.478 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.478 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.478 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.478 DEBUG:boto:Params: {} 2024-04-13T22:22:49.479 DEBUG:boto:Token: None 2024-04-13T22:22:49.479 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000839c3745fb731ec6-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.482 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.482 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.482 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.482 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.482 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:22:49.482 DEBUG:boto:Data: 2024-04-13T22:22:49.483 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.483 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.483 DEBUG:boto:Params: {} 2024-04-13T22:22:49.483 DEBUG:boto:Token: None 2024-04-13T22:22:49.483 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44 2024-04-13T22:22:49.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B04Wggbt8ZRaxeBECzGw+Hk/S44= 2024-04-13T22:22:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B04Wggbt8ZRaxeBECzGw+Hk/S44=', 'Content-Length': '0'} 2024-04-13T22:22:49.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028fde425c523d0ae-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.485 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.485 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.485 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.485 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:22:49.485 DEBUG:boto:Data: 2024-04-13T22:22:49.485 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.486 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.486 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.486 DEBUG:boto:Params: {} 2024-04-13T22:22:49.486 DEBUG:boto:Token: None 2024-04-13T22:22:49.486 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44 2024-04-13T22:22:49.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B04Wggbt8ZRaxeBECzGw+Hk/S44= 2024-04-13T22:22:49.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B04Wggbt8ZRaxeBECzGw+Hk/S44=', 'Content-Length': '0'} 2024-04-13T22:22:49.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435322280083a337-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:22:49.488 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.488 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.488 DEBUG:boto:Method: GET 2024-04-13T22:22:49.489 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:22:49.489 DEBUG:boto:Data: 2024-04-13T22:22:49.489 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.489 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.489 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.489 DEBUG:boto:Params: {} 2024-04-13T22:22:49.489 DEBUG:boto:Token: None 2024-04-13T22:22:49.489 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44 2024-04-13T22:22:49.489 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HkPmCg3SqQxCZ0PIzLx0HzscOoI= 2024-04-13T22:22:49.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HkPmCg3SqQxCZ0PIzLx0HzscOoI=', 'Content-Length': '0'} 2024-04-13T22:22:49.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009127371606888eca-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.491 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:22:49.491 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:22:49.491 DEBUG:boto:Method: GET 2024-04-13T22:22:49.492 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:22:49.492 DEBUG:boto:Data: 2024-04-13T22:22:49.492 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.492 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.492 DEBUG:boto:Params: {} 2024-04-13T22:22:49.492 DEBUG:boto:Token: None 2024-04-13T22:22:49.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj44 2024-04-13T22:22:49.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HkPmCg3SqQxCZ0PIzLx0HzscOoI= 2024-04-13T22:22:49.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HkPmCg3SqQxCZ0PIzLx0HzscOoI=', 'Content-Length': '0'} 2024-04-13T22:22:49.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cec817e62a39692-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.494 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.495 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.495 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.495 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.495 DEBUG:boto:Method: GET 2024-04-13T22:22:49.495 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.495 DEBUG:boto:Data: 2024-04-13T22:22:49.495 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.495 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.495 DEBUG:boto:Params: {} 2024-04-13T22:22:49.495 DEBUG:boto:Token: None 2024-04-13T22:22:49.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adfce8973623404d-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.498 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.498 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.498 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.499 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.499 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.499 DEBUG:boto:Method: GET 2024-04-13T22:22:49.499 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:22:49.499 DEBUG:boto:Data: 2024-04-13T22:22:49.499 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.499 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.499 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.499 DEBUG:boto:Params: {} 2024-04-13T22:22:49.499 DEBUG:boto:Token: None 2024-04-13T22:22:49.500 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987dee90857d4f3a-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.547 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:22:49.547 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.547 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.547 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.548 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.548 DEBUG:boto:Method: GET 2024-04-13T22:22:49.548 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.548 DEBUG:boto:Data: 2024-04-13T22:22:49.548 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.548 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.548 DEBUG:boto:Params: {} 2024-04-13T22:22:49.548 DEBUG:boto:Token: None 2024-04-13T22:22:49.549 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY= 2024-04-13T22:22:49.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY=', 'Content-Length': '0'} 2024-04-13T22:22:49.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5506f057213a571-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.551 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.551 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.551 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.551 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.551 DEBUG:boto:Method: GET 2024-04-13T22:22:49.551 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.551 DEBUG:boto:Data: 2024-04-13T22:22:49.551 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.551 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.552 DEBUG:boto:Params: {} 2024-04-13T22:22:49.552 DEBUG:boto:Token: None 2024-04-13T22:22:49.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY= 2024-04-13T22:22:49.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY=', 'Content-Length': '0'} 2024-04-13T22:22:49.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008089e35bb735da56-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.554 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.554 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.555 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.555 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.555 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.555 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.555 DEBUG:boto:Data: 2024-04-13T22:22:49.555 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.555 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.555 DEBUG:boto:Params: {} 2024-04-13T22:22:49.555 DEBUG:boto:Token: None 2024-04-13T22:22:49.556 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/xO+sSKw9SurP5jwoaZrjC1I/r0= 2024-04-13T22:22:49.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/xO+sSKw9SurP5jwoaZrjC1I/r0=', 'Content-Length': '0'} 2024-04-13T22:22:49.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004556c9f6f471141e-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.558 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.558 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.558 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.558 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.558 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.558 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.558 DEBUG:boto:Data: 2024-04-13T22:22:49.558 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.558 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.558 DEBUG:boto:Params: {} 2024-04-13T22:22:49.559 DEBUG:boto:Token: None 2024-04-13T22:22:49.559 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/xO+sSKw9SurP5jwoaZrjC1I/r0= 2024-04-13T22:22:49.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/xO+sSKw9SurP5jwoaZrjC1I/r0=', 'Content-Length': '0'} 2024-04-13T22:22:49.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053be2832038a90f2-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:22:49.561 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.561 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.561 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.561 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.561 DEBUG:boto:Method: GET 2024-04-13T22:22:49.561 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.561 DEBUG:boto:Data: 2024-04-13T22:22:49.561 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.562 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.562 DEBUG:boto:Params: {} 2024-04-13T22:22:49.562 DEBUG:boto:Token: None 2024-04-13T22:22:49.562 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY= 2024-04-13T22:22:49.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY=', 'Content-Length': '0'} 2024-04-13T22:22:49.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071afcf2a53d65ea4-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.610 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.611 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.611 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.611 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.611 DEBUG:boto:Method: GET 2024-04-13T22:22:49.611 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.611 DEBUG:boto:Data: 2024-04-13T22:22:49.611 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.611 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.611 DEBUG:boto:Params: {} 2024-04-13T22:22:49.612 DEBUG:boto:Token: None 2024-04-13T22:22:49.612 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:22:49.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY= 2024-04-13T22:22:49.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWJpDvL62mYuv8EOIuAhU/HUyqY=', 'Content-Length': '0'} 2024-04-13T22:22:49.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b81f10411e4ebfd6-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.614 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.614 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.614 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.614 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.614 DEBUG:boto:Method: GET 2024-04-13T22:22:49.615 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.615 DEBUG:boto:Data: 2024-04-13T22:22:49.615 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.615 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.615 DEBUG:boto:Params: {} 2024-04-13T22:22:49.615 DEBUG:boto:Token: None 2024-04-13T22:22:49.615 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642f866d47885183-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.618 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.618 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.618 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.618 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.618 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.619 DEBUG:boto:Method: GET 2024-04-13T22:22:49.619 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.619 DEBUG:boto:Data: 2024-04-13T22:22:49.619 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.619 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.619 DEBUG:boto:Params: {} 2024-04-13T22:22:49.619 DEBUG:boto:Token: None 2024-04-13T22:22:49.619 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014e965667a5b1a3a-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.622 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.623 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.623 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.623 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.623 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:22:49.623 DEBUG:boto:Data: 2024-04-13T22:22:49.623 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.624 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.624 DEBUG:boto:Params: {} 2024-04-13T22:22:49.624 DEBUG:boto:Token: None 2024-04-13T22:22:49.624 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51 2024-04-13T22:22:49.624 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aGzmxA0zccqtWf9JccdXT7x1fng= 2024-04-13T22:22:49.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aGzmxA0zccqtWf9JccdXT7x1fng=', 'Content-Length': '0'} 2024-04-13T22:22:49.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6c7123313df1a98-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.626 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.626 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.626 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.626 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:22:49.626 DEBUG:boto:Data: 2024-04-13T22:22:49.626 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.626 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.626 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.627 DEBUG:boto:Params: {} 2024-04-13T22:22:49.627 DEBUG:boto:Token: None 2024-04-13T22:22:49.627 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51 2024-04-13T22:22:49.627 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aGzmxA0zccqtWf9JccdXT7x1fng= 2024-04-13T22:22:49.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aGzmxA0zccqtWf9JccdXT7x1fng=', 'Content-Length': '0'} 2024-04-13T22:22:49.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3eee982a4a12b0-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:22:49.629 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.629 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.629 DEBUG:boto:Method: GET 2024-04-13T22:22:49.629 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:22:49.629 DEBUG:boto:Data: 2024-04-13T22:22:49.629 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.629 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.629 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.629 DEBUG:boto:Params: {} 2024-04-13T22:22:49.630 DEBUG:boto:Token: None 2024-04-13T22:22:49.630 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51 2024-04-13T22:22:49.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DvFSdBrTHO7YPP8x0y+ZLtZGpB0= 2024-04-13T22:22:49.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DvFSdBrTHO7YPP8x0y+ZLtZGpB0=', 'Content-Length': '0'} 2024-04-13T22:22:49.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068f1b39204c51da-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.632 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:22:49.632 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:22:49.632 DEBUG:boto:Method: GET 2024-04-13T22:22:49.632 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:22:49.632 DEBUG:boto:Data: 2024-04-13T22:22:49.632 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.632 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.632 DEBUG:boto:Params: {} 2024-04-13T22:22:49.633 DEBUG:boto:Token: None 2024-04-13T22:22:49.633 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj51 2024-04-13T22:22:49.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DvFSdBrTHO7YPP8x0y+ZLtZGpB0= 2024-04-13T22:22:49.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DvFSdBrTHO7YPP8x0y+ZLtZGpB0=', 'Content-Length': '0'} 2024-04-13T22:22:49.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a39880c8d3fe8873-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.678 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.679 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.679 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.679 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.679 DEBUG:boto:Method: GET 2024-04-13T22:22:49.679 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.679 DEBUG:boto:Data: 2024-04-13T22:22:49.679 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.679 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.679 DEBUG:boto:Params: {} 2024-04-13T22:22:49.680 DEBUG:boto:Token: None 2024-04-13T22:22:49.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c8dd7c0b694078-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.682 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.683 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.683 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.683 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.683 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.683 DEBUG:boto:Method: GET 2024-04-13T22:22:49.683 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:22:49.684 DEBUG:boto:Data: 2024-04-13T22:22:49.684 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.684 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.684 DEBUG:boto:Params: {} 2024-04-13T22:22:49.684 DEBUG:boto:Token: None 2024-04-13T22:22:49.684 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f6c7ef6da55dcc0-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.687 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:22:49.688 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.688 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.688 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.688 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.688 DEBUG:boto:Method: GET 2024-04-13T22:22:49.688 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.688 DEBUG:boto:Data: 2024-04-13T22:22:49.688 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.689 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.689 DEBUG:boto:Params: {} 2024-04-13T22:22:49.689 DEBUG:boto:Token: None 2024-04-13T22:22:49.689 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8= 2024-04-13T22:22:49.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8=', 'Content-Length': '0'} 2024-04-13T22:22:49.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d44454da534bb48-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.691 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.691 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.691 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.691 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.692 DEBUG:boto:Method: GET 2024-04-13T22:22:49.692 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.692 DEBUG:boto:Data: 2024-04-13T22:22:49.692 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.692 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.692 DEBUG:boto:Params: {} 2024-04-13T22:22:49.692 DEBUG:boto:Token: None 2024-04-13T22:22:49.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.693 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8= 2024-04-13T22:22:49.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8=', 'Content-Length': '0'} 2024-04-13T22:22:49.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c402e9e227f53c-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.694 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.695 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.695 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.695 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.695 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.695 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.695 DEBUG:boto:Data: 2024-04-13T22:22:49.695 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.695 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.695 DEBUG:boto:Params: {} 2024-04-13T22:22:49.696 DEBUG:boto:Token: None 2024-04-13T22:22:49.696 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FO/IUAEgAxshRzo9PgttUdlEIW0= 2024-04-13T22:22:49.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FO/IUAEgAxshRzo9PgttUdlEIW0=', 'Content-Length': '0'} 2024-04-13T22:22:49.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007600c224ff92cefa-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.697 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.698 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.698 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.698 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.698 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.698 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.698 DEBUG:boto:Data: 2024-04-13T22:22:49.698 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.698 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.698 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.698 DEBUG:boto:Params: {} 2024-04-13T22:22:49.699 DEBUG:boto:Token: None 2024-04-13T22:22:49.699 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FO/IUAEgAxshRzo9PgttUdlEIW0= 2024-04-13T22:22:49.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FO/IUAEgAxshRzo9PgttUdlEIW0=', 'Content-Length': '0'} 2024-04-13T22:22:49.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279368ca5f3c5447-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:22:49.701 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.701 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.701 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.701 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.701 DEBUG:boto:Method: GET 2024-04-13T22:22:49.701 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.701 DEBUG:boto:Data: 2024-04-13T22:22:49.701 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.701 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.701 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.702 DEBUG:boto:Params: {} 2024-04-13T22:22:49.702 DEBUG:boto:Token: None 2024-04-13T22:22:49.702 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8= 2024-04-13T22:22:49.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8=', 'Content-Length': '0'} 2024-04-13T22:22:49.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c4392d2bfe47b8-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.704 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.704 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.704 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.704 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.704 DEBUG:boto:Method: GET 2024-04-13T22:22:49.704 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.704 DEBUG:boto:Data: 2024-04-13T22:22:49.704 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.705 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.705 DEBUG:boto:Params: {} 2024-04-13T22:22:49.705 DEBUG:boto:Token: None 2024-04-13T22:22:49.705 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:22:49.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8= 2024-04-13T22:22:49.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yMCfGXaCRk/lXDFmZMlCf0LR2n8=', 'Content-Length': '0'} 2024-04-13T22:22:49.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559f2831b19492e0-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.750 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.751 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.751 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.751 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.751 DEBUG:boto:Method: GET 2024-04-13T22:22:49.751 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.751 DEBUG:boto:Data: 2024-04-13T22:22:49.751 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.751 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.752 DEBUG:boto:Params: {} 2024-04-13T22:22:49.752 DEBUG:boto:Token: None 2024-04-13T22:22:49.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000de0b4617f425891-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.755 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.755 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.755 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.755 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.755 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.755 DEBUG:boto:Method: GET 2024-04-13T22:22:49.755 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.756 DEBUG:boto:Data: 2024-04-13T22:22:49.756 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.756 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.756 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.756 DEBUG:boto:Params: {} 2024-04-13T22:22:49.756 DEBUG:boto:Token: None 2024-04-13T22:22:49.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0593692e0ba65b1-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.759 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.759 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.759 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.760 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.760 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:22:49.760 DEBUG:boto:Data: 2024-04-13T22:22:49.760 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.760 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.760 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.760 DEBUG:boto:Params: {} 2024-04-13T22:22:49.760 DEBUG:boto:Token: None 2024-04-13T22:22:49.760 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55 2024-04-13T22:22:49.760 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sBeIk3fiDMywKOkXtK2u7C709VA= 2024-04-13T22:22:49.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sBeIk3fiDMywKOkXtK2u7C709VA=', 'Content-Length': '0'} 2024-04-13T22:22:49.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62731e4d5dfb54a-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.763 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.763 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.763 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.763 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:22:49.763 DEBUG:boto:Data: 2024-04-13T22:22:49.763 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.763 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.763 DEBUG:boto:Params: {} 2024-04-13T22:22:49.764 DEBUG:boto:Token: None 2024-04-13T22:22:49.764 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55 2024-04-13T22:22:49.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sBeIk3fiDMywKOkXtK2u7C709VA= 2024-04-13T22:22:49.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sBeIk3fiDMywKOkXtK2u7C709VA=', 'Content-Length': '0'} 2024-04-13T22:22:49.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcea45dd7d7b597e-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:22:49.766 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.766 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.766 DEBUG:boto:Method: GET 2024-04-13T22:22:49.766 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:22:49.766 DEBUG:boto:Data: 2024-04-13T22:22:49.766 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.766 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.766 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.766 DEBUG:boto:Params: {} 2024-04-13T22:22:49.767 DEBUG:boto:Token: None 2024-04-13T22:22:49.767 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55 2024-04-13T22:22:49.767 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vaHrl6ColUfX25cYrZEGJd5vd+A= 2024-04-13T22:22:49.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vaHrl6ColUfX25cYrZEGJd5vd+A=', 'Content-Length': '0'} 2024-04-13T22:22:49.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4891cc919609798-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.810 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:22:49.811 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:22:49.811 DEBUG:boto:Method: GET 2024-04-13T22:22:49.811 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:22:49.811 DEBUG:boto:Data: 2024-04-13T22:22:49.811 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.811 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.811 DEBUG:boto:Params: {} 2024-04-13T22:22:49.811 DEBUG:boto:Token: None 2024-04-13T22:22:49.811 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj55 2024-04-13T22:22:49.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vaHrl6ColUfX25cYrZEGJd5vd+A= 2024-04-13T22:22:49.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vaHrl6ColUfX25cYrZEGJd5vd+A=', 'Content-Length': '0'} 2024-04-13T22:22:49.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009615d30e8d51d32e-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.814 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.814 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.814 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.814 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.814 DEBUG:boto:Method: GET 2024-04-13T22:22:49.814 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.814 DEBUG:boto:Data: 2024-04-13T22:22:49.814 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.815 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.815 DEBUG:boto:Params: {} 2024-04-13T22:22:49.815 DEBUG:boto:Token: None 2024-04-13T22:22:49.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cff927399d1b28ef-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.817 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.818 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.818 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.818 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.818 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.818 DEBUG:boto:Method: GET 2024-04-13T22:22:49.818 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:22:49.818 DEBUG:boto:Data: 2024-04-13T22:22:49.818 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.819 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.819 DEBUG:boto:Params: {} 2024-04-13T22:22:49.819 DEBUG:boto:Token: None 2024-04-13T22:22:49.819 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19c385527fe7a87-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.822 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:22:49.823 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.823 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.823 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.823 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.823 DEBUG:boto:Method: GET 2024-04-13T22:22:49.823 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.823 DEBUG:boto:Data: 2024-04-13T22:22:49.823 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.823 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.823 DEBUG:boto:Params: {} 2024-04-13T22:22:49.824 DEBUG:boto:Token: None 2024-04-13T22:22:49.824 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg= 2024-04-13T22:22:49.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg=', 'Content-Length': '0'} 2024-04-13T22:22:49.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857455969b6f156f-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.826 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.826 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.826 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.826 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.826 DEBUG:boto:Method: GET 2024-04-13T22:22:49.826 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.826 DEBUG:boto:Data: 2024-04-13T22:22:49.826 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.827 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.827 DEBUG:boto:Params: {} 2024-04-13T22:22:49.827 DEBUG:boto:Token: None 2024-04-13T22:22:49.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg= 2024-04-13T22:22:49.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg=', 'Content-Length': '0'} 2024-04-13T22:22:49.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000894608b204b3f1cd-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.829 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.829 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.829 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.829 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.829 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.830 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.830 DEBUG:boto:Data: 2024-04-13T22:22:49.830 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.830 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.830 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.830 DEBUG:boto:Params: {} 2024-04-13T22:22:49.830 DEBUG:boto:Token: None 2024-04-13T22:22:49.830 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.830 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5XeGStsFMEAamOrEvsdUoIq4OTs= 2024-04-13T22:22:49.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5XeGStsFMEAamOrEvsdUoIq4OTs=', 'Content-Length': '0'} 2024-04-13T22:22:49.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ccf460164bf6d52-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.832 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.832 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.832 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.832 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.833 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.833 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.833 DEBUG:boto:Data: 2024-04-13T22:22:49.833 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.833 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.833 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.833 DEBUG:boto:Params: {} 2024-04-13T22:22:49.833 DEBUG:boto:Token: None 2024-04-13T22:22:49.833 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.833 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5XeGStsFMEAamOrEvsdUoIq4OTs= 2024-04-13T22:22:49.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5XeGStsFMEAamOrEvsdUoIq4OTs=', 'Content-Length': '0'} 2024-04-13T22:22:49.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc57f7c215557b7-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:22:49.835 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.835 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.835 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.835 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.836 DEBUG:boto:Method: GET 2024-04-13T22:22:49.836 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.836 DEBUG:boto:Data: 2024-04-13T22:22:49.836 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.836 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.836 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.836 DEBUG:boto:Params: {} 2024-04-13T22:22:49.836 DEBUG:boto:Token: None 2024-04-13T22:22:49.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg= 2024-04-13T22:22:49.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg=', 'Content-Length': '0'} 2024-04-13T22:22:49.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084402c5d9142feb9-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.838 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.838 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.838 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.839 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.839 DEBUG:boto:Method: GET 2024-04-13T22:22:49.839 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.839 DEBUG:boto:Data: 2024-04-13T22:22:49.839 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.839 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.839 DEBUG:boto:Params: {} 2024-04-13T22:22:49.839 DEBUG:boto:Token: None 2024-04-13T22:22:49.839 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:22:49.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg= 2024-04-13T22:22:49.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:psHXCxYMYJcpIWSrvduy3kvYerg=', 'Content-Length': '0'} 2024-04-13T22:22:49.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851fe5b24dc4cf8b-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.882 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.883 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.883 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.883 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.883 DEBUG:boto:Method: GET 2024-04-13T22:22:49.883 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.883 DEBUG:boto:Data: 2024-04-13T22:22:49.883 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.883 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.883 DEBUG:boto:Params: {} 2024-04-13T22:22:49.884 DEBUG:boto:Token: None 2024-04-13T22:22:49.884 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e8bb690b2da505-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.886 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.887 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.887 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.887 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.887 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.887 DEBUG:boto:Method: GET 2024-04-13T22:22:49.887 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.887 DEBUG:boto:Data: 2024-04-13T22:22:49.888 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.888 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.888 DEBUG:boto:Params: {} 2024-04-13T22:22:49.888 DEBUG:boto:Token: None 2024-04-13T22:22:49.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cea9d451a4c07887-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.891 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.891 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.891 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.891 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.891 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:22:49.891 DEBUG:boto:Data: 2024-04-13T22:22:49.891 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.892 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.892 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.892 DEBUG:boto:Params: {} 2024-04-13T22:22:49.892 DEBUG:boto:Token: None 2024-04-13T22:22:49.892 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57 2024-04-13T22:22:49.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ti3TIEtfUQzuY7kZ8CsRwnhaEjM= 2024-04-13T22:22:49.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ti3TIEtfUQzuY7kZ8CsRwnhaEjM=', 'Content-Length': '0'} 2024-04-13T22:22:49.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005dbcb692a9fcf02-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.894 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.894 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.895 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.895 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:22:49.895 DEBUG:boto:Data: 2024-04-13T22:22:49.895 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.895 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.895 DEBUG:boto:Params: {} 2024-04-13T22:22:49.895 DEBUG:boto:Token: None 2024-04-13T22:22:49.895 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57 2024-04-13T22:22:49.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ti3TIEtfUQzuY7kZ8CsRwnhaEjM= 2024-04-13T22:22:49.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ti3TIEtfUQzuY7kZ8CsRwnhaEjM=', 'Content-Length': '0'} 2024-04-13T22:22:49.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000704d1df5eca1468b-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:22:49.898 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.898 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.898 DEBUG:boto:Method: GET 2024-04-13T22:22:49.898 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:22:49.898 DEBUG:boto:Data: 2024-04-13T22:22:49.898 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.898 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.898 DEBUG:boto:Params: {} 2024-04-13T22:22:49.898 DEBUG:boto:Token: None 2024-04-13T22:22:49.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57 2024-04-13T22:22:49.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ezhU/VscWqeBhKXzafpuAw0Fjho= 2024-04-13T22:22:49.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ezhU/VscWqeBhKXzafpuAw0Fjho=', 'Content-Length': '0'} 2024-04-13T22:22:49.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac04a9579c03776d-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.901 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:22:49.901 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:22:49.901 DEBUG:boto:Method: GET 2024-04-13T22:22:49.901 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:22:49.901 DEBUG:boto:Data: 2024-04-13T22:22:49.901 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.901 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.901 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.901 DEBUG:boto:Params: {} 2024-04-13T22:22:49.902 DEBUG:boto:Token: None 2024-04-13T22:22:49.902 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj57 2024-04-13T22:22:49.902 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ezhU/VscWqeBhKXzafpuAw0Fjho= 2024-04-13T22:22:49.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ezhU/VscWqeBhKXzafpuAw0Fjho=', 'Content-Length': '0'} 2024-04-13T22:22:49.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777a415d18cb3ead-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.904 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.904 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.904 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.904 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.904 DEBUG:boto:Method: GET 2024-04-13T22:22:49.904 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.904 DEBUG:boto:Data: 2024-04-13T22:22:49.904 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.904 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.905 DEBUG:boto:Params: {} 2024-04-13T22:22:49.905 DEBUG:boto:Token: None 2024-04-13T22:22:49.905 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000184a6eb51c2af9e2-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.907 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.908 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.908 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.908 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.908 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.908 DEBUG:boto:Method: GET 2024-04-13T22:22:49.908 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:22:49.908 DEBUG:boto:Data: 2024-04-13T22:22:49.908 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.909 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.909 DEBUG:boto:Params: {} 2024-04-13T22:22:49.909 DEBUG:boto:Token: None 2024-04-13T22:22:49.909 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a38168a07e9ad9-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.912 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:22:49.912 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:49.912 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:49.912 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.913 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.913 DEBUG:boto:Method: GET 2024-04-13T22:22:49.913 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.913 DEBUG:boto:Data: 2024-04-13T22:22:49.913 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.913 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.913 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.913 DEBUG:boto:Params: {} 2024-04-13T22:22:49.913 DEBUG:boto:Token: None 2024-04-13T22:22:49.913 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.914 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek= 2024-04-13T22:22:49.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek=', 'Content-Length': '0'} 2024-04-13T22:22:49.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008aa24d31206e887-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.962 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:49.963 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:49.963 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.963 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.963 DEBUG:boto:Method: GET 2024-04-13T22:22:49.963 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.963 DEBUG:boto:Data: 2024-04-13T22:22:49.963 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.963 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.963 DEBUG:boto:Params: {} 2024-04-13T22:22:49.964 DEBUG:boto:Token: None 2024-04-13T22:22:49.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek= 2024-04-13T22:22:49.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek=', 'Content-Length': '0'} 2024-04-13T22:22:49.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000059a604f1de88b3b-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.966 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:49.966 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:49.966 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.967 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.967 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.967 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.967 DEBUG:boto:Data: 2024-04-13T22:22:49.967 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.967 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.967 DEBUG:boto:Params: {} 2024-04-13T22:22:49.967 DEBUG:boto:Token: None 2024-04-13T22:22:49.967 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6LMsSS7nvh99wlDHKVsCiWhqVeQ= 2024-04-13T22:22:49.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6LMsSS7nvh99wlDHKVsCiWhqVeQ=', 'Content-Length': '0'} 2024-04-13T22:22:49.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab67f542be90a9ba-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.969 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:49.970 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:49.970 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.970 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.970 DEBUG:boto:Method: HEAD 2024-04-13T22:22:49.970 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.970 DEBUG:boto:Data: 2024-04-13T22:22:49.970 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.970 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.970 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.970 DEBUG:boto:Params: {} 2024-04-13T22:22:49.970 DEBUG:boto:Token: None 2024-04-13T22:22:49.971 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6LMsSS7nvh99wlDHKVsCiWhqVeQ= 2024-04-13T22:22:49.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6LMsSS7nvh99wlDHKVsCiWhqVeQ=', 'Content-Length': '0'} 2024-04-13T22:22:49.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a9552b5d87e8ef-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:22:49.973 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:49.973 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:49.973 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.973 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.973 DEBUG:boto:Method: GET 2024-04-13T22:22:49.973 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.973 DEBUG:boto:Data: 2024-04-13T22:22:49.973 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.973 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.974 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.974 DEBUG:boto:Params: {} 2024-04-13T22:22:49.974 DEBUG:boto:Token: None 2024-04-13T22:22:49.974 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.974 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek= 2024-04-13T22:22:49.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek=', 'Content-Length': '0'} 2024-04-13T22:22:49.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8aea64b619812a9-00661b05b9-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.976 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:49.976 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:49.976 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.976 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.976 DEBUG:boto:Method: GET 2024-04-13T22:22:49.976 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.976 DEBUG:boto:Data: 2024-04-13T22:22:49.977 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.977 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.977 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.977 DEBUG:boto:Params: {} 2024-04-13T22:22:49.977 DEBUG:boto:Token: None 2024-04-13T22:22:49.977 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:22:49.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek= 2024-04-13T22:22:49.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xkyUk6i1YtwkZ/ZX1byt/t8VDek=', 'Content-Length': '0'} 2024-04-13T22:22:49.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae635dd541fbe05-00661b05b9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.979 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.979 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.979 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:49.979 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:49.979 DEBUG:boto:Method: GET 2024-04-13T22:22:49.979 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:22:49.979 DEBUG:boto:Data: 2024-04-13T22:22:49.980 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.980 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:49.980 DEBUG:boto:Port: 8001 2024-04-13T22:22:49.980 DEBUG:boto:Params: {} 2024-04-13T22:22:49.980 DEBUG:boto:Token: None 2024-04-13T22:22:49.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7eb0e4012403c0d-00661b05b9-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:49.982 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:49.983 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:49.983 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:49.983 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:49.983 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:49.983 DEBUG:boto:Method: GET 2024-04-13T22:22:49.983 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:22:49.983 DEBUG:boto:Data: 2024-04-13T22:22:49.983 DEBUG:boto:Headers: {} 2024-04-13T22:22:49.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:49.984 DEBUG:boto:Port: 8000 2024-04-13T22:22:49.984 DEBUG:boto:Params: {} 2024-04-13T22:22:49.984 DEBUG:boto:Token: None 2024-04-13T22:22:49.984 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:49 GMT /vwnfns-61/ 2024-04-13T22:22:49.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4= 2024-04-13T22:22:49.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bO2GGbzHDhJLiv+1CR/m7Fv9JJ4=', 'Content-Length': '0'} 2024-04-13T22:22:49.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3080b88de3e3c06-00661b05b9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.034 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.035 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:50.035 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:50.035 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.035 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:22:50.035 DEBUG:boto:Data: 2024-04-13T22:22:50.035 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.036 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.036 DEBUG:boto:Params: {} 2024-04-13T22:22:50.036 DEBUG:boto:Token: None 2024-04-13T22:22:50.036 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj68 2024-04-13T22:22:50.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tycUVZUqQFqt3CYAnXe7WDCv5MY= 2024-04-13T22:22:50.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tycUVZUqQFqt3CYAnXe7WDCv5MY=', 'Content-Length': '0'} 2024-04-13T22:22:50.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0486063fe585bf-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.038 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:50.038 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:50.039 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.039 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:22:50.039 DEBUG:boto:Data: 2024-04-13T22:22:50.039 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.039 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.039 DEBUG:boto:Params: {} 2024-04-13T22:22:50.039 DEBUG:boto:Token: None 2024-04-13T22:22:50.039 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj68 2024-04-13T22:22:50.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tycUVZUqQFqt3CYAnXe7WDCv5MY= 2024-04-13T22:22:50.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tycUVZUqQFqt3CYAnXe7WDCv5MY=', 'Content-Length': '0'} 2024-04-13T22:22:50.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c88b3b97b7c11e7c-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:22:50.041 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:50.041 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:50.042 DEBUG:boto:Method: GET 2024-04-13T22:22:50.042 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:22:50.042 DEBUG:boto:Data: 2024-04-13T22:22:50.042 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.042 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.042 DEBUG:boto:Params: {} 2024-04-13T22:22:50.042 DEBUG:boto:Token: None 2024-04-13T22:22:50.042 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj68 2024-04-13T22:22:50.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sPufJCGFaYdZCIBxQc3CJKW5df4= 2024-04-13T22:22:50.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sPufJCGFaYdZCIBxQc3CJKW5df4=', 'Content-Length': '0'} 2024-04-13T22:22:50.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879dcf5774ba0c71-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.044 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:22:50.044 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:22:50.045 DEBUG:boto:Method: GET 2024-04-13T22:22:50.045 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:22:50.045 DEBUG:boto:Data: 2024-04-13T22:22:50.045 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.045 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.045 DEBUG:boto:Params: {} 2024-04-13T22:22:50.045 DEBUG:boto:Token: None 2024-04-13T22:22:50.045 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj68 2024-04-13T22:22:50.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sPufJCGFaYdZCIBxQc3CJKW5df4= 2024-04-13T22:22:50.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sPufJCGFaYdZCIBxQc3CJKW5df4=', 'Content-Length': '0'} 2024-04-13T22:22:50.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f878d547abaeb2-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.047 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.048 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.048 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:50.048 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:50.048 DEBUG:boto:Method: GET 2024-04-13T22:22:50.048 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:22:50.048 DEBUG:boto:Data: 2024-04-13T22:22:50.048 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.048 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.048 DEBUG:boto:Params: {} 2024-04-13T22:22:50.048 DEBUG:boto:Token: None 2024-04-13T22:22:50.049 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6fd503c654be60f-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.051 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.051 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.051 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.051 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:50.051 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:22:50.052 DEBUG:boto:Method: GET 2024-04-13T22:22:50.052 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:22:50.052 DEBUG:boto:Data: 2024-04-13T22:22:50.052 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.052 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.052 DEBUG:boto:Params: {} 2024-04-13T22:22:50.052 DEBUG:boto:Token: None 2024-04-13T22:22:50.052 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085f4e2ba08d10cb6-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.055 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:22:50.056 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.056 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.056 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.056 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.056 DEBUG:boto:Method: GET 2024-04-13T22:22:50.056 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.056 DEBUG:boto:Data: 2024-04-13T22:22:50.056 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.056 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.056 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.056 DEBUG:boto:Params: {} 2024-04-13T22:22:50.057 DEBUG:boto:Token: None 2024-04-13T22:22:50.057 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.057 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg= 2024-04-13T22:22:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg=', 'Content-Length': '0'} 2024-04-13T22:22:50.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4235dd20b3e4a41-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.059 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.059 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.059 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.059 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.059 DEBUG:boto:Method: GET 2024-04-13T22:22:50.059 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.060 DEBUG:boto:Data: 2024-04-13T22:22:50.060 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.060 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.060 DEBUG:boto:Params: {} 2024-04-13T22:22:50.060 DEBUG:boto:Token: None 2024-04-13T22:22:50.060 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg= 2024-04-13T22:22:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg=', 'Content-Length': '0'} 2024-04-13T22:22:50.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5053525bdb739fa-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.111 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.111 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.111 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.111 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.111 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.111 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.111 DEBUG:boto:Data: 2024-04-13T22:22:50.112 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.112 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.112 DEBUG:boto:Params: {} 2024-04-13T22:22:50.112 DEBUG:boto:Token: None 2024-04-13T22:22:50.112 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7rfZIxpdrLnFOnnxeID1O2pFUIU= 2024-04-13T22:22:50.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7rfZIxpdrLnFOnnxeID1O2pFUIU=', 'Content-Length': '0'} 2024-04-13T22:22:50.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002902733a9bacebc5-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.114 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.114 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.114 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.114 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.115 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.115 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.115 DEBUG:boto:Data: 2024-04-13T22:22:50.115 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.115 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.115 DEBUG:boto:Params: {} 2024-04-13T22:22:50.115 DEBUG:boto:Token: None 2024-04-13T22:22:50.115 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7rfZIxpdrLnFOnnxeID1O2pFUIU= 2024-04-13T22:22:50.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7rfZIxpdrLnFOnnxeID1O2pFUIU=', 'Content-Length': '0'} 2024-04-13T22:22:50.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ba800d977033ed-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:22:50.117 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.118 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.118 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.118 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.118 DEBUG:boto:Method: GET 2024-04-13T22:22:50.118 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.118 DEBUG:boto:Data: 2024-04-13T22:22:50.118 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.118 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.119 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.119 DEBUG:boto:Params: {} 2024-04-13T22:22:50.119 DEBUG:boto:Token: None 2024-04-13T22:22:50.119 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.119 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg= 2024-04-13T22:22:50.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg=', 'Content-Length': '0'} 2024-04-13T22:22:50.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d89a3ea78aac13b-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.121 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.121 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.121 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.121 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.122 DEBUG:boto:Method: GET 2024-04-13T22:22:50.122 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.122 DEBUG:boto:Data: 2024-04-13T22:22:50.122 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.122 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.122 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.122 DEBUG:boto:Params: {} 2024-04-13T22:22:50.122 DEBUG:boto:Token: None 2024-04-13T22:22:50.122 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:22:50.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg= 2024-04-13T22:22:50.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:macBuZG11NCXxXiSbObv3l4c0tg=', 'Content-Length': '0'} 2024-04-13T22:22:50.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce8be6874b9f6ca-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.124 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.124 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.125 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.125 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.125 DEBUG:boto:Method: GET 2024-04-13T22:22:50.125 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.125 DEBUG:boto:Data: 2024-04-13T22:22:50.125 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.125 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.125 DEBUG:boto:Params: {} 2024-04-13T22:22:50.125 DEBUG:boto:Token: None 2024-04-13T22:22:50.125 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b9bea62a40482b7-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.128 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.128 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.129 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.129 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.129 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.129 DEBUG:boto:Method: GET 2024-04-13T22:22:50.129 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.129 DEBUG:boto:Data: 2024-04-13T22:22:50.129 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.129 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.129 DEBUG:boto:Params: {} 2024-04-13T22:22:50.129 DEBUG:boto:Token: None 2024-04-13T22:22:50.130 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.130 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca198a217d4c9af-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.133 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.134 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.134 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.134 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.134 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:22:50.134 DEBUG:boto:Data: 2024-04-13T22:22:50.134 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.134 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.135 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.135 DEBUG:boto:Params: {} 2024-04-13T22:22:50.135 DEBUG:boto:Token: None 2024-04-13T22:22:50.135 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76 2024-04-13T22:22:50.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5HKJgEhE5a7X6WtNHK45mX689vk= 2024-04-13T22:22:50.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5HKJgEhE5a7X6WtNHK45mX689vk=', 'Content-Length': '0'} 2024-04-13T22:22:50.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ec94a5d475f81f-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.137 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.137 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.137 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.137 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:22:50.137 DEBUG:boto:Data: 2024-04-13T22:22:50.138 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.138 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.138 DEBUG:boto:Params: {} 2024-04-13T22:22:50.138 DEBUG:boto:Token: None 2024-04-13T22:22:50.138 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76 2024-04-13T22:22:50.138 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5HKJgEhE5a7X6WtNHK45mX689vk= 2024-04-13T22:22:50.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5HKJgEhE5a7X6WtNHK45mX689vk=', 'Content-Length': '0'} 2024-04-13T22:22:50.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d023beba74f1f765-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:22:50.140 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.141 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.141 DEBUG:boto:Method: GET 2024-04-13T22:22:50.141 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:22:50.141 DEBUG:boto:Data: 2024-04-13T22:22:50.141 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.141 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.141 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.141 DEBUG:boto:Params: {} 2024-04-13T22:22:50.141 DEBUG:boto:Token: None 2024-04-13T22:22:50.141 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76 2024-04-13T22:22:50.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q1s6kyd1O0fsCeunv2HA3edw3zg= 2024-04-13T22:22:50.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q1s6kyd1O0fsCeunv2HA3edw3zg=', 'Content-Length': '0'} 2024-04-13T22:22:50.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7fe0afd6f1253d1-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.144 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:22:50.144 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:22:50.144 DEBUG:boto:Method: GET 2024-04-13T22:22:50.144 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:22:50.144 DEBUG:boto:Data: 2024-04-13T22:22:50.144 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.144 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.144 DEBUG:boto:Params: {} 2024-04-13T22:22:50.145 DEBUG:boto:Token: None 2024-04-13T22:22:50.145 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj76 2024-04-13T22:22:50.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q1s6kyd1O0fsCeunv2HA3edw3zg= 2024-04-13T22:22:50.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q1s6kyd1O0fsCeunv2HA3edw3zg=', 'Content-Length': '0'} 2024-04-13T22:22:50.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b8917727cb3a61-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.147 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.147 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.147 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.147 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.147 DEBUG:boto:Method: GET 2024-04-13T22:22:50.147 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.147 DEBUG:boto:Data: 2024-04-13T22:22:50.147 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.147 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.147 DEBUG:boto:Params: {} 2024-04-13T22:22:50.148 DEBUG:boto:Token: None 2024-04-13T22:22:50.148 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fcf3c269db06f16-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.150 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.151 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.151 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.151 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.151 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.151 DEBUG:boto:Method: GET 2024-04-13T22:22:50.151 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:22:50.151 DEBUG:boto:Data: 2024-04-13T22:22:50.151 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.152 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.152 DEBUG:boto:Params: {} 2024-04-13T22:22:50.152 DEBUG:boto:Token: None 2024-04-13T22:22:50.152 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6813dc903fd1334-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.155 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:22:50.155 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.155 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.155 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.155 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.155 DEBUG:boto:Method: GET 2024-04-13T22:22:50.156 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.156 DEBUG:boto:Data: 2024-04-13T22:22:50.156 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.156 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.156 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.156 DEBUG:boto:Params: {} 2024-04-13T22:22:50.156 DEBUG:boto:Token: None 2024-04-13T22:22:50.156 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0= 2024-04-13T22:22:50.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0=', 'Content-Length': '0'} 2024-04-13T22:22:50.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd65e369e2d6d0b2-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.198 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.199 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.199 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.199 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.199 DEBUG:boto:Method: GET 2024-04-13T22:22:50.199 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.199 DEBUG:boto:Data: 2024-04-13T22:22:50.199 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.199 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.200 DEBUG:boto:Params: {} 2024-04-13T22:22:50.200 DEBUG:boto:Token: None 2024-04-13T22:22:50.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0= 2024-04-13T22:22:50.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0=', 'Content-Length': '0'} 2024-04-13T22:22:50.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecf3ef6c520b800d-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.202 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.202 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.202 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.202 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.203 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.203 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.203 DEBUG:boto:Data: 2024-04-13T22:22:50.203 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.203 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.203 DEBUG:boto:Params: {} 2024-04-13T22:22:50.203 DEBUG:boto:Token: None 2024-04-13T22:22:50.203 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.203 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UPVHd3LHfI2DWmlNjBUyvU4paSo= 2024-04-13T22:22:50.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UPVHd3LHfI2DWmlNjBUyvU4paSo=', 'Content-Length': '0'} 2024-04-13T22:22:50.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f8ca29155fce2c-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.205 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.205 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.205 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.206 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.206 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.206 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.206 DEBUG:boto:Data: 2024-04-13T22:22:50.206 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.206 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.206 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.206 DEBUG:boto:Params: {} 2024-04-13T22:22:50.206 DEBUG:boto:Token: None 2024-04-13T22:22:50.207 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UPVHd3LHfI2DWmlNjBUyvU4paSo= 2024-04-13T22:22:50.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UPVHd3LHfI2DWmlNjBUyvU4paSo=', 'Content-Length': '0'} 2024-04-13T22:22:50.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8a9a63f311f5776-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:22:50.209 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.209 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.209 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.209 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.209 DEBUG:boto:Method: GET 2024-04-13T22:22:50.209 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.209 DEBUG:boto:Data: 2024-04-13T22:22:50.209 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.210 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.210 DEBUG:boto:Params: {} 2024-04-13T22:22:50.210 DEBUG:boto:Token: None 2024-04-13T22:22:50.210 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0= 2024-04-13T22:22:50.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0=', 'Content-Length': '0'} 2024-04-13T22:22:50.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000040e18b9e2ba57ed-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.212 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.212 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.212 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.212 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.212 DEBUG:boto:Method: GET 2024-04-13T22:22:50.212 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.212 DEBUG:boto:Data: 2024-04-13T22:22:50.213 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.213 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.213 DEBUG:boto:Params: {} 2024-04-13T22:22:50.213 DEBUG:boto:Token: None 2024-04-13T22:22:50.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:22:50.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0= 2024-04-13T22:22:50.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+svIxv6o6cz7YEYwMN8go/Ts4S0=', 'Content-Length': '0'} 2024-04-13T22:22:50.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7795daaa4065dd1-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.215 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.215 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.215 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.215 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.215 DEBUG:boto:Method: GET 2024-04-13T22:22:50.216 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.216 DEBUG:boto:Data: 2024-04-13T22:22:50.216 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.216 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.216 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.216 DEBUG:boto:Params: {} 2024-04-13T22:22:50.216 DEBUG:boto:Token: None 2024-04-13T22:22:50.216 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ec05b993284db3-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.219 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.219 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.219 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.219 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.219 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.220 DEBUG:boto:Method: GET 2024-04-13T22:22:50.220 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.220 DEBUG:boto:Data: 2024-04-13T22:22:50.220 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.220 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.220 DEBUG:boto:Params: {} 2024-04-13T22:22:50.220 DEBUG:boto:Token: None 2024-04-13T22:22:50.220 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adab69146a549cc1-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.266 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.267 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.267 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.267 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.267 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:22:50.267 DEBUG:boto:Data: 2024-04-13T22:22:50.268 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.268 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.268 DEBUG:boto:Params: {} 2024-04-13T22:22:50.268 DEBUG:boto:Token: None 2024-04-13T22:22:50.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85 2024-04-13T22:22:50.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LG9AG88KkFFNXpSeS+onlFdbipY= 2024-04-13T22:22:50.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LG9AG88KkFFNXpSeS+onlFdbipY=', 'Content-Length': '0'} 2024-04-13T22:22:50.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f6603760779424-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.270 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.270 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.271 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.271 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:22:50.271 DEBUG:boto:Data: 2024-04-13T22:22:50.271 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.271 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.271 DEBUG:boto:Params: {} 2024-04-13T22:22:50.271 DEBUG:boto:Token: None 2024-04-13T22:22:50.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85 2024-04-13T22:22:50.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LG9AG88KkFFNXpSeS+onlFdbipY= 2024-04-13T22:22:50.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LG9AG88KkFFNXpSeS+onlFdbipY=', 'Content-Length': '0'} 2024-04-13T22:22:50.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7874b9748cf789-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:22:50.273 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.274 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.274 DEBUG:boto:Method: GET 2024-04-13T22:22:50.274 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:22:50.274 DEBUG:boto:Data: 2024-04-13T22:22:50.274 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.274 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.274 DEBUG:boto:Params: {} 2024-04-13T22:22:50.274 DEBUG:boto:Token: None 2024-04-13T22:22:50.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85 2024-04-13T22:22:50.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hGUwzGGXL/Ni3rCqRosJ8BpsCJE= 2024-04-13T22:22:50.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hGUwzGGXL/Ni3rCqRosJ8BpsCJE=', 'Content-Length': '0'} 2024-04-13T22:22:50.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084215e4fc4ea448d-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.277 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:22:50.277 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:22:50.277 DEBUG:boto:Method: GET 2024-04-13T22:22:50.277 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:22:50.277 DEBUG:boto:Data: 2024-04-13T22:22:50.277 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.277 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.278 DEBUG:boto:Params: {} 2024-04-13T22:22:50.278 DEBUG:boto:Token: None 2024-04-13T22:22:50.278 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj85 2024-04-13T22:22:50.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hGUwzGGXL/Ni3rCqRosJ8BpsCJE= 2024-04-13T22:22:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hGUwzGGXL/Ni3rCqRosJ8BpsCJE=', 'Content-Length': '0'} 2024-04-13T22:22:50.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f855370d6577bc-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.280 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.280 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.280 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.280 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.280 DEBUG:boto:Method: GET 2024-04-13T22:22:50.280 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.281 DEBUG:boto:Data: 2024-04-13T22:22:50.281 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.281 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.281 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.281 DEBUG:boto:Params: {} 2024-04-13T22:22:50.281 DEBUG:boto:Token: None 2024-04-13T22:22:50.281 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b53c9416634d1918-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.284 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.284 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.284 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.284 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.284 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.285 DEBUG:boto:Method: GET 2024-04-13T22:22:50.285 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:22:50.285 DEBUG:boto:Data: 2024-04-13T22:22:50.285 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.285 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.285 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.285 DEBUG:boto:Params: {} 2024-04-13T22:22:50.285 DEBUG:boto:Token: None 2024-04-13T22:22:50.285 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a5c4241f4be6f1-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.288 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:22:50.289 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.289 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.289 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.289 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.289 DEBUG:boto:Method: GET 2024-04-13T22:22:50.289 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.289 DEBUG:boto:Data: 2024-04-13T22:22:50.289 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.289 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.289 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.290 DEBUG:boto:Params: {} 2024-04-13T22:22:50.290 DEBUG:boto:Token: None 2024-04-13T22:22:50.290 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.290 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA= 2024-04-13T22:22:50.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA=', 'Content-Length': '0'} 2024-04-13T22:22:50.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2a9270b33e2aea0-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.292 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.292 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.292 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.292 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.292 DEBUG:boto:Method: GET 2024-04-13T22:22:50.293 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.293 DEBUG:boto:Data: 2024-04-13T22:22:50.293 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.293 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.293 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.293 DEBUG:boto:Params: {} 2024-04-13T22:22:50.293 DEBUG:boto:Token: None 2024-04-13T22:22:50.293 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.293 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA= 2024-04-13T22:22:50.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA=', 'Content-Length': '0'} 2024-04-13T22:22:50.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c269d1105bf630-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.295 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.295 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.295 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.295 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.296 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.296 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.296 DEBUG:boto:Data: 2024-04-13T22:22:50.296 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.296 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.296 DEBUG:boto:Params: {} 2024-04-13T22:22:50.296 DEBUG:boto:Token: None 2024-04-13T22:22:50.296 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GDa3i9kskaL30p4qdGoqu6YUpmo= 2024-04-13T22:22:50.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GDa3i9kskaL30p4qdGoqu6YUpmo=', 'Content-Length': '0'} 2024-04-13T22:22:50.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ed43c40642fc399-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.299 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.299 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.299 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.299 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.299 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.299 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.299 DEBUG:boto:Data: 2024-04-13T22:22:50.299 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.299 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.300 DEBUG:boto:Params: {} 2024-04-13T22:22:50.300 DEBUG:boto:Token: None 2024-04-13T22:22:50.300 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GDa3i9kskaL30p4qdGoqu6YUpmo= 2024-04-13T22:22:50.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GDa3i9kskaL30p4qdGoqu6YUpmo=', 'Content-Length': '0'} 2024-04-13T22:22:50.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc196847ffc0101-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:22:50.302 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.302 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.303 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.303 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.303 DEBUG:boto:Method: GET 2024-04-13T22:22:50.303 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.303 DEBUG:boto:Data: 2024-04-13T22:22:50.303 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.303 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.303 DEBUG:boto:Params: {} 2024-04-13T22:22:50.303 DEBUG:boto:Token: None 2024-04-13T22:22:50.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA= 2024-04-13T22:22:50.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA=', 'Content-Length': '0'} 2024-04-13T22:22:50.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062210d71c770a434-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.305 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.305 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.306 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.306 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.306 DEBUG:boto:Method: GET 2024-04-13T22:22:50.306 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.306 DEBUG:boto:Data: 2024-04-13T22:22:50.306 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.306 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.306 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.306 DEBUG:boto:Params: {} 2024-04-13T22:22:50.306 DEBUG:boto:Token: None 2024-04-13T22:22:50.307 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:22:50.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA= 2024-04-13T22:22:50.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bv2oPM09vQWpZbYWAn8+WtpmxuA=', 'Content-Length': '0'} 2024-04-13T22:22:50.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954165369bb489e4-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.351 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.351 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.352 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.352 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.352 DEBUG:boto:Method: GET 2024-04-13T22:22:50.352 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.352 DEBUG:boto:Data: 2024-04-13T22:22:50.352 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.352 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.352 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.353 DEBUG:boto:Params: {} 2024-04-13T22:22:50.353 DEBUG:boto:Token: None 2024-04-13T22:22:50.353 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.353 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f73732fab5b7fb-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.355 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.356 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.356 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.356 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.356 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.356 DEBUG:boto:Method: GET 2024-04-13T22:22:50.356 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.356 DEBUG:boto:Data: 2024-04-13T22:22:50.356 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.356 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.356 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.356 DEBUG:boto:Params: {} 2024-04-13T22:22:50.357 DEBUG:boto:Token: None 2024-04-13T22:22:50.357 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.357 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b422673face4034-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.359 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.360 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.360 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.360 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.360 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:22:50.360 DEBUG:boto:Data: 2024-04-13T22:22:50.360 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.360 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.360 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.360 DEBUG:boto:Params: {} 2024-04-13T22:22:50.361 DEBUG:boto:Token: None 2024-04-13T22:22:50.361 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95 2024-04-13T22:22:50.361 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1AqwdCwto+xHSv9ogOtmfkAEJ18= 2024-04-13T22:22:50.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1AqwdCwto+xHSv9ogOtmfkAEJ18=', 'Content-Length': '0'} 2024-04-13T22:22:50.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071bcea748c972015-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.363 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.363 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.363 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.363 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:22:50.363 DEBUG:boto:Data: 2024-04-13T22:22:50.363 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.363 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.364 DEBUG:boto:Params: {} 2024-04-13T22:22:50.364 DEBUG:boto:Token: None 2024-04-13T22:22:50.364 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95 2024-04-13T22:22:50.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1AqwdCwto+xHSv9ogOtmfkAEJ18= 2024-04-13T22:22:50.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1AqwdCwto+xHSv9ogOtmfkAEJ18=', 'Content-Length': '0'} 2024-04-13T22:22:50.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d1dba846142612-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:22:50.366 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.366 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.366 DEBUG:boto:Method: GET 2024-04-13T22:22:50.366 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:22:50.366 DEBUG:boto:Data: 2024-04-13T22:22:50.366 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.367 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.367 DEBUG:boto:Params: {} 2024-04-13T22:22:50.367 DEBUG:boto:Token: None 2024-04-13T22:22:50.367 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95 2024-04-13T22:22:50.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fWp2LFAoom2PI50xkvMajO03PBk= 2024-04-13T22:22:50.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fWp2LFAoom2PI50xkvMajO03PBk=', 'Content-Length': '0'} 2024-04-13T22:22:50.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4ad75fd9dbc3ca-00661b05ba-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.410 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:22:50.411 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:22:50.411 DEBUG:boto:Method: GET 2024-04-13T22:22:50.411 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:22:50.411 DEBUG:boto:Data: 2024-04-13T22:22:50.411 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.411 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.411 DEBUG:boto:Params: {} 2024-04-13T22:22:50.411 DEBUG:boto:Token: None 2024-04-13T22:22:50.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj95 2024-04-13T22:22:50.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fWp2LFAoom2PI50xkvMajO03PBk= 2024-04-13T22:22:50.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fWp2LFAoom2PI50xkvMajO03PBk=', 'Content-Length': '0'} 2024-04-13T22:22:50.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea91e1d974c1581d-00661b05ba-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.414 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.414 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.414 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.414 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.414 DEBUG:boto:Method: GET 2024-04-13T22:22:50.415 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.415 DEBUG:boto:Data: 2024-04-13T22:22:50.415 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:22:50.415 DEBUG:boto:Port: 8001 2024-04-13T22:22:50.415 DEBUG:boto:Params: {} 2024-04-13T22:22:50.415 DEBUG:boto:Token: None 2024-04-13T22:22:50.415 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.415 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2c29069dc3841b-00661b05ba-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.418 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.418 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.418 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.418 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.418 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.418 DEBUG:boto:Method: GET 2024-04-13T22:22:50.419 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:22:50.419 DEBUG:boto:Data: 2024-04-13T22:22:50.419 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.419 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.419 DEBUG:boto:Params: {} 2024-04-13T22:22:50.419 DEBUG:boto:Token: None 2024-04-13T22:22:50.419 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.419 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8= 2024-04-13T22:22:50.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Okx74QJxhZqzCkazfrU7LkT9WL8=', 'Content-Length': '0'} 2024-04-13T22:22:50.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a31aa705ccf23d-00661b05ba-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.422 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:22:50.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:22:50.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:22:50.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:22:50.423 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:22:50.475 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:22:50.475 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:22:50.520 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:22:50.520 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-61 2024-04-13T22:22:50.527 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 15 2024-04-13T22:22:50.572 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.573 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.573 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.573 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:50.573 DEBUG:boto:Data: 2024-04-13T22:22:50.573 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.573 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.573 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.573 DEBUG:boto:Params: {} 2024-04-13T22:22:50.573 DEBUG:boto:Token: None 2024-04-13T22:22:50.574 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.574 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw= 2024-04-13T22:22:50.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw=', 'Content-Length': '0'} 2024-04-13T22:22:50.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006223f02b424b1a56-00661b05ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.577 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:22:50.577 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:22:50.577 DEBUG:boto:Method: PUT 2024-04-13T22:22:50.578 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:22:50.578 DEBUG:boto:Data: 2024-04-13T22:22:50.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:22:50.578 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.578 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.578 DEBUG:boto:Params: {} 2024-04-13T22:22:50.578 DEBUG:boto:Token: None 2024-04-13T22:22:50.578 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj23 2024-04-13T22:22:50.578 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EGm7tk/jajSll6EWI3AkoTRZp+k= 2024-04-13T22:22:50.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EGm7tk/jajSll6EWI3AkoTRZp+k='} 2024-04-13T22:22:50.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae563b33053af5c1-00661b05ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.584 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.584 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.584 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.584 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:50.584 DEBUG:boto:Data: 2024-04-13T22:22:50.584 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.584 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.585 DEBUG:boto:Params: {} 2024-04-13T22:22:50.585 DEBUG:boto:Token: None 2024-04-13T22:22:50.585 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw= 2024-04-13T22:22:50.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw=', 'Content-Length': '0'} 2024-04-13T22:22:50.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072c65b942e34f2c0-00661b05ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.587 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:22:50.588 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:22:50.588 DEBUG:boto:Method: PUT 2024-04-13T22:22:50.588 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:22:50.588 DEBUG:boto:Data: 2024-04-13T22:22:50.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:22:50.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.588 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.588 DEBUG:boto:Params: {} 2024-04-13T22:22:50.588 DEBUG:boto:Token: None 2024-04-13T22:22:50.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj46 2024-04-13T22:22:50.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yhpq/+1hzCvsSc2Qb7GAaUth3uk= 2024-04-13T22:22:50.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yhpq/+1hzCvsSc2Qb7GAaUth3uk='} 2024-04-13T22:22:50.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064ad99426f9d043c-00661b05ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.632 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.632 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.632 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.632 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:50.632 DEBUG:boto:Data: 2024-04-13T22:22:50.632 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.632 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.633 DEBUG:boto:Params: {} 2024-04-13T22:22:50.633 DEBUG:boto:Token: None 2024-04-13T22:22:50.633 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw= 2024-04-13T22:22:50.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw=', 'Content-Length': '0'} 2024-04-13T22:22:50.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000748c6a0d0add0e8c-00661b05ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.636 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:22:50.636 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:22:50.636 DEBUG:boto:Method: PUT 2024-04-13T22:22:50.636 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:22:50.636 DEBUG:boto:Data: 2024-04-13T22:22:50.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:22:50.637 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.637 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.637 DEBUG:boto:Params: {} 2024-04-13T22:22:50.637 DEBUG:boto:Token: None 2024-04-13T22:22:50.637 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj69 2024-04-13T22:22:50.637 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xdTP/KVaCI+I5MBF2OoV9lWe5C4= 2024-04-13T22:22:50.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xdTP/KVaCI+I5MBF2OoV9lWe5C4='} 2024-04-13T22:22:50.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048b502eaa4d22875-00661b05ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.643 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.643 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.643 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.643 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:50.643 DEBUG:boto:Data: 2024-04-13T22:22:50.643 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.643 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.643 DEBUG:boto:Params: {} 2024-04-13T22:22:50.644 DEBUG:boto:Token: None 2024-04-13T22:22:50.644 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw= 2024-04-13T22:22:50.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw=', 'Content-Length': '0'} 2024-04-13T22:22:50.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d93a5de3a9044cf-00661b05ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.647 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:22:50.647 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:22:50.647 DEBUG:boto:Method: PUT 2024-04-13T22:22:50.647 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:22:50.648 DEBUG:boto:Data: 2024-04-13T22:22:50.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:22:50.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.648 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.648 DEBUG:boto:Params: {} 2024-04-13T22:22:50.648 DEBUG:boto:Token: None 2024-04-13T22:22:50.648 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj92 2024-04-13T22:22:50.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GqZj/w9eC3T83h9YIotcGUAofaI= 2024-04-13T22:22:50.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GqZj/w9eC3T83h9YIotcGUAofaI='} 2024-04-13T22:22:50.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b6d6690a926162-00661b05ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.691 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:22:50.691 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:22:50.691 DEBUG:boto:Method: HEAD 2024-04-13T22:22:50.691 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:22:50.691 DEBUG:boto:Data: 2024-04-13T22:22:50.691 DEBUG:boto:Headers: {} 2024-04-13T22:22:50.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.691 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.691 DEBUG:boto:Params: {} 2024-04-13T22:22:50.692 DEBUG:boto:Token: None 2024-04-13T22:22:50.692 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/ 2024-04-13T22:22:50.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw= 2024-04-13T22:22:50.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lpxaSV0TfyFGdjlYUT7RukkIuqw=', 'Content-Length': '0'} 2024-04-13T22:22:50.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009097a87f67125336-00661b05ba-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.695 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:22:50.695 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:22:50.695 DEBUG:boto:Method: PUT 2024-04-13T22:22:50.695 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:22:50.695 DEBUG:boto:Data: 2024-04-13T22:22:50.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:22:50.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:22:50.695 DEBUG:boto:Port: 8000 2024-04-13T22:22:50.695 DEBUG:boto:Params: {} 2024-04-13T22:22:50.695 DEBUG:boto:Token: None 2024-04-13T22:22:50.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:22:50 GMT /vwnfns-61/obj115 2024-04-13T22:22:50.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rn+75XNe2gKx1hemBe8xZ9+FxLs= 2024-04-13T22:22:50.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:22:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rn+75XNe2gKx1hemBe8xZ9+FxLs='} 2024-04-13T22:22:50.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e169d27ccf5046d6-00661b05ba-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:22:50.701 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:22:50.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:22:50.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:22:50.701 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:22:50.755 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:22:50.756 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:22:50.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:50.787+0000 7fbbb0124780 1 waiting to reach latest gen 3, on 2.. 2024-04-13T22:22:55.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:22:55.786+0000 7fbbb0124780 1 waiting to reach latest gen 3, on 2.. 2024-04-13T22:23:00.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:00.787+0000 7fbbb0124780 1 waiting to reach latest gen 3, on 2.. 2024-04-13T22:23:05.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:05.788+0000 7fbbb0124780 1 waiting to reach latest gen 3, on 2.. 2024-04-13T22:23:10.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:10.790+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:10.794 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:10.794 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:15.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:15.792+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:15.796 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:15.796 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:20.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:20.794+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:20.798 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:20.798 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:25.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:25.796+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:25.800 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:25.800 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:30.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:30.797+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:30.802 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:30.802 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:35.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:35.799+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:35.803 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:35.803 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:40.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:40.800+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:40.805 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:40.805 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:45.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:45.802+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:45.806 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:45.807 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:50.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:50.804+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:50.808 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:50.808 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:23:55.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:23:55.806+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:23:55.810 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:23:55.810 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:00.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:00.808+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:00.812 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , , , 00000000001.122.6, , , , ] 2024-04-13T22:24:00.812 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:05.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:05.810+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:05.814 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , , , 00000000001.122.6, , , , ] 2024-04-13T22:24:05.814 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:10.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:10.811+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:10.816 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:10.816 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:15.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:15.813+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:15.817 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:15.817 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:20.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:20.814+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:20.819 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:20.819 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:25.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:25.816+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:25.821 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:25.821 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:30.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:30.819+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:30.823 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:30.823 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:35.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:35.821+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:35.825 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:35.825 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:40.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:40.823+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:40.827 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:40.827 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:45.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:45.825+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:45.829 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:45.829 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:50.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:50.825+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:50.830 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:50.830 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:55.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:24:55.827+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:24:55.832 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:24:55.832 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:00.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:00.829+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:00.834 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:00.834 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:05.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:05.831+0000 7fbbb0124780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:05.946 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , , , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:05.946 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:10.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:10.834+0000 7fbbb0124780 1 bucket sync caught up with source: 2024-04-13T22:25:10.838 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:10.838 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.100.6, , 00000000002.100.6, , , , , , 00000000001.67.6, , 00000000001.122.6, , , , ] 2024-04-13T22:25:10.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:10.834+0000 7fbbb0124780 0 bucket checkpoint complete 2024-04-13T22:25:10.855 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:25:10.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:25:10.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:25:10.856 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:25:10.918 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:25:10.918 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:25:10.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:10.953+0000 7f5aec66c780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:10.957 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:10.957 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:15.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:15.954+0000 7f5aec66c780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:15.958 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:15.958 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:20.956+0000 7f5aec66c780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:20.960 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:20.960 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:25.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:25.956+0000 7f5aec66c780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:25.961 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000001.2283.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:25.961 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:30.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:30.957+0000 7f5aec66c780 1 bucket sync caught up with source: 2024-04-13T22:25:30.962 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:30.962 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000002.1209.6, 00000000002.1149.6, , , 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:25:30.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:30.957+0000 7f5aec66c780 0 bucket checkpoint complete 2024-04-13T22:25:30.979 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:25:30.979 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:30.980 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:30.980 DEBUG:boto:Method: HEAD 2024-04-13T22:25:30.980 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:30.980 DEBUG:boto:Data: 2024-04-13T22:25:30.980 DEBUG:boto:Headers: {} 2024-04-13T22:25:30.980 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:30.980 DEBUG:boto:Port: 8001 2024-04-13T22:25:30.980 DEBUG:boto:Params: {} 2024-04-13T22:25:30.981 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:25:30.981 DEBUG:boto:Token: None 2024-04-13T22:25:30.981 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:30 GMT /vwnfns-61/ 2024-04-13T22:25:30.981 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GuRpOelOR4eYEon04wQYa0QQwRU= 2024-04-13T22:25:30.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GuRpOelOR4eYEon04wQYa0QQwRU=', 'Content-Length': '0'} 2024-04-13T22:25:30.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c92730da33570bd5-00661b065a-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:30.987 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:30.987 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:30.987 DEBUG:boto:Method: HEAD 2024-04-13T22:25:30.987 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:30.987 DEBUG:boto:Data: 2024-04-13T22:25:30.987 DEBUG:boto:Headers: {} 2024-04-13T22:25:30.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:30.987 DEBUG:boto:Port: 8000 2024-04-13T22:25:30.988 DEBUG:boto:Params: {} 2024-04-13T22:25:30.988 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:25:30.988 DEBUG:boto:Token: None 2024-04-13T22:25:30.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:30 GMT /vwnfns-61/ 2024-04-13T22:25:30.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GuRpOelOR4eYEon04wQYa0QQwRU= 2024-04-13T22:25:30.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GuRpOelOR4eYEon04wQYa0QQwRU=', 'Content-Length': '0'} 2024-04-13T22:25:30.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000277eb04a3ed57f32-00661b065a-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:30.993 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:25:30.993 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:30.993 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:30.993 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:30.993 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:30.993 DEBUG:boto:Method: GET 2024-04-13T22:25:30.993 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:30.993 DEBUG:boto:Data: 2024-04-13T22:25:30.994 DEBUG:boto:Headers: {} 2024-04-13T22:25:30.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:30.994 DEBUG:boto:Port: 8001 2024-04-13T22:25:30.994 DEBUG:boto:Params: {} 2024-04-13T22:25:30.994 DEBUG:boto:Token: None 2024-04-13T22:25:30.994 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:30 GMT /vwnfns-61/?versions 2024-04-13T22:25:30.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:60NmLYrvI3HuioHgg1c8NkHCYT0= 2024-04-13T22:25:30.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:60NmLYrvI3HuioHgg1c8NkHCYT0=', 'Content-Length': '0'} 2024-04-13T22:25:30.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8dda4dabb7c802-00661b065a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:30.997 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:30.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:25:30.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:25:31.000 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:25:31.000 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.000 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.000 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.000 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.001 DEBUG:boto:Method: GET 2024-04-13T22:25:31.001 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.001 DEBUG:boto:Data: 2024-04-13T22:25:31.001 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.001 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.001 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.001 DEBUG:boto:Params: {} 2024-04-13T22:25:31.001 DEBUG:boto:Token: None 2024-04-13T22:25:31.001 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/?versions 2024-04-13T22:25:31.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oGtr87wBCsVXBJRuiGm+q6sZW98= 2024-04-13T22:25:31.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oGtr87wBCsVXBJRuiGm+q6sZW98=', 'Content-Length': '0'} 2024-04-13T22:25:31.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048aa554d2ae46063-00661b065a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.005 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:25:31.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:25:31.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:25:31.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:25:31.008 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.008 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.008 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.008 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.008 DEBUG:boto:Method: GET 2024-04-13T22:25:31.008 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.008 DEBUG:boto:Data: 2024-04-13T22:25:31.008 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.008 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.009 DEBUG:boto:Params: {} 2024-04-13T22:25:31.009 DEBUG:boto:Token: None 2024-04-13T22:25:31.009 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/?versions 2024-04-13T22:25:31.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oGtr87wBCsVXBJRuiGm+q6sZW98= 2024-04-13T22:25:31.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oGtr87wBCsVXBJRuiGm+q6sZW98=', 'Content-Length': '0'} 2024-04-13T22:25:31.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed79920fdb0f77b-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.012 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.013 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.013 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.013 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.013 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.013 DEBUG:boto:Method: GET 2024-04-13T22:25:31.013 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:25:31.013 DEBUG:boto:Data: 2024-04-13T22:25:31.013 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.013 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.013 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.013 DEBUG:boto:Params: {} 2024-04-13T22:25:31.014 DEBUG:boto:Token: None 2024-04-13T22:25:31.014 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/?versions 2024-04-13T22:25:31.014 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oGtr87wBCsVXBJRuiGm+q6sZW98= 2024-04-13T22:25:31.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oGtr87wBCsVXBJRuiGm+q6sZW98=', 'Content-Length': '0'} 2024-04-13T22:25:31.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e4038117b438703-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.017 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:25:31.019 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.019 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.019 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.019 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.019 DEBUG:boto:Method: GET 2024-04-13T22:25:31.019 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.019 DEBUG:boto:Data: 2024-04-13T22:25:31.019 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.020 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.020 DEBUG:boto:Params: {} 2024-04-13T22:25:31.020 DEBUG:boto:Token: None 2024-04-13T22:25:31.020 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY= 2024-04-13T22:25:31.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY=', 'Content-Length': '0'} 2024-04-13T22:25:31.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050073a1054ce92f4-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.022 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.022 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.022 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.022 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.023 DEBUG:boto:Method: GET 2024-04-13T22:25:31.023 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.023 DEBUG:boto:Data: 2024-04-13T22:25:31.023 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.023 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.023 DEBUG:boto:Params: {} 2024-04-13T22:25:31.023 DEBUG:boto:Token: None 2024-04-13T22:25:31.023 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.023 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY= 2024-04-13T22:25:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY=', 'Content-Length': '0'} 2024-04-13T22:25:31.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d87e4d17360af2e-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.026 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.026 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.026 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.026 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.026 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.027 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.027 DEBUG:boto:Data: 2024-04-13T22:25:31.027 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.027 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.027 DEBUG:boto:Params: {} 2024-04-13T22:25:31.027 DEBUG:boto:Token: None 2024-04-13T22:25:31.027 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ypdkpBsO25xT2ESIVG97htG3DSE= 2024-04-13T22:25:31.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ypdkpBsO25xT2ESIVG97htG3DSE=', 'Content-Length': '0'} 2024-04-13T22:25:31.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8463e73b667486-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.029 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.030 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.030 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.030 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.030 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.030 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.030 DEBUG:boto:Data: 2024-04-13T22:25:31.030 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.030 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.030 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.030 DEBUG:boto:Params: {} 2024-04-13T22:25:31.030 DEBUG:boto:Token: None 2024-04-13T22:25:31.031 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.031 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ypdkpBsO25xT2ESIVG97htG3DSE= 2024-04-13T22:25:31.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ypdkpBsO25xT2ESIVG97htG3DSE=', 'Content-Length': '0'} 2024-04-13T22:25:31.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e77b9fa12c85968-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:25:31.033 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.033 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.033 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.033 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.033 DEBUG:boto:Method: GET 2024-04-13T22:25:31.033 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.033 DEBUG:boto:Data: 2024-04-13T22:25:31.033 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.033 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.033 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.033 DEBUG:boto:Params: {} 2024-04-13T22:25:31.034 DEBUG:boto:Token: None 2024-04-13T22:25:31.034 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.034 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY= 2024-04-13T22:25:31.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY=', 'Content-Length': '0'} 2024-04-13T22:25:31.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92c7403263ad32d-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.036 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.036 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.036 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.036 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.036 DEBUG:boto:Method: GET 2024-04-13T22:25:31.036 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.036 DEBUG:boto:Data: 2024-04-13T22:25:31.036 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.036 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.037 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.037 DEBUG:boto:Params: {} 2024-04-13T22:25:31.037 DEBUG:boto:Token: None 2024-04-13T22:25:31.037 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:25:31.037 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY= 2024-04-13T22:25:31.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wVy88oOqp0m4x/Sc1GS7/NFW7iY=', 'Content-Length': '0'} 2024-04-13T22:25:31.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b49d7450cbf26b-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.039 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.039 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.039 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.039 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.039 DEBUG:boto:Method: GET 2024-04-13T22:25:31.039 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.039 DEBUG:boto:Data: 2024-04-13T22:25:31.040 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.040 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.040 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.040 DEBUG:boto:Params: {} 2024-04-13T22:25:31.040 DEBUG:boto:Token: None 2024-04-13T22:25:31.040 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5df0ddf1696dd62-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.042 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.043 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.043 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.043 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.043 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.043 DEBUG:boto:Method: GET 2024-04-13T22:25:31.043 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.044 DEBUG:boto:Data: 2024-04-13T22:25:31.044 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.044 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.044 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.044 DEBUG:boto:Params: {} 2024-04-13T22:25:31.044 DEBUG:boto:Token: None 2024-04-13T22:25:31.044 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007edfe7b3404b9ec5-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.094 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.095 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.095 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.095 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.095 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:25:31.095 DEBUG:boto:Data: 2024-04-13T22:25:31.095 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.096 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.096 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.096 DEBUG:boto:Params: {} 2024-04-13T22:25:31.096 DEBUG:boto:Token: None 2024-04-13T22:25:31.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11 2024-04-13T22:25:31.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6nOQoDObpo3cJXUP4MEBh/nQkwE= 2024-04-13T22:25:31.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6nOQoDObpo3cJXUP4MEBh/nQkwE=', 'Content-Length': '0'} 2024-04-13T22:25:31.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f3a256b882f08f-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.098 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.099 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.099 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.099 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:25:31.099 DEBUG:boto:Data: 2024-04-13T22:25:31.099 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.099 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.099 DEBUG:boto:Params: {} 2024-04-13T22:25:31.099 DEBUG:boto:Token: None 2024-04-13T22:25:31.100 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11 2024-04-13T22:25:31.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6nOQoDObpo3cJXUP4MEBh/nQkwE= 2024-04-13T22:25:31.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6nOQoDObpo3cJXUP4MEBh/nQkwE=', 'Content-Length': '0'} 2024-04-13T22:25:31.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005de8e6315321949-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:25:31.102 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.102 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.102 DEBUG:boto:Method: GET 2024-04-13T22:25:31.102 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:25:31.102 DEBUG:boto:Data: 2024-04-13T22:25:31.102 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.102 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.102 DEBUG:boto:Params: {} 2024-04-13T22:25:31.102 DEBUG:boto:Token: None 2024-04-13T22:25:31.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11 2024-04-13T22:25:31.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s2ssRc0jN/ChLS07+WO47GhaJnk= 2024-04-13T22:25:31.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s2ssRc0jN/ChLS07+WO47GhaJnk=', 'Content-Length': '0'} 2024-04-13T22:25:31.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ac9861c9e0a9dc-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.105 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:25:31.105 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:25:31.105 DEBUG:boto:Method: GET 2024-04-13T22:25:31.105 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:25:31.105 DEBUG:boto:Data: 2024-04-13T22:25:31.105 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.105 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.105 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.105 DEBUG:boto:Params: {} 2024-04-13T22:25:31.105 DEBUG:boto:Token: None 2024-04-13T22:25:31.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj11 2024-04-13T22:25:31.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s2ssRc0jN/ChLS07+WO47GhaJnk= 2024-04-13T22:25:31.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s2ssRc0jN/ChLS07+WO47GhaJnk=', 'Content-Length': '0'} 2024-04-13T22:25:31.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741378e7848128b9-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.108 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.108 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.108 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.108 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.108 DEBUG:boto:Method: GET 2024-04-13T22:25:31.108 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.108 DEBUG:boto:Data: 2024-04-13T22:25:31.108 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.109 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.109 DEBUG:boto:Params: {} 2024-04-13T22:25:31.109 DEBUG:boto:Token: None 2024-04-13T22:25:31.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba26af6d9b11775-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.111 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.112 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.112 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.112 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.112 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.112 DEBUG:boto:Method: GET 2024-04-13T22:25:31.112 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:25:31.112 DEBUG:boto:Data: 2024-04-13T22:25:31.112 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.113 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.113 DEBUG:boto:Params: {} 2024-04-13T22:25:31.113 DEBUG:boto:Token: None 2024-04-13T22:25:31.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000763630c3e6a02627-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.116 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:25:31.117 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.117 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.117 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.117 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.117 DEBUG:boto:Method: GET 2024-04-13T22:25:31.117 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.118 DEBUG:boto:Data: 2024-04-13T22:25:31.118 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.118 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.118 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.118 DEBUG:boto:Params: {} 2024-04-13T22:25:31.118 DEBUG:boto:Token: None 2024-04-13T22:25:31.118 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ= 2024-04-13T22:25:31.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ=', 'Content-Length': '0'} 2024-04-13T22:25:31.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f803ef15b487a54-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.166 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.167 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.167 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.167 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.167 DEBUG:boto:Method: GET 2024-04-13T22:25:31.167 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.167 DEBUG:boto:Data: 2024-04-13T22:25:31.167 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.167 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.167 DEBUG:boto:Params: {} 2024-04-13T22:25:31.168 DEBUG:boto:Token: None 2024-04-13T22:25:31.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ= 2024-04-13T22:25:31.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ=', 'Content-Length': '0'} 2024-04-13T22:25:31.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bdcc72507bbeb1c-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.170 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.170 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.170 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.170 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.171 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.171 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.171 DEBUG:boto:Data: 2024-04-13T22:25:31.171 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.171 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.171 DEBUG:boto:Params: {} 2024-04-13T22:25:31.171 DEBUG:boto:Token: None 2024-04-13T22:25:31.171 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DpvYpyUQJbo8yH9ZJid2EPrdEd8= 2024-04-13T22:25:31.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpvYpyUQJbo8yH9ZJid2EPrdEd8=', 'Content-Length': '0'} 2024-04-13T22:25:31.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd15f307e3b82774-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.173 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.173 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.173 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.173 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.174 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.174 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.174 DEBUG:boto:Data: 2024-04-13T22:25:31.174 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.174 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.174 DEBUG:boto:Params: {} 2024-04-13T22:25:31.174 DEBUG:boto:Token: None 2024-04-13T22:25:31.174 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.174 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DpvYpyUQJbo8yH9ZJid2EPrdEd8= 2024-04-13T22:25:31.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpvYpyUQJbo8yH9ZJid2EPrdEd8=', 'Content-Length': '0'} 2024-04-13T22:25:31.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006999c5043645b3b7-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:25:31.177 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.177 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.177 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.177 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.177 DEBUG:boto:Method: GET 2024-04-13T22:25:31.177 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.177 DEBUG:boto:Data: 2024-04-13T22:25:31.177 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.177 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.178 DEBUG:boto:Params: {} 2024-04-13T22:25:31.178 DEBUG:boto:Token: None 2024-04-13T22:25:31.178 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ= 2024-04-13T22:25:31.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ=', 'Content-Length': '0'} 2024-04-13T22:25:31.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7d8384b696db62-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.180 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.180 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.180 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.180 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.180 DEBUG:boto:Method: GET 2024-04-13T22:25:31.180 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.180 DEBUG:boto:Data: 2024-04-13T22:25:31.180 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.180 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.181 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.181 DEBUG:boto:Params: {} 2024-04-13T22:25:31.181 DEBUG:boto:Token: None 2024-04-13T22:25:31.181 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:25:31.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ= 2024-04-13T22:25:31.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:10j2lXX+ZIbZAh/b+Ubd+BRaVVQ=', 'Content-Length': '0'} 2024-04-13T22:25:31.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d72e5c60ed05ee-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.183 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.183 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.183 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.183 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.184 DEBUG:boto:Method: GET 2024-04-13T22:25:31.184 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.184 DEBUG:boto:Data: 2024-04-13T22:25:31.184 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.184 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.184 DEBUG:boto:Params: {} 2024-04-13T22:25:31.184 DEBUG:boto:Token: None 2024-04-13T22:25:31.184 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000001e52cd8d2359ac-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.187 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.188 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.188 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.188 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.188 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.188 DEBUG:boto:Method: GET 2024-04-13T22:25:31.188 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.188 DEBUG:boto:Data: 2024-04-13T22:25:31.188 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.188 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.188 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.188 DEBUG:boto:Params: {} 2024-04-13T22:25:31.189 DEBUG:boto:Token: None 2024-04-13T22:25:31.189 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.189 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000189386c9989e6d9a-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.192 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.192 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.192 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.193 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.193 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:25:31.193 DEBUG:boto:Data: 2024-04-13T22:25:31.193 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.193 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.193 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.193 DEBUG:boto:Params: {} 2024-04-13T22:25:31.193 DEBUG:boto:Token: None 2024-04-13T22:25:31.193 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115 2024-04-13T22:25:31.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lnIGW0VNtOd13b8rOke59oWjaHs= 2024-04-13T22:25:31.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lnIGW0VNtOd13b8rOke59oWjaHs=', 'Content-Length': '0'} 2024-04-13T22:25:31.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b5c4353df21fe0-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.195 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.195 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.195 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.196 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:25:31.196 DEBUG:boto:Data: 2024-04-13T22:25:31.196 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.196 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.196 DEBUG:boto:Params: {} 2024-04-13T22:25:31.196 DEBUG:boto:Token: None 2024-04-13T22:25:31.196 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115 2024-04-13T22:25:31.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lnIGW0VNtOd13b8rOke59oWjaHs= 2024-04-13T22:25:31.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lnIGW0VNtOd13b8rOke59oWjaHs=', 'Content-Length': '0'} 2024-04-13T22:25:31.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff1044250dd1b63-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:25:31.198 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.198 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.198 DEBUG:boto:Method: GET 2024-04-13T22:25:31.199 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:25:31.199 DEBUG:boto:Data: 2024-04-13T22:25:31.199 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.199 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.199 DEBUG:boto:Params: {} 2024-04-13T22:25:31.199 DEBUG:boto:Token: None 2024-04-13T22:25:31.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115 2024-04-13T22:25:31.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nAxhUY+pP5X9Nz5n8paHyc0JnkI= 2024-04-13T22:25:31.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nAxhUY+pP5X9Nz5n8paHyc0JnkI=', 'Content-Length': '0'} 2024-04-13T22:25:31.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039a702c720b3a29a-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.242 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:25:31.243 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:25:31.243 DEBUG:boto:Method: GET 2024-04-13T22:25:31.243 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:25:31.243 DEBUG:boto:Data: 2024-04-13T22:25:31.243 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.243 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.243 DEBUG:boto:Params: {} 2024-04-13T22:25:31.243 DEBUG:boto:Token: None 2024-04-13T22:25:31.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj115 2024-04-13T22:25:31.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nAxhUY+pP5X9Nz5n8paHyc0JnkI= 2024-04-13T22:25:31.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nAxhUY+pP5X9Nz5n8paHyc0JnkI=', 'Content-Length': '0'} 2024-04-13T22:25:31.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f4e3eaa76c952f-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.246 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.246 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.246 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.246 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.247 DEBUG:boto:Method: GET 2024-04-13T22:25:31.247 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.247 DEBUG:boto:Data: 2024-04-13T22:25:31.247 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.247 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.247 DEBUG:boto:Params: {} 2024-04-13T22:25:31.247 DEBUG:boto:Token: None 2024-04-13T22:25:31.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a4adcac2890ce74-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.250 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.251 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.251 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.251 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.251 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.251 DEBUG:boto:Method: GET 2024-04-13T22:25:31.251 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:25:31.251 DEBUG:boto:Data: 2024-04-13T22:25:31.251 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.251 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.251 DEBUG:boto:Params: {} 2024-04-13T22:25:31.252 DEBUG:boto:Token: None 2024-04-13T22:25:31.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08e3f1b536c12bd-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.255 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:25:31.255 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.255 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.256 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.256 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.256 DEBUG:boto:Method: GET 2024-04-13T22:25:31.256 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.256 DEBUG:boto:Data: 2024-04-13T22:25:31.256 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.256 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.256 DEBUG:boto:Params: {} 2024-04-13T22:25:31.256 DEBUG:boto:Token: None 2024-04-13T22:25:31.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY= 2024-04-13T22:25:31.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY=', 'Content-Length': '0'} 2024-04-13T22:25:31.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db9b98e79a3c5cd0-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.258 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.259 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.259 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.259 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.259 DEBUG:boto:Method: GET 2024-04-13T22:25:31.259 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.259 DEBUG:boto:Data: 2024-04-13T22:25:31.259 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.259 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.259 DEBUG:boto:Params: {} 2024-04-13T22:25:31.259 DEBUG:boto:Token: None 2024-04-13T22:25:31.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY= 2024-04-13T22:25:31.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY=', 'Content-Length': '0'} 2024-04-13T22:25:31.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001588e692478ff596-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.262 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.262 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.262 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.262 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.262 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.262 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.262 DEBUG:boto:Data: 2024-04-13T22:25:31.262 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.262 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.263 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.263 DEBUG:boto:Params: {} 2024-04-13T22:25:31.263 DEBUG:boto:Token: None 2024-04-13T22:25:31.263 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/1xopR8uXuP0U9WFK4ZIxXnjFIo= 2024-04-13T22:25:31.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/1xopR8uXuP0U9WFK4ZIxXnjFIo=', 'Content-Length': '0'} 2024-04-13T22:25:31.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cbebb93c92150b-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.265 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.265 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.265 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.265 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.265 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.265 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.265 DEBUG:boto:Data: 2024-04-13T22:25:31.265 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.265 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.266 DEBUG:boto:Params: {} 2024-04-13T22:25:31.266 DEBUG:boto:Token: None 2024-04-13T22:25:31.266 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.266 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/1xopR8uXuP0U9WFK4ZIxXnjFIo= 2024-04-13T22:25:31.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/1xopR8uXuP0U9WFK4ZIxXnjFIo=', 'Content-Length': '0'} 2024-04-13T22:25:31.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000988f2d225d7e1b80-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:25:31.268 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.268 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.268 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.268 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.269 DEBUG:boto:Method: GET 2024-04-13T22:25:31.269 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.269 DEBUG:boto:Data: 2024-04-13T22:25:31.269 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.269 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.269 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.269 DEBUG:boto:Params: {} 2024-04-13T22:25:31.269 DEBUG:boto:Token: None 2024-04-13T22:25:31.269 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY= 2024-04-13T22:25:31.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY=', 'Content-Length': '0'} 2024-04-13T22:25:31.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c256cf44994014a-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.271 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.272 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.272 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.272 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.272 DEBUG:boto:Method: GET 2024-04-13T22:25:31.272 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.272 DEBUG:boto:Data: 2024-04-13T22:25:31.272 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.272 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.272 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.272 DEBUG:boto:Params: {} 2024-04-13T22:25:31.272 DEBUG:boto:Token: None 2024-04-13T22:25:31.273 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:25:31.273 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY= 2024-04-13T22:25:31.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BfhuIQUh1AKHKVjrTol23suCeFY=', 'Content-Length': '0'} 2024-04-13T22:25:31.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d8790f4e86f578-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.275 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.275 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.275 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.275 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.275 DEBUG:boto:Method: GET 2024-04-13T22:25:31.275 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.275 DEBUG:boto:Data: 2024-04-13T22:25:31.276 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.276 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.276 DEBUG:boto:Params: {} 2024-04-13T22:25:31.276 DEBUG:boto:Token: None 2024-04-13T22:25:31.276 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835e245fa5d5b465-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.318 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.319 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.319 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.319 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.319 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.319 DEBUG:boto:Method: GET 2024-04-13T22:25:31.319 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.319 DEBUG:boto:Data: 2024-04-13T22:25:31.320 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.320 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.320 DEBUG:boto:Params: {} 2024-04-13T22:25:31.320 DEBUG:boto:Token: None 2024-04-13T22:25:31.320 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.320 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1de98c8b28a7d46-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.327 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.327 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.327 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.328 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.328 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:25:31.328 DEBUG:boto:Data: 2024-04-13T22:25:31.328 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.328 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.328 DEBUG:boto:Params: {} 2024-04-13T22:25:31.328 DEBUG:boto:Token: None 2024-04-13T22:25:31.328 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17 2024-04-13T22:25:31.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0FFji/9N5kWTrmQE9OOe8J2iNxw= 2024-04-13T22:25:31.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0FFji/9N5kWTrmQE9OOe8J2iNxw=', 'Content-Length': '0'} 2024-04-13T22:25:31.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b57e1e27ffee2ba-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.331 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.331 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.331 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.331 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:25:31.331 DEBUG:boto:Data: 2024-04-13T22:25:31.331 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.331 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.331 DEBUG:boto:Params: {} 2024-04-13T22:25:31.331 DEBUG:boto:Token: None 2024-04-13T22:25:31.332 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17 2024-04-13T22:25:31.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0FFji/9N5kWTrmQE9OOe8J2iNxw= 2024-04-13T22:25:31.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0FFji/9N5kWTrmQE9OOe8J2iNxw=', 'Content-Length': '0'} 2024-04-13T22:25:31.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003528d2afc19db0b3-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:25:31.333 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.334 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.334 DEBUG:boto:Method: GET 2024-04-13T22:25:31.334 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:25:31.334 DEBUG:boto:Data: 2024-04-13T22:25:31.334 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.334 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.334 DEBUG:boto:Params: {} 2024-04-13T22:25:31.334 DEBUG:boto:Token: None 2024-04-13T22:25:31.335 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17 2024-04-13T22:25:31.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xj3nOrq/9EMBAHnbKxgpOB1mECE= 2024-04-13T22:25:31.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xj3nOrq/9EMBAHnbKxgpOB1mECE=', 'Content-Length': '0'} 2024-04-13T22:25:31.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091dbfc5cbbbceb0a-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.337 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:25:31.337 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:25:31.337 DEBUG:boto:Method: GET 2024-04-13T22:25:31.337 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:25:31.337 DEBUG:boto:Data: 2024-04-13T22:25:31.337 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.337 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.337 DEBUG:boto:Params: {} 2024-04-13T22:25:31.338 DEBUG:boto:Token: None 2024-04-13T22:25:31.338 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj17 2024-04-13T22:25:31.338 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xj3nOrq/9EMBAHnbKxgpOB1mECE= 2024-04-13T22:25:31.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xj3nOrq/9EMBAHnbKxgpOB1mECE=', 'Content-Length': '0'} 2024-04-13T22:25:31.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc1924d1dd3bb1d-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.340 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.340 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.340 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.340 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.340 DEBUG:boto:Method: GET 2024-04-13T22:25:31.340 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.340 DEBUG:boto:Data: 2024-04-13T22:25:31.341 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.341 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.341 DEBUG:boto:Params: {} 2024-04-13T22:25:31.341 DEBUG:boto:Token: None 2024-04-13T22:25:31.341 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.341 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5a0623f92da3e4-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.344 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.344 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.344 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.344 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.344 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.344 DEBUG:boto:Method: GET 2024-04-13T22:25:31.345 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:25:31.345 DEBUG:boto:Data: 2024-04-13T22:25:31.345 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.345 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.345 DEBUG:boto:Params: {} 2024-04-13T22:25:31.345 DEBUG:boto:Token: None 2024-04-13T22:25:31.345 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfb6621d5ef0da93-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.349 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:25:31.349 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.349 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.349 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.349 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.349 DEBUG:boto:Method: GET 2024-04-13T22:25:31.349 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.350 DEBUG:boto:Data: 2024-04-13T22:25:31.350 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.350 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.350 DEBUG:boto:Params: {} 2024-04-13T22:25:31.350 DEBUG:boto:Token: None 2024-04-13T22:25:31.350 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.350 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk= 2024-04-13T22:25:31.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk=', 'Content-Length': '0'} 2024-04-13T22:25:31.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d09ee8c39f94d7-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.394 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.395 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.395 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.395 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.395 DEBUG:boto:Method: GET 2024-04-13T22:25:31.395 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.395 DEBUG:boto:Data: 2024-04-13T22:25:31.395 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.395 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.395 DEBUG:boto:Params: {} 2024-04-13T22:25:31.395 DEBUG:boto:Token: None 2024-04-13T22:25:31.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk= 2024-04-13T22:25:31.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk=', 'Content-Length': '0'} 2024-04-13T22:25:31.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b26a95a01d64f1a-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.398 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.398 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.398 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.398 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.398 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.399 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.399 DEBUG:boto:Data: 2024-04-13T22:25:31.399 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.399 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.399 DEBUG:boto:Params: {} 2024-04-13T22:25:31.399 DEBUG:boto:Token: None 2024-04-13T22:25:31.400 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ee/TcT9y4V9NQa1w1aZdtiQW0Wg= 2024-04-13T22:25:31.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ee/TcT9y4V9NQa1w1aZdtiQW0Wg=', 'Content-Length': '0'} 2024-04-13T22:25:31.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d6765bd090b692-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.402 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.402 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.402 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.402 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.402 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.402 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.402 DEBUG:boto:Data: 2024-04-13T22:25:31.402 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.402 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.402 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.403 DEBUG:boto:Params: {} 2024-04-13T22:25:31.403 DEBUG:boto:Token: None 2024-04-13T22:25:31.403 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.403 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ee/TcT9y4V9NQa1w1aZdtiQW0Wg= 2024-04-13T22:25:31.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ee/TcT9y4V9NQa1w1aZdtiQW0Wg=', 'Content-Length': '0'} 2024-04-13T22:25:31.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc90e191c5520ae0-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:25:31.405 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.405 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.406 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.406 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.406 DEBUG:boto:Method: GET 2024-04-13T22:25:31.406 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.406 DEBUG:boto:Data: 2024-04-13T22:25:31.406 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.406 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.406 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.406 DEBUG:boto:Params: {} 2024-04-13T22:25:31.406 DEBUG:boto:Token: None 2024-04-13T22:25:31.407 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.407 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk= 2024-04-13T22:25:31.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk=', 'Content-Length': '0'} 2024-04-13T22:25:31.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008059f2ada1313dd9-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.408 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.409 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.409 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.409 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.409 DEBUG:boto:Method: GET 2024-04-13T22:25:31.409 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.409 DEBUG:boto:Data: 2024-04-13T22:25:31.409 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.409 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.409 DEBUG:boto:Params: {} 2024-04-13T22:25:31.409 DEBUG:boto:Token: None 2024-04-13T22:25:31.410 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:25:31.410 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk= 2024-04-13T22:25:31.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kv6pjwVQgHP5Ge08Yncw5hYqBAk=', 'Content-Length': '0'} 2024-04-13T22:25:31.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af84c6432dd85997-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.412 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.412 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.412 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.412 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.412 DEBUG:boto:Method: GET 2024-04-13T22:25:31.412 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.412 DEBUG:boto:Data: 2024-04-13T22:25:31.413 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.413 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.413 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.413 DEBUG:boto:Params: {} 2024-04-13T22:25:31.413 DEBUG:boto:Token: None 2024-04-13T22:25:31.413 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de34785314dc676-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.415 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.416 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.416 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.416 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.416 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.417 DEBUG:boto:Method: GET 2024-04-13T22:25:31.417 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.417 DEBUG:boto:Data: 2024-04-13T22:25:31.417 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.417 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.417 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.417 DEBUG:boto:Params: {} 2024-04-13T22:25:31.417 DEBUG:boto:Token: None 2024-04-13T22:25:31.418 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.418 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b8770a3a3cb8dd4-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.421 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.421 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.421 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.421 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.422 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:25:31.422 DEBUG:boto:Data: 2024-04-13T22:25:31.422 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.422 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.422 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.422 DEBUG:boto:Params: {} 2024-04-13T22:25:31.422 DEBUG:boto:Token: None 2024-04-13T22:25:31.422 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19 2024-04-13T22:25:31.422 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:acF+Bw13pxtDnX/u0qA81/A7lI4= 2024-04-13T22:25:31.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:acF+Bw13pxtDnX/u0qA81/A7lI4=', 'Content-Length': '0'} 2024-04-13T22:25:31.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470c9592fa2ec4f7-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.424 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.424 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.424 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.424 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:25:31.425 DEBUG:boto:Data: 2024-04-13T22:25:31.425 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.425 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.425 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.425 DEBUG:boto:Params: {} 2024-04-13T22:25:31.425 DEBUG:boto:Token: None 2024-04-13T22:25:31.425 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19 2024-04-13T22:25:31.425 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:acF+Bw13pxtDnX/u0qA81/A7lI4= 2024-04-13T22:25:31.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:acF+Bw13pxtDnX/u0qA81/A7lI4=', 'Content-Length': '0'} 2024-04-13T22:25:31.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbd8469f38ce192-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:25:31.428 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.428 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.428 DEBUG:boto:Method: GET 2024-04-13T22:25:31.428 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:25:31.428 DEBUG:boto:Data: 2024-04-13T22:25:31.428 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.428 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.428 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.428 DEBUG:boto:Params: {} 2024-04-13T22:25:31.428 DEBUG:boto:Token: None 2024-04-13T22:25:31.429 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19 2024-04-13T22:25:31.429 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PnTvtvuvAgZU5DsMFCbusD0LFGQ= 2024-04-13T22:25:31.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PnTvtvuvAgZU5DsMFCbusD0LFGQ=', 'Content-Length': '0'} 2024-04-13T22:25:31.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002148ff6979cd34c9-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.431 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:25:31.431 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:25:31.431 DEBUG:boto:Method: GET 2024-04-13T22:25:31.431 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:25:31.431 DEBUG:boto:Data: 2024-04-13T22:25:31.431 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.431 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.431 DEBUG:boto:Params: {} 2024-04-13T22:25:31.431 DEBUG:boto:Token: None 2024-04-13T22:25:31.432 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj19 2024-04-13T22:25:31.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PnTvtvuvAgZU5DsMFCbusD0LFGQ= 2024-04-13T22:25:31.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PnTvtvuvAgZU5DsMFCbusD0LFGQ=', 'Content-Length': '0'} 2024-04-13T22:25:31.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1cbdbfd7683b776-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.478 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.479 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.479 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.479 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.479 DEBUG:boto:Method: GET 2024-04-13T22:25:31.479 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.479 DEBUG:boto:Data: 2024-04-13T22:25:31.479 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.479 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.479 DEBUG:boto:Params: {} 2024-04-13T22:25:31.480 DEBUG:boto:Token: None 2024-04-13T22:25:31.480 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89d9b1fc3f3a5ee-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.482 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.483 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.483 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.483 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.483 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.483 DEBUG:boto:Method: GET 2024-04-13T22:25:31.483 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:25:31.483 DEBUG:boto:Data: 2024-04-13T22:25:31.483 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.484 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.484 DEBUG:boto:Params: {} 2024-04-13T22:25:31.484 DEBUG:boto:Token: None 2024-04-13T22:25:31.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090bcaf45e43ddc3d-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.487 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:25:31.487 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.487 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.488 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.488 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.488 DEBUG:boto:Method: GET 2024-04-13T22:25:31.488 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.488 DEBUG:boto:Data: 2024-04-13T22:25:31.488 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.488 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.488 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.488 DEBUG:boto:Params: {} 2024-04-13T22:25:31.488 DEBUG:boto:Token: None 2024-04-13T22:25:31.489 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.489 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg= 2024-04-13T22:25:31.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg=', 'Content-Length': '0'} 2024-04-13T22:25:31.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005819269fd60a6b36-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.490 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.491 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.491 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.491 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.491 DEBUG:boto:Method: GET 2024-04-13T22:25:31.491 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.491 DEBUG:boto:Data: 2024-04-13T22:25:31.491 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.491 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.491 DEBUG:boto:Params: {} 2024-04-13T22:25:31.491 DEBUG:boto:Token: None 2024-04-13T22:25:31.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg= 2024-04-13T22:25:31.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg=', 'Content-Length': '0'} 2024-04-13T22:25:31.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d6cfbdbc68ac29-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.494 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.494 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.494 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.494 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.494 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.494 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.494 DEBUG:boto:Data: 2024-04-13T22:25:31.494 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.494 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.495 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.495 DEBUG:boto:Params: {} 2024-04-13T22:25:31.495 DEBUG:boto:Token: None 2024-04-13T22:25:31.495 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wuwlKm2lk/EvKtlAAY07kmVxmq0= 2024-04-13T22:25:31.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wuwlKm2lk/EvKtlAAY07kmVxmq0=', 'Content-Length': '0'} 2024-04-13T22:25:31.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539f5111b28b0193-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.497 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.497 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.497 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.497 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.497 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.498 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.498 DEBUG:boto:Data: 2024-04-13T22:25:31.498 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.498 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.498 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.498 DEBUG:boto:Params: {} 2024-04-13T22:25:31.498 DEBUG:boto:Token: None 2024-04-13T22:25:31.498 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.498 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wuwlKm2lk/EvKtlAAY07kmVxmq0= 2024-04-13T22:25:31.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wuwlKm2lk/EvKtlAAY07kmVxmq0=', 'Content-Length': '0'} 2024-04-13T22:25:31.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006719bf4a7a561d08-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:25:31.500 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.500 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.500 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.501 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.501 DEBUG:boto:Method: GET 2024-04-13T22:25:31.501 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.501 DEBUG:boto:Data: 2024-04-13T22:25:31.501 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.501 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.501 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.501 DEBUG:boto:Params: {} 2024-04-13T22:25:31.501 DEBUG:boto:Token: None 2024-04-13T22:25:31.501 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.502 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg= 2024-04-13T22:25:31.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg=', 'Content-Length': '0'} 2024-04-13T22:25:31.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc096448290ae486-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.546 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.547 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.547 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.547 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.547 DEBUG:boto:Method: GET 2024-04-13T22:25:31.547 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.547 DEBUG:boto:Data: 2024-04-13T22:25:31.547 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.547 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.547 DEBUG:boto:Params: {} 2024-04-13T22:25:31.548 DEBUG:boto:Token: None 2024-04-13T22:25:31.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:25:31.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg= 2024-04-13T22:25:31.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+BYD3nQ1VdTYI49oBLFdllfgrjg=', 'Content-Length': '0'} 2024-04-13T22:25:31.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebce1fbd23d6a7a4-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.550 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.550 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.550 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.551 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.551 DEBUG:boto:Method: GET 2024-04-13T22:25:31.551 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.551 DEBUG:boto:Data: 2024-04-13T22:25:31.551 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.551 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.551 DEBUG:boto:Params: {} 2024-04-13T22:25:31.551 DEBUG:boto:Token: None 2024-04-13T22:25:31.551 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003299779da892e339-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.554 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.554 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.555 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.555 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.555 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.555 DEBUG:boto:Method: GET 2024-04-13T22:25:31.555 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.555 DEBUG:boto:Data: 2024-04-13T22:25:31.555 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.555 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.555 DEBUG:boto:Params: {} 2024-04-13T22:25:31.556 DEBUG:boto:Token: None 2024-04-13T22:25:31.556 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001370c726d482dbdd-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.559 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.559 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.559 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.560 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.560 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:25:31.560 DEBUG:boto:Data: 2024-04-13T22:25:31.560 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.560 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.560 DEBUG:boto:Params: {} 2024-04-13T22:25:31.560 DEBUG:boto:Token: None 2024-04-13T22:25:31.560 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22 2024-04-13T22:25:31.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Prww98oQ5p/cYdG1QZAxmRfj4r0= 2024-04-13T22:25:31.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Prww98oQ5p/cYdG1QZAxmRfj4r0=', 'Content-Length': '0'} 2024-04-13T22:25:31.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036970ee5144a387c-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.562 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.562 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.562 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.563 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:25:31.563 DEBUG:boto:Data: 2024-04-13T22:25:31.563 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.563 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.563 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.563 DEBUG:boto:Params: {} 2024-04-13T22:25:31.563 DEBUG:boto:Token: None 2024-04-13T22:25:31.563 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22 2024-04-13T22:25:31.563 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Prww98oQ5p/cYdG1QZAxmRfj4r0= 2024-04-13T22:25:31.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Prww98oQ5p/cYdG1QZAxmRfj4r0=', 'Content-Length': '0'} 2024-04-13T22:25:31.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0295164309b0880-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:25:31.566 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.566 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.566 DEBUG:boto:Method: GET 2024-04-13T22:25:31.566 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:25:31.566 DEBUG:boto:Data: 2024-04-13T22:25:31.566 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.566 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.566 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.566 DEBUG:boto:Params: {} 2024-04-13T22:25:31.566 DEBUG:boto:Token: None 2024-04-13T22:25:31.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22 2024-04-13T22:25:31.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YUl0IhpD1rPxNpUvUBrdEacEtXA= 2024-04-13T22:25:31.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YUl0IhpD1rPxNpUvUBrdEacEtXA=', 'Content-Length': '0'} 2024-04-13T22:25:31.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269a5424a1abaaf1-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.569 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:25:31.569 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:25:31.569 DEBUG:boto:Method: GET 2024-04-13T22:25:31.569 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:25:31.569 DEBUG:boto:Data: 2024-04-13T22:25:31.569 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.569 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.569 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.570 DEBUG:boto:Params: {} 2024-04-13T22:25:31.570 DEBUG:boto:Token: None 2024-04-13T22:25:31.570 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj22 2024-04-13T22:25:31.570 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YUl0IhpD1rPxNpUvUBrdEacEtXA= 2024-04-13T22:25:31.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YUl0IhpD1rPxNpUvUBrdEacEtXA=', 'Content-Length': '0'} 2024-04-13T22:25:31.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006424de0fd04bcdf1-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.618 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.619 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.619 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.619 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.619 DEBUG:boto:Method: GET 2024-04-13T22:25:31.619 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.619 DEBUG:boto:Data: 2024-04-13T22:25:31.619 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.619 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.619 DEBUG:boto:Params: {} 2024-04-13T22:25:31.620 DEBUG:boto:Token: None 2024-04-13T22:25:31.620 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e59af42aa9e6056-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.623 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.623 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.623 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.623 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.623 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.624 DEBUG:boto:Method: GET 2024-04-13T22:25:31.624 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:25:31.624 DEBUG:boto:Data: 2024-04-13T22:25:31.624 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.624 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.624 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.624 DEBUG:boto:Params: {} 2024-04-13T22:25:31.624 DEBUG:boto:Token: None 2024-04-13T22:25:31.624 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.624 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000274343e0452dea51-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.628 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:25:31.629 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.629 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.629 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.629 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.629 DEBUG:boto:Method: GET 2024-04-13T22:25:31.629 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.629 DEBUG:boto:Data: 2024-04-13T22:25:31.630 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.630 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.630 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.630 DEBUG:boto:Params: {} 2024-04-13T22:25:31.630 DEBUG:boto:Token: None 2024-04-13T22:25:31.630 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg= 2024-04-13T22:25:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg=', 'Content-Length': '0'} 2024-04-13T22:25:31.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bdb5dfd8e82a478-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.632 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.633 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.633 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.633 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.633 DEBUG:boto:Method: GET 2024-04-13T22:25:31.633 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.633 DEBUG:boto:Data: 2024-04-13T22:25:31.633 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.633 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.633 DEBUG:boto:Params: {} 2024-04-13T22:25:31.633 DEBUG:boto:Token: None 2024-04-13T22:25:31.634 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.634 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg= 2024-04-13T22:25:31.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg=', 'Content-Length': '0'} 2024-04-13T22:25:31.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325d30ebbef99f2d-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.636 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.636 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.636 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.636 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.636 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.636 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.636 DEBUG:boto:Data: 2024-04-13T22:25:31.636 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.637 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.637 DEBUG:boto:Params: {} 2024-04-13T22:25:31.637 DEBUG:boto:Token: None 2024-04-13T22:25:31.637 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.637 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bgQ82NRXnw7pMLvHyo9kVFETxwM= 2024-04-13T22:25:31.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bgQ82NRXnw7pMLvHyo9kVFETxwM=', 'Content-Length': '0'} 2024-04-13T22:25:31.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7e4d9c5e39e39d-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.639 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.639 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.639 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.639 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.639 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.639 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.639 DEBUG:boto:Data: 2024-04-13T22:25:31.639 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.639 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.640 DEBUG:boto:Params: {} 2024-04-13T22:25:31.640 DEBUG:boto:Token: None 2024-04-13T22:25:31.640 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bgQ82NRXnw7pMLvHyo9kVFETxwM= 2024-04-13T22:25:31.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bgQ82NRXnw7pMLvHyo9kVFETxwM=', 'Content-Length': '0'} 2024-04-13T22:25:31.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001431d39355ae06b-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:25:31.642 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.642 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.642 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.643 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.643 DEBUG:boto:Method: GET 2024-04-13T22:25:31.643 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.643 DEBUG:boto:Data: 2024-04-13T22:25:31.643 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.643 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.643 DEBUG:boto:Params: {} 2024-04-13T22:25:31.643 DEBUG:boto:Token: None 2024-04-13T22:25:31.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg= 2024-04-13T22:25:31.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg=', 'Content-Length': '0'} 2024-04-13T22:25:31.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048e094277cbc5b44-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.686 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.686 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.687 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.687 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.687 DEBUG:boto:Method: GET 2024-04-13T22:25:31.687 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.687 DEBUG:boto:Data: 2024-04-13T22:25:31.687 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.687 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.687 DEBUG:boto:Params: {} 2024-04-13T22:25:31.687 DEBUG:boto:Token: None 2024-04-13T22:25:31.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:25:31.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg= 2024-04-13T22:25:31.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LLPQvDeUAKzmCDPquwJaX/tmrmg=', 'Content-Length': '0'} 2024-04-13T22:25:31.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfee1fdc75267102-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.690 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.690 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.690 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.690 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.690 DEBUG:boto:Method: GET 2024-04-13T22:25:31.690 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.690 DEBUG:boto:Data: 2024-04-13T22:25:31.691 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.691 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.691 DEBUG:boto:Params: {} 2024-04-13T22:25:31.691 DEBUG:boto:Token: None 2024-04-13T22:25:31.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d567e3dfd9afa22-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.693 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.694 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.694 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.694 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.694 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.694 DEBUG:boto:Method: GET 2024-04-13T22:25:31.694 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.694 DEBUG:boto:Data: 2024-04-13T22:25:31.695 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.695 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.695 DEBUG:boto:Params: {} 2024-04-13T22:25:31.695 DEBUG:boto:Token: None 2024-04-13T22:25:31.695 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.695 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f44fd2d377f8751-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.698 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.698 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.698 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.699 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.699 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:25:31.699 DEBUG:boto:Data: 2024-04-13T22:25:31.699 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.699 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.699 DEBUG:boto:Params: {} 2024-04-13T22:25:31.699 DEBUG:boto:Token: None 2024-04-13T22:25:31.699 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23 2024-04-13T22:25:31.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x2oCxNnNrGoSwMw3F/Nka2eC4ME= 2024-04-13T22:25:31.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x2oCxNnNrGoSwMw3F/Nka2eC4ME=', 'Content-Length': '0'} 2024-04-13T22:25:31.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000548df24d140751e-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.701 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.701 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.701 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.702 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:25:31.702 DEBUG:boto:Data: 2024-04-13T22:25:31.702 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.702 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.702 DEBUG:boto:Params: {} 2024-04-13T22:25:31.702 DEBUG:boto:Token: None 2024-04-13T22:25:31.702 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23 2024-04-13T22:25:31.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x2oCxNnNrGoSwMw3F/Nka2eC4ME= 2024-04-13T22:25:31.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x2oCxNnNrGoSwMw3F/Nka2eC4ME=', 'Content-Length': '0'} 2024-04-13T22:25:31.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000714687d455e6bd5c-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:25:31.705 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.705 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.705 DEBUG:boto:Method: GET 2024-04-13T22:25:31.705 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:25:31.705 DEBUG:boto:Data: 2024-04-13T22:25:31.705 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.705 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.705 DEBUG:boto:Params: {} 2024-04-13T22:25:31.705 DEBUG:boto:Token: None 2024-04-13T22:25:31.706 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23 2024-04-13T22:25:31.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JMkKEfRe0xyAyGjjNIPBZxTlN3I= 2024-04-13T22:25:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JMkKEfRe0xyAyGjjNIPBZxTlN3I=', 'Content-Length': '0'} 2024-04-13T22:25:31.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad771b601796afa-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.707 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:25:31.707 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:25:31.708 DEBUG:boto:Method: GET 2024-04-13T22:25:31.708 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:25:31.708 DEBUG:boto:Data: 2024-04-13T22:25:31.708 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.708 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.708 DEBUG:boto:Params: {} 2024-04-13T22:25:31.708 DEBUG:boto:Token: None 2024-04-13T22:25:31.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj23 2024-04-13T22:25:31.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JMkKEfRe0xyAyGjjNIPBZxTlN3I= 2024-04-13T22:25:31.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JMkKEfRe0xyAyGjjNIPBZxTlN3I=', 'Content-Length': '0'} 2024-04-13T22:25:31.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007202b534d7843d82-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.710 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.710 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.711 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.711 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.711 DEBUG:boto:Method: GET 2024-04-13T22:25:31.711 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.711 DEBUG:boto:Data: 2024-04-13T22:25:31.711 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.711 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.711 DEBUG:boto:Params: {} 2024-04-13T22:25:31.711 DEBUG:boto:Token: None 2024-04-13T22:25:31.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000684f867cd8d048b9-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.714 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.714 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.715 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.715 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.715 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.715 DEBUG:boto:Method: GET 2024-04-13T22:25:31.715 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:25:31.715 DEBUG:boto:Data: 2024-04-13T22:25:31.715 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.715 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.715 DEBUG:boto:Params: {} 2024-04-13T22:25:31.715 DEBUG:boto:Token: None 2024-04-13T22:25:31.716 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075016e8c0dfd58ba-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.719 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:25:31.719 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.719 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.720 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.720 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.720 DEBUG:boto:Method: GET 2024-04-13T22:25:31.720 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.720 DEBUG:boto:Data: 2024-04-13T22:25:31.720 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.720 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.720 DEBUG:boto:Params: {} 2024-04-13T22:25:31.720 DEBUG:boto:Token: None 2024-04-13T22:25:31.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng= 2024-04-13T22:25:31.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng=', 'Content-Length': '0'} 2024-04-13T22:25:31.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aff00150ac184e1-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.723 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.723 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.723 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.723 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.723 DEBUG:boto:Method: GET 2024-04-13T22:25:31.723 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.723 DEBUG:boto:Data: 2024-04-13T22:25:31.723 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.724 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.724 DEBUG:boto:Params: {} 2024-04-13T22:25:31.724 DEBUG:boto:Token: None 2024-04-13T22:25:31.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng= 2024-04-13T22:25:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng=', 'Content-Length': '0'} 2024-04-13T22:25:31.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aaef8af5fd766e7-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.766 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.767 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.767 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.767 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.767 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.767 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.767 DEBUG:boto:Data: 2024-04-13T22:25:31.767 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.767 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.768 DEBUG:boto:Params: {} 2024-04-13T22:25:31.768 DEBUG:boto:Token: None 2024-04-13T22:25:31.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e2QemqpaUWwjws+WOeT68/wYJPM= 2024-04-13T22:25:31.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e2QemqpaUWwjws+WOeT68/wYJPM=', 'Content-Length': '0'} 2024-04-13T22:25:31.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001189fd13ae6475c6-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.770 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.770 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.770 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.770 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.770 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.770 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.770 DEBUG:boto:Data: 2024-04-13T22:25:31.771 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.771 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.771 DEBUG:boto:Params: {} 2024-04-13T22:25:31.771 DEBUG:boto:Token: None 2024-04-13T22:25:31.771 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.771 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e2QemqpaUWwjws+WOeT68/wYJPM= 2024-04-13T22:25:31.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e2QemqpaUWwjws+WOeT68/wYJPM=', 'Content-Length': '0'} 2024-04-13T22:25:31.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c084dd6fa3bb66b1-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:25:31.773 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.773 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.773 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.773 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.773 DEBUG:boto:Method: GET 2024-04-13T22:25:31.773 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.774 DEBUG:boto:Data: 2024-04-13T22:25:31.774 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.774 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.774 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.774 DEBUG:boto:Params: {} 2024-04-13T22:25:31.774 DEBUG:boto:Token: None 2024-04-13T22:25:31.774 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.774 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng= 2024-04-13T22:25:31.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng=', 'Content-Length': '0'} 2024-04-13T22:25:31.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078af244882335a7d-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.776 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.776 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.777 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.777 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.777 DEBUG:boto:Method: GET 2024-04-13T22:25:31.777 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.777 DEBUG:boto:Data: 2024-04-13T22:25:31.777 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.777 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.777 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.777 DEBUG:boto:Params: {} 2024-04-13T22:25:31.777 DEBUG:boto:Token: None 2024-04-13T22:25:31.777 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:25:31.778 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng= 2024-04-13T22:25:31.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:khnhZHsUsNfQ688WFFvAQ2vT/Ng=', 'Content-Length': '0'} 2024-04-13T22:25:31.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436c4b4de7d7b8a7-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.780 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.780 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.780 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.780 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.780 DEBUG:boto:Method: GET 2024-04-13T22:25:31.780 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.780 DEBUG:boto:Data: 2024-04-13T22:25:31.780 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.781 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.781 DEBUG:boto:Params: {} 2024-04-13T22:25:31.781 DEBUG:boto:Token: None 2024-04-13T22:25:31.781 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.781 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e955b53dce889dc0-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.783 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.784 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.784 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.784 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.784 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.784 DEBUG:boto:Method: GET 2024-04-13T22:25:31.784 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.784 DEBUG:boto:Data: 2024-04-13T22:25:31.784 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.784 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.785 DEBUG:boto:Params: {} 2024-04-13T22:25:31.785 DEBUG:boto:Token: None 2024-04-13T22:25:31.785 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.785 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5ea291ee8bd87bf-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.788 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.788 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.788 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.788 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.788 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:25:31.788 DEBUG:boto:Data: 2024-04-13T22:25:31.788 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.788 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.789 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.789 DEBUG:boto:Params: {} 2024-04-13T22:25:31.789 DEBUG:boto:Token: None 2024-04-13T22:25:31.789 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33 2024-04-13T22:25:31.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MmDjGAVj3LY3gqHUsBuZHHerJpE= 2024-04-13T22:25:31.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MmDjGAVj3LY3gqHUsBuZHHerJpE=', 'Content-Length': '0'} 2024-04-13T22:25:31.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8824ad641f8d413-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.791 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.791 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.791 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.791 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:25:31.791 DEBUG:boto:Data: 2024-04-13T22:25:31.791 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.792 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.792 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.792 DEBUG:boto:Params: {} 2024-04-13T22:25:31.792 DEBUG:boto:Token: None 2024-04-13T22:25:31.792 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33 2024-04-13T22:25:31.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MmDjGAVj3LY3gqHUsBuZHHerJpE= 2024-04-13T22:25:31.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MmDjGAVj3LY3gqHUsBuZHHerJpE=', 'Content-Length': '0'} 2024-04-13T22:25:31.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f08d08ddb66dc1-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:25:31.794 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.794 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.794 DEBUG:boto:Method: GET 2024-04-13T22:25:31.795 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:25:31.795 DEBUG:boto:Data: 2024-04-13T22:25:31.795 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.795 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.795 DEBUG:boto:Params: {} 2024-04-13T22:25:31.795 DEBUG:boto:Token: None 2024-04-13T22:25:31.795 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33 2024-04-13T22:25:31.795 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VcO5kbXLYnLAxqvtumYwPYrU7AA= 2024-04-13T22:25:31.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VcO5kbXLYnLAxqvtumYwPYrU7AA=', 'Content-Length': '0'} 2024-04-13T22:25:31.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d68353f635f0d547-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.838 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:25:31.839 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:25:31.839 DEBUG:boto:Method: GET 2024-04-13T22:25:31.839 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:25:31.839 DEBUG:boto:Data: 2024-04-13T22:25:31.839 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.839 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.839 DEBUG:boto:Params: {} 2024-04-13T22:25:31.839 DEBUG:boto:Token: None 2024-04-13T22:25:31.840 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj33 2024-04-13T22:25:31.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VcO5kbXLYnLAxqvtumYwPYrU7AA= 2024-04-13T22:25:31.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VcO5kbXLYnLAxqvtumYwPYrU7AA=', 'Content-Length': '0'} 2024-04-13T22:25:31.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd6844cb1b8c52f-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.842 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.842 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.842 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.842 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.842 DEBUG:boto:Method: GET 2024-04-13T22:25:31.842 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.842 DEBUG:boto:Data: 2024-04-13T22:25:31.842 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.842 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.843 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.843 DEBUG:boto:Params: {} 2024-04-13T22:25:31.843 DEBUG:boto:Token: None 2024-04-13T22:25:31.843 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.843 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000864b40df53f532c0-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.846 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.846 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.847 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.847 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.847 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.847 DEBUG:boto:Method: GET 2024-04-13T22:25:31.847 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:25:31.847 DEBUG:boto:Data: 2024-04-13T22:25:31.847 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.847 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.847 DEBUG:boto:Params: {} 2024-04-13T22:25:31.848 DEBUG:boto:Token: None 2024-04-13T22:25:31.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000261ed318caef38fa-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.851 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:25:31.851 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.851 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.852 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.852 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.852 DEBUG:boto:Method: GET 2024-04-13T22:25:31.852 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.852 DEBUG:boto:Data: 2024-04-13T22:25:31.852 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.852 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.852 DEBUG:boto:Params: {} 2024-04-13T22:25:31.852 DEBUG:boto:Token: None 2024-04-13T22:25:31.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0= 2024-04-13T22:25:31.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0=', 'Content-Length': '0'} 2024-04-13T22:25:31.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a8efefbd9d20c54-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.855 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.855 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.855 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.855 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.855 DEBUG:boto:Method: GET 2024-04-13T22:25:31.855 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.855 DEBUG:boto:Data: 2024-04-13T22:25:31.855 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.856 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.856 DEBUG:boto:Params: {} 2024-04-13T22:25:31.856 DEBUG:boto:Token: None 2024-04-13T22:25:31.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0= 2024-04-13T22:25:31.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0=', 'Content-Length': '0'} 2024-04-13T22:25:31.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022571cf300ff5dc3-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.902 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.903 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.903 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.903 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.903 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.903 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.903 DEBUG:boto:Data: 2024-04-13T22:25:31.903 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.903 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.904 DEBUG:boto:Params: {} 2024-04-13T22:25:31.904 DEBUG:boto:Token: None 2024-04-13T22:25:31.904 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Uu/PPSD/pBltGIf/7TZBwsW/Wk= 2024-04-13T22:25:31.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Uu/PPSD/pBltGIf/7TZBwsW/Wk=', 'Content-Length': '0'} 2024-04-13T22:25:31.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f5765f948ad41e-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.906 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.906 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.906 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.906 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.906 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.906 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.907 DEBUG:boto:Data: 2024-04-13T22:25:31.907 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.907 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.907 DEBUG:boto:Params: {} 2024-04-13T22:25:31.907 DEBUG:boto:Token: None 2024-04-13T22:25:31.907 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Uu/PPSD/pBltGIf/7TZBwsW/Wk= 2024-04-13T22:25:31.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Uu/PPSD/pBltGIf/7TZBwsW/Wk=', 'Content-Length': '0'} 2024-04-13T22:25:31.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003309086fdfee7805-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:25:31.909 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.909 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.910 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.910 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.910 DEBUG:boto:Method: GET 2024-04-13T22:25:31.910 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.910 DEBUG:boto:Data: 2024-04-13T22:25:31.910 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.910 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.910 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.910 DEBUG:boto:Params: {} 2024-04-13T22:25:31.910 DEBUG:boto:Token: None 2024-04-13T22:25:31.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0= 2024-04-13T22:25:31.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0=', 'Content-Length': '0'} 2024-04-13T22:25:31.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f299eb55e3eed3d7-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.913 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.913 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.913 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.913 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.913 DEBUG:boto:Method: GET 2024-04-13T22:25:31.913 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.913 DEBUG:boto:Data: 2024-04-13T22:25:31.913 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.914 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.914 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.914 DEBUG:boto:Params: {} 2024-04-13T22:25:31.914 DEBUG:boto:Token: None 2024-04-13T22:25:31.914 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:25:31.914 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0= 2024-04-13T22:25:31.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JPXJmCvyDJZ4VZsq7Lr0bgtDQd0=', 'Content-Length': '0'} 2024-04-13T22:25:31.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37cd024209a51e2-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.916 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.916 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.916 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.916 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.916 DEBUG:boto:Method: GET 2024-04-13T22:25:31.916 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.917 DEBUG:boto:Data: 2024-04-13T22:25:31.917 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.917 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.917 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.917 DEBUG:boto:Params: {} 2024-04-13T22:25:31.917 DEBUG:boto:Token: None 2024-04-13T22:25:31.917 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.917 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8a34bf5cf165639-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.919 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.920 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.920 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.920 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.920 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.920 DEBUG:boto:Method: GET 2024-04-13T22:25:31.920 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.920 DEBUG:boto:Data: 2024-04-13T22:25:31.921 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.921 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.921 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.921 DEBUG:boto:Params: {} 2024-04-13T22:25:31.921 DEBUG:boto:Token: None 2024-04-13T22:25:31.921 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6bc0fbc2e6d1cae-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.924 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.924 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.924 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.925 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.925 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:25:31.925 DEBUG:boto:Data: 2024-04-13T22:25:31.925 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.925 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.925 DEBUG:boto:Params: {} 2024-04-13T22:25:31.925 DEBUG:boto:Token: None 2024-04-13T22:25:31.925 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34 2024-04-13T22:25:31.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yK6UT4ymAft56vKT1NL0dhHBZ1g= 2024-04-13T22:25:31.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yK6UT4ymAft56vKT1NL0dhHBZ1g=', 'Content-Length': '0'} 2024-04-13T22:25:31.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e062fe7988a975-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.927 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.927 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.928 DEBUG:boto:Method: HEAD 2024-04-13T22:25:31.928 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:25:31.928 DEBUG:boto:Data: 2024-04-13T22:25:31.928 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.928 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.928 DEBUG:boto:Params: {} 2024-04-13T22:25:31.928 DEBUG:boto:Token: None 2024-04-13T22:25:31.928 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34 2024-04-13T22:25:31.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yK6UT4ymAft56vKT1NL0dhHBZ1g= 2024-04-13T22:25:31.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yK6UT4ymAft56vKT1NL0dhHBZ1g=', 'Content-Length': '0'} 2024-04-13T22:25:31.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe635c866148b32-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:25:31.930 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.930 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.930 DEBUG:boto:Method: GET 2024-04-13T22:25:31.930 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:25:31.931 DEBUG:boto:Data: 2024-04-13T22:25:31.931 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.931 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.931 DEBUG:boto:Params: {} 2024-04-13T22:25:31.931 DEBUG:boto:Token: None 2024-04-13T22:25:31.931 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34 2024-04-13T22:25:31.931 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SpKCqF+R5z/trkAHWnGpM9oL6HM= 2024-04-13T22:25:31.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SpKCqF+R5z/trkAHWnGpM9oL6HM=', 'Content-Length': '0'} 2024-04-13T22:25:31.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aaf8238a6604381-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.974 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:25:31.975 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:25:31.975 DEBUG:boto:Method: GET 2024-04-13T22:25:31.975 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:25:31.975 DEBUG:boto:Data: 2024-04-13T22:25:31.975 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.975 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.975 DEBUG:boto:Params: {} 2024-04-13T22:25:31.975 DEBUG:boto:Token: None 2024-04-13T22:25:31.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj34 2024-04-13T22:25:31.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SpKCqF+R5z/trkAHWnGpM9oL6HM= 2024-04-13T22:25:31.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SpKCqF+R5z/trkAHWnGpM9oL6HM=', 'Content-Length': '0'} 2024-04-13T22:25:31.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea83f30d4f9ae71-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.978 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.978 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.978 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.978 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.978 DEBUG:boto:Method: GET 2024-04-13T22:25:31.979 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.979 DEBUG:boto:Data: 2024-04-13T22:25:31.979 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.979 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.979 DEBUG:boto:Params: {} 2024-04-13T22:25:31.979 DEBUG:boto:Token: None 2024-04-13T22:25:31.979 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.979 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f30c396d834c42-00661b065b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.982 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.982 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:31.982 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:31.982 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.983 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.983 DEBUG:boto:Method: GET 2024-04-13T22:25:31.983 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:25:31.983 DEBUG:boto:Data: 2024-04-13T22:25:31.983 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.983 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.983 DEBUG:boto:Params: {} 2024-04-13T22:25:31.983 DEBUG:boto:Token: None 2024-04-13T22:25:31.983 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/ 2024-04-13T22:25:31.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs= 2024-04-13T22:25:31.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DqWWHf5Cxc9bNuIhNaTqfi30gGs=', 'Content-Length': '0'} 2024-04-13T22:25:31.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010eeaf5808b90a74-00661b065b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.986 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:31.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:25:31.987 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:31.987 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:31.987 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.987 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.987 DEBUG:boto:Method: GET 2024-04-13T22:25:31.987 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.987 DEBUG:boto:Data: 2024-04-13T22:25:31.987 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:31.988 DEBUG:boto:Port: 8001 2024-04-13T22:25:31.988 DEBUG:boto:Params: {} 2024-04-13T22:25:31.988 DEBUG:boto:Token: None 2024-04-13T22:25:31.988 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l4pUcfGy0jQkQ0txH1CkdHUS60U= 2024-04-13T22:25:31.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l4pUcfGy0jQkQ0txH1CkdHUS60U=', 'Content-Length': '0'} 2024-04-13T22:25:31.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8f318d07c2ec7b-00661b065b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:31.990 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:31.990 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:31.990 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.990 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.990 DEBUG:boto:Method: GET 2024-04-13T22:25:31.991 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.991 DEBUG:boto:Data: 2024-04-13T22:25:31.991 DEBUG:boto:Headers: {} 2024-04-13T22:25:31.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:31.991 DEBUG:boto:Port: 8000 2024-04-13T22:25:31.991 DEBUG:boto:Params: {} 2024-04-13T22:25:31.991 DEBUG:boto:Token: None 2024-04-13T22:25:31.991 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:31 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:31.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l4pUcfGy0jQkQ0txH1CkdHUS60U= 2024-04-13T22:25:31.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l4pUcfGy0jQkQ0txH1CkdHUS60U=', 'Content-Length': '0'} 2024-04-13T22:25:31.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c671250c42ad3581-00661b065b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.034 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.034 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.035 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.035 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.035 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.035 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.035 DEBUG:boto:Data: 2024-04-13T22:25:32.035 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.035 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.035 DEBUG:boto:Params: {} 2024-04-13T22:25:32.035 DEBUG:boto:Token: None 2024-04-13T22:25:32.036 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+K80m7UMrSiWiLLa9GwJuQqxLnA= 2024-04-13T22:25:32.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+K80m7UMrSiWiLLa9GwJuQqxLnA=', 'Content-Length': '0'} 2024-04-13T22:25:32.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f568833e6b62b96d-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.038 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.038 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.038 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.038 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.038 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.038 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.038 DEBUG:boto:Data: 2024-04-13T22:25:32.038 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.038 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.038 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.038 DEBUG:boto:Params: {} 2024-04-13T22:25:32.039 DEBUG:boto:Token: None 2024-04-13T22:25:32.039 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+K80m7UMrSiWiLLa9GwJuQqxLnA= 2024-04-13T22:25:32.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+K80m7UMrSiWiLLa9GwJuQqxLnA=', 'Content-Length': '0'} 2024-04-13T22:25:32.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d2f6fccd62ab72-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:25:32.041 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.041 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.041 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.041 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.041 DEBUG:boto:Method: GET 2024-04-13T22:25:32.042 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.042 DEBUG:boto:Data: 2024-04-13T22:25:32.042 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.042 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.042 DEBUG:boto:Params: {} 2024-04-13T22:25:32.042 DEBUG:boto:Token: None 2024-04-13T22:25:32.042 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+kcMRCODWYvtS0N509JWBBB4C7c= 2024-04-13T22:25:32.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+kcMRCODWYvtS0N509JWBBB4C7c=', 'Content-Length': '0'} 2024-04-13T22:25:32.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a65fdf64a709b865-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.045 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.045 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.045 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.045 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.045 DEBUG:boto:Method: GET 2024-04-13T22:25:32.045 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.045 DEBUG:boto:Data: 2024-04-13T22:25:32.045 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.045 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.045 DEBUG:boto:Params: {} 2024-04-13T22:25:32.046 DEBUG:boto:Token: None 2024-04-13T22:25:32.046 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:25:32.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+kcMRCODWYvtS0N509JWBBB4C7c= 2024-04-13T22:25:32.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+kcMRCODWYvtS0N509JWBBB4C7c=', 'Content-Length': '0'} 2024-04-13T22:25:32.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff417e10cc74fbe0-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.048 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.048 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.048 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.048 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.048 DEBUG:boto:Method: GET 2024-04-13T22:25:32.048 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.049 DEBUG:boto:Data: 2024-04-13T22:25:32.049 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.049 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.049 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.049 DEBUG:boto:Params: {} 2024-04-13T22:25:32.049 DEBUG:boto:Token: None 2024-04-13T22:25:32.049 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db0da3d54ccb0d65-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.051 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.052 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.052 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.052 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.052 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.052 DEBUG:boto:Method: GET 2024-04-13T22:25:32.052 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.052 DEBUG:boto:Data: 2024-04-13T22:25:32.053 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.053 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.053 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.053 DEBUG:boto:Params: {} 2024-04-13T22:25:32.053 DEBUG:boto:Token: None 2024-04-13T22:25:32.053 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.053 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f4f7107b3ce2614-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.056 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.056 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.056 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.057 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.057 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:25:32.057 DEBUG:boto:Data: 2024-04-13T22:25:32.057 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.057 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.057 DEBUG:boto:Params: {} 2024-04-13T22:25:32.057 DEBUG:boto:Token: None 2024-04-13T22:25:32.057 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38 2024-04-13T22:25:32.057 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y1qjulX4byWatXrQyNWX9iOIZfc= 2024-04-13T22:25:32.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y1qjulX4byWatXrQyNWX9iOIZfc=', 'Content-Length': '0'} 2024-04-13T22:25:32.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d392853d23b8b0-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.059 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.059 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.059 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.059 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:25:32.060 DEBUG:boto:Data: 2024-04-13T22:25:32.060 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.060 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.060 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.060 DEBUG:boto:Params: {} 2024-04-13T22:25:32.060 DEBUG:boto:Token: None 2024-04-13T22:25:32.060 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38 2024-04-13T22:25:32.060 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y1qjulX4byWatXrQyNWX9iOIZfc= 2024-04-13T22:25:32.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y1qjulX4byWatXrQyNWX9iOIZfc=', 'Content-Length': '0'} 2024-04-13T22:25:32.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f54b75c0851c870-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:25:32.062 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.062 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.062 DEBUG:boto:Method: GET 2024-04-13T22:25:32.062 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:25:32.062 DEBUG:boto:Data: 2024-04-13T22:25:32.063 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.063 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.063 DEBUG:boto:Params: {} 2024-04-13T22:25:32.063 DEBUG:boto:Token: None 2024-04-13T22:25:32.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38 2024-04-13T22:25:32.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q7oAFb4aaJVWIRkRAuAK0vDa4bQ= 2024-04-13T22:25:32.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q7oAFb4aaJVWIRkRAuAK0vDa4bQ=', 'Content-Length': '0'} 2024-04-13T22:25:32.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073aa38808a1fd587-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.106 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:25:32.107 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:25:32.107 DEBUG:boto:Method: GET 2024-04-13T22:25:32.107 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:25:32.107 DEBUG:boto:Data: 2024-04-13T22:25:32.107 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.107 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.107 DEBUG:boto:Params: {} 2024-04-13T22:25:32.108 DEBUG:boto:Token: None 2024-04-13T22:25:32.108 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj38 2024-04-13T22:25:32.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q7oAFb4aaJVWIRkRAuAK0vDa4bQ= 2024-04-13T22:25:32.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q7oAFb4aaJVWIRkRAuAK0vDa4bQ=', 'Content-Length': '0'} 2024-04-13T22:25:32.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000770c21311e7f6d84-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.110 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.110 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.110 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.110 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.110 DEBUG:boto:Method: GET 2024-04-13T22:25:32.110 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.111 DEBUG:boto:Data: 2024-04-13T22:25:32.111 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.111 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.111 DEBUG:boto:Params: {} 2024-04-13T22:25:32.111 DEBUG:boto:Token: None 2024-04-13T22:25:32.111 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8db695276551846-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.113 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.114 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.114 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.114 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.114 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.114 DEBUG:boto:Method: GET 2024-04-13T22:25:32.114 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:25:32.114 DEBUG:boto:Data: 2024-04-13T22:25:32.115 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.115 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.115 DEBUG:boto:Params: {} 2024-04-13T22:25:32.115 DEBUG:boto:Token: None 2024-04-13T22:25:32.115 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.115 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af622de6efe13adc-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.118 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:25:32.119 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.119 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.119 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.119 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.119 DEBUG:boto:Method: GET 2024-04-13T22:25:32.119 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.119 DEBUG:boto:Data: 2024-04-13T22:25:32.119 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.119 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.119 DEBUG:boto:Params: {} 2024-04-13T22:25:32.120 DEBUG:boto:Token: None 2024-04-13T22:25:32.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE= 2024-04-13T22:25:32.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE=', 'Content-Length': '0'} 2024-04-13T22:25:32.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d7ccc4c02c100d-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.122 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.122 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.122 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.122 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.122 DEBUG:boto:Method: GET 2024-04-13T22:25:32.122 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.122 DEBUG:boto:Data: 2024-04-13T22:25:32.122 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.122 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.123 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.123 DEBUG:boto:Params: {} 2024-04-13T22:25:32.123 DEBUG:boto:Token: None 2024-04-13T22:25:32.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE= 2024-04-13T22:25:32.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE=', 'Content-Length': '0'} 2024-04-13T22:25:32.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af53273e3b20d2f2-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.125 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.125 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.125 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.126 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.126 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.126 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.126 DEBUG:boto:Data: 2024-04-13T22:25:32.126 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.126 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.126 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.126 DEBUG:boto:Params: {} 2024-04-13T22:25:32.126 DEBUG:boto:Token: None 2024-04-13T22:25:32.126 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.127 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KmGwDrURiWEQbnhbKhqyEV53DsQ= 2024-04-13T22:25:32.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KmGwDrURiWEQbnhbKhqyEV53DsQ=', 'Content-Length': '0'} 2024-04-13T22:25:32.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000792380762f018a41-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.128 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.128 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.129 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.129 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.129 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.129 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.129 DEBUG:boto:Data: 2024-04-13T22:25:32.129 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.129 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.129 DEBUG:boto:Params: {} 2024-04-13T22:25:32.129 DEBUG:boto:Token: None 2024-04-13T22:25:32.130 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.130 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KmGwDrURiWEQbnhbKhqyEV53DsQ= 2024-04-13T22:25:32.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KmGwDrURiWEQbnhbKhqyEV53DsQ=', 'Content-Length': '0'} 2024-04-13T22:25:32.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca4b3206f4596b03-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:25:32.132 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.132 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.132 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.132 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.132 DEBUG:boto:Method: GET 2024-04-13T22:25:32.132 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.132 DEBUG:boto:Data: 2024-04-13T22:25:32.132 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.132 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.133 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.133 DEBUG:boto:Params: {} 2024-04-13T22:25:32.133 DEBUG:boto:Token: None 2024-04-13T22:25:32.133 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.133 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE= 2024-04-13T22:25:32.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE=', 'Content-Length': '0'} 2024-04-13T22:25:32.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005289c986ddf5b148-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.135 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.135 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.135 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.135 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.135 DEBUG:boto:Method: GET 2024-04-13T22:25:32.135 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.135 DEBUG:boto:Data: 2024-04-13T22:25:32.135 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.136 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.136 DEBUG:boto:Params: {} 2024-04-13T22:25:32.136 DEBUG:boto:Token: None 2024-04-13T22:25:32.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:25:32.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE= 2024-04-13T22:25:32.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:raF0br886PA/XRd+Y1NAZOkcEIE=', 'Content-Length': '0'} 2024-04-13T22:25:32.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f7b0d44b15c974c-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.138 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.138 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.138 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.138 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.138 DEBUG:boto:Method: GET 2024-04-13T22:25:32.139 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.139 DEBUG:boto:Data: 2024-04-13T22:25:32.139 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.139 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.139 DEBUG:boto:Params: {} 2024-04-13T22:25:32.139 DEBUG:boto:Token: None 2024-04-13T22:25:32.139 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9a019c7158f565-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.141 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.142 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.142 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.145 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.146 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.146 DEBUG:boto:Method: GET 2024-04-13T22:25:32.146 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.146 DEBUG:boto:Data: 2024-04-13T22:25:32.146 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.146 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.146 DEBUG:boto:Params: {} 2024-04-13T22:25:32.146 DEBUG:boto:Token: None 2024-04-13T22:25:32.147 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416d66dde5445e3b-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.150 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.150 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.151 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.151 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.151 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:25:32.151 DEBUG:boto:Data: 2024-04-13T22:25:32.151 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.151 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.151 DEBUG:boto:Params: {} 2024-04-13T22:25:32.151 DEBUG:boto:Token: None 2024-04-13T22:25:32.151 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44 2024-04-13T22:25:32.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P5+0JWV/2S2R65brFR4gFb5s+q8= 2024-04-13T22:25:32.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P5+0JWV/2S2R65brFR4gFb5s+q8=', 'Content-Length': '0'} 2024-04-13T22:25:32.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9656006f776f48-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.153 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.153 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.154 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.154 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:25:32.154 DEBUG:boto:Data: 2024-04-13T22:25:32.154 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.154 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.154 DEBUG:boto:Params: {} 2024-04-13T22:25:32.154 DEBUG:boto:Token: None 2024-04-13T22:25:32.154 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44 2024-04-13T22:25:32.154 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P5+0JWV/2S2R65brFR4gFb5s+q8= 2024-04-13T22:25:32.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P5+0JWV/2S2R65brFR4gFb5s+q8=', 'Content-Length': '0'} 2024-04-13T22:25:32.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ef04e1e697c9f2-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:25:32.156 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.156 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.157 DEBUG:boto:Method: GET 2024-04-13T22:25:32.157 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:25:32.157 DEBUG:boto:Data: 2024-04-13T22:25:32.157 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.157 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.157 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.157 DEBUG:boto:Params: {} 2024-04-13T22:25:32.157 DEBUG:boto:Token: None 2024-04-13T22:25:32.157 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44 2024-04-13T22:25:32.157 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hqdnGVIPEuKFUpu3ynn2o60USRI= 2024-04-13T22:25:32.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hqdnGVIPEuKFUpu3ynn2o60USRI=', 'Content-Length': '0'} 2024-04-13T22:25:32.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf18276e99462c6-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.159 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:25:32.160 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:25:32.160 DEBUG:boto:Method: GET 2024-04-13T22:25:32.160 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:25:32.160 DEBUG:boto:Data: 2024-04-13T22:25:32.160 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.160 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.160 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.160 DEBUG:boto:Params: {} 2024-04-13T22:25:32.160 DEBUG:boto:Token: None 2024-04-13T22:25:32.160 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj44 2024-04-13T22:25:32.161 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hqdnGVIPEuKFUpu3ynn2o60USRI= 2024-04-13T22:25:32.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hqdnGVIPEuKFUpu3ynn2o60USRI=', 'Content-Length': '0'} 2024-04-13T22:25:32.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000836a313c53e641ae-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.162 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.163 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.163 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.163 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.163 DEBUG:boto:Method: GET 2024-04-13T22:25:32.163 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.163 DEBUG:boto:Data: 2024-04-13T22:25:32.163 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.163 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.163 DEBUG:boto:Params: {} 2024-04-13T22:25:32.164 DEBUG:boto:Token: None 2024-04-13T22:25:32.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d665e8e4211212-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.207 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.207 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.207 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.207 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.207 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.208 DEBUG:boto:Method: GET 2024-04-13T22:25:32.208 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:25:32.208 DEBUG:boto:Data: 2024-04-13T22:25:32.208 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.208 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.208 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.208 DEBUG:boto:Params: {} 2024-04-13T22:25:32.208 DEBUG:boto:Token: None 2024-04-13T22:25:32.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178003b9cce99b85-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.212 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:25:32.212 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.212 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.212 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.212 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.213 DEBUG:boto:Method: GET 2024-04-13T22:25:32.213 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.213 DEBUG:boto:Data: 2024-04-13T22:25:32.213 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.213 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.213 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.213 DEBUG:boto:Params: {} 2024-04-13T22:25:32.213 DEBUG:boto:Token: None 2024-04-13T22:25:32.213 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.213 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo= 2024-04-13T22:25:32.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo=', 'Content-Length': '0'} 2024-04-13T22:25:32.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007371f4e4af43f98-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.216 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.216 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.216 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.216 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.217 DEBUG:boto:Method: GET 2024-04-13T22:25:32.217 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.217 DEBUG:boto:Data: 2024-04-13T22:25:32.217 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.217 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.217 DEBUG:boto:Params: {} 2024-04-13T22:25:32.217 DEBUG:boto:Token: None 2024-04-13T22:25:32.217 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo= 2024-04-13T22:25:32.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo=', 'Content-Length': '0'} 2024-04-13T22:25:32.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70a1ede56bc34e1-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.262 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.263 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.263 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.263 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.263 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.263 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.263 DEBUG:boto:Data: 2024-04-13T22:25:32.263 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.263 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.264 DEBUG:boto:Params: {} 2024-04-13T22:25:32.264 DEBUG:boto:Token: None 2024-04-13T22:25:32.264 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wcTbs+3wtb140AerHTqIRVmdbB0= 2024-04-13T22:25:32.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wcTbs+3wtb140AerHTqIRVmdbB0=', 'Content-Length': '0'} 2024-04-13T22:25:32.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5dbc29e8633a25-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.267 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.267 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.267 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.267 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.267 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.267 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.267 DEBUG:boto:Data: 2024-04-13T22:25:32.267 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.267 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.268 DEBUG:boto:Params: {} 2024-04-13T22:25:32.268 DEBUG:boto:Token: None 2024-04-13T22:25:32.268 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wcTbs+3wtb140AerHTqIRVmdbB0= 2024-04-13T22:25:32.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wcTbs+3wtb140AerHTqIRVmdbB0=', 'Content-Length': '0'} 2024-04-13T22:25:32.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47d95a94fe67a5b-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:25:32.270 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.270 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.270 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.270 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.270 DEBUG:boto:Method: GET 2024-04-13T22:25:32.271 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.271 DEBUG:boto:Data: 2024-04-13T22:25:32.271 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.271 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.271 DEBUG:boto:Params: {} 2024-04-13T22:25:32.271 DEBUG:boto:Token: None 2024-04-13T22:25:32.271 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo= 2024-04-13T22:25:32.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo=', 'Content-Length': '0'} 2024-04-13T22:25:32.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b77e392821bec6f7-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.273 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.273 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.273 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.273 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.273 DEBUG:boto:Method: GET 2024-04-13T22:25:32.274 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.274 DEBUG:boto:Data: 2024-04-13T22:25:32.274 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.274 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.274 DEBUG:boto:Params: {} 2024-04-13T22:25:32.274 DEBUG:boto:Token: None 2024-04-13T22:25:32.274 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:25:32.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo= 2024-04-13T22:25:32.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:snxRDUK4CheanKftrA2zZFkbrGo=', 'Content-Length': '0'} 2024-04-13T22:25:32.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006accbd5a24157fce-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.276 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.277 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.277 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.277 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.277 DEBUG:boto:Method: GET 2024-04-13T22:25:32.277 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.277 DEBUG:boto:Data: 2024-04-13T22:25:32.277 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.277 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.277 DEBUG:boto:Params: {} 2024-04-13T22:25:32.278 DEBUG:boto:Token: None 2024-04-13T22:25:32.278 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084613196007a6ab0-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.280 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.281 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.281 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.281 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.281 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.281 DEBUG:boto:Method: GET 2024-04-13T22:25:32.281 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.281 DEBUG:boto:Data: 2024-04-13T22:25:32.281 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.281 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.281 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.281 DEBUG:boto:Params: {} 2024-04-13T22:25:32.282 DEBUG:boto:Token: None 2024-04-13T22:25:32.282 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.282 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fefaa917f900009d-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.285 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.285 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.285 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.285 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.285 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:25:32.285 DEBUG:boto:Data: 2024-04-13T22:25:32.285 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.286 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.286 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.286 DEBUG:boto:Params: {} 2024-04-13T22:25:32.286 DEBUG:boto:Token: None 2024-04-13T22:25:32.286 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46 2024-04-13T22:25:32.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZcZ0ckfgPeuRwMdT4PBgn9iUkKA= 2024-04-13T22:25:32.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZcZ0ckfgPeuRwMdT4PBgn9iUkKA=', 'Content-Length': '0'} 2024-04-13T22:25:32.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb37a4997764d11-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.289 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.289 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.290 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.290 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:25:32.290 DEBUG:boto:Data: 2024-04-13T22:25:32.290 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.290 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.290 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.290 DEBUG:boto:Params: {} 2024-04-13T22:25:32.290 DEBUG:boto:Token: None 2024-04-13T22:25:32.290 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46 2024-04-13T22:25:32.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZcZ0ckfgPeuRwMdT4PBgn9iUkKA= 2024-04-13T22:25:32.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZcZ0ckfgPeuRwMdT4PBgn9iUkKA=', 'Content-Length': '0'} 2024-04-13T22:25:32.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f7621c76b7138dd-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:25:32.293 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.293 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.293 DEBUG:boto:Method: GET 2024-04-13T22:25:32.293 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:25:32.293 DEBUG:boto:Data: 2024-04-13T22:25:32.293 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.293 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.293 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.293 DEBUG:boto:Params: {} 2024-04-13T22:25:32.293 DEBUG:boto:Token: None 2024-04-13T22:25:32.294 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46 2024-04-13T22:25:32.294 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TcvXBz4w2g76jyCfoNpxqXn4IIw= 2024-04-13T22:25:32.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TcvXBz4w2g76jyCfoNpxqXn4IIw=', 'Content-Length': '0'} 2024-04-13T22:25:32.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abefde467217116e-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.296 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:25:32.296 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:25:32.296 DEBUG:boto:Method: GET 2024-04-13T22:25:32.296 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:25:32.296 DEBUG:boto:Data: 2024-04-13T22:25:32.296 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.296 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.296 DEBUG:boto:Params: {} 2024-04-13T22:25:32.297 DEBUG:boto:Token: None 2024-04-13T22:25:32.297 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj46 2024-04-13T22:25:32.297 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TcvXBz4w2g76jyCfoNpxqXn4IIw= 2024-04-13T22:25:32.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TcvXBz4w2g76jyCfoNpxqXn4IIw=', 'Content-Length': '0'} 2024-04-13T22:25:32.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2edb752fd89cf6d-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.299 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.299 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.299 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.299 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.299 DEBUG:boto:Method: GET 2024-04-13T22:25:32.299 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.299 DEBUG:boto:Data: 2024-04-13T22:25:32.299 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.300 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.300 DEBUG:boto:Params: {} 2024-04-13T22:25:32.300 DEBUG:boto:Token: None 2024-04-13T22:25:32.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003cac5135b75f607-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.302 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.303 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.303 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.303 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.303 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.303 DEBUG:boto:Method: GET 2024-04-13T22:25:32.303 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:25:32.303 DEBUG:boto:Data: 2024-04-13T22:25:32.303 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.304 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.304 DEBUG:boto:Params: {} 2024-04-13T22:25:32.304 DEBUG:boto:Token: None 2024-04-13T22:25:32.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e6d389b2f8b56e7-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.307 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:25:32.307 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.308 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.308 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.308 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.308 DEBUG:boto:Method: GET 2024-04-13T22:25:32.308 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.308 DEBUG:boto:Data: 2024-04-13T22:25:32.308 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.308 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.308 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.308 DEBUG:boto:Params: {} 2024-04-13T22:25:32.308 DEBUG:boto:Token: None 2024-04-13T22:25:32.309 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.309 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60= 2024-04-13T22:25:32.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60=', 'Content-Length': '0'} 2024-04-13T22:25:32.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561309c91f840dfc-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.354 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.355 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.355 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.355 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.355 DEBUG:boto:Method: GET 2024-04-13T22:25:32.355 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.355 DEBUG:boto:Data: 2024-04-13T22:25:32.355 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.355 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.355 DEBUG:boto:Params: {} 2024-04-13T22:25:32.356 DEBUG:boto:Token: None 2024-04-13T22:25:32.356 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60= 2024-04-13T22:25:32.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60=', 'Content-Length': '0'} 2024-04-13T22:25:32.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f5102fea229a71-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.358 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.358 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.358 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.358 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.358 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.359 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.359 DEBUG:boto:Data: 2024-04-13T22:25:32.359 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.359 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.359 DEBUG:boto:Params: {} 2024-04-13T22:25:32.359 DEBUG:boto:Token: None 2024-04-13T22:25:32.359 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.359 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:42+tnVyB4swaXOlvSaL6HcAAKq8= 2024-04-13T22:25:32.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:42+tnVyB4swaXOlvSaL6HcAAKq8=', 'Content-Length': '0'} 2024-04-13T22:25:32.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82d9d76f4c729b0-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.361 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.361 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.361 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.361 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.361 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.362 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.362 DEBUG:boto:Data: 2024-04-13T22:25:32.362 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.362 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.362 DEBUG:boto:Params: {} 2024-04-13T22:25:32.362 DEBUG:boto:Token: None 2024-04-13T22:25:32.362 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.362 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:42+tnVyB4swaXOlvSaL6HcAAKq8= 2024-04-13T22:25:32.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:42+tnVyB4swaXOlvSaL6HcAAKq8=', 'Content-Length': '0'} 2024-04-13T22:25:32.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180dadb073d5e0d7-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:25:32.364 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.365 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.365 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.365 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.365 DEBUG:boto:Method: GET 2024-04-13T22:25:32.365 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.365 DEBUG:boto:Data: 2024-04-13T22:25:32.365 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.365 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.365 DEBUG:boto:Params: {} 2024-04-13T22:25:32.365 DEBUG:boto:Token: None 2024-04-13T22:25:32.366 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.366 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60= 2024-04-13T22:25:32.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60=', 'Content-Length': '0'} 2024-04-13T22:25:32.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000792a972ed1ae851d-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.367 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.368 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.368 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.368 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.368 DEBUG:boto:Method: GET 2024-04-13T22:25:32.368 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.368 DEBUG:boto:Data: 2024-04-13T22:25:32.368 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.368 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.368 DEBUG:boto:Params: {} 2024-04-13T22:25:32.368 DEBUG:boto:Token: None 2024-04-13T22:25:32.369 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:25:32.369 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60= 2024-04-13T22:25:32.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xOlRA3KI0RaVpoUhYOkfsCntM60=', 'Content-Length': '0'} 2024-04-13T22:25:32.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a171e425f2add620-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.370 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.371 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.371 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.371 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.371 DEBUG:boto:Method: GET 2024-04-13T22:25:32.371 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.371 DEBUG:boto:Data: 2024-04-13T22:25:32.371 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.371 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.371 DEBUG:boto:Params: {} 2024-04-13T22:25:32.372 DEBUG:boto:Token: None 2024-04-13T22:25:32.372 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4195aaee8d7dc4-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.374 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.375 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.375 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.375 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.375 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.375 DEBUG:boto:Method: GET 2024-04-13T22:25:32.375 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.375 DEBUG:boto:Data: 2024-04-13T22:25:32.375 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.375 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.376 DEBUG:boto:Params: {} 2024-04-13T22:25:32.376 DEBUG:boto:Token: None 2024-04-13T22:25:32.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042790b4327c1d525-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.422 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.423 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.423 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.423 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.423 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:25:32.423 DEBUG:boto:Data: 2024-04-13T22:25:32.423 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.424 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.424 DEBUG:boto:Params: {} 2024-04-13T22:25:32.424 DEBUG:boto:Token: None 2024-04-13T22:25:32.424 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51 2024-04-13T22:25:32.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5gEROKaoCCElvw0pm5JLRqri70g= 2024-04-13T22:25:32.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5gEROKaoCCElvw0pm5JLRqri70g=', 'Content-Length': '0'} 2024-04-13T22:25:32.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000438af6692244413a-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.426 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.427 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.427 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.427 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:25:32.427 DEBUG:boto:Data: 2024-04-13T22:25:32.427 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.427 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.427 DEBUG:boto:Params: {} 2024-04-13T22:25:32.427 DEBUG:boto:Token: None 2024-04-13T22:25:32.427 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51 2024-04-13T22:25:32.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5gEROKaoCCElvw0pm5JLRqri70g= 2024-04-13T22:25:32.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5gEROKaoCCElvw0pm5JLRqri70g=', 'Content-Length': '0'} 2024-04-13T22:25:32.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e999974c1ed561b7-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:25:32.429 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.430 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.430 DEBUG:boto:Method: GET 2024-04-13T22:25:32.430 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:25:32.430 DEBUG:boto:Data: 2024-04-13T22:25:32.430 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.430 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.430 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.430 DEBUG:boto:Params: {} 2024-04-13T22:25:32.430 DEBUG:boto:Token: None 2024-04-13T22:25:32.430 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51 2024-04-13T22:25:32.431 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4gg0M3EpiD6oez6ceU0WEK5tZxU= 2024-04-13T22:25:32.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4gg0M3EpiD6oez6ceU0WEK5tZxU=', 'Content-Length': '0'} 2024-04-13T22:25:32.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001951024f4599b519-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.433 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:25:32.433 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:25:32.433 DEBUG:boto:Method: GET 2024-04-13T22:25:32.433 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:25:32.433 DEBUG:boto:Data: 2024-04-13T22:25:32.433 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.433 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.433 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.433 DEBUG:boto:Params: {} 2024-04-13T22:25:32.434 DEBUG:boto:Token: None 2024-04-13T22:25:32.434 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj51 2024-04-13T22:25:32.434 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4gg0M3EpiD6oez6ceU0WEK5tZxU= 2024-04-13T22:25:32.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4gg0M3EpiD6oez6ceU0WEK5tZxU=', 'Content-Length': '0'} 2024-04-13T22:25:32.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df14b24a1350ea7-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.436 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.436 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.436 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.436 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.436 DEBUG:boto:Method: GET 2024-04-13T22:25:32.436 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.436 DEBUG:boto:Data: 2024-04-13T22:25:32.436 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.436 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.437 DEBUG:boto:Params: {} 2024-04-13T22:25:32.437 DEBUG:boto:Token: None 2024-04-13T22:25:32.437 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040940469e90c8bd4-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.439 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.440 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.440 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.440 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.440 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.440 DEBUG:boto:Method: GET 2024-04-13T22:25:32.440 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:25:32.440 DEBUG:boto:Data: 2024-04-13T22:25:32.440 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.440 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.440 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.440 DEBUG:boto:Params: {} 2024-04-13T22:25:32.441 DEBUG:boto:Token: None 2024-04-13T22:25:32.441 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e69f8a064de5d84-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.443 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:25:32.444 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.444 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.444 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.444 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.444 DEBUG:boto:Method: GET 2024-04-13T22:25:32.444 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.445 DEBUG:boto:Data: 2024-04-13T22:25:32.445 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.445 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.445 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.445 DEBUG:boto:Params: {} 2024-04-13T22:25:32.445 DEBUG:boto:Token: None 2024-04-13T22:25:32.445 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c= 2024-04-13T22:25:32.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c=', 'Content-Length': '0'} 2024-04-13T22:25:32.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1eacff4164cbc37-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.447 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.447 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.448 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.448 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.448 DEBUG:boto:Method: GET 2024-04-13T22:25:32.448 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.448 DEBUG:boto:Data: 2024-04-13T22:25:32.448 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.448 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.448 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.448 DEBUG:boto:Params: {} 2024-04-13T22:25:32.448 DEBUG:boto:Token: None 2024-04-13T22:25:32.449 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.449 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c= 2024-04-13T22:25:32.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c=', 'Content-Length': '0'} 2024-04-13T22:25:32.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e4f7116c9f6daa-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.451 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.451 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.451 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.451 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.451 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.451 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.451 DEBUG:boto:Data: 2024-04-13T22:25:32.451 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.451 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.452 DEBUG:boto:Params: {} 2024-04-13T22:25:32.452 DEBUG:boto:Token: None 2024-04-13T22:25:32.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1G3NAll0qreR3DgD4wMluLZPLng= 2024-04-13T22:25:32.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1G3NAll0qreR3DgD4wMluLZPLng=', 'Content-Length': '0'} 2024-04-13T22:25:32.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b45ed894ac7881-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.454 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.454 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.454 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.455 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.455 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.455 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.455 DEBUG:boto:Data: 2024-04-13T22:25:32.455 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.455 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.455 DEBUG:boto:Params: {} 2024-04-13T22:25:32.455 DEBUG:boto:Token: None 2024-04-13T22:25:32.455 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1G3NAll0qreR3DgD4wMluLZPLng= 2024-04-13T22:25:32.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1G3NAll0qreR3DgD4wMluLZPLng=', 'Content-Length': '0'} 2024-04-13T22:25:32.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038524d2f654a5b4c-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:25:32.458 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.458 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.458 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.458 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.458 DEBUG:boto:Method: GET 2024-04-13T22:25:32.458 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.458 DEBUG:boto:Data: 2024-04-13T22:25:32.458 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.459 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.459 DEBUG:boto:Params: {} 2024-04-13T22:25:32.459 DEBUG:boto:Token: None 2024-04-13T22:25:32.459 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c= 2024-04-13T22:25:32.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c=', 'Content-Length': '0'} 2024-04-13T22:25:32.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d68e0c62b9371b-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.502 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.503 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.503 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.503 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.503 DEBUG:boto:Method: GET 2024-04-13T22:25:32.503 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.503 DEBUG:boto:Data: 2024-04-13T22:25:32.503 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.503 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.503 DEBUG:boto:Params: {} 2024-04-13T22:25:32.504 DEBUG:boto:Token: None 2024-04-13T22:25:32.504 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:25:32.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c= 2024-04-13T22:25:32.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hq/GhQqDrKoLAxsRa5ZiAvC0s+c=', 'Content-Length': '0'} 2024-04-13T22:25:32.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a596bdf200d4c41-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.506 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.506 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.506 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.506 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.506 DEBUG:boto:Method: GET 2024-04-13T22:25:32.506 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.507 DEBUG:boto:Data: 2024-04-13T22:25:32.507 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.507 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.507 DEBUG:boto:Params: {} 2024-04-13T22:25:32.507 DEBUG:boto:Token: None 2024-04-13T22:25:32.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b994bcec41ae6d69-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.510 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.510 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.510 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.510 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.510 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.511 DEBUG:boto:Method: GET 2024-04-13T22:25:32.511 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.511 DEBUG:boto:Data: 2024-04-13T22:25:32.511 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.511 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.511 DEBUG:boto:Params: {} 2024-04-13T22:25:32.511 DEBUG:boto:Token: None 2024-04-13T22:25:32.511 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004114ad6264b48dad-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.514 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.515 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.515 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.515 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.515 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:25:32.515 DEBUG:boto:Data: 2024-04-13T22:25:32.515 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.515 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.515 DEBUG:boto:Params: {} 2024-04-13T22:25:32.516 DEBUG:boto:Token: None 2024-04-13T22:25:32.516 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55 2024-04-13T22:25:32.516 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v2mUMgACK/zYKaVtv5jUlVWtrRo= 2024-04-13T22:25:32.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v2mUMgACK/zYKaVtv5jUlVWtrRo=', 'Content-Length': '0'} 2024-04-13T22:25:32.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0612f2eb753734-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.518 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.518 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.518 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.518 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:25:32.518 DEBUG:boto:Data: 2024-04-13T22:25:32.519 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.519 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.519 DEBUG:boto:Params: {} 2024-04-13T22:25:32.519 DEBUG:boto:Token: None 2024-04-13T22:25:32.519 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55 2024-04-13T22:25:32.519 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:v2mUMgACK/zYKaVtv5jUlVWtrRo= 2024-04-13T22:25:32.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:v2mUMgACK/zYKaVtv5jUlVWtrRo=', 'Content-Length': '0'} 2024-04-13T22:25:32.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc0c7d7bfffb9cb-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:25:32.521 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.521 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.521 DEBUG:boto:Method: GET 2024-04-13T22:25:32.522 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:25:32.522 DEBUG:boto:Data: 2024-04-13T22:25:32.522 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.522 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.522 DEBUG:boto:Params: {} 2024-04-13T22:25:32.522 DEBUG:boto:Token: None 2024-04-13T22:25:32.522 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55 2024-04-13T22:25:32.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I1xfFf1ZiHa5PsPqi2gIU7iH57c= 2024-04-13T22:25:32.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I1xfFf1ZiHa5PsPqi2gIU7iH57c=', 'Content-Length': '0'} 2024-04-13T22:25:32.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3074109b66453d7-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.524 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:25:32.525 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:25:32.525 DEBUG:boto:Method: GET 2024-04-13T22:25:32.525 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:25:32.525 DEBUG:boto:Data: 2024-04-13T22:25:32.525 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.525 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.525 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.525 DEBUG:boto:Params: {} 2024-04-13T22:25:32.525 DEBUG:boto:Token: None 2024-04-13T22:25:32.525 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj55 2024-04-13T22:25:32.526 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I1xfFf1ZiHa5PsPqi2gIU7iH57c= 2024-04-13T22:25:32.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I1xfFf1ZiHa5PsPqi2gIU7iH57c=', 'Content-Length': '0'} 2024-04-13T22:25:32.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffc338965467377c-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.527 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.528 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.528 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.528 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.528 DEBUG:boto:Method: GET 2024-04-13T22:25:32.528 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.528 DEBUG:boto:Data: 2024-04-13T22:25:32.528 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.528 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.528 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.528 DEBUG:boto:Params: {} 2024-04-13T22:25:32.529 DEBUG:boto:Token: None 2024-04-13T22:25:32.529 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.529 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d703d8191ecc46e-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.531 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.532 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.532 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.532 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.532 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.532 DEBUG:boto:Method: GET 2024-04-13T22:25:32.532 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:25:32.532 DEBUG:boto:Data: 2024-04-13T22:25:32.532 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.532 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.532 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.533 DEBUG:boto:Params: {} 2024-04-13T22:25:32.533 DEBUG:boto:Token: None 2024-04-13T22:25:32.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a797928401cc9f40-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.536 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:25:32.536 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.536 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.537 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.537 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.537 DEBUG:boto:Method: GET 2024-04-13T22:25:32.537 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.537 DEBUG:boto:Data: 2024-04-13T22:25:32.537 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.537 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.537 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.537 DEBUG:boto:Params: {} 2024-04-13T22:25:32.537 DEBUG:boto:Token: None 2024-04-13T22:25:32.537 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.538 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY= 2024-04-13T22:25:32.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY=', 'Content-Length': '0'} 2024-04-13T22:25:32.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c80c2c720e468ca3-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.539 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.539 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.540 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.540 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.540 DEBUG:boto:Method: GET 2024-04-13T22:25:32.540 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.540 DEBUG:boto:Data: 2024-04-13T22:25:32.540 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.540 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.540 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.540 DEBUG:boto:Params: {} 2024-04-13T22:25:32.540 DEBUG:boto:Token: None 2024-04-13T22:25:32.541 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY= 2024-04-13T22:25:32.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY=', 'Content-Length': '0'} 2024-04-13T22:25:32.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d98d81924f823c-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.543 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.543 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.543 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.543 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.543 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.543 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.543 DEBUG:boto:Data: 2024-04-13T22:25:32.543 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.544 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.544 DEBUG:boto:Params: {} 2024-04-13T22:25:32.544 DEBUG:boto:Token: None 2024-04-13T22:25:32.544 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I/T1pgAEUyrtI2yg2MmWLiC0Kz8= 2024-04-13T22:25:32.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I/T1pgAEUyrtI2yg2MmWLiC0Kz8=', 'Content-Length': '0'} 2024-04-13T22:25:32.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000820ba1938b0d8012-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.546 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.546 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.546 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.546 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.546 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.546 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.546 DEBUG:boto:Data: 2024-04-13T22:25:32.546 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.547 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.547 DEBUG:boto:Params: {} 2024-04-13T22:25:32.547 DEBUG:boto:Token: None 2024-04-13T22:25:32.547 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I/T1pgAEUyrtI2yg2MmWLiC0Kz8= 2024-04-13T22:25:32.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I/T1pgAEUyrtI2yg2MmWLiC0Kz8=', 'Content-Length': '0'} 2024-04-13T22:25:32.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db9dee814cc7d62-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:25:32.549 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.549 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.549 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.549 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.549 DEBUG:boto:Method: GET 2024-04-13T22:25:32.550 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.550 DEBUG:boto:Data: 2024-04-13T22:25:32.550 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.550 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.550 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.550 DEBUG:boto:Params: {} 2024-04-13T22:25:32.550 DEBUG:boto:Token: None 2024-04-13T22:25:32.550 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.553 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY= 2024-04-13T22:25:32.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY=', 'Content-Length': '0'} 2024-04-13T22:25:32.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ff760a173f819d-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.555 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.555 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.556 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.556 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.556 DEBUG:boto:Method: GET 2024-04-13T22:25:32.556 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.556 DEBUG:boto:Data: 2024-04-13T22:25:32.556 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.556 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.556 DEBUG:boto:Params: {} 2024-04-13T22:25:32.556 DEBUG:boto:Token: None 2024-04-13T22:25:32.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:25:32.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY= 2024-04-13T22:25:32.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wpl5zvVYhF2xA9jPzRp6Dek8FbY=', 'Content-Length': '0'} 2024-04-13T22:25:32.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d6b19c44f53058-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.602 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.603 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.603 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.603 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.603 DEBUG:boto:Method: GET 2024-04-13T22:25:32.603 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.603 DEBUG:boto:Data: 2024-04-13T22:25:32.603 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.603 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.603 DEBUG:boto:Params: {} 2024-04-13T22:25:32.604 DEBUG:boto:Token: None 2024-04-13T22:25:32.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e6fbdd55a8e294-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.606 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.607 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.607 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.607 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.607 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.607 DEBUG:boto:Method: GET 2024-04-13T22:25:32.607 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.607 DEBUG:boto:Data: 2024-04-13T22:25:32.607 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.607 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.607 DEBUG:boto:Params: {} 2024-04-13T22:25:32.608 DEBUG:boto:Token: None 2024-04-13T22:25:32.608 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b36a23f12d1e33e-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.611 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.612 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.612 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.612 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.612 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:25:32.612 DEBUG:boto:Data: 2024-04-13T22:25:32.612 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.612 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.612 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.612 DEBUG:boto:Params: {} 2024-04-13T22:25:32.613 DEBUG:boto:Token: None 2024-04-13T22:25:32.613 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57 2024-04-13T22:25:32.613 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/rLjzmd8ra6aBHeMArVLOsepkxM= 2024-04-13T22:25:32.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/rLjzmd8ra6aBHeMArVLOsepkxM=', 'Content-Length': '0'} 2024-04-13T22:25:32.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004776f76108fb74f0-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.615 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.615 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.615 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.615 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:25:32.615 DEBUG:boto:Data: 2024-04-13T22:25:32.615 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.615 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.615 DEBUG:boto:Params: {} 2024-04-13T22:25:32.615 DEBUG:boto:Token: None 2024-04-13T22:25:32.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57 2024-04-13T22:25:32.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/rLjzmd8ra6aBHeMArVLOsepkxM= 2024-04-13T22:25:32.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/rLjzmd8ra6aBHeMArVLOsepkxM=', 'Content-Length': '0'} 2024-04-13T22:25:32.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093392dc5d0f5efca-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:25:32.618 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.618 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.618 DEBUG:boto:Method: GET 2024-04-13T22:25:32.618 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:25:32.618 DEBUG:boto:Data: 2024-04-13T22:25:32.618 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.618 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.618 DEBUG:boto:Params: {} 2024-04-13T22:25:32.618 DEBUG:boto:Token: None 2024-04-13T22:25:32.619 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57 2024-04-13T22:25:32.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NePGbD1tsaeHv/tkgOgsjo8J1iY= 2024-04-13T22:25:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NePGbD1tsaeHv/tkgOgsjo8J1iY=', 'Content-Length': '0'} 2024-04-13T22:25:32.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009e88e74de1f58a2-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.662 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:25:32.663 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:25:32.663 DEBUG:boto:Method: GET 2024-04-13T22:25:32.663 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:25:32.663 DEBUG:boto:Data: 2024-04-13T22:25:32.663 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.663 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.663 DEBUG:boto:Params: {} 2024-04-13T22:25:32.664 DEBUG:boto:Token: None 2024-04-13T22:25:32.664 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj57 2024-04-13T22:25:32.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NePGbD1tsaeHv/tkgOgsjo8J1iY= 2024-04-13T22:25:32.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NePGbD1tsaeHv/tkgOgsjo8J1iY=', 'Content-Length': '0'} 2024-04-13T22:25:32.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3d488f1bd873832-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.666 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.666 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.666 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.666 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.666 DEBUG:boto:Method: GET 2024-04-13T22:25:32.666 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.666 DEBUG:boto:Data: 2024-04-13T22:25:32.667 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.667 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.667 DEBUG:boto:Params: {} 2024-04-13T22:25:32.667 DEBUG:boto:Token: None 2024-04-13T22:25:32.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a2251181da6d57-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.670 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.671 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.671 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.671 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.671 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.671 DEBUG:boto:Method: GET 2024-04-13T22:25:32.671 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:25:32.672 DEBUG:boto:Data: 2024-04-13T22:25:32.672 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.672 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.672 DEBUG:boto:Params: {} 2024-04-13T22:25:32.672 DEBUG:boto:Token: None 2024-04-13T22:25:32.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b20ea594f6ff842-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.675 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:25:32.676 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.676 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.676 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.676 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.676 DEBUG:boto:Method: GET 2024-04-13T22:25:32.676 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.676 DEBUG:boto:Data: 2024-04-13T22:25:32.677 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.677 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.677 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.677 DEBUG:boto:Params: {} 2024-04-13T22:25:32.677 DEBUG:boto:Token: None 2024-04-13T22:25:32.677 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8= 2024-04-13T22:25:32.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8=', 'Content-Length': '0'} 2024-04-13T22:25:32.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd29b96f080f57d-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.679 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.679 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.679 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.679 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.679 DEBUG:boto:Method: GET 2024-04-13T22:25:32.679 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.680 DEBUG:boto:Data: 2024-04-13T22:25:32.680 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.680 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.680 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.680 DEBUG:boto:Params: {} 2024-04-13T22:25:32.680 DEBUG:boto:Token: None 2024-04-13T22:25:32.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8= 2024-04-13T22:25:32.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8=', 'Content-Length': '0'} 2024-04-13T22:25:32.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027151d229a03bedb-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.722 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.723 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.723 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.723 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.723 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.723 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.723 DEBUG:boto:Data: 2024-04-13T22:25:32.723 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.723 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.723 DEBUG:boto:Params: {} 2024-04-13T22:25:32.724 DEBUG:boto:Token: None 2024-04-13T22:25:32.724 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bvY1KVBKJMBMTborc/90LHuaqvk= 2024-04-13T22:25:32.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bvY1KVBKJMBMTborc/90LHuaqvk=', 'Content-Length': '0'} 2024-04-13T22:25:32.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfab176c14746eb8-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.726 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.726 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.726 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.726 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.726 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.726 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.727 DEBUG:boto:Data: 2024-04-13T22:25:32.727 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.727 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.727 DEBUG:boto:Params: {} 2024-04-13T22:25:32.727 DEBUG:boto:Token: None 2024-04-13T22:25:32.727 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bvY1KVBKJMBMTborc/90LHuaqvk= 2024-04-13T22:25:32.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bvY1KVBKJMBMTborc/90LHuaqvk=', 'Content-Length': '0'} 2024-04-13T22:25:32.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000581bdfcabb94f2a7-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:25:32.729 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.730 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.730 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.730 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.730 DEBUG:boto:Method: GET 2024-04-13T22:25:32.730 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.730 DEBUG:boto:Data: 2024-04-13T22:25:32.730 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.730 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.730 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.730 DEBUG:boto:Params: {} 2024-04-13T22:25:32.730 DEBUG:boto:Token: None 2024-04-13T22:25:32.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.731 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8= 2024-04-13T22:25:32.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8=', 'Content-Length': '0'} 2024-04-13T22:25:32.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb30d1f6b79135a9-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.732 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.733 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.733 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.733 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.733 DEBUG:boto:Method: GET 2024-04-13T22:25:32.733 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.733 DEBUG:boto:Data: 2024-04-13T22:25:32.733 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.733 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.733 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.733 DEBUG:boto:Params: {} 2024-04-13T22:25:32.734 DEBUG:boto:Token: None 2024-04-13T22:25:32.734 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:25:32.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8= 2024-04-13T22:25:32.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PRtb4W7pc3A1rXlTMriCbIHLrH8=', 'Content-Length': '0'} 2024-04-13T22:25:32.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051eb335c09464737-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.736 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.736 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.736 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.736 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.736 DEBUG:boto:Method: GET 2024-04-13T22:25:32.736 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.737 DEBUG:boto:Data: 2024-04-13T22:25:32.737 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.737 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.737 DEBUG:boto:Params: {} 2024-04-13T22:25:32.737 DEBUG:boto:Token: None 2024-04-13T22:25:32.737 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9a46ea22e5f001-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.740 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.740 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.741 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.741 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.741 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.741 DEBUG:boto:Method: GET 2024-04-13T22:25:32.741 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.741 DEBUG:boto:Data: 2024-04-13T22:25:32.741 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.741 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.741 DEBUG:boto:Params: {} 2024-04-13T22:25:32.742 DEBUG:boto:Token: None 2024-04-13T22:25:32.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000053cfebe39fd23f1-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.745 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.746 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.746 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.747 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.747 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:25:32.747 DEBUG:boto:Data: 2024-04-13T22:25:32.747 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.747 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.747 DEBUG:boto:Params: {} 2024-04-13T22:25:32.747 DEBUG:boto:Token: None 2024-04-13T22:25:32.747 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68 2024-04-13T22:25:32.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YWUmpJMugaHSdKUleJwSb/elhxc= 2024-04-13T22:25:32.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YWUmpJMugaHSdKUleJwSb/elhxc=', 'Content-Length': '0'} 2024-04-13T22:25:32.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008feb0996dfde3bf-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.750 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.750 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.750 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.751 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:25:32.751 DEBUG:boto:Data: 2024-04-13T22:25:32.751 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.751 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.751 DEBUG:boto:Params: {} 2024-04-13T22:25:32.752 DEBUG:boto:Token: None 2024-04-13T22:25:32.752 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68 2024-04-13T22:25:32.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YWUmpJMugaHSdKUleJwSb/elhxc= 2024-04-13T22:25:32.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YWUmpJMugaHSdKUleJwSb/elhxc=', 'Content-Length': '0'} 2024-04-13T22:25:32.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e2e579ef041b5d-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:25:32.754 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.754 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.754 DEBUG:boto:Method: GET 2024-04-13T22:25:32.754 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:25:32.754 DEBUG:boto:Data: 2024-04-13T22:25:32.754 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.755 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.755 DEBUG:boto:Params: {} 2024-04-13T22:25:32.755 DEBUG:boto:Token: None 2024-04-13T22:25:32.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68 2024-04-13T22:25:32.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d6g5hyazOlY5zsb47W4b/i9Fhtg= 2024-04-13T22:25:32.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d6g5hyazOlY5zsb47W4b/i9Fhtg=', 'Content-Length': '0'} 2024-04-13T22:25:32.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac4c69b6e83588e-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.802 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:25:32.803 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:25:32.803 DEBUG:boto:Method: GET 2024-04-13T22:25:32.803 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:25:32.803 DEBUG:boto:Data: 2024-04-13T22:25:32.803 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.803 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.803 DEBUG:boto:Params: {} 2024-04-13T22:25:32.803 DEBUG:boto:Token: None 2024-04-13T22:25:32.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj68 2024-04-13T22:25:32.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d6g5hyazOlY5zsb47W4b/i9Fhtg= 2024-04-13T22:25:32.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d6g5hyazOlY5zsb47W4b/i9Fhtg=', 'Content-Length': '0'} 2024-04-13T22:25:32.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d783b261b05d23e1-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.806 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.806 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.806 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.806 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.806 DEBUG:boto:Method: GET 2024-04-13T22:25:32.806 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.806 DEBUG:boto:Data: 2024-04-13T22:25:32.806 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.807 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.807 DEBUG:boto:Params: {} 2024-04-13T22:25:32.807 DEBUG:boto:Token: None 2024-04-13T22:25:32.807 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e267de3298bdcc77-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.809 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.810 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.810 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.810 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.810 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.810 DEBUG:boto:Method: GET 2024-04-13T22:25:32.810 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:25:32.810 DEBUG:boto:Data: 2024-04-13T22:25:32.810 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.810 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.811 DEBUG:boto:Params: {} 2024-04-13T22:25:32.811 DEBUG:boto:Token: None 2024-04-13T22:25:32.811 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e01b6aa7cf8442ed-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.814 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:25:32.815 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.815 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.815 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.815 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.815 DEBUG:boto:Method: GET 2024-04-13T22:25:32.815 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.815 DEBUG:boto:Data: 2024-04-13T22:25:32.815 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.816 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.816 DEBUG:boto:Params: {} 2024-04-13T22:25:32.816 DEBUG:boto:Token: None 2024-04-13T22:25:32.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k= 2024-04-13T22:25:32.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k=', 'Content-Length': '0'} 2024-04-13T22:25:32.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a83400cb3b83c23-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.819 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.820 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.820 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.820 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.820 DEBUG:boto:Method: GET 2024-04-13T22:25:32.820 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.820 DEBUG:boto:Data: 2024-04-13T22:25:32.820 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.820 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.820 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.820 DEBUG:boto:Params: {} 2024-04-13T22:25:32.821 DEBUG:boto:Token: None 2024-04-13T22:25:32.821 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.821 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k= 2024-04-13T22:25:32.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k=', 'Content-Length': '0'} 2024-04-13T22:25:32.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b1f4dd61c093fa-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.823 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.823 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.823 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.823 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.823 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.823 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.823 DEBUG:boto:Data: 2024-04-13T22:25:32.823 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.824 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.824 DEBUG:boto:Params: {} 2024-04-13T22:25:32.824 DEBUG:boto:Token: None 2024-04-13T22:25:32.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D3emyVoPedqcHyVNOAmAJwlzBJE= 2024-04-13T22:25:32.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D3emyVoPedqcHyVNOAmAJwlzBJE=', 'Content-Length': '0'} 2024-04-13T22:25:32.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005532ca769e4a87bd-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.826 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.826 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.826 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.827 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.827 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.827 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.827 DEBUG:boto:Data: 2024-04-13T22:25:32.827 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.827 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.827 DEBUG:boto:Params: {} 2024-04-13T22:25:32.828 DEBUG:boto:Token: None 2024-04-13T22:25:32.828 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D3emyVoPedqcHyVNOAmAJwlzBJE= 2024-04-13T22:25:32.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D3emyVoPedqcHyVNOAmAJwlzBJE=', 'Content-Length': '0'} 2024-04-13T22:25:32.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008324e7d64b4fd7eb-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:25:32.830 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.830 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.830 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.830 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.830 DEBUG:boto:Method: GET 2024-04-13T22:25:32.830 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.830 DEBUG:boto:Data: 2024-04-13T22:25:32.831 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.831 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.831 DEBUG:boto:Params: {} 2024-04-13T22:25:32.831 DEBUG:boto:Token: None 2024-04-13T22:25:32.831 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.831 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k= 2024-04-13T22:25:32.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k=', 'Content-Length': '0'} 2024-04-13T22:25:32.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002580f490546d0b63-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.833 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.833 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.833 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.834 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.834 DEBUG:boto:Method: GET 2024-04-13T22:25:32.834 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.834 DEBUG:boto:Data: 2024-04-13T22:25:32.834 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.834 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.834 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.834 DEBUG:boto:Params: {} 2024-04-13T22:25:32.834 DEBUG:boto:Token: None 2024-04-13T22:25:32.834 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:25:32.835 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k= 2024-04-13T22:25:32.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ftt6TxpH98fEi4hjLQZsnHonh5k=', 'Content-Length': '0'} 2024-04-13T22:25:32.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050cffc1fca249abd-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.837 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.837 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.837 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.837 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.837 DEBUG:boto:Method: GET 2024-04-13T22:25:32.837 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.837 DEBUG:boto:Data: 2024-04-13T22:25:32.837 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.837 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.837 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.837 DEBUG:boto:Params: {} 2024-04-13T22:25:32.838 DEBUG:boto:Token: None 2024-04-13T22:25:32.838 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.838 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000797237f20da6c6c4-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.840 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.841 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.841 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.841 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.841 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.841 DEBUG:boto:Method: GET 2024-04-13T22:25:32.841 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.841 DEBUG:boto:Data: 2024-04-13T22:25:32.841 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.841 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.842 DEBUG:boto:Params: {} 2024-04-13T22:25:32.842 DEBUG:boto:Token: None 2024-04-13T22:25:32.842 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.842 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b842cd53726a480-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.845 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.845 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.846 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.846 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.846 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:25:32.846 DEBUG:boto:Data: 2024-04-13T22:25:32.846 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.846 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.846 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.846 DEBUG:boto:Params: {} 2024-04-13T22:25:32.846 DEBUG:boto:Token: None 2024-04-13T22:25:32.846 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69 2024-04-13T22:25:32.847 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1fU1arSgYooet5183FnfMDZ5N/k= 2024-04-13T22:25:32.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1fU1arSgYooet5183FnfMDZ5N/k=', 'Content-Length': '0'} 2024-04-13T22:25:32.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a540621bb6e9f7d-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.848 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.849 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.849 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.849 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:25:32.849 DEBUG:boto:Data: 2024-04-13T22:25:32.849 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.849 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.849 DEBUG:boto:Params: {} 2024-04-13T22:25:32.849 DEBUG:boto:Token: None 2024-04-13T22:25:32.849 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69 2024-04-13T22:25:32.850 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1fU1arSgYooet5183FnfMDZ5N/k= 2024-04-13T22:25:32.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1fU1arSgYooet5183FnfMDZ5N/k=', 'Content-Length': '0'} 2024-04-13T22:25:32.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfde946bfc4d90ed-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:25:32.851 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.852 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.852 DEBUG:boto:Method: GET 2024-04-13T22:25:32.852 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:25:32.852 DEBUG:boto:Data: 2024-04-13T22:25:32.852 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.852 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.852 DEBUG:boto:Params: {} 2024-04-13T22:25:32.852 DEBUG:boto:Token: None 2024-04-13T22:25:32.852 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69 2024-04-13T22:25:32.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3RXDKHuHRqS0K2AzAvunVHKt4Gc= 2024-04-13T22:25:32.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3RXDKHuHRqS0K2AzAvunVHKt4Gc=', 'Content-Length': '0'} 2024-04-13T22:25:32.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa292d79625cc32-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.855 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:25:32.855 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:25:32.855 DEBUG:boto:Method: GET 2024-04-13T22:25:32.856 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:25:32.856 DEBUG:boto:Data: 2024-04-13T22:25:32.856 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.856 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.856 DEBUG:boto:Params: {} 2024-04-13T22:25:32.856 DEBUG:boto:Token: None 2024-04-13T22:25:32.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj69 2024-04-13T22:25:32.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3RXDKHuHRqS0K2AzAvunVHKt4Gc= 2024-04-13T22:25:32.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3RXDKHuHRqS0K2AzAvunVHKt4Gc=', 'Content-Length': '0'} 2024-04-13T22:25:32.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018d6a41d0406a282-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.902 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.903 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.903 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.903 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.903 DEBUG:boto:Method: GET 2024-04-13T22:25:32.903 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.903 DEBUG:boto:Data: 2024-04-13T22:25:32.903 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.903 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.903 DEBUG:boto:Params: {} 2024-04-13T22:25:32.904 DEBUG:boto:Token: None 2024-04-13T22:25:32.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3562f955146bb64-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.906 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.907 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.907 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.907 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.907 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.907 DEBUG:boto:Method: GET 2024-04-13T22:25:32.907 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:25:32.907 DEBUG:boto:Data: 2024-04-13T22:25:32.907 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.908 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.908 DEBUG:boto:Params: {} 2024-04-13T22:25:32.908 DEBUG:boto:Token: None 2024-04-13T22:25:32.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000820ecd5a2b5ebaf6-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.911 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:25:32.912 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.912 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.912 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.912 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.912 DEBUG:boto:Method: GET 2024-04-13T22:25:32.912 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.912 DEBUG:boto:Data: 2024-04-13T22:25:32.912 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.913 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.913 DEBUG:boto:Params: {} 2024-04-13T22:25:32.913 DEBUG:boto:Token: None 2024-04-13T22:25:32.913 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4= 2024-04-13T22:25:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4=', 'Content-Length': '0'} 2024-04-13T22:25:32.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab0e1cd8aad9dcba-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.915 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.915 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.915 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.915 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.915 DEBUG:boto:Method: GET 2024-04-13T22:25:32.915 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.915 DEBUG:boto:Data: 2024-04-13T22:25:32.915 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.916 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.916 DEBUG:boto:Params: {} 2024-04-13T22:25:32.916 DEBUG:boto:Token: None 2024-04-13T22:25:32.916 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4= 2024-04-13T22:25:32.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4=', 'Content-Length': '0'} 2024-04-13T22:25:32.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e60c233eb5a787e-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.918 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.918 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.918 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.918 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.918 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.918 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.919 DEBUG:boto:Data: 2024-04-13T22:25:32.919 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.919 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.919 DEBUG:boto:Params: {} 2024-04-13T22:25:32.919 DEBUG:boto:Token: None 2024-04-13T22:25:32.919 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/sp2vwhchbRSmVIEhh3to8t834w= 2024-04-13T22:25:32.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/sp2vwhchbRSmVIEhh3to8t834w=', 'Content-Length': '0'} 2024-04-13T22:25:32.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15d4b67029043e4-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.921 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.921 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.922 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.922 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.922 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.922 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.922 DEBUG:boto:Data: 2024-04-13T22:25:32.922 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.922 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.922 DEBUG:boto:Params: {} 2024-04-13T22:25:32.922 DEBUG:boto:Token: None 2024-04-13T22:25:32.922 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/sp2vwhchbRSmVIEhh3to8t834w= 2024-04-13T22:25:32.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/sp2vwhchbRSmVIEhh3to8t834w=', 'Content-Length': '0'} 2024-04-13T22:25:32.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d6e0c1c5209621-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:25:32.924 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.925 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.925 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.925 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.925 DEBUG:boto:Method: GET 2024-04-13T22:25:32.925 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.925 DEBUG:boto:Data: 2024-04-13T22:25:32.925 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.925 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.925 DEBUG:boto:Params: {} 2024-04-13T22:25:32.925 DEBUG:boto:Token: None 2024-04-13T22:25:32.926 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4= 2024-04-13T22:25:32.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4=', 'Content-Length': '0'} 2024-04-13T22:25:32.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ee4abb191f72fcc-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.970 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.971 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.971 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.971 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.971 DEBUG:boto:Method: GET 2024-04-13T22:25:32.971 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.971 DEBUG:boto:Data: 2024-04-13T22:25:32.971 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.971 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.972 DEBUG:boto:Params: {} 2024-04-13T22:25:32.972 DEBUG:boto:Token: None 2024-04-13T22:25:32.972 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:25:32.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4= 2024-04-13T22:25:32.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DX2SItRwR+WeoFWc3D5G//rzde4=', 'Content-Length': '0'} 2024-04-13T22:25:32.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c21e004576d5ce5-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.974 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.974 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.975 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:32.975 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:32.975 DEBUG:boto:Method: GET 2024-04-13T22:25:32.975 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:25:32.975 DEBUG:boto:Data: 2024-04-13T22:25:32.975 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.975 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.975 DEBUG:boto:Params: {} 2024-04-13T22:25:32.975 DEBUG:boto:Token: None 2024-04-13T22:25:32.976 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a69e787d419ef3e-00661b065c-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.978 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.979 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:32.979 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:32.979 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:32.979 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:32.979 DEBUG:boto:Method: GET 2024-04-13T22:25:32.980 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:25:32.980 DEBUG:boto:Data: 2024-04-13T22:25:32.980 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.980 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.980 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.980 DEBUG:boto:Params: {} 2024-04-13T22:25:32.980 DEBUG:boto:Token: None 2024-04-13T22:25:32.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/ 2024-04-13T22:25:32.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg= 2024-04-13T22:25:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+k046IGFdIKtzwCNLIqGq/Ppbpg=', 'Content-Length': '0'} 2024-04-13T22:25:32.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95439448baa5cc5-00661b065c-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.983 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:32.984 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.984 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.984 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.984 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:25:32.984 DEBUG:boto:Data: 2024-04-13T22:25:32.984 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.984 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.984 DEBUG:boto:Params: {} 2024-04-13T22:25:32.984 DEBUG:boto:Token: None 2024-04-13T22:25:32.985 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76 2024-04-13T22:25:32.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sqfFkYwYaRdyQwJf7g3QEv27LjE= 2024-04-13T22:25:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sqfFkYwYaRdyQwJf7g3QEv27LjE=', 'Content-Length': '0'} 2024-04-13T22:25:32.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa09e597375b5e40-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.986 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.987 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.987 DEBUG:boto:Method: HEAD 2024-04-13T22:25:32.987 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:25:32.987 DEBUG:boto:Data: 2024-04-13T22:25:32.987 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.987 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.987 DEBUG:boto:Params: {} 2024-04-13T22:25:32.988 DEBUG:boto:Token: None 2024-04-13T22:25:32.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76 2024-04-13T22:25:32.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sqfFkYwYaRdyQwJf7g3QEv27LjE= 2024-04-13T22:25:32.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sqfFkYwYaRdyQwJf7g3QEv27LjE=', 'Content-Length': '0'} 2024-04-13T22:25:32.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e666d61b394d7345-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:25:32.990 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.990 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.990 DEBUG:boto:Method: GET 2024-04-13T22:25:32.990 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:25:32.990 DEBUG:boto:Data: 2024-04-13T22:25:32.990 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:32.991 DEBUG:boto:Port: 8001 2024-04-13T22:25:32.991 DEBUG:boto:Params: {} 2024-04-13T22:25:32.991 DEBUG:boto:Token: None 2024-04-13T22:25:32.992 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76 2024-04-13T22:25:32.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xhyxFgPc4VgQLbOlEYdjgIp9OHI= 2024-04-13T22:25:32.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xhyxFgPc4VgQLbOlEYdjgIp9OHI=', 'Content-Length': '0'} 2024-04-13T22:25:32.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a41dc966aeae60a8-00661b065c-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:32.994 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:25:32.994 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:25:32.995 DEBUG:boto:Method: GET 2024-04-13T22:25:32.995 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:25:32.995 DEBUG:boto:Data: 2024-04-13T22:25:32.995 DEBUG:boto:Headers: {} 2024-04-13T22:25:32.995 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:32.996 DEBUG:boto:Port: 8000 2024-04-13T22:25:32.996 DEBUG:boto:Params: {} 2024-04-13T22:25:32.996 DEBUG:boto:Token: None 2024-04-13T22:25:32.996 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:32 GMT /vwnfns-61/obj76 2024-04-13T22:25:32.997 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xhyxFgPc4VgQLbOlEYdjgIp9OHI= 2024-04-13T22:25:32.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xhyxFgPc4VgQLbOlEYdjgIp9OHI=', 'Content-Length': '0'} 2024-04-13T22:25:32.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f8239651f3cc75-00661b065c-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.046 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.047 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.047 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:33.047 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:33.047 DEBUG:boto:Method: GET 2024-04-13T22:25:33.047 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:25:33.047 DEBUG:boto:Data: 2024-04-13T22:25:33.047 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.047 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.048 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.048 DEBUG:boto:Params: {} 2024-04-13T22:25:33.048 DEBUG:boto:Token: None 2024-04-13T22:25:33.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c5a1eb14fc41fba-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.051 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.051 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.051 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.051 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:33.051 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:25:33.052 DEBUG:boto:Method: GET 2024-04-13T22:25:33.052 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:25:33.052 DEBUG:boto:Data: 2024-04-13T22:25:33.052 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.052 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.052 DEBUG:boto:Params: {} 2024-04-13T22:25:33.052 DEBUG:boto:Token: None 2024-04-13T22:25:33.052 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.053 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8a190928bd3660-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.055 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:25:33.056 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.056 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.056 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.056 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.056 DEBUG:boto:Method: GET 2024-04-13T22:25:33.056 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.056 DEBUG:boto:Data: 2024-04-13T22:25:33.056 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.056 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.057 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.057 DEBUG:boto:Params: {} 2024-04-13T22:25:33.057 DEBUG:boto:Token: None 2024-04-13T22:25:33.057 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.057 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU= 2024-04-13T22:25:33.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU=', 'Content-Length': '0'} 2024-04-13T22:25:33.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e1cfcc3a9b9cd9b-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.060 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.060 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.061 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.061 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.061 DEBUG:boto:Method: GET 2024-04-13T22:25:33.061 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.061 DEBUG:boto:Data: 2024-04-13T22:25:33.062 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.062 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.062 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.062 DEBUG:boto:Params: {} 2024-04-13T22:25:33.062 DEBUG:boto:Token: None 2024-04-13T22:25:33.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU= 2024-04-13T22:25:33.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU=', 'Content-Length': '0'} 2024-04-13T22:25:33.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8ec68f2ee23c98-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.066 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.066 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.066 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.067 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.067 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.067 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.067 DEBUG:boto:Data: 2024-04-13T22:25:33.068 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.068 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.068 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.068 DEBUG:boto:Params: {} 2024-04-13T22:25:33.068 DEBUG:boto:Token: None 2024-04-13T22:25:33.069 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.069 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Adj9Ebw5XSLwWDIPttAeutHkCAs= 2024-04-13T22:25:33.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Adj9Ebw5XSLwWDIPttAeutHkCAs=', 'Content-Length': '0'} 2024-04-13T22:25:33.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11ac037221b1bdf-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.071 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.071 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.071 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.071 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.072 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.072 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.072 DEBUG:boto:Data: 2024-04-13T22:25:33.072 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.072 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.073 DEBUG:boto:Params: {} 2024-04-13T22:25:33.073 DEBUG:boto:Token: None 2024-04-13T22:25:33.073 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.073 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Adj9Ebw5XSLwWDIPttAeutHkCAs= 2024-04-13T22:25:33.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Adj9Ebw5XSLwWDIPttAeutHkCAs=', 'Content-Length': '0'} 2024-04-13T22:25:33.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093131c08cae11801-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:25:33.075 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.076 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.076 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.076 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.076 DEBUG:boto:Method: GET 2024-04-13T22:25:33.076 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.076 DEBUG:boto:Data: 2024-04-13T22:25:33.076 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.076 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.076 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.076 DEBUG:boto:Params: {} 2024-04-13T22:25:33.077 DEBUG:boto:Token: None 2024-04-13T22:25:33.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU= 2024-04-13T22:25:33.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU=', 'Content-Length': '0'} 2024-04-13T22:25:33.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000297c422293c37352-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.122 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.123 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.123 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.123 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.123 DEBUG:boto:Method: GET 2024-04-13T22:25:33.123 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.123 DEBUG:boto:Data: 2024-04-13T22:25:33.123 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.123 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.124 DEBUG:boto:Params: {} 2024-04-13T22:25:33.124 DEBUG:boto:Token: None 2024-04-13T22:25:33.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:25:33.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU= 2024-04-13T22:25:33.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ElaEsomElRv9aBpnBi7RU7GFxaU=', 'Content-Length': '0'} 2024-04-13T22:25:33.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf35d9bc53ff374f-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.126 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.126 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.126 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.127 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.127 DEBUG:boto:Method: GET 2024-04-13T22:25:33.127 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.127 DEBUG:boto:Data: 2024-04-13T22:25:33.127 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.127 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.127 DEBUG:boto:Params: {} 2024-04-13T22:25:33.128 DEBUG:boto:Token: None 2024-04-13T22:25:33.128 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.128 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055bf208696e598ed-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.130 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.131 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.131 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.131 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.131 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.131 DEBUG:boto:Method: GET 2024-04-13T22:25:33.131 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.131 DEBUG:boto:Data: 2024-04-13T22:25:33.131 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.132 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.132 DEBUG:boto:Params: {} 2024-04-13T22:25:33.132 DEBUG:boto:Token: None 2024-04-13T22:25:33.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377e5086adb9183e-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.135 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.135 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.135 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.136 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.136 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:25:33.136 DEBUG:boto:Data: 2024-04-13T22:25:33.136 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.136 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.136 DEBUG:boto:Params: {} 2024-04-13T22:25:33.136 DEBUG:boto:Token: None 2024-04-13T22:25:33.137 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85 2024-04-13T22:25:33.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8S+a2Jc9/L8g70F0Tz7twvrSfqQ= 2024-04-13T22:25:33.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8S+a2Jc9/L8g70F0Tz7twvrSfqQ=', 'Content-Length': '0'} 2024-04-13T22:25:33.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992e33a5ebcb48a6-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.139 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.140 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.140 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.140 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:25:33.140 DEBUG:boto:Data: 2024-04-13T22:25:33.140 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.141 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.141 DEBUG:boto:Params: {} 2024-04-13T22:25:33.141 DEBUG:boto:Token: None 2024-04-13T22:25:33.141 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85 2024-04-13T22:25:33.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8S+a2Jc9/L8g70F0Tz7twvrSfqQ= 2024-04-13T22:25:33.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8S+a2Jc9/L8g70F0Tz7twvrSfqQ=', 'Content-Length': '0'} 2024-04-13T22:25:33.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6b3275e7d0ae89-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:25:33.145 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.145 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.145 DEBUG:boto:Method: GET 2024-04-13T22:25:33.145 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:25:33.145 DEBUG:boto:Data: 2024-04-13T22:25:33.146 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.146 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.146 DEBUG:boto:Params: {} 2024-04-13T22:25:33.147 DEBUG:boto:Token: None 2024-04-13T22:25:33.147 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85 2024-04-13T22:25:33.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pj03rzOLxF7T0pSVp8as6I1qeLo= 2024-04-13T22:25:33.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pj03rzOLxF7T0pSVp8as6I1qeLo=', 'Content-Length': '0'} 2024-04-13T22:25:33.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5696173fd837ab-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.150 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:25:33.150 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:25:33.150 DEBUG:boto:Method: GET 2024-04-13T22:25:33.150 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:25:33.151 DEBUG:boto:Data: 2024-04-13T22:25:33.151 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.151 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.151 DEBUG:boto:Params: {} 2024-04-13T22:25:33.152 DEBUG:boto:Token: None 2024-04-13T22:25:33.152 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj85 2024-04-13T22:25:33.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Pj03rzOLxF7T0pSVp8as6I1qeLo= 2024-04-13T22:25:33.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Pj03rzOLxF7T0pSVp8as6I1qeLo=', 'Content-Length': '0'} 2024-04-13T22:25:33.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f735569ec0c236ab-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.198 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.199 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.199 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.199 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.199 DEBUG:boto:Method: GET 2024-04-13T22:25:33.199 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.199 DEBUG:boto:Data: 2024-04-13T22:25:33.199 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.199 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.199 DEBUG:boto:Params: {} 2024-04-13T22:25:33.200 DEBUG:boto:Token: None 2024-04-13T22:25:33.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f10defda4fc7ee0e-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.202 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.203 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.203 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.203 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.203 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.204 DEBUG:boto:Method: GET 2024-04-13T22:25:33.204 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:25:33.204 DEBUG:boto:Data: 2024-04-13T22:25:33.204 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.204 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.204 DEBUG:boto:Params: {} 2024-04-13T22:25:33.204 DEBUG:boto:Token: None 2024-04-13T22:25:33.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.205 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921f672f4ee383a8-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.207 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:25:33.208 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.208 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.208 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.208 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.208 DEBUG:boto:Method: GET 2024-04-13T22:25:33.208 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.208 DEBUG:boto:Data: 2024-04-13T22:25:33.209 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.209 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.209 DEBUG:boto:Params: {} 2024-04-13T22:25:33.209 DEBUG:boto:Token: None 2024-04-13T22:25:33.209 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.209 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k= 2024-04-13T22:25:33.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k=', 'Content-Length': '0'} 2024-04-13T22:25:33.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a14274259eabd1f-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.254 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.255 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.255 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.255 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.255 DEBUG:boto:Method: GET 2024-04-13T22:25:33.255 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.255 DEBUG:boto:Data: 2024-04-13T22:25:33.255 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.255 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.255 DEBUG:boto:Params: {} 2024-04-13T22:25:33.255 DEBUG:boto:Token: None 2024-04-13T22:25:33.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k= 2024-04-13T22:25:33.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k=', 'Content-Length': '0'} 2024-04-13T22:25:33.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fd953f6eacc19e7-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.258 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.258 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.258 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.258 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.258 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.258 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.259 DEBUG:boto:Data: 2024-04-13T22:25:33.259 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.259 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.259 DEBUG:boto:Params: {} 2024-04-13T22:25:33.259 DEBUG:boto:Token: None 2024-04-13T22:25:33.259 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Nhb4SOuXOSG8BAVcUbeEN+Kwb0= 2024-04-13T22:25:33.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Nhb4SOuXOSG8BAVcUbeEN+Kwb0=', 'Content-Length': '0'} 2024-04-13T22:25:33.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7902df457342fa7-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.261 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.261 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.261 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.261 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.261 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.262 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.262 DEBUG:boto:Data: 2024-04-13T22:25:33.262 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.262 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.262 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.262 DEBUG:boto:Params: {} 2024-04-13T22:25:33.262 DEBUG:boto:Token: None 2024-04-13T22:25:33.262 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Nhb4SOuXOSG8BAVcUbeEN+Kwb0= 2024-04-13T22:25:33.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Nhb4SOuXOSG8BAVcUbeEN+Kwb0=', 'Content-Length': '0'} 2024-04-13T22:25:33.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9229104d4411b2-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:25:33.264 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.264 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.264 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.264 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.265 DEBUG:boto:Method: GET 2024-04-13T22:25:33.265 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.265 DEBUG:boto:Data: 2024-04-13T22:25:33.265 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.265 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.265 DEBUG:boto:Params: {} 2024-04-13T22:25:33.265 DEBUG:boto:Token: None 2024-04-13T22:25:33.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k= 2024-04-13T22:25:33.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k=', 'Content-Length': '0'} 2024-04-13T22:25:33.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c7b6a90c3361450-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.267 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.267 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.267 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.268 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.268 DEBUG:boto:Method: GET 2024-04-13T22:25:33.268 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.268 DEBUG:boto:Data: 2024-04-13T22:25:33.268 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.268 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.268 DEBUG:boto:Params: {} 2024-04-13T22:25:33.268 DEBUG:boto:Token: None 2024-04-13T22:25:33.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:25:33.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k= 2024-04-13T22:25:33.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jhGn2QFQG+o65icmvVo5nQyjL0k=', 'Content-Length': '0'} 2024-04-13T22:25:33.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef11e757c492aecc-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.270 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.271 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.271 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.271 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.271 DEBUG:boto:Method: GET 2024-04-13T22:25:33.271 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.271 DEBUG:boto:Data: 2024-04-13T22:25:33.271 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.271 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.271 DEBUG:boto:Params: {} 2024-04-13T22:25:33.271 DEBUG:boto:Token: None 2024-04-13T22:25:33.272 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97d8d6cbdac21e7-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.274 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.275 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.275 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.275 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.275 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.275 DEBUG:boto:Method: GET 2024-04-13T22:25:33.275 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.275 DEBUG:boto:Data: 2024-04-13T22:25:33.275 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.275 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.276 DEBUG:boto:Params: {} 2024-04-13T22:25:33.276 DEBUG:boto:Token: None 2024-04-13T22:25:33.276 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004acbdf81724f3340-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.279 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.280 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.280 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.280 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.280 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:25:33.280 DEBUG:boto:Data: 2024-04-13T22:25:33.280 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.280 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.280 DEBUG:boto:Params: {} 2024-04-13T22:25:33.281 DEBUG:boto:Token: None 2024-04-13T22:25:33.281 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92 2024-04-13T22:25:33.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fY/Dt9I7SV1g20hO/+4EhOUAut8= 2024-04-13T22:25:33.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fY/Dt9I7SV1g20hO/+4EhOUAut8=', 'Content-Length': '0'} 2024-04-13T22:25:33.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de0fc53c4284c94-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.282 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.283 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.283 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.283 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:25:33.283 DEBUG:boto:Data: 2024-04-13T22:25:33.283 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.283 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.283 DEBUG:boto:Params: {} 2024-04-13T22:25:33.283 DEBUG:boto:Token: None 2024-04-13T22:25:33.283 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92 2024-04-13T22:25:33.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fY/Dt9I7SV1g20hO/+4EhOUAut8= 2024-04-13T22:25:33.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fY/Dt9I7SV1g20hO/+4EhOUAut8=', 'Content-Length': '0'} 2024-04-13T22:25:33.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45387941e89570f-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:25:33.286 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.286 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.286 DEBUG:boto:Method: GET 2024-04-13T22:25:33.286 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:25:33.286 DEBUG:boto:Data: 2024-04-13T22:25:33.286 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.286 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.286 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.286 DEBUG:boto:Params: {} 2024-04-13T22:25:33.286 DEBUG:boto:Token: None 2024-04-13T22:25:33.287 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92 2024-04-13T22:25:33.287 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BNK54RcXMJp+DcuXpLr6z9sqGc8= 2024-04-13T22:25:33.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BNK54RcXMJp+DcuXpLr6z9sqGc8=', 'Content-Length': '0'} 2024-04-13T22:25:33.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785b66568fcb9c33-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.334 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:25:33.335 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:25:33.335 DEBUG:boto:Method: GET 2024-04-13T22:25:33.335 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:25:33.335 DEBUG:boto:Data: 2024-04-13T22:25:33.335 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.335 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.335 DEBUG:boto:Params: {} 2024-04-13T22:25:33.336 DEBUG:boto:Token: None 2024-04-13T22:25:33.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj92 2024-04-13T22:25:33.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BNK54RcXMJp+DcuXpLr6z9sqGc8= 2024-04-13T22:25:33.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BNK54RcXMJp+DcuXpLr6z9sqGc8=', 'Content-Length': '0'} 2024-04-13T22:25:33.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c4edbd0697363e-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.339 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.339 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.339 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.339 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.339 DEBUG:boto:Method: GET 2024-04-13T22:25:33.340 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.340 DEBUG:boto:Data: 2024-04-13T22:25:33.340 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.340 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.340 DEBUG:boto:Params: {} 2024-04-13T22:25:33.340 DEBUG:boto:Token: None 2024-04-13T22:25:33.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067f6bb424f2699f5-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.342 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.343 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.343 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.343 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.343 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.343 DEBUG:boto:Method: GET 2024-04-13T22:25:33.343 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:25:33.343 DEBUG:boto:Data: 2024-04-13T22:25:33.343 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.344 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.344 DEBUG:boto:Params: {} 2024-04-13T22:25:33.344 DEBUG:boto:Token: None 2024-04-13T22:25:33.344 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806c055965f865fe-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.394 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:25:33.395 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.395 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.395 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.395 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.395 DEBUG:boto:Method: GET 2024-04-13T22:25:33.396 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.396 DEBUG:boto:Data: 2024-04-13T22:25:33.396 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.396 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.396 DEBUG:boto:Params: {} 2024-04-13T22:25:33.396 DEBUG:boto:Token: None 2024-04-13T22:25:33.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0= 2024-04-13T22:25:33.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0=', 'Content-Length': '0'} 2024-04-13T22:25:33.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba8173e627e0e5c-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.399 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.399 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.399 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.399 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.399 DEBUG:boto:Method: GET 2024-04-13T22:25:33.399 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.399 DEBUG:boto:Data: 2024-04-13T22:25:33.399 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.400 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.400 DEBUG:boto:Params: {} 2024-04-13T22:25:33.400 DEBUG:boto:Token: None 2024-04-13T22:25:33.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0= 2024-04-13T22:25:33.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0=', 'Content-Length': '0'} 2024-04-13T22:25:33.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d11302072994cd4-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.402 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.402 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.402 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.402 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.403 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.403 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.403 DEBUG:boto:Data: 2024-04-13T22:25:33.403 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.404 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.404 DEBUG:boto:Params: {} 2024-04-13T22:25:33.404 DEBUG:boto:Token: None 2024-04-13T22:25:33.404 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lQKAXUrZCk8Sos/Vg6yBfDzog9w= 2024-04-13T22:25:33.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lQKAXUrZCk8Sos/Vg6yBfDzog9w=', 'Content-Length': '0'} 2024-04-13T22:25:33.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7dd45eac0bfeea8-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.407 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.407 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.407 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.407 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.407 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.407 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.408 DEBUG:boto:Data: 2024-04-13T22:25:33.408 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.408 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.408 DEBUG:boto:Params: {} 2024-04-13T22:25:33.408 DEBUG:boto:Token: None 2024-04-13T22:25:33.408 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lQKAXUrZCk8Sos/Vg6yBfDzog9w= 2024-04-13T22:25:33.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lQKAXUrZCk8Sos/Vg6yBfDzog9w=', 'Content-Length': '0'} 2024-04-13T22:25:33.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004685b01fcbc65689-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:25:33.410 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.410 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.410 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.410 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.410 DEBUG:boto:Method: GET 2024-04-13T22:25:33.411 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.411 DEBUG:boto:Data: 2024-04-13T22:25:33.411 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.411 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.411 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.411 DEBUG:boto:Params: {} 2024-04-13T22:25:33.411 DEBUG:boto:Token: None 2024-04-13T22:25:33.411 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0= 2024-04-13T22:25:33.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0=', 'Content-Length': '0'} 2024-04-13T22:25:33.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092cda51b2af0eced-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.454 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.455 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.455 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.455 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.455 DEBUG:boto:Method: GET 2024-04-13T22:25:33.455 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.455 DEBUG:boto:Data: 2024-04-13T22:25:33.455 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.455 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.455 DEBUG:boto:Params: {} 2024-04-13T22:25:33.456 DEBUG:boto:Token: None 2024-04-13T22:25:33.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:25:33.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0= 2024-04-13T22:25:33.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0d8NxkZrteKhAy8X0wOm1eNZ7E0=', 'Content-Length': '0'} 2024-04-13T22:25:33.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb28bcb2525964cc-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.458 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.458 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.458 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.458 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.458 DEBUG:boto:Method: GET 2024-04-13T22:25:33.458 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.458 DEBUG:boto:Data: 2024-04-13T22:25:33.458 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.459 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.459 DEBUG:boto:Params: {} 2024-04-13T22:25:33.459 DEBUG:boto:Token: None 2024-04-13T22:25:33.459 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000125208e146424a20-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.461 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.462 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.462 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.462 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.462 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.462 DEBUG:boto:Method: GET 2024-04-13T22:25:33.462 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.463 DEBUG:boto:Data: 2024-04-13T22:25:33.463 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.463 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.463 DEBUG:boto:Params: {} 2024-04-13T22:25:33.463 DEBUG:boto:Token: None 2024-04-13T22:25:33.463 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.463 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650a3896efeb5088-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.466 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.467 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.467 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.467 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.467 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:25:33.467 DEBUG:boto:Data: 2024-04-13T22:25:33.467 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.467 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.467 DEBUG:boto:Params: {} 2024-04-13T22:25:33.468 DEBUG:boto:Token: None 2024-04-13T22:25:33.468 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95 2024-04-13T22:25:33.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cw2hCq2AlNVob9NzOWssv1i3RAk= 2024-04-13T22:25:33.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cw2hCq2AlNVob9NzOWssv1i3RAk=', 'Content-Length': '0'} 2024-04-13T22:25:33.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb7ab5f67cc39f2-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.470 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.470 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.470 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.470 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:25:33.470 DEBUG:boto:Data: 2024-04-13T22:25:33.470 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.470 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.471 DEBUG:boto:Params: {} 2024-04-13T22:25:33.471 DEBUG:boto:Token: None 2024-04-13T22:25:33.471 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95 2024-04-13T22:25:33.471 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cw2hCq2AlNVob9NzOWssv1i3RAk= 2024-04-13T22:25:33.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cw2hCq2AlNVob9NzOWssv1i3RAk=', 'Content-Length': '0'} 2024-04-13T22:25:33.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee701d564b6f8537-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:25:33.473 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.473 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.473 DEBUG:boto:Method: GET 2024-04-13T22:25:33.473 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:25:33.473 DEBUG:boto:Data: 2024-04-13T22:25:33.473 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.473 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.474 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.474 DEBUG:boto:Params: {} 2024-04-13T22:25:33.474 DEBUG:boto:Token: None 2024-04-13T22:25:33.474 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95 2024-04-13T22:25:33.474 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3L4oz2eFRB+Vji6PZF/m1Oa5ZVU= 2024-04-13T22:25:33.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3L4oz2eFRB+Vji6PZF/m1Oa5ZVU=', 'Content-Length': '0'} 2024-04-13T22:25:33.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b54c20935cafac4a-00661b065d-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.476 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:25:33.476 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:25:33.476 DEBUG:boto:Method: GET 2024-04-13T22:25:33.476 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:25:33.477 DEBUG:boto:Data: 2024-04-13T22:25:33.477 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.477 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.477 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.477 DEBUG:boto:Params: {} 2024-04-13T22:25:33.477 DEBUG:boto:Token: None 2024-04-13T22:25:33.477 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj95 2024-04-13T22:25:33.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3L4oz2eFRB+Vji6PZF/m1Oa5ZVU= 2024-04-13T22:25:33.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3L4oz2eFRB+Vji6PZF/m1Oa5ZVU=', 'Content-Length': '0'} 2024-04-13T22:25:33.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f88e8f70ccc1b2b-00661b065d-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.479 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.479 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.479 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.479 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.479 DEBUG:boto:Method: GET 2024-04-13T22:25:33.480 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.480 DEBUG:boto:Data: 2024-04-13T22:25:33.480 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:25:33.480 DEBUG:boto:Port: 8001 2024-04-13T22:25:33.480 DEBUG:boto:Params: {} 2024-04-13T22:25:33.480 DEBUG:boto:Token: None 2024-04-13T22:25:33.480 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.481 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e2e834766a1e68d-00661b065d-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.483 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.483 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.483 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.483 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.484 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.484 DEBUG:boto:Method: GET 2024-04-13T22:25:33.484 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:25:33.484 DEBUG:boto:Data: 2024-04-13T22:25:33.484 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.484 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.484 DEBUG:boto:Params: {} 2024-04-13T22:25:33.484 DEBUG:boto:Token: None 2024-04-13T22:25:33.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs= 2024-04-13T22:25:33.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DH2mSKAN5qVQH1DPQVAEmmhsHxs=', 'Content-Length': '0'} 2024-04-13T22:25:33.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f034ddedfcdcb952-00661b065d-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.534 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:25:33.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:25:33.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:25:33.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:25:33.536 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:25:33.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:25:33.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:25:33.634 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:25:33.635 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-61 2024-04-13T22:25:33.644 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 20 2024-04-13T22:25:33.693 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.694 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.694 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.694 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:33.694 DEBUG:boto:Data: 2024-04-13T22:25:33.694 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.694 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.694 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.694 DEBUG:boto:Params: {} 2024-04-13T22:25:33.694 DEBUG:boto:Token: None 2024-04-13T22:25:33.695 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.695 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y= 2024-04-13T22:25:33.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y=', 'Content-Length': '0'} 2024-04-13T22:25:33.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2acc790901365bb-00661b065d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.699 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:25:33.699 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:25:33.699 DEBUG:boto:Method: PUT 2024-04-13T22:25:33.699 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:25:33.699 DEBUG:boto:Data: 2024-04-13T22:25:33.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:25:33.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.699 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.699 DEBUG:boto:Params: {} 2024-04-13T22:25:33.699 DEBUG:boto:Token: None 2024-04-13T22:25:33.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj29 2024-04-13T22:25:33.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aX0VDL2+5fWvatgL2panzD+tK/0= 2024-04-13T22:25:33.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aX0VDL2+5fWvatgL2panzD+tK/0='} 2024-04-13T22:25:33.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca99bb1cbb330c0a-00661b065d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.705 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.705 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.705 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.705 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:33.705 DEBUG:boto:Data: 2024-04-13T22:25:33.706 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.706 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.706 DEBUG:boto:Params: {} 2024-04-13T22:25:33.706 DEBUG:boto:Token: None 2024-04-13T22:25:33.706 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.707 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y= 2024-04-13T22:25:33.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y=', 'Content-Length': '0'} 2024-04-13T22:25:33.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d270efb49f61b49-00661b065d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.710 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:25:33.710 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:25:33.710 DEBUG:boto:Method: PUT 2024-04-13T22:25:33.710 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:25:33.710 DEBUG:boto:Data: 2024-04-13T22:25:33.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:25:33.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.710 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.710 DEBUG:boto:Params: {} 2024-04-13T22:25:33.711 DEBUG:boto:Token: None 2024-04-13T22:25:33.711 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj58 2024-04-13T22:25:33.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6CRU33BSo9Ehod7PvY9BJ01NIzY= 2024-04-13T22:25:33.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6CRU33BSo9Ehod7PvY9BJ01NIzY='} 2024-04-13T22:25:33.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c582da3f0d8ce30-00661b065d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.715 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.716 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.716 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.716 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:33.716 DEBUG:boto:Data: 2024-04-13T22:25:33.716 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.716 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.716 DEBUG:boto:Params: {} 2024-04-13T22:25:33.716 DEBUG:boto:Token: None 2024-04-13T22:25:33.717 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y= 2024-04-13T22:25:33.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y=', 'Content-Length': '0'} 2024-04-13T22:25:33.719 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088535fa1920ba08f-00661b065d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.720 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:25:33.720 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:25:33.720 DEBUG:boto:Method: PUT 2024-04-13T22:25:33.720 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:25:33.720 DEBUG:boto:Data: 2024-04-13T22:25:33.720 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:25:33.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.720 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.721 DEBUG:boto:Params: {} 2024-04-13T22:25:33.721 DEBUG:boto:Token: None 2024-04-13T22:25:33.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj87 2024-04-13T22:25:33.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:REtZRQIqDM18lFKGeEOdDIFWMZs= 2024-04-13T22:25:33.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:REtZRQIqDM18lFKGeEOdDIFWMZs='} 2024-04-13T22:25:33.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000704767137e64818c-00661b065d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.767 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.767 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.767 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.767 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:33.767 DEBUG:boto:Data: 2024-04-13T22:25:33.767 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.768 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.768 DEBUG:boto:Params: {} 2024-04-13T22:25:33.768 DEBUG:boto:Token: None 2024-04-13T22:25:33.768 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y= 2024-04-13T22:25:33.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y=', 'Content-Length': '0'} 2024-04-13T22:25:33.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d7a52f4ddd01c2d-00661b065d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.772 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:25:33.772 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:25:33.772 DEBUG:boto:Method: PUT 2024-04-13T22:25:33.772 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:25:33.773 DEBUG:boto:Data: 2024-04-13T22:25:33.773 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:25:33.773 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.773 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.773 DEBUG:boto:Params: {} 2024-04-13T22:25:33.773 DEBUG:boto:Token: None 2024-04-13T22:25:33.774 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj116 2024-04-13T22:25:33.774 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5pWHieZ8JbhX1ELw8H8G0K08lDo= 2024-04-13T22:25:33.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5pWHieZ8JbhX1ELw8H8G0K08lDo='} 2024-04-13T22:25:33.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e99f1e73f49d4de-00661b065d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.782 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:25:33.782 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:25:33.783 DEBUG:boto:Method: HEAD 2024-04-13T22:25:33.783 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:25:33.783 DEBUG:boto:Data: 2024-04-13T22:25:33.783 DEBUG:boto:Headers: {} 2024-04-13T22:25:33.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.783 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.783 DEBUG:boto:Params: {} 2024-04-13T22:25:33.783 DEBUG:boto:Token: None 2024-04-13T22:25:33.783 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/ 2024-04-13T22:25:33.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y= 2024-04-13T22:25:33.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7dwB/3FPfw5OZoCRnRkf6ZuoH2Y=', 'Content-Length': '0'} 2024-04-13T22:25:33.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063e5980edd2f0e66-00661b065d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.787 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:25:33.787 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:25:33.787 DEBUG:boto:Method: PUT 2024-04-13T22:25:33.787 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:25:33.787 DEBUG:boto:Data: 2024-04-13T22:25:33.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:25:33.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:25:33.787 DEBUG:boto:Port: 8000 2024-04-13T22:25:33.787 DEBUG:boto:Params: {} 2024-04-13T22:25:33.788 DEBUG:boto:Token: None 2024-04-13T22:25:33.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:25:33 GMT /vwnfns-61/obj145 2024-04-13T22:25:33.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DcshCAMxUyIBCTFV0X6JM7nkjkM= 2024-04-13T22:25:33.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:25:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DcshCAMxUyIBCTFV0X6JM7nkjkM='} 2024-04-13T22:25:33.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001006b78011d7dd16-00661b065d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:25:33.835 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:25:33.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:25:33.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:25:33.836 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:25:33.890 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:25:33.890 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:25:33.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:33.921+0000 7f8f07885780 1 waiting to reach latest gen 4, on 3.. 2024-04-13T22:25:38.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:38.922+0000 7f8f07885780 1 waiting to reach latest gen 4, on 3.. 2024-04-13T22:25:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:43.922+0000 7f8f07885780 1 waiting to reach latest gen 4, on 3.. 2024-04-13T22:25:48.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:48.926+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:48.929 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:25:48.930 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:25:53.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:53.928+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:53.932 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:25:53.932 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:25:58.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:25:58.930+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:25:58.934 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:25:58.934 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:03.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:03.931+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:03.936 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:03.936 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:08.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:08.933+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:08.938 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:08.938 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:13.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:13.935+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:13.940 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:13.940 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:18.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:18.937+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:18.942 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:18.942 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:23.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:23.939+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:23.944 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:23.944 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:28.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:28.943+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:28.946 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:28.947 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:33.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:33.945+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:33.949 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:33.949 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:38.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:38.947+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:38.951 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:26:38.951 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:43.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:43.948+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:43.953 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:43.953 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:48.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:48.950+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:48.955 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:48.955 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:53.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:53.953+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:53.957 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:53.957 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:58.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:26:58.955+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:26:58.959 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:26:58.959 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:03.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:03.958+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:03.962 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:03.962 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:08.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:08.961+0000 7f8f07885780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:09.007 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:09.007 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:13.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:13.963+0000 7f8f07885780 1 bucket sync caught up with source: 2024-04-13T22:27:13.966 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:13.966 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [00000000002.48.6, 00000000001.73.6, , , , , 00000000001.54.6, , , , , , , , , , , 00000000002.129.6, 00000000001.54.6, , , , , , , , , , ] 2024-04-13T22:27:13.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:13.963+0000 7f8f07885780 0 bucket checkpoint complete 2024-04-13T22:27:13.985 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:27:13.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:27:13.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:27:13.986 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:27:14.047 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:27:14.048 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:27:14.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:14.082+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:14.085 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:14.086 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:19.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:19.083+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:19.087 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:19.087 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:24.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:24.083+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:24.088 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:24.088 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:29.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:29.085+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:29.089 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:29.089 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:34.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:34.086+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:34.090 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:34.090 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:39.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:39.087+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:39.091 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:39.091 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:44.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:44.089+0000 7f62a7e9e780 1 waiting for incremental sync to catch up: 2024-04-13T22:27:44.093 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000003.1153.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:44.093 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:49.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:49.091+0000 7f62a7e9e780 1 bucket sync caught up with source: 2024-04-13T22:27:49.094 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:49.095 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:27:49.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:49.091+0000 7f62a7e9e780 0 bucket checkpoint complete 2024-04-13T22:27:49.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:27:49.111 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.112 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.112 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.112 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:49.112 DEBUG:boto:Data: 2024-04-13T22:27:49.112 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.112 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.112 DEBUG:boto:Params: {} 2024-04-13T22:27:49.113 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:27:49.113 DEBUG:boto:Token: None 2024-04-13T22:27:49.113 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5j+FJz774uHnt4m6f2NaEdzobs= 2024-04-13T22:27:49.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5j+FJz774uHnt4m6f2NaEdzobs=', 'Content-Length': '0'} 2024-04-13T22:27:49.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004005503d399a343-00661b06e5-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.119 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.119 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.119 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.119 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:49.119 DEBUG:boto:Data: 2024-04-13T22:27:49.119 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.119 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.119 DEBUG:boto:Params: {} 2024-04-13T22:27:49.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:27:49.120 DEBUG:boto:Token: None 2024-04-13T22:27:49.120 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o5j+FJz774uHnt4m6f2NaEdzobs= 2024-04-13T22:27:49.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5j+FJz774uHnt4m6f2NaEdzobs=', 'Content-Length': '0'} 2024-04-13T22:27:49.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f447bbbb51e7112-00661b06e5-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.125 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:27:49.125 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.125 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.125 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.125 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.125 DEBUG:boto:Method: GET 2024-04-13T22:27:49.125 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.125 DEBUG:boto:Data: 2024-04-13T22:27:49.125 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.126 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.126 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.126 DEBUG:boto:Params: {} 2024-04-13T22:27:49.126 DEBUG:boto:Token: None 2024-04-13T22:27:49.126 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/?versions 2024-04-13T22:27:49.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ= 2024-04-13T22:27:49.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad60a90f94768c02-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.129 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:27:49.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:27:49.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:27:49.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:27:49.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:27:49.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:27:49.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:27:49.133 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:27:49.133 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.133 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.133 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.133 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.133 DEBUG:boto:Method: GET 2024-04-13T22:27:49.134 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.134 DEBUG:boto:Data: 2024-04-13T22:27:49.134 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.134 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.134 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.134 DEBUG:boto:Params: {} 2024-04-13T22:27:49.134 DEBUG:boto:Token: None 2024-04-13T22:27:49.134 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/?versions 2024-04-13T22:27:49.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ= 2024-04-13T22:27:49.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfc52777f233364b-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.138 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:27:49.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:27:49.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:27:49.142 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.142 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.142 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.142 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.142 DEBUG:boto:Method: GET 2024-04-13T22:27:49.142 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.142 DEBUG:boto:Data: 2024-04-13T22:27:49.142 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.142 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.143 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.143 DEBUG:boto:Params: {} 2024-04-13T22:27:49.143 DEBUG:boto:Token: None 2024-04-13T22:27:49.143 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/?versions 2024-04-13T22:27:49.143 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ= 2024-04-13T22:27:49.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000847c53256a7e4b21-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.146 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.147 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.147 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.147 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.147 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.147 DEBUG:boto:Method: GET 2024-04-13T22:27:49.148 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:27:49.148 DEBUG:boto:Data: 2024-04-13T22:27:49.148 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.148 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.148 DEBUG:boto:Params: {} 2024-04-13T22:27:49.148 DEBUG:boto:Token: None 2024-04-13T22:27:49.148 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/?versions 2024-04-13T22:27:49.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ= 2024-04-13T22:27:49.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qW3Jecd9RpL+XeTc0gDAM4n+niQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b2c40661fe5f65-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.152 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:27:49.153 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.154 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.154 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.154 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.154 DEBUG:boto:Method: GET 2024-04-13T22:27:49.154 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.154 DEBUG:boto:Data: 2024-04-13T22:27:49.154 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.154 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.154 DEBUG:boto:Params: {} 2024-04-13T22:27:49.155 DEBUG:boto:Token: None 2024-04-13T22:27:49.155 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ= 2024-04-13T22:27:49.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805c233ac0ab6d9d-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.157 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.157 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.157 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.157 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.157 DEBUG:boto:Method: GET 2024-04-13T22:27:49.157 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.158 DEBUG:boto:Data: 2024-04-13T22:27:49.158 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.158 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.158 DEBUG:boto:Params: {} 2024-04-13T22:27:49.158 DEBUG:boto:Token: None 2024-04-13T22:27:49.158 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.158 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ= 2024-04-13T22:27:49.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000854ba96ebf94d9b5-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.160 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.160 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.161 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.161 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.161 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.161 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.161 DEBUG:boto:Data: 2024-04-13T22:27:49.161 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.161 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.161 DEBUG:boto:Params: {} 2024-04-13T22:27:49.161 DEBUG:boto:Token: None 2024-04-13T22:27:49.161 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.162 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xil8th+TTXtzRsy7YNopnW+hUgU= 2024-04-13T22:27:49.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xil8th+TTXtzRsy7YNopnW+hUgU=', 'Content-Length': '0'} 2024-04-13T22:27:49.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000962fec33efbb48a0-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.164 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.164 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.164 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.164 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.164 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.164 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.164 DEBUG:boto:Data: 2024-04-13T22:27:49.164 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.164 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.164 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.164 DEBUG:boto:Params: {} 2024-04-13T22:27:49.165 DEBUG:boto:Token: None 2024-04-13T22:27:49.165 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xil8th+TTXtzRsy7YNopnW+hUgU= 2024-04-13T22:27:49.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xil8th+TTXtzRsy7YNopnW+hUgU=', 'Content-Length': '0'} 2024-04-13T22:27:49.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef7d351c4c62b787-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:27:49.167 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.167 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.167 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.167 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.167 DEBUG:boto:Method: GET 2024-04-13T22:27:49.167 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.167 DEBUG:boto:Data: 2024-04-13T22:27:49.167 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.168 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.168 DEBUG:boto:Params: {} 2024-04-13T22:27:49.168 DEBUG:boto:Token: None 2024-04-13T22:27:49.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ= 2024-04-13T22:27:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044fca581b9ba694a-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.170 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.170 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.170 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.170 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.170 DEBUG:boto:Method: GET 2024-04-13T22:27:49.171 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.171 DEBUG:boto:Data: 2024-04-13T22:27:49.171 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.171 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.171 DEBUG:boto:Params: {} 2024-04-13T22:27:49.171 DEBUG:boto:Token: None 2024-04-13T22:27:49.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:27:49.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ= 2024-04-13T22:27:49.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fNcGfEn/iERP+rOetHMtzsJPUfQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59acde3f61e9b7f-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.214 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.215 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.215 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.215 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.215 DEBUG:boto:Method: GET 2024-04-13T22:27:49.215 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.215 DEBUG:boto:Data: 2024-04-13T22:27:49.215 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.215 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.215 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.215 DEBUG:boto:Params: {} 2024-04-13T22:27:49.215 DEBUG:boto:Token: None 2024-04-13T22:27:49.216 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.216 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de88532995f9fce4-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.218 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.219 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.219 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.219 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.219 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.219 DEBUG:boto:Method: GET 2024-04-13T22:27:49.219 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.219 DEBUG:boto:Data: 2024-04-13T22:27:49.219 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.220 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.220 DEBUG:boto:Params: {} 2024-04-13T22:27:49.220 DEBUG:boto:Token: None 2024-04-13T22:27:49.220 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f5c9781f24fcdc-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.224 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.224 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.224 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.224 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.224 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:27:49.224 DEBUG:boto:Data: 2024-04-13T22:27:49.224 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.225 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.225 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.225 DEBUG:boto:Params: {} 2024-04-13T22:27:49.225 DEBUG:boto:Token: None 2024-04-13T22:27:49.225 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11 2024-04-13T22:27:49.225 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yCWU1PjpfgidvND71vgvlA4WuNs= 2024-04-13T22:27:49.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yCWU1PjpfgidvND71vgvlA4WuNs=', 'Content-Length': '0'} 2024-04-13T22:27:49.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb09bbfbec060e11-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.227 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.227 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.227 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.227 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:27:49.227 DEBUG:boto:Data: 2024-04-13T22:27:49.227 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.228 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.228 DEBUG:boto:Params: {} 2024-04-13T22:27:49.228 DEBUG:boto:Token: None 2024-04-13T22:27:49.228 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11 2024-04-13T22:27:49.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yCWU1PjpfgidvND71vgvlA4WuNs= 2024-04-13T22:27:49.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yCWU1PjpfgidvND71vgvlA4WuNs=', 'Content-Length': '0'} 2024-04-13T22:27:49.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ec5630666d2aa9-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:27:49.230 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.230 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.230 DEBUG:boto:Method: GET 2024-04-13T22:27:49.231 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:27:49.231 DEBUG:boto:Data: 2024-04-13T22:27:49.231 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.231 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.231 DEBUG:boto:Params: {} 2024-04-13T22:27:49.231 DEBUG:boto:Token: None 2024-04-13T22:27:49.231 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11 2024-04-13T22:27:49.231 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YuDFwRCu37gCWpkOy65hC9sJsdo= 2024-04-13T22:27:49.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YuDFwRCu37gCWpkOy65hC9sJsdo=', 'Content-Length': '0'} 2024-04-13T22:27:49.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000505077c2e48cc058-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.234 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:27:49.234 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:27:49.234 DEBUG:boto:Method: GET 2024-04-13T22:27:49.234 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:27:49.234 DEBUG:boto:Data: 2024-04-13T22:27:49.234 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.234 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.234 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.234 DEBUG:boto:Params: {} 2024-04-13T22:27:49.234 DEBUG:boto:Token: None 2024-04-13T22:27:49.235 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj11 2024-04-13T22:27:49.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YuDFwRCu37gCWpkOy65hC9sJsdo= 2024-04-13T22:27:49.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YuDFwRCu37gCWpkOy65hC9sJsdo=', 'Content-Length': '0'} 2024-04-13T22:27:49.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eaf72ece687431a-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.237 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.237 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.237 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.237 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.237 DEBUG:boto:Method: GET 2024-04-13T22:27:49.237 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.237 DEBUG:boto:Data: 2024-04-13T22:27:49.238 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.238 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.238 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.238 DEBUG:boto:Params: {} 2024-04-13T22:27:49.238 DEBUG:boto:Token: None 2024-04-13T22:27:49.238 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.238 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc4d392092c5124-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.241 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.241 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.241 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.241 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.241 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.241 DEBUG:boto:Method: GET 2024-04-13T22:27:49.241 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:27:49.242 DEBUG:boto:Data: 2024-04-13T22:27:49.242 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.242 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.242 DEBUG:boto:Params: {} 2024-04-13T22:27:49.242 DEBUG:boto:Token: None 2024-04-13T22:27:49.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005724ae18f5b50eb-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.245 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:27:49.246 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.246 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.246 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.246 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.246 DEBUG:boto:Method: GET 2024-04-13T22:27:49.246 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.246 DEBUG:boto:Data: 2024-04-13T22:27:49.247 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.247 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.247 DEBUG:boto:Params: {} 2024-04-13T22:27:49.247 DEBUG:boto:Token: None 2024-04-13T22:27:49.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k= 2024-04-13T22:27:49.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k=', 'Content-Length': '0'} 2024-04-13T22:27:49.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a1c2695f408feb-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.249 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.249 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.250 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.250 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.250 DEBUG:boto:Method: GET 2024-04-13T22:27:49.250 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.250 DEBUG:boto:Data: 2024-04-13T22:27:49.250 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.250 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.250 DEBUG:boto:Params: {} 2024-04-13T22:27:49.251 DEBUG:boto:Token: None 2024-04-13T22:27:49.251 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k= 2024-04-13T22:27:49.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k=', 'Content-Length': '0'} 2024-04-13T22:27:49.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd2e0821ddace29-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.253 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.253 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.253 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.253 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.253 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.254 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.254 DEBUG:boto:Data: 2024-04-13T22:27:49.254 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.254 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.254 DEBUG:boto:Params: {} 2024-04-13T22:27:49.254 DEBUG:boto:Token: None 2024-04-13T22:27:49.254 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QZ4y+It3cVvaLYugkrpuuJJHMj0= 2024-04-13T22:27:49.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QZ4y+It3cVvaLYugkrpuuJJHMj0=', 'Content-Length': '0'} 2024-04-13T22:27:49.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020d69a6f4f10161-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.256 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.256 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.256 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.257 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.257 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.257 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.257 DEBUG:boto:Data: 2024-04-13T22:27:49.257 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.257 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.257 DEBUG:boto:Params: {} 2024-04-13T22:27:49.257 DEBUG:boto:Token: None 2024-04-13T22:27:49.257 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QZ4y+It3cVvaLYugkrpuuJJHMj0= 2024-04-13T22:27:49.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QZ4y+It3cVvaLYugkrpuuJJHMj0=', 'Content-Length': '0'} 2024-04-13T22:27:49.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1f4088b30f2326-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:27:49.260 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.260 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.260 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.260 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.260 DEBUG:boto:Method: GET 2024-04-13T22:27:49.260 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.260 DEBUG:boto:Data: 2024-04-13T22:27:49.261 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.261 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.261 DEBUG:boto:Params: {} 2024-04-13T22:27:49.261 DEBUG:boto:Token: None 2024-04-13T22:27:49.261 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k= 2024-04-13T22:27:49.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k=', 'Content-Length': '0'} 2024-04-13T22:27:49.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd56f53cdacf92a-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.263 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.263 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.263 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.263 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.263 DEBUG:boto:Method: GET 2024-04-13T22:27:49.263 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.264 DEBUG:boto:Data: 2024-04-13T22:27:49.264 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.264 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.264 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.264 DEBUG:boto:Params: {} 2024-04-13T22:27:49.264 DEBUG:boto:Token: None 2024-04-13T22:27:49.264 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:27:49.264 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k= 2024-04-13T22:27:49.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZLt6S9wUxgTeORiLmvSzNx5Q85k=', 'Content-Length': '0'} 2024-04-13T22:27:49.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed57030721b73a9b-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.306 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.306 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.307 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.307 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.307 DEBUG:boto:Method: GET 2024-04-13T22:27:49.307 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.307 DEBUG:boto:Data: 2024-04-13T22:27:49.307 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.307 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.307 DEBUG:boto:Params: {} 2024-04-13T22:27:49.307 DEBUG:boto:Token: None 2024-04-13T22:27:49.308 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054520abb5b3669c9-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.310 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.311 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.311 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.311 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.311 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.311 DEBUG:boto:Method: GET 2024-04-13T22:27:49.311 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.311 DEBUG:boto:Data: 2024-04-13T22:27:49.311 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.312 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.312 DEBUG:boto:Params: {} 2024-04-13T22:27:49.312 DEBUG:boto:Token: None 2024-04-13T22:27:49.312 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3fefc9202f7c64-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.315 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.316 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.316 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.316 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.316 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:27:49.316 DEBUG:boto:Data: 2024-04-13T22:27:49.316 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.317 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.317 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.317 DEBUG:boto:Params: {} 2024-04-13T22:27:49.317 DEBUG:boto:Token: None 2024-04-13T22:27:49.317 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115 2024-04-13T22:27:49.317 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iQpTTlVbkdtZwHCgTvyCAMvMrxI= 2024-04-13T22:27:49.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iQpTTlVbkdtZwHCgTvyCAMvMrxI=', 'Content-Length': '0'} 2024-04-13T22:27:49.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eba2966e5007069-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.319 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.319 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.320 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.320 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:27:49.320 DEBUG:boto:Data: 2024-04-13T22:27:49.320 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.320 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.320 DEBUG:boto:Params: {} 2024-04-13T22:27:49.320 DEBUG:boto:Token: None 2024-04-13T22:27:49.320 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115 2024-04-13T22:27:49.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iQpTTlVbkdtZwHCgTvyCAMvMrxI= 2024-04-13T22:27:49.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iQpTTlVbkdtZwHCgTvyCAMvMrxI=', 'Content-Length': '0'} 2024-04-13T22:27:49.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef817a5efd0fc444-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:27:49.323 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.323 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.323 DEBUG:boto:Method: GET 2024-04-13T22:27:49.323 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:27:49.323 DEBUG:boto:Data: 2024-04-13T22:27:49.323 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.323 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.323 DEBUG:boto:Params: {} 2024-04-13T22:27:49.323 DEBUG:boto:Token: None 2024-04-13T22:27:49.324 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115 2024-04-13T22:27:49.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jAX/SYsnwZK853HS5b5DIoVPeNo= 2024-04-13T22:27:49.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jAX/SYsnwZK853HS5b5DIoVPeNo=', 'Content-Length': '0'} 2024-04-13T22:27:49.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060666df8c7e39e74-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.325 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:27:49.326 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:27:49.326 DEBUG:boto:Method: GET 2024-04-13T22:27:49.326 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:27:49.326 DEBUG:boto:Data: 2024-04-13T22:27:49.326 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.326 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.326 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.326 DEBUG:boto:Params: {} 2024-04-13T22:27:49.326 DEBUG:boto:Token: None 2024-04-13T22:27:49.327 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj115 2024-04-13T22:27:49.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jAX/SYsnwZK853HS5b5DIoVPeNo= 2024-04-13T22:27:49.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jAX/SYsnwZK853HS5b5DIoVPeNo=', 'Content-Length': '0'} 2024-04-13T22:27:49.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011421059cb79ed8-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.370 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.370 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.371 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.371 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.371 DEBUG:boto:Method: GET 2024-04-13T22:27:49.371 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.371 DEBUG:boto:Data: 2024-04-13T22:27:49.371 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.371 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.371 DEBUG:boto:Params: {} 2024-04-13T22:27:49.371 DEBUG:boto:Token: None 2024-04-13T22:27:49.372 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e6e28ebaacb0645-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.374 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.375 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.375 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.375 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.375 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.375 DEBUG:boto:Method: GET 2024-04-13T22:27:49.375 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:27:49.375 DEBUG:boto:Data: 2024-04-13T22:27:49.375 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.376 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.376 DEBUG:boto:Params: {} 2024-04-13T22:27:49.376 DEBUG:boto:Token: None 2024-04-13T22:27:49.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e86df303734719c-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.379 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:27:49.380 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.380 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.380 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.380 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.380 DEBUG:boto:Method: GET 2024-04-13T22:27:49.380 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.380 DEBUG:boto:Data: 2024-04-13T22:27:49.380 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.380 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.380 DEBUG:boto:Params: {} 2024-04-13T22:27:49.381 DEBUG:boto:Token: None 2024-04-13T22:27:49.381 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8= 2024-04-13T22:27:49.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8=', 'Content-Length': '0'} 2024-04-13T22:27:49.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005bf272efa4ea821-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.383 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.383 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.383 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.383 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.383 DEBUG:boto:Method: GET 2024-04-13T22:27:49.383 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.384 DEBUG:boto:Data: 2024-04-13T22:27:49.384 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.384 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.384 DEBUG:boto:Params: {} 2024-04-13T22:27:49.384 DEBUG:boto:Token: None 2024-04-13T22:27:49.384 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8= 2024-04-13T22:27:49.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8=', 'Content-Length': '0'} 2024-04-13T22:27:49.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002110f5d03cb12356-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.387 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.387 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.387 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.387 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.387 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.387 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.387 DEBUG:boto:Data: 2024-04-13T22:27:49.387 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.387 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.387 DEBUG:boto:Params: {} 2024-04-13T22:27:49.388 DEBUG:boto:Token: None 2024-04-13T22:27:49.388 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cGGxfdWyFGXxSDj//eFGA9kdcgo= 2024-04-13T22:27:49.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cGGxfdWyFGXxSDj//eFGA9kdcgo=', 'Content-Length': '0'} 2024-04-13T22:27:49.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001848268f7096ee89-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.389 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.390 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.390 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.390 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.390 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.390 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.390 DEBUG:boto:Data: 2024-04-13T22:27:49.390 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.390 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.390 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.390 DEBUG:boto:Params: {} 2024-04-13T22:27:49.391 DEBUG:boto:Token: None 2024-04-13T22:27:49.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cGGxfdWyFGXxSDj//eFGA9kdcgo= 2024-04-13T22:27:49.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cGGxfdWyFGXxSDj//eFGA9kdcgo=', 'Content-Length': '0'} 2024-04-13T22:27:49.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007350b8f223fb0a7a-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:27:49.393 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.393 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.393 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.393 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.393 DEBUG:boto:Method: GET 2024-04-13T22:27:49.393 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.393 DEBUG:boto:Data: 2024-04-13T22:27:49.393 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.393 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.394 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.394 DEBUG:boto:Params: {} 2024-04-13T22:27:49.394 DEBUG:boto:Token: None 2024-04-13T22:27:49.394 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.394 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8= 2024-04-13T22:27:49.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8=', 'Content-Length': '0'} 2024-04-13T22:27:49.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003de5df4705b4a24f-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.438 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.439 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.439 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.439 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.439 DEBUG:boto:Method: GET 2024-04-13T22:27:49.439 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.439 DEBUG:boto:Data: 2024-04-13T22:27:49.439 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.439 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.439 DEBUG:boto:Params: {} 2024-04-13T22:27:49.439 DEBUG:boto:Token: None 2024-04-13T22:27:49.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:27:49.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8= 2024-04-13T22:27:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uSQ7qWxwBQVgBacc0vQvaHHEVM8=', 'Content-Length': '0'} 2024-04-13T22:27:49.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9779b801128e28d-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.442 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.442 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.442 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.442 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.442 DEBUG:boto:Method: GET 2024-04-13T22:27:49.442 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.442 DEBUG:boto:Data: 2024-04-13T22:27:49.442 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.442 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.443 DEBUG:boto:Params: {} 2024-04-13T22:27:49.443 DEBUG:boto:Token: None 2024-04-13T22:27:49.443 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36d173d50704793-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.445 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.446 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.446 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.446 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.446 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.446 DEBUG:boto:Method: GET 2024-04-13T22:27:49.446 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.447 DEBUG:boto:Data: 2024-04-13T22:27:49.447 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.447 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.447 DEBUG:boto:Params: {} 2024-04-13T22:27:49.447 DEBUG:boto:Token: None 2024-04-13T22:27:49.447 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063be710f48e909e3-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.450 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.451 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.451 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.451 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.451 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:27:49.451 DEBUG:boto:Data: 2024-04-13T22:27:49.451 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.451 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.451 DEBUG:boto:Params: {} 2024-04-13T22:27:49.452 DEBUG:boto:Token: None 2024-04-13T22:27:49.452 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116 2024-04-13T22:27:49.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoxRIuVDF6D2waK/LTEcIjm2Log= 2024-04-13T22:27:49.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoxRIuVDF6D2waK/LTEcIjm2Log=', 'Content-Length': '0'} 2024-04-13T22:27:49.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ecb6926bdc14d0-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.453 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.454 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.454 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.454 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:27:49.454 DEBUG:boto:Data: 2024-04-13T22:27:49.454 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.454 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.454 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.454 DEBUG:boto:Params: {} 2024-04-13T22:27:49.454 DEBUG:boto:Token: None 2024-04-13T22:27:49.455 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116 2024-04-13T22:27:49.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoxRIuVDF6D2waK/LTEcIjm2Log= 2024-04-13T22:27:49.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoxRIuVDF6D2waK/LTEcIjm2Log=', 'Content-Length': '0'} 2024-04-13T22:27:49.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e9047886a53e18-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:27:49.457 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.457 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.457 DEBUG:boto:Method: GET 2024-04-13T22:27:49.457 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:27:49.457 DEBUG:boto:Data: 2024-04-13T22:27:49.457 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.457 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.457 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.457 DEBUG:boto:Params: {} 2024-04-13T22:27:49.457 DEBUG:boto:Token: None 2024-04-13T22:27:49.458 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116 2024-04-13T22:27:49.458 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ng9ZWac4sqMVbq+MnCyvMvBOf1c= 2024-04-13T22:27:49.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ng9ZWac4sqMVbq+MnCyvMvBOf1c=', 'Content-Length': '0'} 2024-04-13T22:27:49.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20aadbe6e017861-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.460 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:27:49.460 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:27:49.460 DEBUG:boto:Method: GET 2024-04-13T22:27:49.460 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:27:49.460 DEBUG:boto:Data: 2024-04-13T22:27:49.460 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.461 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.461 DEBUG:boto:Params: {} 2024-04-13T22:27:49.461 DEBUG:boto:Token: None 2024-04-13T22:27:49.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj116 2024-04-13T22:27:49.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ng9ZWac4sqMVbq+MnCyvMvBOf1c= 2024-04-13T22:27:49.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ng9ZWac4sqMVbq+MnCyvMvBOf1c=', 'Content-Length': '0'} 2024-04-13T22:27:49.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b28eb244a2ff6a-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.463 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.463 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.463 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.463 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.463 DEBUG:boto:Method: GET 2024-04-13T22:27:49.463 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.464 DEBUG:boto:Data: 2024-04-13T22:27:49.464 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.464 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.464 DEBUG:boto:Params: {} 2024-04-13T22:27:49.464 DEBUG:boto:Token: None 2024-04-13T22:27:49.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aae9d7d911328fb-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.466 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.467 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.467 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.467 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.467 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.467 DEBUG:boto:Method: GET 2024-04-13T22:27:49.467 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:27:49.467 DEBUG:boto:Data: 2024-04-13T22:27:49.468 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.468 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.468 DEBUG:boto:Params: {} 2024-04-13T22:27:49.468 DEBUG:boto:Token: None 2024-04-13T22:27:49.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003608bbd716a60bde-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.471 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:27:49.472 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.472 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.472 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.472 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.472 DEBUG:boto:Method: GET 2024-04-13T22:27:49.472 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.472 DEBUG:boto:Data: 2024-04-13T22:27:49.472 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.473 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.473 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.473 DEBUG:boto:Params: {} 2024-04-13T22:27:49.473 DEBUG:boto:Token: None 2024-04-13T22:27:49.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk= 2024-04-13T22:27:49.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk=', 'Content-Length': '0'} 2024-04-13T22:27:49.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514d29e57cbfe6d4-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.475 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.475 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.475 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.475 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.475 DEBUG:boto:Method: GET 2024-04-13T22:27:49.476 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.476 DEBUG:boto:Data: 2024-04-13T22:27:49.476 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.476 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.476 DEBUG:boto:Params: {} 2024-04-13T22:27:49.476 DEBUG:boto:Token: None 2024-04-13T22:27:49.476 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.476 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk= 2024-04-13T22:27:49.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk=', 'Content-Length': '0'} 2024-04-13T22:27:49.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb3696dfad32641-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.518 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.518 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.519 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.519 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.519 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.519 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.519 DEBUG:boto:Data: 2024-04-13T22:27:49.519 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.519 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.519 DEBUG:boto:Params: {} 2024-04-13T22:27:49.519 DEBUG:boto:Token: None 2024-04-13T22:27:49.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y2Gfv1dLk2Fr6tOvKoOCEi3n7wU= 2024-04-13T22:27:49.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y2Gfv1dLk2Fr6tOvKoOCEi3n7wU=', 'Content-Length': '0'} 2024-04-13T22:27:49.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900b14f74ef11556-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.522 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.522 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.522 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.522 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.522 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.522 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.522 DEBUG:boto:Data: 2024-04-13T22:27:49.522 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.523 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.523 DEBUG:boto:Params: {} 2024-04-13T22:27:49.523 DEBUG:boto:Token: None 2024-04-13T22:27:49.523 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.523 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y2Gfv1dLk2Fr6tOvKoOCEi3n7wU= 2024-04-13T22:27:49.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y2Gfv1dLk2Fr6tOvKoOCEi3n7wU=', 'Content-Length': '0'} 2024-04-13T22:27:49.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db8609730b9805e7-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:27:49.525 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.525 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.525 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.525 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.526 DEBUG:boto:Method: GET 2024-04-13T22:27:49.526 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.526 DEBUG:boto:Data: 2024-04-13T22:27:49.526 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.526 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.526 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.526 DEBUG:boto:Params: {} 2024-04-13T22:27:49.526 DEBUG:boto:Token: None 2024-04-13T22:27:49.526 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk= 2024-04-13T22:27:49.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk=', 'Content-Length': '0'} 2024-04-13T22:27:49.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def785a39cf3029a-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.528 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.529 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.529 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.529 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.529 DEBUG:boto:Method: GET 2024-04-13T22:27:49.529 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.529 DEBUG:boto:Data: 2024-04-13T22:27:49.529 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.529 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.529 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.529 DEBUG:boto:Params: {} 2024-04-13T22:27:49.530 DEBUG:boto:Token: None 2024-04-13T22:27:49.530 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:27:49.530 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk= 2024-04-13T22:27:49.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sKj7DQH3oWbjw9/WHXoYF0oIaSk=', 'Content-Length': '0'} 2024-04-13T22:27:49.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051b4d757287cb5a-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.532 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.532 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.532 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.532 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.532 DEBUG:boto:Method: GET 2024-04-13T22:27:49.532 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.533 DEBUG:boto:Data: 2024-04-13T22:27:49.533 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.533 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.533 DEBUG:boto:Params: {} 2024-04-13T22:27:49.533 DEBUG:boto:Token: None 2024-04-13T22:27:49.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78b37d1cbd16229-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.536 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.536 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.536 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.536 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.536 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.536 DEBUG:boto:Method: GET 2024-04-13T22:27:49.536 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.537 DEBUG:boto:Data: 2024-04-13T22:27:49.537 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.537 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.537 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.537 DEBUG:boto:Params: {} 2024-04-13T22:27:49.537 DEBUG:boto:Token: None 2024-04-13T22:27:49.537 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.537 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f472bbb15f408d8-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.540 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.541 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.541 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.541 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.541 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:27:49.541 DEBUG:boto:Data: 2024-04-13T22:27:49.541 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.541 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.541 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.541 DEBUG:boto:Params: {} 2024-04-13T22:27:49.542 DEBUG:boto:Token: None 2024-04-13T22:27:49.542 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145 2024-04-13T22:27:49.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LFNolAPIPEI1OgfHcxAn71w/ydU= 2024-04-13T22:27:49.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LFNolAPIPEI1OgfHcxAn71w/ydU=', 'Content-Length': '0'} 2024-04-13T22:27:49.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deb5b14e5a43e11f-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.544 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.544 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.544 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.544 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:27:49.544 DEBUG:boto:Data: 2024-04-13T22:27:49.544 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.544 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.544 DEBUG:boto:Params: {} 2024-04-13T22:27:49.544 DEBUG:boto:Token: None 2024-04-13T22:27:49.545 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145 2024-04-13T22:27:49.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LFNolAPIPEI1OgfHcxAn71w/ydU= 2024-04-13T22:27:49.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LFNolAPIPEI1OgfHcxAn71w/ydU=', 'Content-Length': '0'} 2024-04-13T22:27:49.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b75f884ae6e75a-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:27:49.547 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.547 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.547 DEBUG:boto:Method: GET 2024-04-13T22:27:49.547 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:27:49.547 DEBUG:boto:Data: 2024-04-13T22:27:49.547 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.547 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.547 DEBUG:boto:Params: {} 2024-04-13T22:27:49.547 DEBUG:boto:Token: None 2024-04-13T22:27:49.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145 2024-04-13T22:27:49.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aQ9q6yxhAvecL1YdrQOcRN8Iyik= 2024-04-13T22:27:49.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aQ9q6yxhAvecL1YdrQOcRN8Iyik=', 'Content-Length': '0'} 2024-04-13T22:27:49.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c766e1a249367bd9-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.590 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:27:49.591 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:27:49.591 DEBUG:boto:Method: GET 2024-04-13T22:27:49.591 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:27:49.591 DEBUG:boto:Data: 2024-04-13T22:27:49.591 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.591 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.591 DEBUG:boto:Params: {} 2024-04-13T22:27:49.591 DEBUG:boto:Token: None 2024-04-13T22:27:49.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj145 2024-04-13T22:27:49.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aQ9q6yxhAvecL1YdrQOcRN8Iyik= 2024-04-13T22:27:49.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aQ9q6yxhAvecL1YdrQOcRN8Iyik=', 'Content-Length': '0'} 2024-04-13T22:27:49.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056b2d46c12660abe-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.594 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.594 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.594 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.594 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.595 DEBUG:boto:Method: GET 2024-04-13T22:27:49.595 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.595 DEBUG:boto:Data: 2024-04-13T22:27:49.595 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.595 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.595 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.595 DEBUG:boto:Params: {} 2024-04-13T22:27:49.595 DEBUG:boto:Token: None 2024-04-13T22:27:49.595 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a5166b3c32dc05-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.598 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.598 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.598 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.598 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.599 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.599 DEBUG:boto:Method: GET 2024-04-13T22:27:49.599 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:27:49.599 DEBUG:boto:Data: 2024-04-13T22:27:49.599 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.599 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.599 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.599 DEBUG:boto:Params: {} 2024-04-13T22:27:49.600 DEBUG:boto:Token: None 2024-04-13T22:27:49.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2db516c6e12d383-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.603 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:27:49.603 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.603 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.603 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.604 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.604 DEBUG:boto:Method: GET 2024-04-13T22:27:49.604 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.604 DEBUG:boto:Data: 2024-04-13T22:27:49.604 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.604 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.604 DEBUG:boto:Params: {} 2024-04-13T22:27:49.604 DEBUG:boto:Token: None 2024-04-13T22:27:49.605 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.605 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM= 2024-04-13T22:27:49.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM=', 'Content-Length': '0'} 2024-04-13T22:27:49.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a91d06c2bdb256-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.607 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.607 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.607 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.607 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.607 DEBUG:boto:Method: GET 2024-04-13T22:27:49.607 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.607 DEBUG:boto:Data: 2024-04-13T22:27:49.607 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.607 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.607 DEBUG:boto:Params: {} 2024-04-13T22:27:49.608 DEBUG:boto:Token: None 2024-04-13T22:27:49.608 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM= 2024-04-13T22:27:49.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM=', 'Content-Length': '0'} 2024-04-13T22:27:49.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4420d97c842f6f-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.650 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.651 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.651 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.651 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.651 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.651 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.651 DEBUG:boto:Data: 2024-04-13T22:27:49.651 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.651 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.651 DEBUG:boto:Params: {} 2024-04-13T22:27:49.651 DEBUG:boto:Token: None 2024-04-13T22:27:49.652 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fRRM6RjatWtDdIyhdQOfey79XYw= 2024-04-13T22:27:49.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fRRM6RjatWtDdIyhdQOfey79XYw=', 'Content-Length': '0'} 2024-04-13T22:27:49.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af0305b922d19d1-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.654 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.654 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.654 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.654 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.654 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.654 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.654 DEBUG:boto:Data: 2024-04-13T22:27:49.655 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.655 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.655 DEBUG:boto:Params: {} 2024-04-13T22:27:49.655 DEBUG:boto:Token: None 2024-04-13T22:27:49.655 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fRRM6RjatWtDdIyhdQOfey79XYw= 2024-04-13T22:27:49.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fRRM6RjatWtDdIyhdQOfey79XYw=', 'Content-Length': '0'} 2024-04-13T22:27:49.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3c1060f1c11ad8-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:27:49.658 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.658 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.658 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.658 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.658 DEBUG:boto:Method: GET 2024-04-13T22:27:49.658 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.658 DEBUG:boto:Data: 2024-04-13T22:27:49.658 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.658 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.659 DEBUG:boto:Params: {} 2024-04-13T22:27:49.659 DEBUG:boto:Token: None 2024-04-13T22:27:49.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM= 2024-04-13T22:27:49.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM=', 'Content-Length': '0'} 2024-04-13T22:27:49.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007686654508ad56c3-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.661 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.661 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.661 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.661 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.661 DEBUG:boto:Method: GET 2024-04-13T22:27:49.661 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.661 DEBUG:boto:Data: 2024-04-13T22:27:49.661 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.662 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.662 DEBUG:boto:Params: {} 2024-04-13T22:27:49.662 DEBUG:boto:Token: None 2024-04-13T22:27:49.662 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:27:49.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM= 2024-04-13T22:27:49.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VxkzkC0Jrt8H8aWtF2CxOes7nIM=', 'Content-Length': '0'} 2024-04-13T22:27:49.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512472e51b55bf05-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.664 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.664 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.664 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.664 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.664 DEBUG:boto:Method: GET 2024-04-13T22:27:49.664 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.664 DEBUG:boto:Data: 2024-04-13T22:27:49.665 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.665 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.665 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.665 DEBUG:boto:Params: {} 2024-04-13T22:27:49.665 DEBUG:boto:Token: None 2024-04-13T22:27:49.665 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.665 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cbee6d39f4eab6c-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.668 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.668 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.668 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.668 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.668 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.669 DEBUG:boto:Method: GET 2024-04-13T22:27:49.669 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.669 DEBUG:boto:Data: 2024-04-13T22:27:49.669 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.669 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.669 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.669 DEBUG:boto:Params: {} 2024-04-13T22:27:49.669 DEBUG:boto:Token: None 2024-04-13T22:27:49.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a24132d043ffba8-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.673 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.673 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.673 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.673 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.673 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:27:49.673 DEBUG:boto:Data: 2024-04-13T22:27:49.673 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.673 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.674 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.674 DEBUG:boto:Params: {} 2024-04-13T22:27:49.674 DEBUG:boto:Token: None 2024-04-13T22:27:49.674 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17 2024-04-13T22:27:49.674 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/keie1JWDfMVpvabIBORjmQyW0E= 2024-04-13T22:27:49.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/keie1JWDfMVpvabIBORjmQyW0E=', 'Content-Length': '0'} 2024-04-13T22:27:49.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99a5f0485417ff5-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.676 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.676 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.676 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.676 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:27:49.676 DEBUG:boto:Data: 2024-04-13T22:27:49.676 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.676 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.677 DEBUG:boto:Params: {} 2024-04-13T22:27:49.677 DEBUG:boto:Token: None 2024-04-13T22:27:49.677 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17 2024-04-13T22:27:49.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/keie1JWDfMVpvabIBORjmQyW0E= 2024-04-13T22:27:49.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/keie1JWDfMVpvabIBORjmQyW0E=', 'Content-Length': '0'} 2024-04-13T22:27:49.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097bad3b8765d113-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:27:49.679 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.679 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.679 DEBUG:boto:Method: GET 2024-04-13T22:27:49.679 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:27:49.679 DEBUG:boto:Data: 2024-04-13T22:27:49.679 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.679 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.679 DEBUG:boto:Params: {} 2024-04-13T22:27:49.680 DEBUG:boto:Token: None 2024-04-13T22:27:49.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17 2024-04-13T22:27:49.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p+T999fnE+r6v6HouQcmxSMnSyc= 2024-04-13T22:27:49.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p+T999fnE+r6v6HouQcmxSMnSyc=', 'Content-Length': '0'} 2024-04-13T22:27:49.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7bafc3b4b222559-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.682 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:27:49.682 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:27:49.682 DEBUG:boto:Method: GET 2024-04-13T22:27:49.682 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:27:49.682 DEBUG:boto:Data: 2024-04-13T22:27:49.682 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.682 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.682 DEBUG:boto:Params: {} 2024-04-13T22:27:49.682 DEBUG:boto:Token: None 2024-04-13T22:27:49.683 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj17 2024-04-13T22:27:49.683 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p+T999fnE+r6v6HouQcmxSMnSyc= 2024-04-13T22:27:49.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p+T999fnE+r6v6HouQcmxSMnSyc=', 'Content-Length': '0'} 2024-04-13T22:27:49.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006560a961c49f8ae9-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.730 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.730 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.731 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.731 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.731 DEBUG:boto:Method: GET 2024-04-13T22:27:49.731 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.731 DEBUG:boto:Data: 2024-04-13T22:27:49.731 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.731 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.731 DEBUG:boto:Params: {} 2024-04-13T22:27:49.731 DEBUG:boto:Token: None 2024-04-13T22:27:49.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ef2df9afac4d98-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.734 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.734 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.734 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.735 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.735 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.735 DEBUG:boto:Method: GET 2024-04-13T22:27:49.735 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:27:49.735 DEBUG:boto:Data: 2024-04-13T22:27:49.735 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.735 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.735 DEBUG:boto:Params: {} 2024-04-13T22:27:49.735 DEBUG:boto:Token: None 2024-04-13T22:27:49.735 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227012ae4df516ea-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.739 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:27:49.740 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.740 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.740 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.740 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.740 DEBUG:boto:Method: GET 2024-04-13T22:27:49.740 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.740 DEBUG:boto:Data: 2024-04-13T22:27:49.740 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.740 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.741 DEBUG:boto:Params: {} 2024-04-13T22:27:49.741 DEBUG:boto:Token: None 2024-04-13T22:27:49.741 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw= 2024-04-13T22:27:49.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw=', 'Content-Length': '0'} 2024-04-13T22:27:49.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114696b49d86410b-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.743 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.743 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.743 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.743 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.743 DEBUG:boto:Method: GET 2024-04-13T22:27:49.743 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.743 DEBUG:boto:Data: 2024-04-13T22:27:49.743 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.744 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.744 DEBUG:boto:Params: {} 2024-04-13T22:27:49.744 DEBUG:boto:Token: None 2024-04-13T22:27:49.744 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw= 2024-04-13T22:27:49.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw=', 'Content-Length': '0'} 2024-04-13T22:27:49.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1d3eb5978ba119-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.746 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.746 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.746 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.747 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.747 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.747 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.747 DEBUG:boto:Data: 2024-04-13T22:27:49.747 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.747 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.747 DEBUG:boto:Params: {} 2024-04-13T22:27:49.747 DEBUG:boto:Token: None 2024-04-13T22:27:49.747 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nqDmkzUpuxUslBdYBKXKRJc1y3w= 2024-04-13T22:27:49.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nqDmkzUpuxUslBdYBKXKRJc1y3w=', 'Content-Length': '0'} 2024-04-13T22:27:49.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129fbadd9ecbb147-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.749 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.749 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.749 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.750 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.750 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.750 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.750 DEBUG:boto:Data: 2024-04-13T22:27:49.750 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.750 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.750 DEBUG:boto:Params: {} 2024-04-13T22:27:49.750 DEBUG:boto:Token: None 2024-04-13T22:27:49.750 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nqDmkzUpuxUslBdYBKXKRJc1y3w= 2024-04-13T22:27:49.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nqDmkzUpuxUslBdYBKXKRJc1y3w=', 'Content-Length': '0'} 2024-04-13T22:27:49.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d2d6967ec35dc3-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:27:49.752 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.753 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.753 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.753 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.753 DEBUG:boto:Method: GET 2024-04-13T22:27:49.753 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.753 DEBUG:boto:Data: 2024-04-13T22:27:49.753 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.753 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.753 DEBUG:boto:Params: {} 2024-04-13T22:27:49.754 DEBUG:boto:Token: None 2024-04-13T22:27:49.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw= 2024-04-13T22:27:49.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw=', 'Content-Length': '0'} 2024-04-13T22:27:49.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6cd99ffad3f9ec3-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.798 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.798 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.799 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.799 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.799 DEBUG:boto:Method: GET 2024-04-13T22:27:49.799 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.799 DEBUG:boto:Data: 2024-04-13T22:27:49.799 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.799 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.799 DEBUG:boto:Params: {} 2024-04-13T22:27:49.799 DEBUG:boto:Token: None 2024-04-13T22:27:49.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:27:49.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw= 2024-04-13T22:27:49.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jy2lgmtNpK1dfJJcr6EnkhYpoTw=', 'Content-Length': '0'} 2024-04-13T22:27:49.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aefea473e3c5a638-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.801 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.802 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.802 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.802 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.802 DEBUG:boto:Method: GET 2024-04-13T22:27:49.802 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.802 DEBUG:boto:Data: 2024-04-13T22:27:49.802 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.802 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.802 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.802 DEBUG:boto:Params: {} 2024-04-13T22:27:49.802 DEBUG:boto:Token: None 2024-04-13T22:27:49.803 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0bb768de3de660d-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.805 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.805 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.805 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.805 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.806 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.806 DEBUG:boto:Method: GET 2024-04-13T22:27:49.806 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.806 DEBUG:boto:Data: 2024-04-13T22:27:49.806 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.806 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.806 DEBUG:boto:Params: {} 2024-04-13T22:27:49.806 DEBUG:boto:Token: None 2024-04-13T22:27:49.806 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c0684860fb39130-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.810 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.810 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.811 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.811 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.811 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:27:49.811 DEBUG:boto:Data: 2024-04-13T22:27:49.811 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.811 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.811 DEBUG:boto:Params: {} 2024-04-13T22:27:49.811 DEBUG:boto:Token: None 2024-04-13T22:27:49.811 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19 2024-04-13T22:27:49.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lq4Dw24DoA3DSwnU+KT/hyURRMQ= 2024-04-13T22:27:49.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lq4Dw24DoA3DSwnU+KT/hyURRMQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309b7b919d2de1b6-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.813 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.813 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.814 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.814 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:27:49.814 DEBUG:boto:Data: 2024-04-13T22:27:49.814 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.814 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.814 DEBUG:boto:Params: {} 2024-04-13T22:27:49.814 DEBUG:boto:Token: None 2024-04-13T22:27:49.814 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19 2024-04-13T22:27:49.814 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lq4Dw24DoA3DSwnU+KT/hyURRMQ= 2024-04-13T22:27:49.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lq4Dw24DoA3DSwnU+KT/hyURRMQ=', 'Content-Length': '0'} 2024-04-13T22:27:49.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf4a67f65ee1ced-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:27:49.816 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.817 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.817 DEBUG:boto:Method: GET 2024-04-13T22:27:49.817 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:27:49.817 DEBUG:boto:Data: 2024-04-13T22:27:49.817 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.817 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.817 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.817 DEBUG:boto:Params: {} 2024-04-13T22:27:49.817 DEBUG:boto:Token: None 2024-04-13T22:27:49.817 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19 2024-04-13T22:27:49.818 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FbbGHiBSf6L91wh69H2Q9hdxB4o= 2024-04-13T22:27:49.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FbbGHiBSf6L91wh69H2Q9hdxB4o=', 'Content-Length': '0'} 2024-04-13T22:27:49.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9bee46c40140685-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.820 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:27:49.820 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:27:49.820 DEBUG:boto:Method: GET 2024-04-13T22:27:49.820 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:27:49.820 DEBUG:boto:Data: 2024-04-13T22:27:49.820 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.820 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.820 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.820 DEBUG:boto:Params: {} 2024-04-13T22:27:49.821 DEBUG:boto:Token: None 2024-04-13T22:27:49.821 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj19 2024-04-13T22:27:49.821 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FbbGHiBSf6L91wh69H2Q9hdxB4o= 2024-04-13T22:27:49.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FbbGHiBSf6L91wh69H2Q9hdxB4o=', 'Content-Length': '0'} 2024-04-13T22:27:49.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58c15ad2998926a-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.822 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.823 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.823 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.823 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.823 DEBUG:boto:Method: GET 2024-04-13T22:27:49.823 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.823 DEBUG:boto:Data: 2024-04-13T22:27:49.823 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.823 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.823 DEBUG:boto:Params: {} 2024-04-13T22:27:49.824 DEBUG:boto:Token: None 2024-04-13T22:27:49.824 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028144188f364b902-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.826 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.827 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.827 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.827 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.827 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.827 DEBUG:boto:Method: GET 2024-04-13T22:27:49.827 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:27:49.827 DEBUG:boto:Data: 2024-04-13T22:27:49.827 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.827 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.827 DEBUG:boto:Params: {} 2024-04-13T22:27:49.828 DEBUG:boto:Token: None 2024-04-13T22:27:49.828 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bec6fa22f78707fe-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.874 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:27:49.875 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.875 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.875 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.875 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.875 DEBUG:boto:Method: GET 2024-04-13T22:27:49.875 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.875 DEBUG:boto:Data: 2024-04-13T22:27:49.875 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.875 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.875 DEBUG:boto:Params: {} 2024-04-13T22:27:49.876 DEBUG:boto:Token: None 2024-04-13T22:27:49.876 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo= 2024-04-13T22:27:49.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo=', 'Content-Length': '0'} 2024-04-13T22:27:49.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df10f83d29f2af93-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.878 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.878 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.878 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.878 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.879 DEBUG:boto:Method: GET 2024-04-13T22:27:49.879 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.879 DEBUG:boto:Data: 2024-04-13T22:27:49.879 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.879 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.879 DEBUG:boto:Params: {} 2024-04-13T22:27:49.879 DEBUG:boto:Token: None 2024-04-13T22:27:49.879 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.879 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo= 2024-04-13T22:27:49.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo=', 'Content-Length': '0'} 2024-04-13T22:27:49.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad04ae2368793f6c-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.881 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.881 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.882 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.882 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.882 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.882 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.882 DEBUG:boto:Data: 2024-04-13T22:27:49.882 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.882 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.882 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.882 DEBUG:boto:Params: {} 2024-04-13T22:27:49.882 DEBUG:boto:Token: None 2024-04-13T22:27:49.882 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iVHKuGkU/Vf0zkYI1ytCIzr1SjA= 2024-04-13T22:27:49.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iVHKuGkU/Vf0zkYI1ytCIzr1SjA=', 'Content-Length': '0'} 2024-04-13T22:27:49.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e1bfa4e0542713-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.884 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.884 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.884 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.884 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.885 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.885 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.885 DEBUG:boto:Data: 2024-04-13T22:27:49.885 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.885 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.885 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.885 DEBUG:boto:Params: {} 2024-04-13T22:27:49.885 DEBUG:boto:Token: None 2024-04-13T22:27:49.885 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.885 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iVHKuGkU/Vf0zkYI1ytCIzr1SjA= 2024-04-13T22:27:49.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iVHKuGkU/Vf0zkYI1ytCIzr1SjA=', 'Content-Length': '0'} 2024-04-13T22:27:49.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc49e2257c20fb7-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:27:49.887 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.888 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.888 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.888 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.888 DEBUG:boto:Method: GET 2024-04-13T22:27:49.888 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.888 DEBUG:boto:Data: 2024-04-13T22:27:49.888 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.888 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.888 DEBUG:boto:Params: {} 2024-04-13T22:27:49.888 DEBUG:boto:Token: None 2024-04-13T22:27:49.889 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo= 2024-04-13T22:27:49.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo=', 'Content-Length': '0'} 2024-04-13T22:27:49.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f6cf942f91ba1a-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.891 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.891 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.891 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.891 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.891 DEBUG:boto:Method: GET 2024-04-13T22:27:49.891 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.891 DEBUG:boto:Data: 2024-04-13T22:27:49.891 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.892 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.892 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.892 DEBUG:boto:Params: {} 2024-04-13T22:27:49.892 DEBUG:boto:Token: None 2024-04-13T22:27:49.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:27:49.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo= 2024-04-13T22:27:49.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ln0VaOyKtDwyhUAmIzp25eC2Oyo=', 'Content-Length': '0'} 2024-04-13T22:27:49.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005466cad2ff81f729-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.894 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.894 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.894 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.894 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.894 DEBUG:boto:Method: GET 2024-04-13T22:27:49.894 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.894 DEBUG:boto:Data: 2024-04-13T22:27:49.894 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.894 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.895 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.895 DEBUG:boto:Params: {} 2024-04-13T22:27:49.895 DEBUG:boto:Token: None 2024-04-13T22:27:49.895 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.895 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066bc3df4061f7627-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.897 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.898 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.898 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.898 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.898 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.898 DEBUG:boto:Method: GET 2024-04-13T22:27:49.898 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.898 DEBUG:boto:Data: 2024-04-13T22:27:49.898 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.898 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.899 DEBUG:boto:Params: {} 2024-04-13T22:27:49.899 DEBUG:boto:Token: None 2024-04-13T22:27:49.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a3628dcaaa2e4c-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.902 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.902 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.902 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.903 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.903 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:27:49.903 DEBUG:boto:Data: 2024-04-13T22:27:49.903 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.903 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.903 DEBUG:boto:Params: {} 2024-04-13T22:27:49.903 DEBUG:boto:Token: None 2024-04-13T22:27:49.903 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22 2024-04-13T22:27:49.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2MN3BKktoWS+wjzFj3wVon3eAdk= 2024-04-13T22:27:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2MN3BKktoWS+wjzFj3wVon3eAdk=', 'Content-Length': '0'} 2024-04-13T22:27:49.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb1ff1290fa711b1-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.905 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.905 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.905 DEBUG:boto:Method: HEAD 2024-04-13T22:27:49.905 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:27:49.906 DEBUG:boto:Data: 2024-04-13T22:27:49.906 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.906 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.906 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.906 DEBUG:boto:Params: {} 2024-04-13T22:27:49.906 DEBUG:boto:Token: None 2024-04-13T22:27:49.906 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22 2024-04-13T22:27:49.906 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2MN3BKktoWS+wjzFj3wVon3eAdk= 2024-04-13T22:27:49.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2MN3BKktoWS+wjzFj3wVon3eAdk=', 'Content-Length': '0'} 2024-04-13T22:27:49.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b362f144fc12d223-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:27:49.908 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.909 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.909 DEBUG:boto:Method: GET 2024-04-13T22:27:49.909 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:27:49.909 DEBUG:boto:Data: 2024-04-13T22:27:49.909 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.909 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.909 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.909 DEBUG:boto:Params: {} 2024-04-13T22:27:49.909 DEBUG:boto:Token: None 2024-04-13T22:27:49.909 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22 2024-04-13T22:27:49.910 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QRLJl3ubb/ZyOo/jQq/cgcjzVG4= 2024-04-13T22:27:49.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QRLJl3ubb/ZyOo/jQq/cgcjzVG4=', 'Content-Length': '0'} 2024-04-13T22:27:49.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7319a15f633292b-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.911 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:27:49.911 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:27:49.911 DEBUG:boto:Method: GET 2024-04-13T22:27:49.912 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:27:49.912 DEBUG:boto:Data: 2024-04-13T22:27:49.912 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.912 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.912 DEBUG:boto:Params: {} 2024-04-13T22:27:49.912 DEBUG:boto:Token: None 2024-04-13T22:27:49.912 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj22 2024-04-13T22:27:49.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QRLJl3ubb/ZyOo/jQq/cgcjzVG4= 2024-04-13T22:27:49.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QRLJl3ubb/ZyOo/jQq/cgcjzVG4=', 'Content-Length': '0'} 2024-04-13T22:27:49.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9decc629d16c87-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.914 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.914 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.915 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.915 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.915 DEBUG:boto:Method: GET 2024-04-13T22:27:49.915 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.915 DEBUG:boto:Data: 2024-04-13T22:27:49.915 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.915 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.915 DEBUG:boto:Params: {} 2024-04-13T22:27:49.915 DEBUG:boto:Token: None 2024-04-13T22:27:49.915 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3c71933798c4c1-00661b06e5-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.962 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.963 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:49.963 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:49.963 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.963 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.963 DEBUG:boto:Method: GET 2024-04-13T22:27:49.963 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:27:49.963 DEBUG:boto:Data: 2024-04-13T22:27:49.963 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.964 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.964 DEBUG:boto:Params: {} 2024-04-13T22:27:49.964 DEBUG:boto:Token: None 2024-04-13T22:27:49.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/ 2024-04-13T22:27:49.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc= 2024-04-13T22:27:49.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6oM+Q+AS/DY1FibQsPxISHaVQuc=', 'Content-Length': '0'} 2024-04-13T22:27:49.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000372c6d4eaec7c1ba-00661b06e5-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.967 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:49.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:27:49.968 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:49.968 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:49.968 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.968 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.968 DEBUG:boto:Method: GET 2024-04-13T22:27:49.968 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.968 DEBUG:boto:Data: 2024-04-13T22:27:49.968 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:49.968 DEBUG:boto:Port: 8001 2024-04-13T22:27:49.969 DEBUG:boto:Params: {} 2024-04-13T22:27:49.969 DEBUG:boto:Token: None 2024-04-13T22:27:49.969 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.969 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Vv3rtH7dFT2OmP1GwZooSVEiDA= 2024-04-13T22:27:49.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Vv3rtH7dFT2OmP1GwZooSVEiDA=', 'Content-Length': '0'} 2024-04-13T22:27:49.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000133eb248255ab633-00661b06e5-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:49.971 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:49.971 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:49.971 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.971 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.972 DEBUG:boto:Method: GET 2024-04-13T22:27:49.972 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.972 DEBUG:boto:Data: 2024-04-13T22:27:49.972 DEBUG:boto:Headers: {} 2024-04-13T22:27:49.972 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:49.972 DEBUG:boto:Port: 8000 2024-04-13T22:27:49.972 DEBUG:boto:Params: {} 2024-04-13T22:27:49.972 DEBUG:boto:Token: None 2024-04-13T22:27:49.972 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:49 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:49.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4Vv3rtH7dFT2OmP1GwZooSVEiDA= 2024-04-13T22:27:49.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:49 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Vv3rtH7dFT2OmP1GwZooSVEiDA=', 'Content-Length': '0'} 2024-04-13T22:27:49.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c435fa0a56a6cc6-00661b06e5-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.018 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.019 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.019 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.019 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.019 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.019 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.019 DEBUG:boto:Data: 2024-04-13T22:27:50.019 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.019 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.020 DEBUG:boto:Params: {} 2024-04-13T22:27:50.020 DEBUG:boto:Token: None 2024-04-13T22:27:50.020 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hOKlPzO7/jBn/SXsVg5QmBRHdyc= 2024-04-13T22:27:50.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hOKlPzO7/jBn/SXsVg5QmBRHdyc=', 'Content-Length': '0'} 2024-04-13T22:27:50.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687d562ad7545b34-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.022 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.022 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.022 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.022 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.023 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.023 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.023 DEBUG:boto:Data: 2024-04-13T22:27:50.023 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.023 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.023 DEBUG:boto:Params: {} 2024-04-13T22:27:50.023 DEBUG:boto:Token: None 2024-04-13T22:27:50.023 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.023 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hOKlPzO7/jBn/SXsVg5QmBRHdyc= 2024-04-13T22:27:50.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hOKlPzO7/jBn/SXsVg5QmBRHdyc=', 'Content-Length': '0'} 2024-04-13T22:27:50.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c681b72bc2696b-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:27:50.026 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.026 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.026 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.026 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.026 DEBUG:boto:Method: GET 2024-04-13T22:27:50.026 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.026 DEBUG:boto:Data: 2024-04-13T22:27:50.026 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.026 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.026 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.026 DEBUG:boto:Params: {} 2024-04-13T22:27:50.027 DEBUG:boto:Token: None 2024-04-13T22:27:50.027 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8hWWlq+px/P9no3i2HyoAUcZ5bY= 2024-04-13T22:27:50.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8hWWlq+px/P9no3i2HyoAUcZ5bY=', 'Content-Length': '0'} 2024-04-13T22:27:50.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db3fc3070147a1b-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.029 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.029 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.029 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.029 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.029 DEBUG:boto:Method: GET 2024-04-13T22:27:50.029 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.029 DEBUG:boto:Data: 2024-04-13T22:27:50.029 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.030 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.030 DEBUG:boto:Params: {} 2024-04-13T22:27:50.030 DEBUG:boto:Token: None 2024-04-13T22:27:50.030 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:27:50.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8hWWlq+px/P9no3i2HyoAUcZ5bY= 2024-04-13T22:27:50.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8hWWlq+px/P9no3i2HyoAUcZ5bY=', 'Content-Length': '0'} 2024-04-13T22:27:50.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce10ab08188db92-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.032 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.032 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.032 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.032 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.032 DEBUG:boto:Method: GET 2024-04-13T22:27:50.033 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.033 DEBUG:boto:Data: 2024-04-13T22:27:50.033 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.033 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.033 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.033 DEBUG:boto:Params: {} 2024-04-13T22:27:50.033 DEBUG:boto:Token: None 2024-04-13T22:27:50.033 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.034 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63e326ee1ccddbe-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.036 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.036 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.036 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.036 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.037 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.037 DEBUG:boto:Method: GET 2024-04-13T22:27:50.037 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.037 DEBUG:boto:Data: 2024-04-13T22:27:50.037 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.037 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.037 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.037 DEBUG:boto:Params: {} 2024-04-13T22:27:50.037 DEBUG:boto:Token: None 2024-04-13T22:27:50.037 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.038 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ae2673dc0480627-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.041 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.041 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.041 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.042 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.042 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:27:50.042 DEBUG:boto:Data: 2024-04-13T22:27:50.042 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.042 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.042 DEBUG:boto:Params: {} 2024-04-13T22:27:50.042 DEBUG:boto:Token: None 2024-04-13T22:27:50.042 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23 2024-04-13T22:27:50.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HGNjL681jiCXc/aDmwy+rXZsuic= 2024-04-13T22:27:50.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HGNjL681jiCXc/aDmwy+rXZsuic=', 'Content-Length': '0'} 2024-04-13T22:27:50.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e09293f5ec191df3-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.044 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.045 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.045 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.045 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:27:50.045 DEBUG:boto:Data: 2024-04-13T22:27:50.045 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.045 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.045 DEBUG:boto:Params: {} 2024-04-13T22:27:50.045 DEBUG:boto:Token: None 2024-04-13T22:27:50.045 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23 2024-04-13T22:27:50.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HGNjL681jiCXc/aDmwy+rXZsuic= 2024-04-13T22:27:50.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HGNjL681jiCXc/aDmwy+rXZsuic=', 'Content-Length': '0'} 2024-04-13T22:27:50.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024481c6763619348-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:27:50.048 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.048 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.048 DEBUG:boto:Method: GET 2024-04-13T22:27:50.048 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:27:50.048 DEBUG:boto:Data: 2024-04-13T22:27:50.048 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.048 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.048 DEBUG:boto:Params: {} 2024-04-13T22:27:50.048 DEBUG:boto:Token: None 2024-04-13T22:27:50.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23 2024-04-13T22:27:50.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:djSE3mkBGM5MUistIxNYGUpTMdQ= 2024-04-13T22:27:50.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:djSE3mkBGM5MUistIxNYGUpTMdQ=', 'Content-Length': '0'} 2024-04-13T22:27:50.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561724ecba975269-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.050 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:27:50.050 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:27:50.050 DEBUG:boto:Method: GET 2024-04-13T22:27:50.051 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:27:50.051 DEBUG:boto:Data: 2024-04-13T22:27:50.051 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.051 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.051 DEBUG:boto:Params: {} 2024-04-13T22:27:50.051 DEBUG:boto:Token: None 2024-04-13T22:27:50.051 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj23 2024-04-13T22:27:50.051 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:djSE3mkBGM5MUistIxNYGUpTMdQ= 2024-04-13T22:27:50.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:djSE3mkBGM5MUistIxNYGUpTMdQ=', 'Content-Length': '0'} 2024-04-13T22:27:50.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b17921e649842f1-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.053 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.053 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.054 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.054 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.054 DEBUG:boto:Method: GET 2024-04-13T22:27:50.054 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.054 DEBUG:boto:Data: 2024-04-13T22:27:50.054 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.054 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.054 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.054 DEBUG:boto:Params: {} 2024-04-13T22:27:50.054 DEBUG:boto:Token: None 2024-04-13T22:27:50.054 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c244347f0e978177-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.057 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.057 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.058 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.058 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.058 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.058 DEBUG:boto:Method: GET 2024-04-13T22:27:50.058 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:27:50.058 DEBUG:boto:Data: 2024-04-13T22:27:50.058 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.058 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.058 DEBUG:boto:Params: {} 2024-04-13T22:27:50.059 DEBUG:boto:Token: None 2024-04-13T22:27:50.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000362c718ae6f3ae0b-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.062 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:27:50.063 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.063 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.063 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.063 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.063 DEBUG:boto:Method: GET 2024-04-13T22:27:50.063 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.063 DEBUG:boto:Data: 2024-04-13T22:27:50.063 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.063 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.064 DEBUG:boto:Params: {} 2024-04-13T22:27:50.064 DEBUG:boto:Token: None 2024-04-13T22:27:50.064 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.064 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA= 2024-04-13T22:27:50.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA=', 'Content-Length': '0'} 2024-04-13T22:27:50.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa608c1e479fef46-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.066 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.066 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.066 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.066 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.067 DEBUG:boto:Method: GET 2024-04-13T22:27:50.067 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.067 DEBUG:boto:Data: 2024-04-13T22:27:50.067 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.067 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.067 DEBUG:boto:Params: {} 2024-04-13T22:27:50.067 DEBUG:boto:Token: None 2024-04-13T22:27:50.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA= 2024-04-13T22:27:50.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA=', 'Content-Length': '0'} 2024-04-13T22:27:50.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a60bcfb00da09d-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.069 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.069 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.069 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.070 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.070 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.070 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.070 DEBUG:boto:Data: 2024-04-13T22:27:50.070 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.070 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.070 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.070 DEBUG:boto:Params: {} 2024-04-13T22:27:50.070 DEBUG:boto:Token: None 2024-04-13T22:27:50.070 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RUko8PkMtnHUeKpy3xIquE/20Aw= 2024-04-13T22:27:50.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RUko8PkMtnHUeKpy3xIquE/20Aw=', 'Content-Length': '0'} 2024-04-13T22:27:50.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3b5aedc6b032d5b-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.072 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.073 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.073 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.073 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.073 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.073 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.073 DEBUG:boto:Data: 2024-04-13T22:27:50.073 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.073 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.073 DEBUG:boto:Params: {} 2024-04-13T22:27:50.073 DEBUG:boto:Token: None 2024-04-13T22:27:50.074 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RUko8PkMtnHUeKpy3xIquE/20Aw= 2024-04-13T22:27:50.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RUko8PkMtnHUeKpy3xIquE/20Aw=', 'Content-Length': '0'} 2024-04-13T22:27:50.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8eda0144db3fe8-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:27:50.076 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.076 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.076 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.076 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.076 DEBUG:boto:Method: GET 2024-04-13T22:27:50.076 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.077 DEBUG:boto:Data: 2024-04-13T22:27:50.077 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.077 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.077 DEBUG:boto:Params: {} 2024-04-13T22:27:50.077 DEBUG:boto:Token: None 2024-04-13T22:27:50.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA= 2024-04-13T22:27:50.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA=', 'Content-Length': '0'} 2024-04-13T22:27:50.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092e7acf18479f489-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.079 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.079 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.079 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.079 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.079 DEBUG:boto:Method: GET 2024-04-13T22:27:50.079 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.080 DEBUG:boto:Data: 2024-04-13T22:27:50.080 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.080 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.080 DEBUG:boto:Params: {} 2024-04-13T22:27:50.080 DEBUG:boto:Token: None 2024-04-13T22:27:50.080 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:27:50.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA= 2024-04-13T22:27:50.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9UNGlZtRxV7M9q1vWKwosZLwKA=', 'Content-Length': '0'} 2024-04-13T22:27:50.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003279c1fc6576ff8f-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.082 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.082 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.082 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.082 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.082 DEBUG:boto:Method: GET 2024-04-13T22:27:50.083 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.083 DEBUG:boto:Data: 2024-04-13T22:27:50.083 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.083 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.083 DEBUG:boto:Params: {} 2024-04-13T22:27:50.083 DEBUG:boto:Token: None 2024-04-13T22:27:50.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7b9911b111e6d3-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.086 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.086 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.086 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.086 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.086 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.086 DEBUG:boto:Method: GET 2024-04-13T22:27:50.087 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.087 DEBUG:boto:Data: 2024-04-13T22:27:50.087 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.087 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.087 DEBUG:boto:Params: {} 2024-04-13T22:27:50.087 DEBUG:boto:Token: None 2024-04-13T22:27:50.087 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.087 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e01829b5247b0a7-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.091 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.091 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.091 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.091 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.091 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:27:50.091 DEBUG:boto:Data: 2024-04-13T22:27:50.091 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.092 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.092 DEBUG:boto:Params: {} 2024-04-13T22:27:50.092 DEBUG:boto:Token: None 2024-04-13T22:27:50.092 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29 2024-04-13T22:27:50.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eVoHVRaThnghw/xd/6XUINsA/Y4= 2024-04-13T22:27:50.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eVoHVRaThnghw/xd/6XUINsA/Y4=', 'Content-Length': '0'} 2024-04-13T22:27:50.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f14e3cbccc4b6a-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.094 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.094 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.094 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.094 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:27:50.094 DEBUG:boto:Data: 2024-04-13T22:27:50.094 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.095 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.095 DEBUG:boto:Params: {} 2024-04-13T22:27:50.095 DEBUG:boto:Token: None 2024-04-13T22:27:50.095 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29 2024-04-13T22:27:50.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eVoHVRaThnghw/xd/6XUINsA/Y4= 2024-04-13T22:27:50.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eVoHVRaThnghw/xd/6XUINsA/Y4=', 'Content-Length': '0'} 2024-04-13T22:27:50.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25599d2da793ab3-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:27:50.097 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.097 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.097 DEBUG:boto:Method: GET 2024-04-13T22:27:50.097 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:27:50.097 DEBUG:boto:Data: 2024-04-13T22:27:50.097 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.097 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.098 DEBUG:boto:Params: {} 2024-04-13T22:27:50.098 DEBUG:boto:Token: None 2024-04-13T22:27:50.098 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29 2024-04-13T22:27:50.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWcmBP8uI+FNxuhlEl8FKJMD/CE= 2024-04-13T22:27:50.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWcmBP8uI+FNxuhlEl8FKJMD/CE=', 'Content-Length': '0'} 2024-04-13T22:27:50.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000767de2502b345e8e-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.100 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:27:50.100 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:27:50.100 DEBUG:boto:Method: GET 2024-04-13T22:27:50.100 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:27:50.100 DEBUG:boto:Data: 2024-04-13T22:27:50.100 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.101 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.101 DEBUG:boto:Params: {} 2024-04-13T22:27:50.101 DEBUG:boto:Token: None 2024-04-13T22:27:50.101 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj29 2024-04-13T22:27:50.101 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gWcmBP8uI+FNxuhlEl8FKJMD/CE= 2024-04-13T22:27:50.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWcmBP8uI+FNxuhlEl8FKJMD/CE=', 'Content-Length': '0'} 2024-04-13T22:27:50.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c624149ebcfe6d0b-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.150 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.150 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.151 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.151 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.151 DEBUG:boto:Method: GET 2024-04-13T22:27:50.151 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.151 DEBUG:boto:Data: 2024-04-13T22:27:50.151 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.151 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.151 DEBUG:boto:Params: {} 2024-04-13T22:27:50.153 DEBUG:boto:Token: None 2024-04-13T22:27:50.153 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.153 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c64c8ef4ff4c0db6-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.155 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.156 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.156 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.156 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.156 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.156 DEBUG:boto:Method: GET 2024-04-13T22:27:50.156 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:27:50.156 DEBUG:boto:Data: 2024-04-13T22:27:50.156 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.156 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.157 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.157 DEBUG:boto:Params: {} 2024-04-13T22:27:50.157 DEBUG:boto:Token: None 2024-04-13T22:27:50.157 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.157 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da22fe26f641533-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.161 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:27:50.161 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.161 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.161 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.161 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.161 DEBUG:boto:Method: GET 2024-04-13T22:27:50.162 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.162 DEBUG:boto:Data: 2024-04-13T22:27:50.162 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.162 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.162 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.162 DEBUG:boto:Params: {} 2024-04-13T22:27:50.162 DEBUG:boto:Token: None 2024-04-13T22:27:50.162 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.162 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro= 2024-04-13T22:27:50.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro=', 'Content-Length': '0'} 2024-04-13T22:27:50.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999875ae6a4cb908-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.206 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.206 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.207 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.207 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.207 DEBUG:boto:Method: GET 2024-04-13T22:27:50.207 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.207 DEBUG:boto:Data: 2024-04-13T22:27:50.207 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.207 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.207 DEBUG:boto:Params: {} 2024-04-13T22:27:50.207 DEBUG:boto:Token: None 2024-04-13T22:27:50.207 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro= 2024-04-13T22:27:50.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro=', 'Content-Length': '0'} 2024-04-13T22:27:50.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f60818c643088b61-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.210 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.210 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.210 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.210 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.210 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.210 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.210 DEBUG:boto:Data: 2024-04-13T22:27:50.210 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.211 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.211 DEBUG:boto:Params: {} 2024-04-13T22:27:50.211 DEBUG:boto:Token: None 2024-04-13T22:27:50.211 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.211 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wyaadiy8g7DjzzHiBB/+eGVIoAY= 2024-04-13T22:27:50.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wyaadiy8g7DjzzHiBB/+eGVIoAY=', 'Content-Length': '0'} 2024-04-13T22:27:50.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f1a00e1cc11edb-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.213 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.213 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.213 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.213 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.213 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.213 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.213 DEBUG:boto:Data: 2024-04-13T22:27:50.214 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.214 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.214 DEBUG:boto:Params: {} 2024-04-13T22:27:50.214 DEBUG:boto:Token: None 2024-04-13T22:27:50.214 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.214 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wyaadiy8g7DjzzHiBB/+eGVIoAY= 2024-04-13T22:27:50.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wyaadiy8g7DjzzHiBB/+eGVIoAY=', 'Content-Length': '0'} 2024-04-13T22:27:50.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a8a4bcf11020ac-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:27:50.216 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.216 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.217 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.217 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.217 DEBUG:boto:Method: GET 2024-04-13T22:27:50.217 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.217 DEBUG:boto:Data: 2024-04-13T22:27:50.217 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.217 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.217 DEBUG:boto:Params: {} 2024-04-13T22:27:50.217 DEBUG:boto:Token: None 2024-04-13T22:27:50.218 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro= 2024-04-13T22:27:50.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro=', 'Content-Length': '0'} 2024-04-13T22:27:50.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93ddfbd75641499-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.220 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.220 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.220 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.220 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.220 DEBUG:boto:Method: GET 2024-04-13T22:27:50.220 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.220 DEBUG:boto:Data: 2024-04-13T22:27:50.220 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.220 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.221 DEBUG:boto:Params: {} 2024-04-13T22:27:50.221 DEBUG:boto:Token: None 2024-04-13T22:27:50.221 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:27:50.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro= 2024-04-13T22:27:50.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ogk5nVdp+i+HNj0w1Yv01boTHro=', 'Content-Length': '0'} 2024-04-13T22:27:50.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a407ea4af730d01f-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.223 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.223 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.223 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.223 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.223 DEBUG:boto:Method: GET 2024-04-13T22:27:50.223 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.223 DEBUG:boto:Data: 2024-04-13T22:27:50.223 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.224 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.224 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.224 DEBUG:boto:Params: {} 2024-04-13T22:27:50.224 DEBUG:boto:Token: None 2024-04-13T22:27:50.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397d34eb30468cd7-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.226 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.227 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.227 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.227 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.227 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.227 DEBUG:boto:Method: GET 2024-04-13T22:27:50.227 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.227 DEBUG:boto:Data: 2024-04-13T22:27:50.227 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.228 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.228 DEBUG:boto:Params: {} 2024-04-13T22:27:50.228 DEBUG:boto:Token: None 2024-04-13T22:27:50.228 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdf6759c076fa32-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.231 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.232 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.232 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.232 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.232 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:27:50.232 DEBUG:boto:Data: 2024-04-13T22:27:50.232 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.232 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.232 DEBUG:boto:Params: {} 2024-04-13T22:27:50.232 DEBUG:boto:Token: None 2024-04-13T22:27:50.233 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33 2024-04-13T22:27:50.233 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oasnFupVuSEOpT8xpRmoxoigboE= 2024-04-13T22:27:50.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oasnFupVuSEOpT8xpRmoxoigboE=', 'Content-Length': '0'} 2024-04-13T22:27:50.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f169f2a67fcf2164-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.235 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.235 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.235 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.235 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:27:50.235 DEBUG:boto:Data: 2024-04-13T22:27:50.235 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.235 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.235 DEBUG:boto:Params: {} 2024-04-13T22:27:50.236 DEBUG:boto:Token: None 2024-04-13T22:27:50.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33 2024-04-13T22:27:50.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oasnFupVuSEOpT8xpRmoxoigboE= 2024-04-13T22:27:50.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oasnFupVuSEOpT8xpRmoxoigboE=', 'Content-Length': '0'} 2024-04-13T22:27:50.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4859c7f2dc0cf1-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:27:50.238 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.238 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.238 DEBUG:boto:Method: GET 2024-04-13T22:27:50.238 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:27:50.238 DEBUG:boto:Data: 2024-04-13T22:27:50.238 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.239 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.239 DEBUG:boto:Params: {} 2024-04-13T22:27:50.239 DEBUG:boto:Token: None 2024-04-13T22:27:50.239 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33 2024-04-13T22:27:50.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6wV/Tk+sYoxwAodbzEUoew/jqI= 2024-04-13T22:27:50.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6wV/Tk+sYoxwAodbzEUoew/jqI=', 'Content-Length': '0'} 2024-04-13T22:27:50.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecad272d851d608e-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.286 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:27:50.286 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:27:50.287 DEBUG:boto:Method: GET 2024-04-13T22:27:50.287 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:27:50.287 DEBUG:boto:Data: 2024-04-13T22:27:50.287 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.287 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.287 DEBUG:boto:Params: {} 2024-04-13T22:27:50.287 DEBUG:boto:Token: None 2024-04-13T22:27:50.287 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj33 2024-04-13T22:27:50.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J6wV/Tk+sYoxwAodbzEUoew/jqI= 2024-04-13T22:27:50.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J6wV/Tk+sYoxwAodbzEUoew/jqI=', 'Content-Length': '0'} 2024-04-13T22:27:50.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052920ada19c60c4e-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.290 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.290 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.290 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.290 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.290 DEBUG:boto:Method: GET 2024-04-13T22:27:50.290 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.290 DEBUG:boto:Data: 2024-04-13T22:27:50.290 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.290 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.290 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.290 DEBUG:boto:Params: {} 2024-04-13T22:27:50.291 DEBUG:boto:Token: None 2024-04-13T22:27:50.291 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.291 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f820299a324a9f-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.293 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.294 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.294 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.294 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.294 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.294 DEBUG:boto:Method: GET 2024-04-13T22:27:50.294 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:27:50.294 DEBUG:boto:Data: 2024-04-13T22:27:50.294 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.294 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.294 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.295 DEBUG:boto:Params: {} 2024-04-13T22:27:50.295 DEBUG:boto:Token: None 2024-04-13T22:27:50.295 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c824719100a964b2-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.298 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:27:50.299 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.299 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.299 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.299 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.299 DEBUG:boto:Method: GET 2024-04-13T22:27:50.299 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.299 DEBUG:boto:Data: 2024-04-13T22:27:50.299 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.300 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.300 DEBUG:boto:Params: {} 2024-04-13T22:27:50.300 DEBUG:boto:Token: None 2024-04-13T22:27:50.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw= 2024-04-13T22:27:50.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw=', 'Content-Length': '0'} 2024-04-13T22:27:50.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fae39e74ffe9590-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.302 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.302 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.302 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.302 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.302 DEBUG:boto:Method: GET 2024-04-13T22:27:50.302 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.303 DEBUG:boto:Data: 2024-04-13T22:27:50.303 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.303 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.303 DEBUG:boto:Params: {} 2024-04-13T22:27:50.303 DEBUG:boto:Token: None 2024-04-13T22:27:50.303 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.303 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw= 2024-04-13T22:27:50.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw=', 'Content-Length': '0'} 2024-04-13T22:27:50.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7480006130112a-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.305 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.305 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.305 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.306 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.306 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.306 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.306 DEBUG:boto:Data: 2024-04-13T22:27:50.306 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.306 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.306 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.306 DEBUG:boto:Params: {} 2024-04-13T22:27:50.306 DEBUG:boto:Token: None 2024-04-13T22:27:50.306 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4eyV7VOTpanJ9n0cZxalnBCy8cc= 2024-04-13T22:27:50.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4eyV7VOTpanJ9n0cZxalnBCy8cc=', 'Content-Length': '0'} 2024-04-13T22:27:50.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f91f0d80e8a19a-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.308 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.308 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.309 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.309 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.309 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.309 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.309 DEBUG:boto:Data: 2024-04-13T22:27:50.309 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.309 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.309 DEBUG:boto:Params: {} 2024-04-13T22:27:50.309 DEBUG:boto:Token: None 2024-04-13T22:27:50.310 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4eyV7VOTpanJ9n0cZxalnBCy8cc= 2024-04-13T22:27:50.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4eyV7VOTpanJ9n0cZxalnBCy8cc=', 'Content-Length': '0'} 2024-04-13T22:27:50.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec21253b37cb6ab-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:27:50.312 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.312 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.312 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.312 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.312 DEBUG:boto:Method: GET 2024-04-13T22:27:50.312 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.313 DEBUG:boto:Data: 2024-04-13T22:27:50.313 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.313 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.313 DEBUG:boto:Params: {} 2024-04-13T22:27:50.313 DEBUG:boto:Token: None 2024-04-13T22:27:50.313 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.313 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw= 2024-04-13T22:27:50.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw=', 'Content-Length': '0'} 2024-04-13T22:27:50.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad4f4ce5ad62504-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.315 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.315 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.315 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.315 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.315 DEBUG:boto:Method: GET 2024-04-13T22:27:50.316 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.316 DEBUG:boto:Data: 2024-04-13T22:27:50.316 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.316 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.316 DEBUG:boto:Params: {} 2024-04-13T22:27:50.316 DEBUG:boto:Token: None 2024-04-13T22:27:50.316 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:27:50.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw= 2024-04-13T22:27:50.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H5PofS1Jz4P03WGkgD2LTGFR/iw=', 'Content-Length': '0'} 2024-04-13T22:27:50.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19c9834992a33c2-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.318 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.318 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.318 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.318 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.319 DEBUG:boto:Method: GET 2024-04-13T22:27:50.319 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.319 DEBUG:boto:Data: 2024-04-13T22:27:50.319 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.319 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.319 DEBUG:boto:Params: {} 2024-04-13T22:27:50.319 DEBUG:boto:Token: None 2024-04-13T22:27:50.319 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764e551c4833f155-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.322 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.322 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.322 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.322 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.322 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.322 DEBUG:boto:Method: GET 2024-04-13T22:27:50.322 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.323 DEBUG:boto:Data: 2024-04-13T22:27:50.323 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.323 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.323 DEBUG:boto:Params: {} 2024-04-13T22:27:50.323 DEBUG:boto:Token: None 2024-04-13T22:27:50.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.323 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c447ddc785d3b2-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.326 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.327 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.327 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.327 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.327 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:27:50.327 DEBUG:boto:Data: 2024-04-13T22:27:50.327 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.327 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.327 DEBUG:boto:Params: {} 2024-04-13T22:27:50.328 DEBUG:boto:Token: None 2024-04-13T22:27:50.328 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34 2024-04-13T22:27:50.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VVP0e8hC+0dbqRxVITOj8tO8ocA= 2024-04-13T22:27:50.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VVP0e8hC+0dbqRxVITOj8tO8ocA=', 'Content-Length': '0'} 2024-04-13T22:27:50.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090440a4004ec1605-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.330 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.330 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.330 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.330 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:27:50.330 DEBUG:boto:Data: 2024-04-13T22:27:50.330 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.331 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.331 DEBUG:boto:Params: {} 2024-04-13T22:27:50.331 DEBUG:boto:Token: None 2024-04-13T22:27:50.331 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34 2024-04-13T22:27:50.331 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VVP0e8hC+0dbqRxVITOj8tO8ocA= 2024-04-13T22:27:50.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VVP0e8hC+0dbqRxVITOj8tO8ocA=', 'Content-Length': '0'} 2024-04-13T22:27:50.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a08d7b9ed282c9-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:27:50.333 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.333 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.333 DEBUG:boto:Method: GET 2024-04-13T22:27:50.333 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:27:50.333 DEBUG:boto:Data: 2024-04-13T22:27:50.333 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.333 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.333 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.334 DEBUG:boto:Params: {} 2024-04-13T22:27:50.334 DEBUG:boto:Token: None 2024-04-13T22:27:50.334 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34 2024-04-13T22:27:50.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RE0hJap7MqaS5VeUPTxyaFtJkBk= 2024-04-13T22:27:50.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RE0hJap7MqaS5VeUPTxyaFtJkBk=', 'Content-Length': '0'} 2024-04-13T22:27:50.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058fe18cbc024f0d9-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.336 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:27:50.336 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:27:50.336 DEBUG:boto:Method: GET 2024-04-13T22:27:50.336 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:27:50.336 DEBUG:boto:Data: 2024-04-13T22:27:50.336 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.336 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.336 DEBUG:boto:Params: {} 2024-04-13T22:27:50.337 DEBUG:boto:Token: None 2024-04-13T22:27:50.337 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj34 2024-04-13T22:27:50.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RE0hJap7MqaS5VeUPTxyaFtJkBk= 2024-04-13T22:27:50.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RE0hJap7MqaS5VeUPTxyaFtJkBk=', 'Content-Length': '0'} 2024-04-13T22:27:50.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d26796c1d5a9ca-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.339 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.339 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.339 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.339 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.339 DEBUG:boto:Method: GET 2024-04-13T22:27:50.339 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.339 DEBUG:boto:Data: 2024-04-13T22:27:50.339 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.339 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.340 DEBUG:boto:Params: {} 2024-04-13T22:27:50.340 DEBUG:boto:Token: None 2024-04-13T22:27:50.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008899b62424a56a10-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.342 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.343 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.343 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.343 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.343 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.343 DEBUG:boto:Method: GET 2024-04-13T22:27:50.343 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:27:50.343 DEBUG:boto:Data: 2024-04-13T22:27:50.344 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.344 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.344 DEBUG:boto:Params: {} 2024-04-13T22:27:50.344 DEBUG:boto:Token: None 2024-04-13T22:27:50.344 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.344 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ed89fb8645a482e-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.347 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:27:50.348 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.348 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.348 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.348 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.348 DEBUG:boto:Method: GET 2024-04-13T22:27:50.348 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.348 DEBUG:boto:Data: 2024-04-13T22:27:50.348 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.348 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.349 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.349 DEBUG:boto:Params: {} 2024-04-13T22:27:50.349 DEBUG:boto:Token: None 2024-04-13T22:27:50.349 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.349 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo= 2024-04-13T22:27:50.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo=', 'Content-Length': '0'} 2024-04-13T22:27:50.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a7911297acad45-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.351 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.351 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.351 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.351 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.351 DEBUG:boto:Method: GET 2024-04-13T22:27:50.351 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.352 DEBUG:boto:Data: 2024-04-13T22:27:50.352 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.352 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.352 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.352 DEBUG:boto:Params: {} 2024-04-13T22:27:50.352 DEBUG:boto:Token: None 2024-04-13T22:27:50.352 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.352 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo= 2024-04-13T22:27:50.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo=', 'Content-Length': '0'} 2024-04-13T22:27:50.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc8b36265d2a484-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.394 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.394 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.395 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.395 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.395 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.395 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.395 DEBUG:boto:Data: 2024-04-13T22:27:50.395 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.395 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.395 DEBUG:boto:Params: {} 2024-04-13T22:27:50.395 DEBUG:boto:Token: None 2024-04-13T22:27:50.396 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qi4EZCkv5/3mqHsQiqv4JEC/Q4U= 2024-04-13T22:27:50.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qi4EZCkv5/3mqHsQiqv4JEC/Q4U=', 'Content-Length': '0'} 2024-04-13T22:27:50.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086850e1d78b1828d-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.398 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.398 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.398 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.398 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.398 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.398 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.398 DEBUG:boto:Data: 2024-04-13T22:27:50.398 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.398 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.399 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.399 DEBUG:boto:Params: {} 2024-04-13T22:27:50.399 DEBUG:boto:Token: None 2024-04-13T22:27:50.399 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.399 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qi4EZCkv5/3mqHsQiqv4JEC/Q4U= 2024-04-13T22:27:50.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qi4EZCkv5/3mqHsQiqv4JEC/Q4U=', 'Content-Length': '0'} 2024-04-13T22:27:50.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6c7719a44cb5cc6-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:27:50.401 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.401 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.402 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.402 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.402 DEBUG:boto:Method: GET 2024-04-13T22:27:50.402 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.402 DEBUG:boto:Data: 2024-04-13T22:27:50.402 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.402 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.402 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.402 DEBUG:boto:Params: {} 2024-04-13T22:27:50.402 DEBUG:boto:Token: None 2024-04-13T22:27:50.403 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.403 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo= 2024-04-13T22:27:50.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo=', 'Content-Length': '0'} 2024-04-13T22:27:50.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95f7db972f4160c-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.404 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.405 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.405 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.405 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.405 DEBUG:boto:Method: GET 2024-04-13T22:27:50.405 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.405 DEBUG:boto:Data: 2024-04-13T22:27:50.405 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.405 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.405 DEBUG:boto:Params: {} 2024-04-13T22:27:50.406 DEBUG:boto:Token: None 2024-04-13T22:27:50.406 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:27:50.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo= 2024-04-13T22:27:50.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pVh8Z3z0LqZAbsPxZVHqndccGqo=', 'Content-Length': '0'} 2024-04-13T22:27:50.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1fa9c909092ed33-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.408 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.408 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.408 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.408 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.408 DEBUG:boto:Method: GET 2024-04-13T22:27:50.408 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.408 DEBUG:boto:Data: 2024-04-13T22:27:50.408 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.409 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.409 DEBUG:boto:Params: {} 2024-04-13T22:27:50.409 DEBUG:boto:Token: None 2024-04-13T22:27:50.409 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.409 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a1996b943bc41d-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.411 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.412 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.412 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.412 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.412 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.412 DEBUG:boto:Method: GET 2024-04-13T22:27:50.412 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.412 DEBUG:boto:Data: 2024-04-13T22:27:50.412 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.412 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.413 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.413 DEBUG:boto:Params: {} 2024-04-13T22:27:50.413 DEBUG:boto:Token: None 2024-04-13T22:27:50.413 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c303c5194d12d55-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.417 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.417 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.417 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.417 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.417 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:27:50.417 DEBUG:boto:Data: 2024-04-13T22:27:50.417 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.417 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.418 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.418 DEBUG:boto:Params: {} 2024-04-13T22:27:50.418 DEBUG:boto:Token: None 2024-04-13T22:27:50.418 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38 2024-04-13T22:27:50.418 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGt96fQFo/Df4yjI8uA5NvghIS0= 2024-04-13T22:27:50.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGt96fQFo/Df4yjI8uA5NvghIS0=', 'Content-Length': '0'} 2024-04-13T22:27:50.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014c8a2542d1080e5-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.420 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.420 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.420 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.420 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:27:50.420 DEBUG:boto:Data: 2024-04-13T22:27:50.420 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.420 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.420 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.420 DEBUG:boto:Params: {} 2024-04-13T22:27:50.421 DEBUG:boto:Token: None 2024-04-13T22:27:50.421 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38 2024-04-13T22:27:50.421 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGt96fQFo/Df4yjI8uA5NvghIS0= 2024-04-13T22:27:50.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGt96fQFo/Df4yjI8uA5NvghIS0=', 'Content-Length': '0'} 2024-04-13T22:27:50.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc667f6f88e239f-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:27:50.423 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.423 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.423 DEBUG:boto:Method: GET 2024-04-13T22:27:50.423 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:27:50.423 DEBUG:boto:Data: 2024-04-13T22:27:50.423 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.423 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.423 DEBUG:boto:Params: {} 2024-04-13T22:27:50.423 DEBUG:boto:Token: None 2024-04-13T22:27:50.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38 2024-04-13T22:27:50.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BGGMidQK70Yqtr4s6e4TvoWHHrQ= 2024-04-13T22:27:50.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BGGMidQK70Yqtr4s6e4TvoWHHrQ=', 'Content-Length': '0'} 2024-04-13T22:27:50.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3dc42c8340df6d-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.466 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:27:50.466 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:27:50.466 DEBUG:boto:Method: GET 2024-04-13T22:27:50.467 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:27:50.467 DEBUG:boto:Data: 2024-04-13T22:27:50.467 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.467 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.467 DEBUG:boto:Params: {} 2024-04-13T22:27:50.467 DEBUG:boto:Token: None 2024-04-13T22:27:50.467 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj38 2024-04-13T22:27:50.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BGGMidQK70Yqtr4s6e4TvoWHHrQ= 2024-04-13T22:27:50.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BGGMidQK70Yqtr4s6e4TvoWHHrQ=', 'Content-Length': '0'} 2024-04-13T22:27:50.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005031f1333cde2a98-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.469 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.469 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.469 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.469 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.470 DEBUG:boto:Method: GET 2024-04-13T22:27:50.470 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.470 DEBUG:boto:Data: 2024-04-13T22:27:50.470 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.470 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.470 DEBUG:boto:Params: {} 2024-04-13T22:27:50.470 DEBUG:boto:Token: None 2024-04-13T22:27:50.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a24ea3a729c017-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.473 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.473 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.473 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.473 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.473 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.473 DEBUG:boto:Method: GET 2024-04-13T22:27:50.474 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:27:50.474 DEBUG:boto:Data: 2024-04-13T22:27:50.474 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.474 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.474 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.474 DEBUG:boto:Params: {} 2024-04-13T22:27:50.474 DEBUG:boto:Token: None 2024-04-13T22:27:50.474 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.474 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78ff5ccba68c454-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.478 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:27:50.478 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.478 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.478 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.478 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.478 DEBUG:boto:Method: GET 2024-04-13T22:27:50.478 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.479 DEBUG:boto:Data: 2024-04-13T22:27:50.479 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.479 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.479 DEBUG:boto:Params: {} 2024-04-13T22:27:50.479 DEBUG:boto:Token: None 2024-04-13T22:27:50.479 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4= 2024-04-13T22:27:50.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4=', 'Content-Length': '0'} 2024-04-13T22:27:50.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3a74b830a3fec4-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.481 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.481 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.481 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.481 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.482 DEBUG:boto:Method: GET 2024-04-13T22:27:50.482 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.482 DEBUG:boto:Data: 2024-04-13T22:27:50.482 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.482 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.482 DEBUG:boto:Params: {} 2024-04-13T22:27:50.482 DEBUG:boto:Token: None 2024-04-13T22:27:50.482 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4= 2024-04-13T22:27:50.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4=', 'Content-Length': '0'} 2024-04-13T22:27:50.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026d3c9b2996f5038-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.484 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.484 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.485 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.485 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.485 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.485 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.485 DEBUG:boto:Data: 2024-04-13T22:27:50.485 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.485 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.485 DEBUG:boto:Params: {} 2024-04-13T22:27:50.485 DEBUG:boto:Token: None 2024-04-13T22:27:50.485 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BeouZrs8IudFKX16H5zUU12H6aU= 2024-04-13T22:27:50.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BeouZrs8IudFKX16H5zUU12H6aU=', 'Content-Length': '0'} 2024-04-13T22:27:50.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cf9d5fab8f19f7-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.487 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.487 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.488 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.488 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.488 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.488 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.488 DEBUG:boto:Data: 2024-04-13T22:27:50.488 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.488 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.488 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.488 DEBUG:boto:Params: {} 2024-04-13T22:27:50.488 DEBUG:boto:Token: None 2024-04-13T22:27:50.489 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.489 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BeouZrs8IudFKX16H5zUU12H6aU= 2024-04-13T22:27:50.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BeouZrs8IudFKX16H5zUU12H6aU=', 'Content-Length': '0'} 2024-04-13T22:27:50.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f52107dec5bd2e-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:27:50.491 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.491 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.491 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.491 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.491 DEBUG:boto:Method: GET 2024-04-13T22:27:50.491 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.492 DEBUG:boto:Data: 2024-04-13T22:27:50.492 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.492 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.492 DEBUG:boto:Params: {} 2024-04-13T22:27:50.492 DEBUG:boto:Token: None 2024-04-13T22:27:50.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4= 2024-04-13T22:27:50.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4=', 'Content-Length': '0'} 2024-04-13T22:27:50.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c94c3925b14a144e-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.494 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.494 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.494 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.494 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.495 DEBUG:boto:Method: GET 2024-04-13T22:27:50.495 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.495 DEBUG:boto:Data: 2024-04-13T22:27:50.495 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.495 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.495 DEBUG:boto:Params: {} 2024-04-13T22:27:50.495 DEBUG:boto:Token: None 2024-04-13T22:27:50.495 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:27:50.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4= 2024-04-13T22:27:50.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dEdzni2mRC8fSyIliygiSqeacD4=', 'Content-Length': '0'} 2024-04-13T22:27:50.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf67c7576f0db44-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.543 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.543 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.543 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.543 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.543 DEBUG:boto:Method: GET 2024-04-13T22:27:50.543 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.543 DEBUG:boto:Data: 2024-04-13T22:27:50.543 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.544 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.544 DEBUG:boto:Params: {} 2024-04-13T22:27:50.544 DEBUG:boto:Token: None 2024-04-13T22:27:50.544 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000853a0d2b378723a0-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.547 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.547 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.547 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.547 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.547 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.547 DEBUG:boto:Method: GET 2024-04-13T22:27:50.547 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.547 DEBUG:boto:Data: 2024-04-13T22:27:50.548 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.548 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.548 DEBUG:boto:Params: {} 2024-04-13T22:27:50.548 DEBUG:boto:Token: None 2024-04-13T22:27:50.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100bb653a1afc0be-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.552 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.552 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.552 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.553 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.553 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:27:50.553 DEBUG:boto:Data: 2024-04-13T22:27:50.553 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.553 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.553 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.553 DEBUG:boto:Params: {} 2024-04-13T22:27:50.553 DEBUG:boto:Token: None 2024-04-13T22:27:50.553 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44 2024-04-13T22:27:50.553 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+6Q1ySi65TXOp7Fdo8rLBHqNEDc= 2024-04-13T22:27:50.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+6Q1ySi65TXOp7Fdo8rLBHqNEDc=', 'Content-Length': '0'} 2024-04-13T22:27:50.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008816d3088498b117-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.555 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.555 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.555 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.555 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:27:50.556 DEBUG:boto:Data: 2024-04-13T22:27:50.556 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.556 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.556 DEBUG:boto:Params: {} 2024-04-13T22:27:50.556 DEBUG:boto:Token: None 2024-04-13T22:27:50.556 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44 2024-04-13T22:27:50.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+6Q1ySi65TXOp7Fdo8rLBHqNEDc= 2024-04-13T22:27:50.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+6Q1ySi65TXOp7Fdo8rLBHqNEDc=', 'Content-Length': '0'} 2024-04-13T22:27:50.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f6cb55b165e773-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:27:50.558 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.558 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.559 DEBUG:boto:Method: GET 2024-04-13T22:27:50.559 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:27:50.559 DEBUG:boto:Data: 2024-04-13T22:27:50.559 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.559 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.559 DEBUG:boto:Params: {} 2024-04-13T22:27:50.559 DEBUG:boto:Token: None 2024-04-13T22:27:50.559 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44 2024-04-13T22:27:50.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ts1kxhyT3gI+ENn16Xc7EQRfMI0= 2024-04-13T22:27:50.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ts1kxhyT3gI+ENn16Xc7EQRfMI0=', 'Content-Length': '0'} 2024-04-13T22:27:50.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6bb5377ee12bbd-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.602 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:27:50.602 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:27:50.603 DEBUG:boto:Method: GET 2024-04-13T22:27:50.603 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:27:50.603 DEBUG:boto:Data: 2024-04-13T22:27:50.603 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.603 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.603 DEBUG:boto:Params: {} 2024-04-13T22:27:50.603 DEBUG:boto:Token: None 2024-04-13T22:27:50.603 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj44 2024-04-13T22:27:50.603 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ts1kxhyT3gI+ENn16Xc7EQRfMI0= 2024-04-13T22:27:50.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ts1kxhyT3gI+ENn16Xc7EQRfMI0=', 'Content-Length': '0'} 2024-04-13T22:27:50.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078753587cc47d51d-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.606 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.606 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.606 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.606 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.606 DEBUG:boto:Method: GET 2024-04-13T22:27:50.606 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.606 DEBUG:boto:Data: 2024-04-13T22:27:50.606 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.606 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.606 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.606 DEBUG:boto:Params: {} 2024-04-13T22:27:50.607 DEBUG:boto:Token: None 2024-04-13T22:27:50.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad0874575831b86-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.609 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.609 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.609 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.610 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.610 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.610 DEBUG:boto:Method: GET 2024-04-13T22:27:50.610 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:27:50.610 DEBUG:boto:Data: 2024-04-13T22:27:50.610 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.610 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.610 DEBUG:boto:Params: {} 2024-04-13T22:27:50.610 DEBUG:boto:Token: None 2024-04-13T22:27:50.611 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d42e192cec525cf-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.614 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:27:50.615 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.615 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.615 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.615 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.615 DEBUG:boto:Method: GET 2024-04-13T22:27:50.615 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.615 DEBUG:boto:Data: 2024-04-13T22:27:50.615 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.615 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.615 DEBUG:boto:Params: {} 2024-04-13T22:27:50.616 DEBUG:boto:Token: None 2024-04-13T22:27:50.616 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE= 2024-04-13T22:27:50.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE=', 'Content-Length': '0'} 2024-04-13T22:27:50.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000657bc6d4ac87e743-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.618 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.618 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.618 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.618 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.618 DEBUG:boto:Method: GET 2024-04-13T22:27:50.618 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.619 DEBUG:boto:Data: 2024-04-13T22:27:50.619 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.619 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.619 DEBUG:boto:Params: {} 2024-04-13T22:27:50.619 DEBUG:boto:Token: None 2024-04-13T22:27:50.619 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE= 2024-04-13T22:27:50.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE=', 'Content-Length': '0'} 2024-04-13T22:27:50.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edffeb9cc18ef6ff-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.621 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.621 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.621 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.621 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.621 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.621 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.622 DEBUG:boto:Data: 2024-04-13T22:27:50.622 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.622 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.622 DEBUG:boto:Params: {} 2024-04-13T22:27:50.622 DEBUG:boto:Token: None 2024-04-13T22:27:50.622 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NEokk0xyoJodPp9XhoNfn/Awy48= 2024-04-13T22:27:50.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NEokk0xyoJodPp9XhoNfn/Awy48=', 'Content-Length': '0'} 2024-04-13T22:27:50.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e35636b0d512d66-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.624 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.624 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.624 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.624 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.624 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.625 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.625 DEBUG:boto:Data: 2024-04-13T22:27:50.625 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.625 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.625 DEBUG:boto:Params: {} 2024-04-13T22:27:50.625 DEBUG:boto:Token: None 2024-04-13T22:27:50.625 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NEokk0xyoJodPp9XhoNfn/Awy48= 2024-04-13T22:27:50.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NEokk0xyoJodPp9XhoNfn/Awy48=', 'Content-Length': '0'} 2024-04-13T22:27:50.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acde8bff59faf4dd-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:27:50.628 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.628 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.628 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.628 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.628 DEBUG:boto:Method: GET 2024-04-13T22:27:50.628 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.628 DEBUG:boto:Data: 2024-04-13T22:27:50.628 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.628 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.628 DEBUG:boto:Params: {} 2024-04-13T22:27:50.629 DEBUG:boto:Token: None 2024-04-13T22:27:50.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE= 2024-04-13T22:27:50.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE=', 'Content-Length': '0'} 2024-04-13T22:27:50.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019f82b517feaaa2-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.630 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.631 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.631 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.631 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.631 DEBUG:boto:Method: GET 2024-04-13T22:27:50.631 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.631 DEBUG:boto:Data: 2024-04-13T22:27:50.631 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.631 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.632 DEBUG:boto:Params: {} 2024-04-13T22:27:50.632 DEBUG:boto:Token: None 2024-04-13T22:27:50.632 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:27:50.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE= 2024-04-13T22:27:50.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nbQyJT2nlDf5gM84wNUNjWEbHTE=', 'Content-Length': '0'} 2024-04-13T22:27:50.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95482ba3156b93c-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.675 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.675 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.675 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.675 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.675 DEBUG:boto:Method: GET 2024-04-13T22:27:50.675 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.675 DEBUG:boto:Data: 2024-04-13T22:27:50.675 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.675 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.676 DEBUG:boto:Params: {} 2024-04-13T22:27:50.676 DEBUG:boto:Token: None 2024-04-13T22:27:50.676 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c23931ab6b9719-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.678 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.678 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.679 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.679 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.679 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.679 DEBUG:boto:Method: GET 2024-04-13T22:27:50.679 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.679 DEBUG:boto:Data: 2024-04-13T22:27:50.679 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.679 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.679 DEBUG:boto:Params: {} 2024-04-13T22:27:50.679 DEBUG:boto:Token: None 2024-04-13T22:27:50.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030ef2eeb0a26866b-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.683 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.684 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.684 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.684 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.684 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:27:50.684 DEBUG:boto:Data: 2024-04-13T22:27:50.684 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.684 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.684 DEBUG:boto:Params: {} 2024-04-13T22:27:50.684 DEBUG:boto:Token: None 2024-04-13T22:27:50.685 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46 2024-04-13T22:27:50.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Le5APilcpCMOkIuJgBB4JfoWJSQ= 2024-04-13T22:27:50.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Le5APilcpCMOkIuJgBB4JfoWJSQ=', 'Content-Length': '0'} 2024-04-13T22:27:50.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f0f9f18f9099a1-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.687 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.687 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.687 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.687 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:27:50.687 DEBUG:boto:Data: 2024-04-13T22:27:50.687 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.687 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.688 DEBUG:boto:Params: {} 2024-04-13T22:27:50.688 DEBUG:boto:Token: None 2024-04-13T22:27:50.688 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46 2024-04-13T22:27:50.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Le5APilcpCMOkIuJgBB4JfoWJSQ= 2024-04-13T22:27:50.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Le5APilcpCMOkIuJgBB4JfoWJSQ=', 'Content-Length': '0'} 2024-04-13T22:27:50.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887c7d8fa83f1185-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:27:50.690 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.690 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.690 DEBUG:boto:Method: GET 2024-04-13T22:27:50.690 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:27:50.690 DEBUG:boto:Data: 2024-04-13T22:27:50.690 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.690 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.690 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.691 DEBUG:boto:Params: {} 2024-04-13T22:27:50.691 DEBUG:boto:Token: None 2024-04-13T22:27:50.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46 2024-04-13T22:27:50.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dQDnMOnZlCeoqGsW5McCFG/UIE= 2024-04-13T22:27:50.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dQDnMOnZlCeoqGsW5McCFG/UIE=', 'Content-Length': '0'} 2024-04-13T22:27:50.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e960827fd8f803-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.693 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:27:50.693 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:27:50.693 DEBUG:boto:Method: GET 2024-04-13T22:27:50.694 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:27:50.694 DEBUG:boto:Data: 2024-04-13T22:27:50.694 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.694 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.694 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.694 DEBUG:boto:Params: {} 2024-04-13T22:27:50.694 DEBUG:boto:Token: None 2024-04-13T22:27:50.694 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj46 2024-04-13T22:27:50.694 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dQDnMOnZlCeoqGsW5McCFG/UIE= 2024-04-13T22:27:50.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dQDnMOnZlCeoqGsW5McCFG/UIE=', 'Content-Length': '0'} 2024-04-13T22:27:50.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000593872a1acfd782-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.696 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.697 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.697 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.697 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.697 DEBUG:boto:Method: GET 2024-04-13T22:27:50.697 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.697 DEBUG:boto:Data: 2024-04-13T22:27:50.697 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.697 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.697 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.697 DEBUG:boto:Params: {} 2024-04-13T22:27:50.697 DEBUG:boto:Token: None 2024-04-13T22:27:50.698 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.698 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000939e9f05d73c6cb2-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.700 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.700 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.700 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.700 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.701 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.701 DEBUG:boto:Method: GET 2024-04-13T22:27:50.701 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:27:50.701 DEBUG:boto:Data: 2024-04-13T22:27:50.701 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.701 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.701 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.701 DEBUG:boto:Params: {} 2024-04-13T22:27:50.701 DEBUG:boto:Token: None 2024-04-13T22:27:50.701 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016fea12ef215fdd2-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.705 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:27:50.705 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.705 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.705 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.705 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.705 DEBUG:boto:Method: GET 2024-04-13T22:27:50.706 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.706 DEBUG:boto:Data: 2024-04-13T22:27:50.706 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.706 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.706 DEBUG:boto:Params: {} 2024-04-13T22:27:50.706 DEBUG:boto:Token: None 2024-04-13T22:27:50.706 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo= 2024-04-13T22:27:50.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo=', 'Content-Length': '0'} 2024-04-13T22:27:50.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8a2ed4b3e8904e7-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.708 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.708 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.708 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.708 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.709 DEBUG:boto:Method: GET 2024-04-13T22:27:50.709 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.709 DEBUG:boto:Data: 2024-04-13T22:27:50.709 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.709 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.709 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.709 DEBUG:boto:Params: {} 2024-04-13T22:27:50.709 DEBUG:boto:Token: None 2024-04-13T22:27:50.709 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.710 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo= 2024-04-13T22:27:50.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo=', 'Content-Length': '0'} 2024-04-13T22:27:50.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cbf953818e965a-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.711 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.711 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.711 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.711 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.712 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.712 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.712 DEBUG:boto:Data: 2024-04-13T22:27:50.712 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.712 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.712 DEBUG:boto:Params: {} 2024-04-13T22:27:50.712 DEBUG:boto:Token: None 2024-04-13T22:27:50.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.713 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w+FxXhYtJ25nzgC/0MIkBX0WR+Q= 2024-04-13T22:27:50.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w+FxXhYtJ25nzgC/0MIkBX0WR+Q=', 'Content-Length': '0'} 2024-04-13T22:27:50.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d538ea9dc28ecee-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.715 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.715 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.715 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.715 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.715 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.715 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.715 DEBUG:boto:Data: 2024-04-13T22:27:50.715 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.715 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.715 DEBUG:boto:Params: {} 2024-04-13T22:27:50.716 DEBUG:boto:Token: None 2024-04-13T22:27:50.716 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w+FxXhYtJ25nzgC/0MIkBX0WR+Q= 2024-04-13T22:27:50.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w+FxXhYtJ25nzgC/0MIkBX0WR+Q=', 'Content-Length': '0'} 2024-04-13T22:27:50.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1f118a90f6fbc4-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:27:50.718 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.718 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.718 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.719 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.719 DEBUG:boto:Method: GET 2024-04-13T22:27:50.719 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.719 DEBUG:boto:Data: 2024-04-13T22:27:50.719 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.719 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.719 DEBUG:boto:Params: {} 2024-04-13T22:27:50.719 DEBUG:boto:Token: None 2024-04-13T22:27:50.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo= 2024-04-13T22:27:50.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo=', 'Content-Length': '0'} 2024-04-13T22:27:50.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006741e87293017ac1-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.766 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.766 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.766 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.767 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.767 DEBUG:boto:Method: GET 2024-04-13T22:27:50.767 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.767 DEBUG:boto:Data: 2024-04-13T22:27:50.767 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.767 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.767 DEBUG:boto:Params: {} 2024-04-13T22:27:50.767 DEBUG:boto:Token: None 2024-04-13T22:27:50.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:27:50.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo= 2024-04-13T22:27:50.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YCevscmkYEKXTTmdvKObZAY0Cbo=', 'Content-Length': '0'} 2024-04-13T22:27:50.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f78b2fe3c7d0f88-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.770 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.770 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.770 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.770 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.770 DEBUG:boto:Method: GET 2024-04-13T22:27:50.771 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.771 DEBUG:boto:Data: 2024-04-13T22:27:50.771 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.773 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.773 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.773 DEBUG:boto:Params: {} 2024-04-13T22:27:50.773 DEBUG:boto:Token: None 2024-04-13T22:27:50.773 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.773 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b5007fd393a5ba-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.776 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.776 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.776 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.776 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.776 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.776 DEBUG:boto:Method: GET 2024-04-13T22:27:50.777 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.777 DEBUG:boto:Data: 2024-04-13T22:27:50.777 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.777 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.777 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.777 DEBUG:boto:Params: {} 2024-04-13T22:27:50.777 DEBUG:boto:Token: None 2024-04-13T22:27:50.777 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.777 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5faf09ce3ad60c5-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.781 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.781 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.781 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.781 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.781 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:27:50.782 DEBUG:boto:Data: 2024-04-13T22:27:50.782 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.782 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.782 DEBUG:boto:Params: {} 2024-04-13T22:27:50.782 DEBUG:boto:Token: None 2024-04-13T22:27:50.782 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51 2024-04-13T22:27:50.782 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RaK2Fx/LxD3oZOP0QBXD+LAJIjg= 2024-04-13T22:27:50.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RaK2Fx/LxD3oZOP0QBXD+LAJIjg=', 'Content-Length': '0'} 2024-04-13T22:27:50.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092430ac395e58894-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.784 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.784 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.784 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.784 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:27:50.785 DEBUG:boto:Data: 2024-04-13T22:27:50.785 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.785 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.785 DEBUG:boto:Params: {} 2024-04-13T22:27:50.785 DEBUG:boto:Token: None 2024-04-13T22:27:50.785 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51 2024-04-13T22:27:50.785 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RaK2Fx/LxD3oZOP0QBXD+LAJIjg= 2024-04-13T22:27:50.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RaK2Fx/LxD3oZOP0QBXD+LAJIjg=', 'Content-Length': '0'} 2024-04-13T22:27:50.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005694eee73d5a6f-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:27:50.787 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.787 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.787 DEBUG:boto:Method: GET 2024-04-13T22:27:50.788 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:27:50.788 DEBUG:boto:Data: 2024-04-13T22:27:50.788 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.788 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.788 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.788 DEBUG:boto:Params: {} 2024-04-13T22:27:50.788 DEBUG:boto:Token: None 2024-04-13T22:27:50.788 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51 2024-04-13T22:27:50.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/8PV83z+n6fgVOaar9CBwu9lWGo= 2024-04-13T22:27:50.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/8PV83z+n6fgVOaar9CBwu9lWGo=', 'Content-Length': '0'} 2024-04-13T22:27:50.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5cc268e84c3ec2-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.790 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:27:50.790 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:27:50.790 DEBUG:boto:Method: GET 2024-04-13T22:27:50.790 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:27:50.790 DEBUG:boto:Data: 2024-04-13T22:27:50.791 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.791 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.791 DEBUG:boto:Params: {} 2024-04-13T22:27:50.791 DEBUG:boto:Token: None 2024-04-13T22:27:50.791 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj51 2024-04-13T22:27:50.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/8PV83z+n6fgVOaar9CBwu9lWGo= 2024-04-13T22:27:50.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/8PV83z+n6fgVOaar9CBwu9lWGo=', 'Content-Length': '0'} 2024-04-13T22:27:50.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdec7eb2eb3508be-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.793 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.793 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.793 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.793 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.793 DEBUG:boto:Method: GET 2024-04-13T22:27:50.793 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.793 DEBUG:boto:Data: 2024-04-13T22:27:50.794 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.794 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.794 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.794 DEBUG:boto:Params: {} 2024-04-13T22:27:50.794 DEBUG:boto:Token: None 2024-04-13T22:27:50.794 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.794 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d2e7ea87a762a3-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.796 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.797 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.797 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.797 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.797 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.797 DEBUG:boto:Method: GET 2024-04-13T22:27:50.797 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:27:50.797 DEBUG:boto:Data: 2024-04-13T22:27:50.798 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.798 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.798 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.798 DEBUG:boto:Params: {} 2024-04-13T22:27:50.798 DEBUG:boto:Token: None 2024-04-13T22:27:50.798 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004670174a3852e83c-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.802 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:27:50.802 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.802 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.802 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.803 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.803 DEBUG:boto:Method: GET 2024-04-13T22:27:50.803 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.803 DEBUG:boto:Data: 2024-04-13T22:27:50.803 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.803 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.803 DEBUG:boto:Params: {} 2024-04-13T22:27:50.803 DEBUG:boto:Token: None 2024-04-13T22:27:50.803 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA= 2024-04-13T22:27:50.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA=', 'Content-Length': '0'} 2024-04-13T22:27:50.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009acdfb64a731c487-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.805 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.805 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.805 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.806 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.806 DEBUG:boto:Method: GET 2024-04-13T22:27:50.806 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.806 DEBUG:boto:Data: 2024-04-13T22:27:50.806 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.806 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.806 DEBUG:boto:Params: {} 2024-04-13T22:27:50.806 DEBUG:boto:Token: None 2024-04-13T22:27:50.806 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA= 2024-04-13T22:27:50.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA=', 'Content-Length': '0'} 2024-04-13T22:27:50.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc27dfbf0b08dc85-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.808 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.808 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.808 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.809 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.809 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.809 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.809 DEBUG:boto:Data: 2024-04-13T22:27:50.809 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.809 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.809 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.809 DEBUG:boto:Params: {} 2024-04-13T22:27:50.809 DEBUG:boto:Token: None 2024-04-13T22:27:50.809 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FpWjbpVSsxYS/+MbfyTUxwblKB4= 2024-04-13T22:27:50.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FpWjbpVSsxYS/+MbfyTUxwblKB4=', 'Content-Length': '0'} 2024-04-13T22:27:50.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64d8d5841c19b35-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.811 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.811 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.811 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.811 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.811 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.812 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.812 DEBUG:boto:Data: 2024-04-13T22:27:50.812 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.812 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.812 DEBUG:boto:Params: {} 2024-04-13T22:27:50.812 DEBUG:boto:Token: None 2024-04-13T22:27:50.812 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FpWjbpVSsxYS/+MbfyTUxwblKB4= 2024-04-13T22:27:50.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FpWjbpVSsxYS/+MbfyTUxwblKB4=', 'Content-Length': '0'} 2024-04-13T22:27:50.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6e79f56a5a3e14-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:27:50.814 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.814 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.814 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.815 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.815 DEBUG:boto:Method: GET 2024-04-13T22:27:50.815 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.815 DEBUG:boto:Data: 2024-04-13T22:27:50.815 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.815 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.815 DEBUG:boto:Params: {} 2024-04-13T22:27:50.815 DEBUG:boto:Token: None 2024-04-13T22:27:50.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA= 2024-04-13T22:27:50.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA=', 'Content-Length': '0'} 2024-04-13T22:27:50.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39b486d60f6c42a-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.858 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.858 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.859 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.859 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.859 DEBUG:boto:Method: GET 2024-04-13T22:27:50.859 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.859 DEBUG:boto:Data: 2024-04-13T22:27:50.859 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.859 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.859 DEBUG:boto:Params: {} 2024-04-13T22:27:50.859 DEBUG:boto:Token: None 2024-04-13T22:27:50.860 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:27:50.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA= 2024-04-13T22:27:50.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7G6T/bGsbBrBMkGkQFw5JgckLkA=', 'Content-Length': '0'} 2024-04-13T22:27:50.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083add1e602a9b118-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.862 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.862 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.862 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.862 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.862 DEBUG:boto:Method: GET 2024-04-13T22:27:50.863 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.863 DEBUG:boto:Data: 2024-04-13T22:27:50.863 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.863 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.863 DEBUG:boto:Params: {} 2024-04-13T22:27:50.863 DEBUG:boto:Token: None 2024-04-13T22:27:50.863 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a74e8ebd9143de-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.866 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.866 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.866 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.866 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.866 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.867 DEBUG:boto:Method: GET 2024-04-13T22:27:50.867 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.867 DEBUG:boto:Data: 2024-04-13T22:27:50.867 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.867 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.867 DEBUG:boto:Params: {} 2024-04-13T22:27:50.867 DEBUG:boto:Token: None 2024-04-13T22:27:50.867 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d3e9087d830dd9-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.870 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.871 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.871 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.871 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.871 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:27:50.871 DEBUG:boto:Data: 2024-04-13T22:27:50.871 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.871 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.871 DEBUG:boto:Params: {} 2024-04-13T22:27:50.872 DEBUG:boto:Token: None 2024-04-13T22:27:50.872 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55 2024-04-13T22:27:50.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t5RrvoKAO/7jc9wvM9oDeIElfMg= 2024-04-13T22:27:50.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t5RrvoKAO/7jc9wvM9oDeIElfMg=', 'Content-Length': '0'} 2024-04-13T22:27:50.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff8fd7f67b350dd-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.874 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.874 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.874 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.874 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:27:50.874 DEBUG:boto:Data: 2024-04-13T22:27:50.874 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.874 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.874 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.874 DEBUG:boto:Params: {} 2024-04-13T22:27:50.875 DEBUG:boto:Token: None 2024-04-13T22:27:50.875 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55 2024-04-13T22:27:50.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t5RrvoKAO/7jc9wvM9oDeIElfMg= 2024-04-13T22:27:50.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t5RrvoKAO/7jc9wvM9oDeIElfMg=', 'Content-Length': '0'} 2024-04-13T22:27:50.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ba97d799a6e78b-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:27:50.877 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.877 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.877 DEBUG:boto:Method: GET 2024-04-13T22:27:50.877 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:27:50.877 DEBUG:boto:Data: 2024-04-13T22:27:50.877 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.877 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.878 DEBUG:boto:Params: {} 2024-04-13T22:27:50.878 DEBUG:boto:Token: None 2024-04-13T22:27:50.878 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55 2024-04-13T22:27:50.878 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cQCBCQKYmsumwltwRNqsxnrfnOU= 2024-04-13T22:27:50.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cQCBCQKYmsumwltwRNqsxnrfnOU=', 'Content-Length': '0'} 2024-04-13T22:27:50.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65bac3cffa6bf0e-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.880 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:27:50.880 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:27:50.880 DEBUG:boto:Method: GET 2024-04-13T22:27:50.880 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:27:50.881 DEBUG:boto:Data: 2024-04-13T22:27:50.881 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.881 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.881 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.881 DEBUG:boto:Params: {} 2024-04-13T22:27:50.881 DEBUG:boto:Token: None 2024-04-13T22:27:50.881 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj55 2024-04-13T22:27:50.881 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cQCBCQKYmsumwltwRNqsxnrfnOU= 2024-04-13T22:27:50.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cQCBCQKYmsumwltwRNqsxnrfnOU=', 'Content-Length': '0'} 2024-04-13T22:27:50.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba1f13a5340c2a71-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.927 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.927 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.927 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.927 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.927 DEBUG:boto:Method: GET 2024-04-13T22:27:50.927 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.927 DEBUG:boto:Data: 2024-04-13T22:27:50.927 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.927 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.928 DEBUG:boto:Params: {} 2024-04-13T22:27:50.928 DEBUG:boto:Token: None 2024-04-13T22:27:50.928 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f66c683ab4b020f5-00661b06e6-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.931 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.931 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:50.931 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:50.931 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.931 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.931 DEBUG:boto:Method: GET 2024-04-13T22:27:50.931 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:27:50.931 DEBUG:boto:Data: 2024-04-13T22:27:50.932 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.932 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.932 DEBUG:boto:Params: {} 2024-04-13T22:27:50.932 DEBUG:boto:Token: None 2024-04-13T22:27:50.932 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/ 2024-04-13T22:27:50.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0= 2024-04-13T22:27:50.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b+3eTRiRiB/RWlAL/eyG9oWTU/0=', 'Content-Length': '0'} 2024-04-13T22:27:50.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003963fa8ac01c4b58-00661b06e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.935 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:50.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:27:50.936 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:50.936 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:50.936 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.936 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.936 DEBUG:boto:Method: GET 2024-04-13T22:27:50.936 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.937 DEBUG:boto:Data: 2024-04-13T22:27:50.937 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.937 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.937 DEBUG:boto:Params: {} 2024-04-13T22:27:50.937 DEBUG:boto:Token: None 2024-04-13T22:27:50.937 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA= 2024-04-13T22:27:50.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA=', 'Content-Length': '0'} 2024-04-13T22:27:50.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a074516ae74bba0e-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.939 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:50.939 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:50.939 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.939 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.939 DEBUG:boto:Method: GET 2024-04-13T22:27:50.940 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.940 DEBUG:boto:Data: 2024-04-13T22:27:50.940 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.940 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.940 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.940 DEBUG:boto:Params: {} 2024-04-13T22:27:50.940 DEBUG:boto:Token: None 2024-04-13T22:27:50.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA= 2024-04-13T22:27:50.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA=', 'Content-Length': '0'} 2024-04-13T22:27:50.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004609b6c5b7573406-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.942 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:50.942 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:50.943 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.943 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.943 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.943 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.943 DEBUG:boto:Data: 2024-04-13T22:27:50.943 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.943 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.943 DEBUG:boto:Params: {} 2024-04-13T22:27:50.943 DEBUG:boto:Token: None 2024-04-13T22:27:50.943 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.944 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eAF7FNIaB7y5AycvqckVwiaQWaw= 2024-04-13T22:27:50.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eAF7FNIaB7y5AycvqckVwiaQWaw=', 'Content-Length': '0'} 2024-04-13T22:27:50.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d9719d53ff7fb3-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.945 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:50.945 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:50.945 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.946 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.946 DEBUG:boto:Method: HEAD 2024-04-13T22:27:50.946 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.946 DEBUG:boto:Data: 2024-04-13T22:27:50.946 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.946 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.946 DEBUG:boto:Params: {} 2024-04-13T22:27:50.947 DEBUG:boto:Token: None 2024-04-13T22:27:50.947 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eAF7FNIaB7y5AycvqckVwiaQWaw= 2024-04-13T22:27:50.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eAF7FNIaB7y5AycvqckVwiaQWaw=', 'Content-Length': '0'} 2024-04-13T22:27:50.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f02fdd8dec4c35e-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:27:50.949 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:50.949 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:50.949 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.949 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.949 DEBUG:boto:Method: GET 2024-04-13T22:27:50.949 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.950 DEBUG:boto:Data: 2024-04-13T22:27:50.950 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.950 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:50.950 DEBUG:boto:Port: 8001 2024-04-13T22:27:50.950 DEBUG:boto:Params: {} 2024-04-13T22:27:50.950 DEBUG:boto:Token: None 2024-04-13T22:27:50.950 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.950 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA= 2024-04-13T22:27:50.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA=', 'Content-Length': '0'} 2024-04-13T22:27:50.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5cd23312683dda-00661b06e6-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:50.998 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:50.998 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:50.999 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.999 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.999 DEBUG:boto:Method: GET 2024-04-13T22:27:50.999 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:50.999 DEBUG:boto:Data: 2024-04-13T22:27:50.999 DEBUG:boto:Headers: {} 2024-04-13T22:27:50.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:50.999 DEBUG:boto:Port: 8000 2024-04-13T22:27:50.999 DEBUG:boto:Params: {} 2024-04-13T22:27:50.999 DEBUG:boto:Token: None 2024-04-13T22:27:50.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:50 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:27:51.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA= 2024-04-13T22:27:51.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sl5OlxHml03QnuOeECPvM9jNObA=', 'Content-Length': '0'} 2024-04-13T22:27:51.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140aad5241c7a78e-00661b06e6-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.002 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.002 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.002 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.002 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.002 DEBUG:boto:Method: GET 2024-04-13T22:27:51.002 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.002 DEBUG:boto:Data: 2024-04-13T22:27:51.002 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.002 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.002 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.003 DEBUG:boto:Params: {} 2024-04-13T22:27:51.003 DEBUG:boto:Token: None 2024-04-13T22:27:51.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50660f61779b693-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.005 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.006 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.006 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.006 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.006 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.006 DEBUG:boto:Method: GET 2024-04-13T22:27:51.006 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.006 DEBUG:boto:Data: 2024-04-13T22:27:51.006 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.006 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.007 DEBUG:boto:Params: {} 2024-04-13T22:27:51.007 DEBUG:boto:Token: None 2024-04-13T22:27:51.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34eac5176ee38c2-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.010 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.010 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:51.010 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:51.011 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.011 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:27:51.011 DEBUG:boto:Data: 2024-04-13T22:27:51.011 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.011 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.011 DEBUG:boto:Params: {} 2024-04-13T22:27:51.011 DEBUG:boto:Token: None 2024-04-13T22:27:51.011 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj57 2024-04-13T22:27:51.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wDqaspYFXyJwcrmdNl6L1DX0vfM= 2024-04-13T22:27:51.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wDqaspYFXyJwcrmdNl6L1DX0vfM=', 'Content-Length': '0'} 2024-04-13T22:27:51.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a5df34cf2c6330-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.013 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:51.013 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:51.013 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.013 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:27:51.014 DEBUG:boto:Data: 2024-04-13T22:27:51.014 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.014 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.014 DEBUG:boto:Params: {} 2024-04-13T22:27:51.014 DEBUG:boto:Token: None 2024-04-13T22:27:51.014 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj57 2024-04-13T22:27:51.014 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wDqaspYFXyJwcrmdNl6L1DX0vfM= 2024-04-13T22:27:51.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wDqaspYFXyJwcrmdNl6L1DX0vfM=', 'Content-Length': '0'} 2024-04-13T22:27:51.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f0ed00c2ec001a0-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:27:51.016 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:51.016 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:51.016 DEBUG:boto:Method: GET 2024-04-13T22:27:51.016 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:27:51.016 DEBUG:boto:Data: 2024-04-13T22:27:51.017 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.017 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.017 DEBUG:boto:Params: {} 2024-04-13T22:27:51.017 DEBUG:boto:Token: None 2024-04-13T22:27:51.017 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj57 2024-04-13T22:27:51.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rI2pfhywBkl9BwyUAv6dsh+kVzk= 2024-04-13T22:27:51.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rI2pfhywBkl9BwyUAv6dsh+kVzk=', 'Content-Length': '0'} 2024-04-13T22:27:51.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2520ee2a9598cc8-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.019 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:27:51.019 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:27:51.019 DEBUG:boto:Method: GET 2024-04-13T22:27:51.019 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:27:51.019 DEBUG:boto:Data: 2024-04-13T22:27:51.019 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.020 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.020 DEBUG:boto:Params: {} 2024-04-13T22:27:51.020 DEBUG:boto:Token: None 2024-04-13T22:27:51.020 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj57 2024-04-13T22:27:51.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rI2pfhywBkl9BwyUAv6dsh+kVzk= 2024-04-13T22:27:51.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rI2pfhywBkl9BwyUAv6dsh+kVzk=', 'Content-Length': '0'} 2024-04-13T22:27:51.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b2879949012811-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.022 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.022 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.022 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.022 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.022 DEBUG:boto:Method: GET 2024-04-13T22:27:51.022 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.022 DEBUG:boto:Data: 2024-04-13T22:27:51.023 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.023 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.023 DEBUG:boto:Params: {} 2024-04-13T22:27:51.023 DEBUG:boto:Token: None 2024-04-13T22:27:51.023 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.023 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b48fcb4346971269-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.025 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.026 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.026 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.026 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.026 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.026 DEBUG:boto:Method: GET 2024-04-13T22:27:51.026 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:27:51.026 DEBUG:boto:Data: 2024-04-13T22:27:51.027 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.027 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.027 DEBUG:boto:Params: {} 2024-04-13T22:27:51.027 DEBUG:boto:Token: None 2024-04-13T22:27:51.027 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cadb7865f210600d-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.030 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:27:51.031 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.031 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.031 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.031 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.031 DEBUG:boto:Method: GET 2024-04-13T22:27:51.031 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.031 DEBUG:boto:Data: 2024-04-13T22:27:51.031 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.031 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.031 DEBUG:boto:Params: {} 2024-04-13T22:27:51.032 DEBUG:boto:Token: None 2024-04-13T22:27:51.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0= 2024-04-13T22:27:51.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0=', 'Content-Length': '0'} 2024-04-13T22:27:51.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801ea46736fe89bb-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.034 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.034 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.034 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.034 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.034 DEBUG:boto:Method: GET 2024-04-13T22:27:51.034 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.035 DEBUG:boto:Data: 2024-04-13T22:27:51.035 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.035 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.035 DEBUG:boto:Params: {} 2024-04-13T22:27:51.035 DEBUG:boto:Token: None 2024-04-13T22:27:51.035 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.035 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0= 2024-04-13T22:27:51.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0=', 'Content-Length': '0'} 2024-04-13T22:27:51.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e673aa6b4da7c272-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.078 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.079 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.079 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.079 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.079 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.079 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.079 DEBUG:boto:Data: 2024-04-13T22:27:51.079 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.079 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.079 DEBUG:boto:Params: {} 2024-04-13T22:27:51.080 DEBUG:boto:Token: None 2024-04-13T22:27:51.080 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2N8Ec/wmI923bDrLaCZcbP6vuxo= 2024-04-13T22:27:51.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2N8Ec/wmI923bDrLaCZcbP6vuxo=', 'Content-Length': '0'} 2024-04-13T22:27:51.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a67709050a6ab2-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.082 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.082 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.082 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.082 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.082 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.082 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.082 DEBUG:boto:Data: 2024-04-13T22:27:51.082 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.083 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.083 DEBUG:boto:Params: {} 2024-04-13T22:27:51.083 DEBUG:boto:Token: None 2024-04-13T22:27:51.083 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2N8Ec/wmI923bDrLaCZcbP6vuxo= 2024-04-13T22:27:51.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2N8Ec/wmI923bDrLaCZcbP6vuxo=', 'Content-Length': '0'} 2024-04-13T22:27:51.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e83826f3dc33417-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:27:51.085 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.085 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.085 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.086 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.086 DEBUG:boto:Method: GET 2024-04-13T22:27:51.086 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.086 DEBUG:boto:Data: 2024-04-13T22:27:51.086 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.086 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.086 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.086 DEBUG:boto:Params: {} 2024-04-13T22:27:51.086 DEBUG:boto:Token: None 2024-04-13T22:27:51.086 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.087 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0= 2024-04-13T22:27:51.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0=', 'Content-Length': '0'} 2024-04-13T22:27:51.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b8776d406f795a3-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.088 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.088 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.089 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.089 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.089 DEBUG:boto:Method: GET 2024-04-13T22:27:51.089 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.089 DEBUG:boto:Data: 2024-04-13T22:27:51.089 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.089 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.089 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.089 DEBUG:boto:Params: {} 2024-04-13T22:27:51.089 DEBUG:boto:Token: None 2024-04-13T22:27:51.090 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:27:51.090 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0= 2024-04-13T22:27:51.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DynsKLEfd6YpdON4kU7JCftpFC0=', 'Content-Length': '0'} 2024-04-13T22:27:51.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2fae2db1e9f6fac-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.092 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.092 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.092 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.092 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.092 DEBUG:boto:Method: GET 2024-04-13T22:27:51.092 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.092 DEBUG:boto:Data: 2024-04-13T22:27:51.093 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.093 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.093 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.093 DEBUG:boto:Params: {} 2024-04-13T22:27:51.093 DEBUG:boto:Token: None 2024-04-13T22:27:51.093 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.093 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f52904a45b3f52a8-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.096 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.096 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.096 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.096 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.096 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.096 DEBUG:boto:Method: GET 2024-04-13T22:27:51.097 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.097 DEBUG:boto:Data: 2024-04-13T22:27:51.097 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.097 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.097 DEBUG:boto:Params: {} 2024-04-13T22:27:51.097 DEBUG:boto:Token: None 2024-04-13T22:27:51.097 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.097 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ec0df15b5e92ff7-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.101 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.101 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.101 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.102 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.102 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:27:51.102 DEBUG:boto:Data: 2024-04-13T22:27:51.102 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.102 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.102 DEBUG:boto:Params: {} 2024-04-13T22:27:51.105 DEBUG:boto:Token: None 2024-04-13T22:27:51.106 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58 2024-04-13T22:27:51.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FXn9FhiJ+Om0sBN35lRPTSmWb5s= 2024-04-13T22:27:51.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FXn9FhiJ+Om0sBN35lRPTSmWb5s=', 'Content-Length': '0'} 2024-04-13T22:27:51.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d056a77a90c37f19-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.108 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.108 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.108 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.108 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:27:51.108 DEBUG:boto:Data: 2024-04-13T22:27:51.108 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.108 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.108 DEBUG:boto:Params: {} 2024-04-13T22:27:51.108 DEBUG:boto:Token: None 2024-04-13T22:27:51.109 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58 2024-04-13T22:27:51.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FXn9FhiJ+Om0sBN35lRPTSmWb5s= 2024-04-13T22:27:51.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FXn9FhiJ+Om0sBN35lRPTSmWb5s=', 'Content-Length': '0'} 2024-04-13T22:27:51.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077119e80ef3596aa-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:27:51.111 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.111 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.111 DEBUG:boto:Method: GET 2024-04-13T22:27:51.111 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:27:51.111 DEBUG:boto:Data: 2024-04-13T22:27:51.111 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.111 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.112 DEBUG:boto:Params: {} 2024-04-13T22:27:51.112 DEBUG:boto:Token: None 2024-04-13T22:27:51.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58 2024-04-13T22:27:51.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UUPFWMYyJiRtpFXVL1QlXLytHx4= 2024-04-13T22:27:51.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UUPFWMYyJiRtpFXVL1QlXLytHx4=', 'Content-Length': '0'} 2024-04-13T22:27:51.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659bb7478dce41a2-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.154 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:27:51.154 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:27:51.155 DEBUG:boto:Method: GET 2024-04-13T22:27:51.155 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:27:51.155 DEBUG:boto:Data: 2024-04-13T22:27:51.155 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.155 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.155 DEBUG:boto:Params: {} 2024-04-13T22:27:51.155 DEBUG:boto:Token: None 2024-04-13T22:27:51.155 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj58 2024-04-13T22:27:51.156 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UUPFWMYyJiRtpFXVL1QlXLytHx4= 2024-04-13T22:27:51.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UUPFWMYyJiRtpFXVL1QlXLytHx4=', 'Content-Length': '0'} 2024-04-13T22:27:51.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334d5c0d6a783c52-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.158 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.158 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.158 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.158 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.159 DEBUG:boto:Method: GET 2024-04-13T22:27:51.159 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.159 DEBUG:boto:Data: 2024-04-13T22:27:51.159 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.159 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.159 DEBUG:boto:Params: {} 2024-04-13T22:27:51.159 DEBUG:boto:Token: None 2024-04-13T22:27:51.160 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc26084cd9409b70-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.162 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.162 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.162 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.163 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.163 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.163 DEBUG:boto:Method: GET 2024-04-13T22:27:51.163 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:27:51.163 DEBUG:boto:Data: 2024-04-13T22:27:51.163 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.163 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.163 DEBUG:boto:Params: {} 2024-04-13T22:27:51.163 DEBUG:boto:Token: None 2024-04-13T22:27:51.163 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006533b1ceb4f3585-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.167 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:27:51.167 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.167 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.168 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.168 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.168 DEBUG:boto:Method: GET 2024-04-13T22:27:51.168 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.168 DEBUG:boto:Data: 2024-04-13T22:27:51.168 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.168 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.168 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.168 DEBUG:boto:Params: {} 2024-04-13T22:27:51.168 DEBUG:boto:Token: None 2024-04-13T22:27:51.169 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE= 2024-04-13T22:27:51.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE=', 'Content-Length': '0'} 2024-04-13T22:27:51.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259a02b4ade7cf8a-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.170 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.171 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.171 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.171 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.171 DEBUG:boto:Method: GET 2024-04-13T22:27:51.171 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.171 DEBUG:boto:Data: 2024-04-13T22:27:51.171 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.171 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.171 DEBUG:boto:Params: {} 2024-04-13T22:27:51.172 DEBUG:boto:Token: None 2024-04-13T22:27:51.172 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE= 2024-04-13T22:27:51.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE=', 'Content-Length': '0'} 2024-04-13T22:27:51.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097f22172531da9ff-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.218 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.218 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.219 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.219 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.219 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.219 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.219 DEBUG:boto:Data: 2024-04-13T22:27:51.219 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.219 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.219 DEBUG:boto:Params: {} 2024-04-13T22:27:51.220 DEBUG:boto:Token: None 2024-04-13T22:27:51.220 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PaZXIlnAXnimLXKoLadCDXQ+NtA= 2024-04-13T22:27:51.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PaZXIlnAXnimLXKoLadCDXQ+NtA=', 'Content-Length': '0'} 2024-04-13T22:27:51.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ee44ac0c069936-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.222 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.222 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.222 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.222 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.222 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.222 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.222 DEBUG:boto:Data: 2024-04-13T22:27:51.222 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.222 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.222 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.222 DEBUG:boto:Params: {} 2024-04-13T22:27:51.223 DEBUG:boto:Token: None 2024-04-13T22:27:51.223 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.223 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PaZXIlnAXnimLXKoLadCDXQ+NtA= 2024-04-13T22:27:51.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PaZXIlnAXnimLXKoLadCDXQ+NtA=', 'Content-Length': '0'} 2024-04-13T22:27:51.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d9271ad9c98ee5-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:27:51.225 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.225 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.225 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.225 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.225 DEBUG:boto:Method: GET 2024-04-13T22:27:51.225 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.225 DEBUG:boto:Data: 2024-04-13T22:27:51.226 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.226 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.226 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.226 DEBUG:boto:Params: {} 2024-04-13T22:27:51.226 DEBUG:boto:Token: None 2024-04-13T22:27:51.226 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.226 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE= 2024-04-13T22:27:51.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE=', 'Content-Length': '0'} 2024-04-13T22:27:51.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115810f514229942-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.228 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.228 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.228 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.228 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.229 DEBUG:boto:Method: GET 2024-04-13T22:27:51.229 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.229 DEBUG:boto:Data: 2024-04-13T22:27:51.229 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.229 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.229 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.229 DEBUG:boto:Params: {} 2024-04-13T22:27:51.229 DEBUG:boto:Token: None 2024-04-13T22:27:51.229 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:27:51.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE= 2024-04-13T22:27:51.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e0IJXIlNT53IL4jXY6P0zmmJVGE=', 'Content-Length': '0'} 2024-04-13T22:27:51.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6e38cedef22de6-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.231 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.231 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.231 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.231 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.232 DEBUG:boto:Method: GET 2024-04-13T22:27:51.232 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.232 DEBUG:boto:Data: 2024-04-13T22:27:51.232 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.232 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.232 DEBUG:boto:Params: {} 2024-04-13T22:27:51.232 DEBUG:boto:Token: None 2024-04-13T22:27:51.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0828773310eb8aa-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.234 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.235 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.235 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.235 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.235 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.235 DEBUG:boto:Method: GET 2024-04-13T22:27:51.235 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.235 DEBUG:boto:Data: 2024-04-13T22:27:51.235 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.236 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.236 DEBUG:boto:Params: {} 2024-04-13T22:27:51.236 DEBUG:boto:Token: None 2024-04-13T22:27:51.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e9f6ebf0fc379ec-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.239 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.239 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.239 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.239 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.240 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:27:51.240 DEBUG:boto:Data: 2024-04-13T22:27:51.240 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.240 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.240 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.240 DEBUG:boto:Params: {} 2024-04-13T22:27:51.240 DEBUG:boto:Token: None 2024-04-13T22:27:51.240 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68 2024-04-13T22:27:51.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sM1zAu0Kqo6NNpAbcKtrl/V09uk= 2024-04-13T22:27:51.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sM1zAu0Kqo6NNpAbcKtrl/V09uk=', 'Content-Length': '0'} 2024-04-13T22:27:51.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001283c2f2c6d907e2-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.242 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.242 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.242 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.242 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:27:51.242 DEBUG:boto:Data: 2024-04-13T22:27:51.243 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.243 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.243 DEBUG:boto:Params: {} 2024-04-13T22:27:51.243 DEBUG:boto:Token: None 2024-04-13T22:27:51.243 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68 2024-04-13T22:27:51.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sM1zAu0Kqo6NNpAbcKtrl/V09uk= 2024-04-13T22:27:51.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sM1zAu0Kqo6NNpAbcKtrl/V09uk=', 'Content-Length': '0'} 2024-04-13T22:27:51.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2384f9d55076006-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:27:51.245 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.245 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.245 DEBUG:boto:Method: GET 2024-04-13T22:27:51.245 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:27:51.245 DEBUG:boto:Data: 2024-04-13T22:27:51.245 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.246 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.246 DEBUG:boto:Params: {} 2024-04-13T22:27:51.246 DEBUG:boto:Token: None 2024-04-13T22:27:51.246 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68 2024-04-13T22:27:51.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RyVhXWlWz9W+aJE5AjJ/5GB3fxU= 2024-04-13T22:27:51.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RyVhXWlWz9W+aJE5AjJ/5GB3fxU=', 'Content-Length': '0'} 2024-04-13T22:27:51.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7f76fd14bde8cd-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.248 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:27:51.248 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:27:51.248 DEBUG:boto:Method: GET 2024-04-13T22:27:51.248 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:27:51.248 DEBUG:boto:Data: 2024-04-13T22:27:51.248 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.249 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.249 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.249 DEBUG:boto:Params: {} 2024-04-13T22:27:51.249 DEBUG:boto:Token: None 2024-04-13T22:27:51.249 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj68 2024-04-13T22:27:51.249 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RyVhXWlWz9W+aJE5AjJ/5GB3fxU= 2024-04-13T22:27:51.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RyVhXWlWz9W+aJE5AjJ/5GB3fxU=', 'Content-Length': '0'} 2024-04-13T22:27:51.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb9bd730da34f77-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.294 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.295 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.295 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.295 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.295 DEBUG:boto:Method: GET 2024-04-13T22:27:51.295 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.295 DEBUG:boto:Data: 2024-04-13T22:27:51.295 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.295 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.295 DEBUG:boto:Params: {} 2024-04-13T22:27:51.295 DEBUG:boto:Token: None 2024-04-13T22:27:51.296 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004814e27d05f0bfaa-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.298 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.299 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.299 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.299 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.299 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.299 DEBUG:boto:Method: GET 2024-04-13T22:27:51.299 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:27:51.299 DEBUG:boto:Data: 2024-04-13T22:27:51.299 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.299 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.299 DEBUG:boto:Params: {} 2024-04-13T22:27:51.300 DEBUG:boto:Token: None 2024-04-13T22:27:51.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000236f0c7d917e7c6b-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.303 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:27:51.304 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.304 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.304 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.304 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.304 DEBUG:boto:Method: GET 2024-04-13T22:27:51.304 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.304 DEBUG:boto:Data: 2024-04-13T22:27:51.304 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.304 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.304 DEBUG:boto:Params: {} 2024-04-13T22:27:51.304 DEBUG:boto:Token: None 2024-04-13T22:27:51.305 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg= 2024-04-13T22:27:51.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg=', 'Content-Length': '0'} 2024-04-13T22:27:51.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000358d5bdb50de9a2d-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.307 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.307 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.307 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.307 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.307 DEBUG:boto:Method: GET 2024-04-13T22:27:51.307 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.307 DEBUG:boto:Data: 2024-04-13T22:27:51.307 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.307 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.308 DEBUG:boto:Params: {} 2024-04-13T22:27:51.308 DEBUG:boto:Token: None 2024-04-13T22:27:51.308 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg= 2024-04-13T22:27:51.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg=', 'Content-Length': '0'} 2024-04-13T22:27:51.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b1a1a283114175-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.310 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.310 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.310 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.311 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.311 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.311 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.311 DEBUG:boto:Data: 2024-04-13T22:27:51.311 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.311 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.311 DEBUG:boto:Params: {} 2024-04-13T22:27:51.311 DEBUG:boto:Token: None 2024-04-13T22:27:51.311 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/nZre0nJdwblwB2DPpUU8eme4D8= 2024-04-13T22:27:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/nZre0nJdwblwB2DPpUU8eme4D8=', 'Content-Length': '0'} 2024-04-13T22:27:51.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbebe29a4728023b-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.313 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.313 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.313 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.314 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.314 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.314 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.314 DEBUG:boto:Data: 2024-04-13T22:27:51.314 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.314 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.314 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.314 DEBUG:boto:Params: {} 2024-04-13T22:27:51.314 DEBUG:boto:Token: None 2024-04-13T22:27:51.314 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/nZre0nJdwblwB2DPpUU8eme4D8= 2024-04-13T22:27:51.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/nZre0nJdwblwB2DPpUU8eme4D8=', 'Content-Length': '0'} 2024-04-13T22:27:51.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5d307102ba8477-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:27:51.316 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.316 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.317 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.317 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.317 DEBUG:boto:Method: GET 2024-04-13T22:27:51.317 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.317 DEBUG:boto:Data: 2024-04-13T22:27:51.317 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.317 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.317 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.317 DEBUG:boto:Params: {} 2024-04-13T22:27:51.317 DEBUG:boto:Token: None 2024-04-13T22:27:51.318 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.318 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg= 2024-04-13T22:27:51.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg=', 'Content-Length': '0'} 2024-04-13T22:27:51.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ffc2a8668cd439-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.362 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.362 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.363 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.363 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.363 DEBUG:boto:Method: GET 2024-04-13T22:27:51.363 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.363 DEBUG:boto:Data: 2024-04-13T22:27:51.363 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.363 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.363 DEBUG:boto:Params: {} 2024-04-13T22:27:51.363 DEBUG:boto:Token: None 2024-04-13T22:27:51.364 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:27:51.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg= 2024-04-13T22:27:51.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OjJK5m6/XpcskUAgwn9SMrqoaGg=', 'Content-Length': '0'} 2024-04-13T22:27:51.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6df199246b7dd6-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.366 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.366 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.366 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.366 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.366 DEBUG:boto:Method: GET 2024-04-13T22:27:51.366 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.366 DEBUG:boto:Data: 2024-04-13T22:27:51.366 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.367 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.367 DEBUG:boto:Params: {} 2024-04-13T22:27:51.367 DEBUG:boto:Token: None 2024-04-13T22:27:51.367 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cda20146577ed54b-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.369 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.370 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.370 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.370 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.370 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.370 DEBUG:boto:Method: GET 2024-04-13T22:27:51.370 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.370 DEBUG:boto:Data: 2024-04-13T22:27:51.370 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.370 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.371 DEBUG:boto:Params: {} 2024-04-13T22:27:51.371 DEBUG:boto:Token: None 2024-04-13T22:27:51.371 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f20263232d8acbd-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.376 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.376 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.376 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.376 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.376 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:27:51.377 DEBUG:boto:Data: 2024-04-13T22:27:51.377 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.377 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.377 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.377 DEBUG:boto:Params: {} 2024-04-13T22:27:51.377 DEBUG:boto:Token: None 2024-04-13T22:27:51.377 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69 2024-04-13T22:27:51.377 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bQrsNPPtem4cczMYl75zdPfgPs0= 2024-04-13T22:27:51.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bQrsNPPtem4cczMYl75zdPfgPs0=', 'Content-Length': '0'} 2024-04-13T22:27:51.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095bca70cd20655e4-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.379 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.379 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.379 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.379 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:27:51.379 DEBUG:boto:Data: 2024-04-13T22:27:51.380 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.380 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.380 DEBUG:boto:Params: {} 2024-04-13T22:27:51.380 DEBUG:boto:Token: None 2024-04-13T22:27:51.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69 2024-04-13T22:27:51.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bQrsNPPtem4cczMYl75zdPfgPs0= 2024-04-13T22:27:51.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bQrsNPPtem4cczMYl75zdPfgPs0=', 'Content-Length': '0'} 2024-04-13T22:27:51.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7646d5e071b35b-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:27:51.382 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.382 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.385 DEBUG:boto:Method: GET 2024-04-13T22:27:51.385 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:27:51.385 DEBUG:boto:Data: 2024-04-13T22:27:51.385 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.385 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.385 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.385 DEBUG:boto:Params: {} 2024-04-13T22:27:51.386 DEBUG:boto:Token: None 2024-04-13T22:27:51.386 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69 2024-04-13T22:27:51.386 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g8zOsaou6QEmHYfB7yigJcLRf6Y= 2024-04-13T22:27:51.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g8zOsaou6QEmHYfB7yigJcLRf6Y=', 'Content-Length': '0'} 2024-04-13T22:27:51.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa53ca421ccc691-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.388 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:27:51.388 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:27:51.388 DEBUG:boto:Method: GET 2024-04-13T22:27:51.388 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:27:51.388 DEBUG:boto:Data: 2024-04-13T22:27:51.389 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.389 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.389 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.389 DEBUG:boto:Params: {} 2024-04-13T22:27:51.389 DEBUG:boto:Token: None 2024-04-13T22:27:51.389 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj69 2024-04-13T22:27:51.389 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g8zOsaou6QEmHYfB7yigJcLRf6Y= 2024-04-13T22:27:51.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g8zOsaou6QEmHYfB7yigJcLRf6Y=', 'Content-Length': '0'} 2024-04-13T22:27:51.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000874ff05d851da9dd-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.434 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.434 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.435 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.435 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.435 DEBUG:boto:Method: GET 2024-04-13T22:27:51.435 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.435 DEBUG:boto:Data: 2024-04-13T22:27:51.435 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.435 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.435 DEBUG:boto:Params: {} 2024-04-13T22:27:51.435 DEBUG:boto:Token: None 2024-04-13T22:27:51.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced47d578593666f-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.438 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.439 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.439 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.439 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.439 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.439 DEBUG:boto:Method: GET 2024-04-13T22:27:51.439 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:27:51.439 DEBUG:boto:Data: 2024-04-13T22:27:51.439 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.440 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.440 DEBUG:boto:Params: {} 2024-04-13T22:27:51.440 DEBUG:boto:Token: None 2024-04-13T22:27:51.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004664067a083b9c10-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.443 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:27:51.443 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.444 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.444 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.444 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.444 DEBUG:boto:Method: GET 2024-04-13T22:27:51.444 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.444 DEBUG:boto:Data: 2024-04-13T22:27:51.444 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.444 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.444 DEBUG:boto:Params: {} 2024-04-13T22:27:51.444 DEBUG:boto:Token: None 2024-04-13T22:27:51.444 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4= 2024-04-13T22:27:51.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4=', 'Content-Length': '0'} 2024-04-13T22:27:51.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c9438c91c12775-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.486 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.486 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.486 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.486 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.487 DEBUG:boto:Method: GET 2024-04-13T22:27:51.487 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.487 DEBUG:boto:Data: 2024-04-13T22:27:51.487 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.487 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.487 DEBUG:boto:Params: {} 2024-04-13T22:27:51.487 DEBUG:boto:Token: None 2024-04-13T22:27:51.487 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4= 2024-04-13T22:27:51.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4=', 'Content-Length': '0'} 2024-04-13T22:27:51.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410a5e689b500d04-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.490 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.490 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.490 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.490 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.490 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.490 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.490 DEBUG:boto:Data: 2024-04-13T22:27:51.490 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.491 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.491 DEBUG:boto:Params: {} 2024-04-13T22:27:51.491 DEBUG:boto:Token: None 2024-04-13T22:27:51.491 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rFippr7DG8hB9pzMjise8UkTgH0= 2024-04-13T22:27:51.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rFippr7DG8hB9pzMjise8UkTgH0=', 'Content-Length': '0'} 2024-04-13T22:27:51.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9353055411e8461-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.493 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.493 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.493 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.493 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.493 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.493 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.493 DEBUG:boto:Data: 2024-04-13T22:27:51.493 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.493 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.494 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.494 DEBUG:boto:Params: {} 2024-04-13T22:27:51.494 DEBUG:boto:Token: None 2024-04-13T22:27:51.494 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.494 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rFippr7DG8hB9pzMjise8UkTgH0= 2024-04-13T22:27:51.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rFippr7DG8hB9pzMjise8UkTgH0=', 'Content-Length': '0'} 2024-04-13T22:27:51.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006957021f793e6fe6-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:27:51.496 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.496 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.496 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.496 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.497 DEBUG:boto:Method: GET 2024-04-13T22:27:51.497 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.497 DEBUG:boto:Data: 2024-04-13T22:27:51.497 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.497 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.497 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.497 DEBUG:boto:Params: {} 2024-04-13T22:27:51.497 DEBUG:boto:Token: None 2024-04-13T22:27:51.497 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.498 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4= 2024-04-13T22:27:51.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4=', 'Content-Length': '0'} 2024-04-13T22:27:51.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8e87698e7fcb41-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.499 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.499 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.500 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.500 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.500 DEBUG:boto:Method: GET 2024-04-13T22:27:51.500 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.500 DEBUG:boto:Data: 2024-04-13T22:27:51.500 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.501 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.501 DEBUG:boto:Params: {} 2024-04-13T22:27:51.501 DEBUG:boto:Token: None 2024-04-13T22:27:51.501 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:27:51.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4= 2024-04-13T22:27:51.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g2hF1yK6VPVoxldpW+4qYH5fYJ4=', 'Content-Length': '0'} 2024-04-13T22:27:51.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001df8198c1ecaa881-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.503 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.503 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.504 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.504 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.504 DEBUG:boto:Method: GET 2024-04-13T22:27:51.504 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.504 DEBUG:boto:Data: 2024-04-13T22:27:51.504 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.504 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.504 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.504 DEBUG:boto:Params: {} 2024-04-13T22:27:51.505 DEBUG:boto:Token: None 2024-04-13T22:27:51.505 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.505 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f186c21fc64e4721-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.507 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.508 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.508 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.508 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.508 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.508 DEBUG:boto:Method: GET 2024-04-13T22:27:51.508 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.508 DEBUG:boto:Data: 2024-04-13T22:27:51.508 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.508 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.509 DEBUG:boto:Params: {} 2024-04-13T22:27:51.509 DEBUG:boto:Token: None 2024-04-13T22:27:51.509 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc85f65dac03870-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.512 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.512 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.512 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.513 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.513 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:27:51.513 DEBUG:boto:Data: 2024-04-13T22:27:51.513 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.513 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.513 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.513 DEBUG:boto:Params: {} 2024-04-13T22:27:51.513 DEBUG:boto:Token: None 2024-04-13T22:27:51.513 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76 2024-04-13T22:27:51.514 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xYInpM8/nIfH/AE6b9DuzWMpyZs= 2024-04-13T22:27:51.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xYInpM8/nIfH/AE6b9DuzWMpyZs=', 'Content-Length': '0'} 2024-04-13T22:27:51.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a0255c8b6a7c8f-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.516 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.516 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.516 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.516 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:27:51.516 DEBUG:boto:Data: 2024-04-13T22:27:51.516 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.516 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.516 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.516 DEBUG:boto:Params: {} 2024-04-13T22:27:51.516 DEBUG:boto:Token: None 2024-04-13T22:27:51.517 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76 2024-04-13T22:27:51.517 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xYInpM8/nIfH/AE6b9DuzWMpyZs= 2024-04-13T22:27:51.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xYInpM8/nIfH/AE6b9DuzWMpyZs=', 'Content-Length': '0'} 2024-04-13T22:27:51.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e7c38bc60f7e7c-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:27:51.519 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.519 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.519 DEBUG:boto:Method: GET 2024-04-13T22:27:51.519 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:27:51.519 DEBUG:boto:Data: 2024-04-13T22:27:51.519 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.519 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.519 DEBUG:boto:Params: {} 2024-04-13T22:27:51.519 DEBUG:boto:Token: None 2024-04-13T22:27:51.520 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76 2024-04-13T22:27:51.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F2j4x+JZ69JWp4bn2vEIHtr+PYo= 2024-04-13T22:27:51.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F2j4x+JZ69JWp4bn2vEIHtr+PYo=', 'Content-Length': '0'} 2024-04-13T22:27:51.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ed0eff6d33330d8-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.566 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:27:51.566 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:27:51.566 DEBUG:boto:Method: GET 2024-04-13T22:27:51.567 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:27:51.567 DEBUG:boto:Data: 2024-04-13T22:27:51.567 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.567 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.567 DEBUG:boto:Params: {} 2024-04-13T22:27:51.567 DEBUG:boto:Token: None 2024-04-13T22:27:51.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj76 2024-04-13T22:27:51.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F2j4x+JZ69JWp4bn2vEIHtr+PYo= 2024-04-13T22:27:51.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F2j4x+JZ69JWp4bn2vEIHtr+PYo=', 'Content-Length': '0'} 2024-04-13T22:27:51.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae234cca668291d8-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.569 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.569 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.570 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.570 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.570 DEBUG:boto:Method: GET 2024-04-13T22:27:51.570 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.570 DEBUG:boto:Data: 2024-04-13T22:27:51.570 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.570 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.570 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.570 DEBUG:boto:Params: {} 2024-04-13T22:27:51.570 DEBUG:boto:Token: None 2024-04-13T22:27:51.571 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046925bfacb6114e4-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.573 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.573 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.574 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.574 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.574 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.574 DEBUG:boto:Method: GET 2024-04-13T22:27:51.574 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:27:51.574 DEBUG:boto:Data: 2024-04-13T22:27:51.574 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.574 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.574 DEBUG:boto:Params: {} 2024-04-13T22:27:51.575 DEBUG:boto:Token: None 2024-04-13T22:27:51.575 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd0d2dc6e4521b4-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.579 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:27:51.579 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.579 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.579 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.579 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.580 DEBUG:boto:Method: GET 2024-04-13T22:27:51.580 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.580 DEBUG:boto:Data: 2024-04-13T22:27:51.580 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.580 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.580 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.580 DEBUG:boto:Params: {} 2024-04-13T22:27:51.580 DEBUG:boto:Token: None 2024-04-13T22:27:51.580 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI= 2024-04-13T22:27:51.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI=', 'Content-Length': '0'} 2024-04-13T22:27:51.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8d7001a1336b403-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.582 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.582 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.583 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.583 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.583 DEBUG:boto:Method: GET 2024-04-13T22:27:51.583 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.583 DEBUG:boto:Data: 2024-04-13T22:27:51.583 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.583 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.583 DEBUG:boto:Params: {} 2024-04-13T22:27:51.583 DEBUG:boto:Token: None 2024-04-13T22:27:51.584 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI= 2024-04-13T22:27:51.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI=', 'Content-Length': '0'} 2024-04-13T22:27:51.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f43a46306ae8c1c-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.586 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.586 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.586 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.586 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.586 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.586 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.586 DEBUG:boto:Data: 2024-04-13T22:27:51.586 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.587 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.587 DEBUG:boto:Params: {} 2024-04-13T22:27:51.587 DEBUG:boto:Token: None 2024-04-13T22:27:51.587 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ckW/2fmK7MXgOOaL9rJVV5lLzJo= 2024-04-13T22:27:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ckW/2fmK7MXgOOaL9rJVV5lLzJo=', 'Content-Length': '0'} 2024-04-13T22:27:51.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003873c65d0ff9ab15-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.589 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.589 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.589 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.589 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.589 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.590 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.590 DEBUG:boto:Data: 2024-04-13T22:27:51.590 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.590 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.590 DEBUG:boto:Params: {} 2024-04-13T22:27:51.590 DEBUG:boto:Token: None 2024-04-13T22:27:51.590 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.590 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ckW/2fmK7MXgOOaL9rJVV5lLzJo= 2024-04-13T22:27:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ckW/2fmK7MXgOOaL9rJVV5lLzJo=', 'Content-Length': '0'} 2024-04-13T22:27:51.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ba6a14c02ed6d7-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:27:51.593 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.593 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.593 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.593 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.593 DEBUG:boto:Method: GET 2024-04-13T22:27:51.593 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.593 DEBUG:boto:Data: 2024-04-13T22:27:51.594 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.594 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.594 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.594 DEBUG:boto:Params: {} 2024-04-13T22:27:51.594 DEBUG:boto:Token: None 2024-04-13T22:27:51.594 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI= 2024-04-13T22:27:51.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI=', 'Content-Length': '0'} 2024-04-13T22:27:51.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011812261a3ec2849-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.596 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.596 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.596 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.596 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.596 DEBUG:boto:Method: GET 2024-04-13T22:27:51.596 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.597 DEBUG:boto:Data: 2024-04-13T22:27:51.597 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.597 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.597 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.597 DEBUG:boto:Params: {} 2024-04-13T22:27:51.597 DEBUG:boto:Token: None 2024-04-13T22:27:51.597 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:27:51.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI= 2024-04-13T22:27:51.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:enZymiWnHrSpO+XfO5zX3gsC8DI=', 'Content-Length': '0'} 2024-04-13T22:27:51.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b540c46a895a845-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.599 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.600 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.600 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.600 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.600 DEBUG:boto:Method: GET 2024-04-13T22:27:51.600 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.600 DEBUG:boto:Data: 2024-04-13T22:27:51.600 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.600 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.600 DEBUG:boto:Params: {} 2024-04-13T22:27:51.600 DEBUG:boto:Token: None 2024-04-13T22:27:51.601 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23e6355b2e0df5b-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.603 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.603 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.603 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.603 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.604 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.604 DEBUG:boto:Method: GET 2024-04-13T22:27:51.604 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.604 DEBUG:boto:Data: 2024-04-13T22:27:51.604 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.604 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.604 DEBUG:boto:Params: {} 2024-04-13T22:27:51.604 DEBUG:boto:Token: None 2024-04-13T22:27:51.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.605 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003efcf54326a133eb-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.608 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.608 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.608 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.608 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.608 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:27:51.608 DEBUG:boto:Data: 2024-04-13T22:27:51.608 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.609 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.609 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.609 DEBUG:boto:Params: {} 2024-04-13T22:27:51.609 DEBUG:boto:Token: None 2024-04-13T22:27:51.609 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85 2024-04-13T22:27:51.609 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SJ1AbeCCVoBGiWouIIrsbmpiD8o= 2024-04-13T22:27:51.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SJ1AbeCCVoBGiWouIIrsbmpiD8o=', 'Content-Length': '0'} 2024-04-13T22:27:51.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9e43569635bfad-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.611 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.611 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.611 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.611 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:27:51.611 DEBUG:boto:Data: 2024-04-13T22:27:51.611 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.611 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.611 DEBUG:boto:Params: {} 2024-04-13T22:27:51.612 DEBUG:boto:Token: None 2024-04-13T22:27:51.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85 2024-04-13T22:27:51.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SJ1AbeCCVoBGiWouIIrsbmpiD8o= 2024-04-13T22:27:51.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SJ1AbeCCVoBGiWouIIrsbmpiD8o=', 'Content-Length': '0'} 2024-04-13T22:27:51.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004977a7aef766578a-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:27:51.614 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.614 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.614 DEBUG:boto:Method: GET 2024-04-13T22:27:51.614 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:27:51.614 DEBUG:boto:Data: 2024-04-13T22:27:51.614 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.614 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.614 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.614 DEBUG:boto:Params: {} 2024-04-13T22:27:51.615 DEBUG:boto:Token: None 2024-04-13T22:27:51.615 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85 2024-04-13T22:27:51.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SsJZ42OilqJB2u83G8a/qGzmxDM= 2024-04-13T22:27:51.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SsJZ42OilqJB2u83G8a/qGzmxDM=', 'Content-Length': '0'} 2024-04-13T22:27:51.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb8a577e00d90f4-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.658 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:27:51.658 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:27:51.658 DEBUG:boto:Method: GET 2024-04-13T22:27:51.659 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:27:51.659 DEBUG:boto:Data: 2024-04-13T22:27:51.659 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.659 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.659 DEBUG:boto:Params: {} 2024-04-13T22:27:51.659 DEBUG:boto:Token: None 2024-04-13T22:27:51.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj85 2024-04-13T22:27:51.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SsJZ42OilqJB2u83G8a/qGzmxDM= 2024-04-13T22:27:51.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SsJZ42OilqJB2u83G8a/qGzmxDM=', 'Content-Length': '0'} 2024-04-13T22:27:51.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6ada9ba2c114e9-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.661 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.662 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.662 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.662 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.662 DEBUG:boto:Method: GET 2024-04-13T22:27:51.662 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.662 DEBUG:boto:Data: 2024-04-13T22:27:51.662 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.662 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.662 DEBUG:boto:Params: {} 2024-04-13T22:27:51.663 DEBUG:boto:Token: None 2024-04-13T22:27:51.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000897aca4593026126-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.665 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.666 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.666 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.666 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.666 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.666 DEBUG:boto:Method: GET 2024-04-13T22:27:51.666 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:27:51.666 DEBUG:boto:Data: 2024-04-13T22:27:51.666 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.667 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.667 DEBUG:boto:Params: {} 2024-04-13T22:27:51.667 DEBUG:boto:Token: None 2024-04-13T22:27:51.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000301470e2dbad7f3b-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.670 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:27:51.671 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.671 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.671 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.671 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.671 DEBUG:boto:Method: GET 2024-04-13T22:27:51.671 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.671 DEBUG:boto:Data: 2024-04-13T22:27:51.671 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.671 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.671 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.672 DEBUG:boto:Params: {} 2024-04-13T22:27:51.672 DEBUG:boto:Token: None 2024-04-13T22:27:51.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY= 2024-04-13T22:27:51.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY=', 'Content-Length': '0'} 2024-04-13T22:27:51.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009822812222106311-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.674 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.674 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.674 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.674 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.674 DEBUG:boto:Method: GET 2024-04-13T22:27:51.674 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.674 DEBUG:boto:Data: 2024-04-13T22:27:51.675 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.675 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.675 DEBUG:boto:Params: {} 2024-04-13T22:27:51.675 DEBUG:boto:Token: None 2024-04-13T22:27:51.675 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.675 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY= 2024-04-13T22:27:51.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY=', 'Content-Length': '0'} 2024-04-13T22:27:51.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a452b581d62535-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.677 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.677 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.677 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.677 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.677 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.678 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.678 DEBUG:boto:Data: 2024-04-13T22:27:51.678 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.678 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.678 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.678 DEBUG:boto:Params: {} 2024-04-13T22:27:51.678 DEBUG:boto:Token: None 2024-04-13T22:27:51.678 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.678 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ni/wMP+abPbx6WuPqg9ALBuZnBI= 2024-04-13T22:27:51.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ni/wMP+abPbx6WuPqg9ALBuZnBI=', 'Content-Length': '0'} 2024-04-13T22:27:51.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1810df61976527e-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.680 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.680 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.681 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.681 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.681 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.681 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.681 DEBUG:boto:Data: 2024-04-13T22:27:51.681 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.681 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.681 DEBUG:boto:Params: {} 2024-04-13T22:27:51.681 DEBUG:boto:Token: None 2024-04-13T22:27:51.681 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ni/wMP+abPbx6WuPqg9ALBuZnBI= 2024-04-13T22:27:51.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ni/wMP+abPbx6WuPqg9ALBuZnBI=', 'Content-Length': '0'} 2024-04-13T22:27:51.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c99aee8c902198-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:27:51.683 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.684 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.684 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.684 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.684 DEBUG:boto:Method: GET 2024-04-13T22:27:51.684 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.684 DEBUG:boto:Data: 2024-04-13T22:27:51.684 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.684 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.684 DEBUG:boto:Params: {} 2024-04-13T22:27:51.685 DEBUG:boto:Token: None 2024-04-13T22:27:51.685 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY= 2024-04-13T22:27:51.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY=', 'Content-Length': '0'} 2024-04-13T22:27:51.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030683bb7d4b12023-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.687 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.687 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.687 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.687 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.687 DEBUG:boto:Method: GET 2024-04-13T22:27:51.687 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.687 DEBUG:boto:Data: 2024-04-13T22:27:51.687 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.688 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.688 DEBUG:boto:Params: {} 2024-04-13T22:27:51.688 DEBUG:boto:Token: None 2024-04-13T22:27:51.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:27:51.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY= 2024-04-13T22:27:51.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DNGooW0IxjeY+XKK3oTUPYgJZnY=', 'Content-Length': '0'} 2024-04-13T22:27:51.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046725a3fe13de3b5-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.690 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.690 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.690 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.690 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.690 DEBUG:boto:Method: GET 2024-04-13T22:27:51.690 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.690 DEBUG:boto:Data: 2024-04-13T22:27:51.690 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.690 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.691 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.691 DEBUG:boto:Params: {} 2024-04-13T22:27:51.691 DEBUG:boto:Token: None 2024-04-13T22:27:51.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcfecb4a43b15635-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.693 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.694 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.694 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.694 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.694 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.694 DEBUG:boto:Method: GET 2024-04-13T22:27:51.694 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.694 DEBUG:boto:Data: 2024-04-13T22:27:51.694 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.695 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.695 DEBUG:boto:Params: {} 2024-04-13T22:27:51.695 DEBUG:boto:Token: None 2024-04-13T22:27:51.695 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.695 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a37556824b4ef50-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.698 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.699 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.699 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.699 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.699 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:27:51.699 DEBUG:boto:Data: 2024-04-13T22:27:51.699 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.699 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.700 DEBUG:boto:Params: {} 2024-04-13T22:27:51.700 DEBUG:boto:Token: None 2024-04-13T22:27:51.700 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87 2024-04-13T22:27:51.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qdIAdfr3Ad6H+r3GK7BbxmMQ7SU= 2024-04-13T22:27:51.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qdIAdfr3Ad6H+r3GK7BbxmMQ7SU=', 'Content-Length': '0'} 2024-04-13T22:27:51.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f571d348321a6a7-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.702 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.702 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.702 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.702 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:27:51.702 DEBUG:boto:Data: 2024-04-13T22:27:51.702 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.702 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.702 DEBUG:boto:Params: {} 2024-04-13T22:27:51.703 DEBUG:boto:Token: None 2024-04-13T22:27:51.703 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87 2024-04-13T22:27:51.703 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qdIAdfr3Ad6H+r3GK7BbxmMQ7SU= 2024-04-13T22:27:51.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qdIAdfr3Ad6H+r3GK7BbxmMQ7SU=', 'Content-Length': '0'} 2024-04-13T22:27:51.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2acc3db6e59012-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:27:51.705 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.705 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.705 DEBUG:boto:Method: GET 2024-04-13T22:27:51.705 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:27:51.705 DEBUG:boto:Data: 2024-04-13T22:27:51.705 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.705 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.705 DEBUG:boto:Params: {} 2024-04-13T22:27:51.705 DEBUG:boto:Token: None 2024-04-13T22:27:51.705 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87 2024-04-13T22:27:51.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MZuQoIldERlNOm/DGhpYPfovpyY= 2024-04-13T22:27:51.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MZuQoIldERlNOm/DGhpYPfovpyY=', 'Content-Length': '0'} 2024-04-13T22:27:51.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221544903c3c27e1-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.707 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:27:51.707 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:27:51.707 DEBUG:boto:Method: GET 2024-04-13T22:27:51.707 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:27:51.708 DEBUG:boto:Data: 2024-04-13T22:27:51.708 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.708 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.708 DEBUG:boto:Params: {} 2024-04-13T22:27:51.708 DEBUG:boto:Token: None 2024-04-13T22:27:51.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj87 2024-04-13T22:27:51.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MZuQoIldERlNOm/DGhpYPfovpyY= 2024-04-13T22:27:51.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MZuQoIldERlNOm/DGhpYPfovpyY=', 'Content-Length': '0'} 2024-04-13T22:27:51.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008074814bbfd951a0-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.710 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.710 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.710 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.711 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.711 DEBUG:boto:Method: GET 2024-04-13T22:27:51.711 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.711 DEBUG:boto:Data: 2024-04-13T22:27:51.711 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.711 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.712 DEBUG:boto:Params: {} 2024-04-13T22:27:51.712 DEBUG:boto:Token: None 2024-04-13T22:27:51.712 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ef6bf483736cc76-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.714 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.715 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.715 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.715 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.715 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.715 DEBUG:boto:Method: GET 2024-04-13T22:27:51.715 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:27:51.715 DEBUG:boto:Data: 2024-04-13T22:27:51.715 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.716 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.716 DEBUG:boto:Params: {} 2024-04-13T22:27:51.716 DEBUG:boto:Token: None 2024-04-13T22:27:51.716 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541daa38feafbc5a-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.719 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:27:51.720 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.720 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.720 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.720 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.720 DEBUG:boto:Method: GET 2024-04-13T22:27:51.720 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.720 DEBUG:boto:Data: 2024-04-13T22:27:51.720 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.720 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.720 DEBUG:boto:Params: {} 2024-04-13T22:27:51.721 DEBUG:boto:Token: None 2024-04-13T22:27:51.721 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg= 2024-04-13T22:27:51.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg=', 'Content-Length': '0'} 2024-04-13T22:27:51.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023f17d5776f2a9f5-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.766 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.767 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.767 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.767 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.767 DEBUG:boto:Method: GET 2024-04-13T22:27:51.767 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.767 DEBUG:boto:Data: 2024-04-13T22:27:51.767 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.767 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.767 DEBUG:boto:Params: {} 2024-04-13T22:27:51.768 DEBUG:boto:Token: None 2024-04-13T22:27:51.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg= 2024-04-13T22:27:51.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg=', 'Content-Length': '0'} 2024-04-13T22:27:51.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8bdef2d784e25ed-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.770 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.770 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.770 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.770 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.770 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.770 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.770 DEBUG:boto:Data: 2024-04-13T22:27:51.771 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.771 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.771 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.771 DEBUG:boto:Params: {} 2024-04-13T22:27:51.771 DEBUG:boto:Token: None 2024-04-13T22:27:51.771 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.771 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:onSJ7YNdBh0XxsBkmtlJbUcIdAk= 2024-04-13T22:27:51.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:onSJ7YNdBh0XxsBkmtlJbUcIdAk=', 'Content-Length': '0'} 2024-04-13T22:27:51.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268db4aba86aa6ea-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.773 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.773 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.773 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.773 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.773 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.773 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.773 DEBUG:boto:Data: 2024-04-13T22:27:51.774 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.774 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.774 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.774 DEBUG:boto:Params: {} 2024-04-13T22:27:51.774 DEBUG:boto:Token: None 2024-04-13T22:27:51.774 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.774 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:onSJ7YNdBh0XxsBkmtlJbUcIdAk= 2024-04-13T22:27:51.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:onSJ7YNdBh0XxsBkmtlJbUcIdAk=', 'Content-Length': '0'} 2024-04-13T22:27:51.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff07d7b8bbb7eba-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:27:51.776 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.776 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.776 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.776 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.776 DEBUG:boto:Method: GET 2024-04-13T22:27:51.776 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.777 DEBUG:boto:Data: 2024-04-13T22:27:51.777 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.777 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.777 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.777 DEBUG:boto:Params: {} 2024-04-13T22:27:51.777 DEBUG:boto:Token: None 2024-04-13T22:27:51.777 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.777 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg= 2024-04-13T22:27:51.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg=', 'Content-Length': '0'} 2024-04-13T22:27:51.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b25556c1381a4c5-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.779 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.779 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.779 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.779 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.779 DEBUG:boto:Method: GET 2024-04-13T22:27:51.780 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.780 DEBUG:boto:Data: 2024-04-13T22:27:51.780 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.780 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.780 DEBUG:boto:Params: {} 2024-04-13T22:27:51.780 DEBUG:boto:Token: None 2024-04-13T22:27:51.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:27:51.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg= 2024-04-13T22:27:51.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iW28PUK9uhkjA9odWcWu8s775Rg=', 'Content-Length': '0'} 2024-04-13T22:27:51.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4dfe2f9b9e10585-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.782 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.782 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.783 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.783 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.783 DEBUG:boto:Method: GET 2024-04-13T22:27:51.783 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.783 DEBUG:boto:Data: 2024-04-13T22:27:51.783 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.783 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.783 DEBUG:boto:Params: {} 2024-04-13T22:27:51.783 DEBUG:boto:Token: None 2024-04-13T22:27:51.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021de7bde23b30e35-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.786 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.786 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.786 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.786 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.787 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.787 DEBUG:boto:Method: GET 2024-04-13T22:27:51.787 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.787 DEBUG:boto:Data: 2024-04-13T22:27:51.787 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.787 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.787 DEBUG:boto:Params: {} 2024-04-13T22:27:51.787 DEBUG:boto:Token: None 2024-04-13T22:27:51.787 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.788 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000104baa976e2b28a8-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.791 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.791 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.791 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.791 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.791 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:27:51.791 DEBUG:boto:Data: 2024-04-13T22:27:51.791 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.791 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.792 DEBUG:boto:Params: {} 2024-04-13T22:27:51.792 DEBUG:boto:Token: None 2024-04-13T22:27:51.792 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92 2024-04-13T22:27:51.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F7g1EQkMVE9EEXr5dHWHh8u/GGE= 2024-04-13T22:27:51.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F7g1EQkMVE9EEXr5dHWHh8u/GGE=', 'Content-Length': '0'} 2024-04-13T22:27:51.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b2e12584eb5a925-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.794 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.794 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.794 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.794 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:27:51.794 DEBUG:boto:Data: 2024-04-13T22:27:51.794 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.794 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.795 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.795 DEBUG:boto:Params: {} 2024-04-13T22:27:51.795 DEBUG:boto:Token: None 2024-04-13T22:27:51.795 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92 2024-04-13T22:27:51.795 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F7g1EQkMVE9EEXr5dHWHh8u/GGE= 2024-04-13T22:27:51.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F7g1EQkMVE9EEXr5dHWHh8u/GGE=', 'Content-Length': '0'} 2024-04-13T22:27:51.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006591a5ea64026410-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:27:51.797 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.797 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.797 DEBUG:boto:Method: GET 2024-04-13T22:27:51.797 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:27:51.797 DEBUG:boto:Data: 2024-04-13T22:27:51.797 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.797 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.798 DEBUG:boto:Params: {} 2024-04-13T22:27:51.798 DEBUG:boto:Token: None 2024-04-13T22:27:51.798 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92 2024-04-13T22:27:51.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OrRAv5RmMR2NeNB4B5qhGNyQ5DE= 2024-04-13T22:27:51.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OrRAv5RmMR2NeNB4B5qhGNyQ5DE=', 'Content-Length': '0'} 2024-04-13T22:27:51.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8e17e7ed516ea2-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.800 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:27:51.800 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:27:51.800 DEBUG:boto:Method: GET 2024-04-13T22:27:51.800 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:27:51.800 DEBUG:boto:Data: 2024-04-13T22:27:51.801 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.801 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.801 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.801 DEBUG:boto:Params: {} 2024-04-13T22:27:51.801 DEBUG:boto:Token: None 2024-04-13T22:27:51.801 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj92 2024-04-13T22:27:51.801 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OrRAv5RmMR2NeNB4B5qhGNyQ5DE= 2024-04-13T22:27:51.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OrRAv5RmMR2NeNB4B5qhGNyQ5DE=', 'Content-Length': '0'} 2024-04-13T22:27:51.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecf46c9b123e7afe-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.803 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.803 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.803 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.803 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.803 DEBUG:boto:Method: GET 2024-04-13T22:27:51.803 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.803 DEBUG:boto:Data: 2024-04-13T22:27:51.804 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.804 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.804 DEBUG:boto:Params: {} 2024-04-13T22:27:51.804 DEBUG:boto:Token: None 2024-04-13T22:27:51.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f7e3525efdb323-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.806 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.807 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.807 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.807 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.807 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.807 DEBUG:boto:Method: GET 2024-04-13T22:27:51.807 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:27:51.807 DEBUG:boto:Data: 2024-04-13T22:27:51.807 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.807 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.807 DEBUG:boto:Params: {} 2024-04-13T22:27:51.807 DEBUG:boto:Token: None 2024-04-13T22:27:51.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b721b752b0914007-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.854 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:27:51.855 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.855 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.855 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.855 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.855 DEBUG:boto:Method: GET 2024-04-13T22:27:51.855 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.855 DEBUG:boto:Data: 2024-04-13T22:27:51.855 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.856 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.856 DEBUG:boto:Params: {} 2024-04-13T22:27:51.856 DEBUG:boto:Token: None 2024-04-13T22:27:51.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc= 2024-04-13T22:27:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc=', 'Content-Length': '0'} 2024-04-13T22:27:51.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d3bb3ef207c376-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.858 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.858 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.858 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.858 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.858 DEBUG:boto:Method: GET 2024-04-13T22:27:51.859 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.859 DEBUG:boto:Data: 2024-04-13T22:27:51.859 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.859 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.859 DEBUG:boto:Params: {} 2024-04-13T22:27:51.859 DEBUG:boto:Token: None 2024-04-13T22:27:51.859 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc= 2024-04-13T22:27:51.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc=', 'Content-Length': '0'} 2024-04-13T22:27:51.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001709a59d9889d14e-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.861 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.861 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.861 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.862 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.862 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.862 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.862 DEBUG:boto:Data: 2024-04-13T22:27:51.862 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.862 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.862 DEBUG:boto:Params: {} 2024-04-13T22:27:51.862 DEBUG:boto:Token: None 2024-04-13T22:27:51.862 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zy4292B1Js80pPOJ2f0GzRPd7YY= 2024-04-13T22:27:51.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zy4292B1Js80pPOJ2f0GzRPd7YY=', 'Content-Length': '0'} 2024-04-13T22:27:51.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f490d9dcd1c843-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.864 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.864 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.864 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.865 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.865 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.865 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.865 DEBUG:boto:Data: 2024-04-13T22:27:51.865 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.865 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.865 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.865 DEBUG:boto:Params: {} 2024-04-13T22:27:51.865 DEBUG:boto:Token: None 2024-04-13T22:27:51.865 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.866 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zy4292B1Js80pPOJ2f0GzRPd7YY= 2024-04-13T22:27:51.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zy4292B1Js80pPOJ2f0GzRPd7YY=', 'Content-Length': '0'} 2024-04-13T22:27:51.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019837e192989dbb1-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:27:51.867 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.868 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.868 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.868 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.868 DEBUG:boto:Method: GET 2024-04-13T22:27:51.868 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.868 DEBUG:boto:Data: 2024-04-13T22:27:51.868 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.868 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.868 DEBUG:boto:Params: {} 2024-04-13T22:27:51.868 DEBUG:boto:Token: None 2024-04-13T22:27:51.869 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc= 2024-04-13T22:27:51.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc=', 'Content-Length': '0'} 2024-04-13T22:27:51.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbe58de8c82492b-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.870 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.871 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.871 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.871 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.871 DEBUG:boto:Method: GET 2024-04-13T22:27:51.871 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.871 DEBUG:boto:Data: 2024-04-13T22:27:51.871 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.871 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.871 DEBUG:boto:Params: {} 2024-04-13T22:27:51.871 DEBUG:boto:Token: None 2024-04-13T22:27:51.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:27:51.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc= 2024-04-13T22:27:51.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XxTtuOuKxnrHvqEu3NGlRvVcflc=', 'Content-Length': '0'} 2024-04-13T22:27:51.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431f8308072a43c2-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.874 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.874 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.874 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.874 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.875 DEBUG:boto:Method: GET 2024-04-13T22:27:51.875 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.875 DEBUG:boto:Data: 2024-04-13T22:27:51.875 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.875 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.875 DEBUG:boto:Params: {} 2024-04-13T22:27:51.875 DEBUG:boto:Token: None 2024-04-13T22:27:51.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005693b8ee9e569e3-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.922 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.923 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.923 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.923 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.923 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.923 DEBUG:boto:Method: GET 2024-04-13T22:27:51.923 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.923 DEBUG:boto:Data: 2024-04-13T22:27:51.924 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.924 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.924 DEBUG:boto:Params: {} 2024-04-13T22:27:51.924 DEBUG:boto:Token: None 2024-04-13T22:27:51.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33bd744ae77b075-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.927 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.928 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.928 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.928 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.928 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:27:51.928 DEBUG:boto:Data: 2024-04-13T22:27:51.928 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.928 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.928 DEBUG:boto:Params: {} 2024-04-13T22:27:51.929 DEBUG:boto:Token: None 2024-04-13T22:27:51.929 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95 2024-04-13T22:27:51.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kWMtMgT0Vy0pRarmlU9VZgk2HWw= 2024-04-13T22:27:51.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kWMtMgT0Vy0pRarmlU9VZgk2HWw=', 'Content-Length': '0'} 2024-04-13T22:27:51.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c45700914695906-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.931 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.931 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.931 DEBUG:boto:Method: HEAD 2024-04-13T22:27:51.931 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:27:51.931 DEBUG:boto:Data: 2024-04-13T22:27:51.931 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.932 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.932 DEBUG:boto:Params: {} 2024-04-13T22:27:51.932 DEBUG:boto:Token: None 2024-04-13T22:27:51.932 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95 2024-04-13T22:27:51.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kWMtMgT0Vy0pRarmlU9VZgk2HWw= 2024-04-13T22:27:51.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kWMtMgT0Vy0pRarmlU9VZgk2HWw=', 'Content-Length': '0'} 2024-04-13T22:27:51.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8ae4dfa9453710-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:27:51.934 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.934 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.934 DEBUG:boto:Method: GET 2024-04-13T22:27:51.934 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:27:51.934 DEBUG:boto:Data: 2024-04-13T22:27:51.934 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.935 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.935 DEBUG:boto:Params: {} 2024-04-13T22:27:51.935 DEBUG:boto:Token: None 2024-04-13T22:27:51.935 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95 2024-04-13T22:27:51.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yh3KFQ6hcbnRPy/MaIRf4XSZGB4= 2024-04-13T22:27:51.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yh3KFQ6hcbnRPy/MaIRf4XSZGB4=', 'Content-Length': '0'} 2024-04-13T22:27:51.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a553fccc44a62dbd-00661b06e7-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.937 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:27:51.937 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:27:51.937 DEBUG:boto:Method: GET 2024-04-13T22:27:51.937 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:27:51.937 DEBUG:boto:Data: 2024-04-13T22:27:51.938 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.938 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.938 DEBUG:boto:Params: {} 2024-04-13T22:27:51.938 DEBUG:boto:Token: None 2024-04-13T22:27:51.938 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/obj95 2024-04-13T22:27:51.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yh3KFQ6hcbnRPy/MaIRf4XSZGB4= 2024-04-13T22:27:51.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yh3KFQ6hcbnRPy/MaIRf4XSZGB4=', 'Content-Length': '0'} 2024-04-13T22:27:51.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8edc93b02f5607-00661b06e7-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.940 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.940 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.940 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.940 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.940 DEBUG:boto:Method: GET 2024-04-13T22:27:51.940 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.941 DEBUG:boto:Data: 2024-04-13T22:27:51.941 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:27:51.941 DEBUG:boto:Port: 8001 2024-04-13T22:27:51.941 DEBUG:boto:Params: {} 2024-04-13T22:27:51.941 DEBUG:boto:Token: None 2024-04-13T22:27:51.941 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.941 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a618ce28b4c0446-00661b06e7-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.943 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.944 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:51.944 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:51.944 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.944 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.944 DEBUG:boto:Method: GET 2024-04-13T22:27:51.944 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:27:51.944 DEBUG:boto:Data: 2024-04-13T22:27:51.945 DEBUG:boto:Headers: {} 2024-04-13T22:27:51.945 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:51.945 DEBUG:boto:Port: 8000 2024-04-13T22:27:51.945 DEBUG:boto:Params: {} 2024-04-13T22:27:51.945 DEBUG:boto:Token: None 2024-04-13T22:27:51.945 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:27:51 GMT /vwnfns-61/ 2024-04-13T22:27:51.945 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI= 2024-04-13T22:27:51.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:51 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aYeh3Rg0GGwpVQbifk1eHSkAULI=', 'Content-Length': '0'} 2024-04-13T22:27:51.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda8a94b01661697-00661b06e7-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:51.948 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:27:51.949 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:27:51.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:27:51.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:27:51.949 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:27:52.001 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:27:52.002 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:27:52.048 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:27:52.048 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-61 2024-04-13T22:27:52.058 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 25 2024-04-13T22:27:52.115 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:52.115 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:52.115 DEBUG:boto:Method: HEAD 2024-04-13T22:27:52.115 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:52.115 DEBUG:boto:Data: 2024-04-13T22:27:52.116 DEBUG:boto:Headers: {} 2024-04-13T22:27:52.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.116 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.116 DEBUG:boto:Params: {} 2024-04-13T22:27:52.116 DEBUG:boto:Token: None 2024-04-13T22:27:52.116 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/ 2024-04-13T22:27:52.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs= 2024-04-13T22:27:52.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs=', 'Content-Length': '0'} 2024-04-13T22:27:52.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095c1180b1920f954-00661b06e8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.120 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:27:52.120 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:27:52.121 DEBUG:boto:Method: PUT 2024-04-13T22:27:52.121 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:27:52.121 DEBUG:boto:Data: 2024-04-13T22:27:52.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:27:52.121 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.121 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.121 DEBUG:boto:Params: {} 2024-04-13T22:27:52.121 DEBUG:boto:Token: None 2024-04-13T22:27:52.121 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/obj31 2024-04-13T22:27:52.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oIgav1UxHDpw83tmwurVE+vOfH8= 2024-04-13T22:27:52.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oIgav1UxHDpw83tmwurVE+vOfH8='} 2024-04-13T22:27:52.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037d399f06af14d94-00661b06e8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.167 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:52.167 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:52.167 DEBUG:boto:Method: HEAD 2024-04-13T22:27:52.167 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:52.167 DEBUG:boto:Data: 2024-04-13T22:27:52.167 DEBUG:boto:Headers: {} 2024-04-13T22:27:52.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.167 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.167 DEBUG:boto:Params: {} 2024-04-13T22:27:52.168 DEBUG:boto:Token: None 2024-04-13T22:27:52.168 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/ 2024-04-13T22:27:52.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs= 2024-04-13T22:27:52.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs=', 'Content-Length': '0'} 2024-04-13T22:27:52.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000781217667735fa88-00661b06e8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.171 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:27:52.172 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:27:52.172 DEBUG:boto:Method: PUT 2024-04-13T22:27:52.172 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:27:52.172 DEBUG:boto:Data: 2024-04-13T22:27:52.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:27:52.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.172 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.172 DEBUG:boto:Params: {} 2024-04-13T22:27:52.172 DEBUG:boto:Token: None 2024-04-13T22:27:52.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/obj62 2024-04-13T22:27:52.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5Qs4I33xnrahAw8kR1beWQz8zcQ= 2024-04-13T22:27:52.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5Qs4I33xnrahAw8kR1beWQz8zcQ='} 2024-04-13T22:27:52.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b65229b6bf1b7c1a-00661b06e8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.178 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:52.178 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:52.178 DEBUG:boto:Method: HEAD 2024-04-13T22:27:52.178 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:52.178 DEBUG:boto:Data: 2024-04-13T22:27:52.179 DEBUG:boto:Headers: {} 2024-04-13T22:27:52.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.179 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.179 DEBUG:boto:Params: {} 2024-04-13T22:27:52.179 DEBUG:boto:Token: None 2024-04-13T22:27:52.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/ 2024-04-13T22:27:52.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs= 2024-04-13T22:27:52.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs=', 'Content-Length': '0'} 2024-04-13T22:27:52.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce06f6149df3033e-00661b06e8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.182 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:27:52.183 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:27:52.183 DEBUG:boto:Method: PUT 2024-04-13T22:27:52.183 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:27:52.183 DEBUG:boto:Data: 2024-04-13T22:27:52.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:27:52.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.183 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.183 DEBUG:boto:Params: {} 2024-04-13T22:27:52.183 DEBUG:boto:Token: None 2024-04-13T22:27:52.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/obj93 2024-04-13T22:27:52.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ah7Vv+DxU+koCznSzJraBvud0YI= 2024-04-13T22:27:52.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ah7Vv+DxU+koCznSzJraBvud0YI='} 2024-04-13T22:27:52.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd0a2be7f96bcbde-00661b06e8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.231 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:52.231 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:52.232 DEBUG:boto:Method: HEAD 2024-04-13T22:27:52.232 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:52.232 DEBUG:boto:Data: 2024-04-13T22:27:52.232 DEBUG:boto:Headers: {} 2024-04-13T22:27:52.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.232 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.232 DEBUG:boto:Params: {} 2024-04-13T22:27:52.233 DEBUG:boto:Token: None 2024-04-13T22:27:52.233 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/ 2024-04-13T22:27:52.233 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs= 2024-04-13T22:27:52.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs=', 'Content-Length': '0'} 2024-04-13T22:27:52.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fa94876c7043986-00661b06e8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.236 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:27:52.236 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:27:52.236 DEBUG:boto:Method: PUT 2024-04-13T22:27:52.236 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:27:52.236 DEBUG:boto:Data: 2024-04-13T22:27:52.236 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:27:52.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.237 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.237 DEBUG:boto:Params: {} 2024-04-13T22:27:52.237 DEBUG:boto:Token: None 2024-04-13T22:27:52.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/obj124 2024-04-13T22:27:52.237 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E2R3yaakHA9sOYPgt/dzFZscJnc= 2024-04-13T22:27:52.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E2R3yaakHA9sOYPgt/dzFZscJnc='} 2024-04-13T22:27:52.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cf268ec69c4bfcc-00661b06e8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.243 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:27:52.243 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:27:52.244 DEBUG:boto:Method: HEAD 2024-04-13T22:27:52.244 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:27:52.244 DEBUG:boto:Data: 2024-04-13T22:27:52.244 DEBUG:boto:Headers: {} 2024-04-13T22:27:52.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.245 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.245 DEBUG:boto:Params: {} 2024-04-13T22:27:52.245 DEBUG:boto:Token: None 2024-04-13T22:27:52.245 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/ 2024-04-13T22:27:52.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs= 2024-04-13T22:27:52.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Hb14I+bdbwawetJeewJxKYg5HRs=', 'Content-Length': '0'} 2024-04-13T22:27:52.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030deea6db2668193-00661b06e8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.250 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:27:52.250 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:27:52.250 DEBUG:boto:Method: PUT 2024-04-13T22:27:52.250 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:27:52.251 DEBUG:boto:Data: 2024-04-13T22:27:52.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:27:52.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:27:52.251 DEBUG:boto:Port: 8000 2024-04-13T22:27:52.251 DEBUG:boto:Params: {} 2024-04-13T22:27:52.252 DEBUG:boto:Token: None 2024-04-13T22:27:52.252 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:27:52 GMT /vwnfns-61/obj155 2024-04-13T22:27:52.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pQu0yvwcgGLHI6/KO2NF1sxmauU= 2024-04-13T22:27:52.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:27:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pQu0yvwcgGLHI6/KO2NF1sxmauU='} 2024-04-13T22:27:52.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4aea8fbfaf9898c-00661b06e8-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:27:52.299 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:27:52.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:27:52.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:27:52.300 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:27:52.355 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:27:52.355 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:27:52.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:52.388+0000 7f4029a13780 1 waiting to reach latest gen 5, on 4.. 2024-04-13T22:27:57.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:27:57.389+0000 7f4029a13780 1 waiting to reach latest gen 5, on 4.. 2024-04-13T22:28:02.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:02.388+0000 7f4029a13780 1 waiting to reach latest gen 5, on 4.. 2024-04-13T22:28:07.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:07.389+0000 7f4029a13780 1 waiting to reach latest gen 5, on 4.. 2024-04-13T22:28:12.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:12.392+0000 7f4029a13780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:12.396 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:28:12.396 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:17.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:17.394+0000 7f4029a13780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:17.398 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:28:17.398 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:22.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:22.397+0000 7f4029a13780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:22.400 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:28:22.400 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:27.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:27.399+0000 7f4029a13780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:27.403 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-13T22:28:27.403 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:32.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:32.402+0000 7f4029a13780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:32.405 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , 00000000001.63.6, , , , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , ] 2024-04-13T22:28:32.405 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:37.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:37.404+0000 7f4029a13780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:37.407 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , 00000000001.63.6, , , , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , ] 2024-04-13T22:28:37.407 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:42.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:42.405+0000 7f4029a13780 1 bucket sync caught up with source: 2024-04-13T22:28:42.409 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:42.409 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , 00000000002.58.6, , 00000000001.63.6, , 00000000001.99.6, , , , , , , , , , 00000000001.80.6, , , , , , , , , , , , , , 00000000001.101.6] 2024-04-13T22:28:42.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:42.405+0000 7f4029a13780 0 bucket checkpoint complete 2024-04-13T22:28:42.426 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:28:42.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:28:42.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:28:42.427 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:28:42.486 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:28:42.486 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:28:42.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:42.519+0000 7fa61c92f780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:42.523 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000002.2050.6, 00000000002.2294.6, 00000000001.3328.6, 00000000003.1213.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000002.624.6, 00000000005.1904.6, 00000000004.1750.6] 2024-04-13T22:28:42.523 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000004.1217.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000003.635.6, 00000000006.1910.6, 00000000004.1750.6] 2024-04-13T22:28:47.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:47.520+0000 7fa61c92f780 1 waiting for incremental sync to catch up: 2024-04-13T22:28:47.524 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000003.2072.6, 00000000002.2294.6, 00000000001.3328.6, 00000000004.1217.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000003.635.6, 00000000006.1910.6, 00000000004.1750.6] 2024-04-13T22:28:47.525 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000004.1217.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000003.635.6, 00000000006.1910.6, 00000000004.1750.6] 2024-04-13T22:28:52.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:52.522+0000 7fa61c92f780 1 bucket sync caught up with source: 2024-04-13T22:28:52.526 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000004.1217.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000003.635.6, 00000000006.1910.6, 00000000004.1750.6] 2024-04-13T22:28:52.526 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000004.1217.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000003.635.6, 00000000006.1910.6, 00000000004.1750.6] 2024-04-13T22:28:52.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:52.522+0000 7fa61c92f780 0 bucket checkpoint complete 2024-04-13T22:28:52.542 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:28:52.542 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.542 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.542 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.543 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:52.543 DEBUG:boto:Data: 2024-04-13T22:28:52.543 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.543 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.543 DEBUG:boto:Params: {} 2024-04-13T22:28:52.543 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:28:52.544 DEBUG:boto:Token: None 2024-04-13T22:28:52.544 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MVWUndI1TREgyLVU4H6adzTbiJY= 2024-04-13T22:28:52.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MVWUndI1TREgyLVU4H6adzTbiJY=', 'Content-Length': '0'} 2024-04-13T22:28:52.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa2ec520a06758a5-00661b0724-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.549 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.549 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.549 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.550 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:52.550 DEBUG:boto:Data: 2024-04-13T22:28:52.550 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.550 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.550 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.550 DEBUG:boto:Params: {} 2024-04-13T22:28:52.550 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:28:52.550 DEBUG:boto:Token: None 2024-04-13T22:28:52.551 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MVWUndI1TREgyLVU4H6adzTbiJY= 2024-04-13T22:28:52.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MVWUndI1TREgyLVU4H6adzTbiJY=', 'Content-Length': '0'} 2024-04-13T22:28:52.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edc7e1c08450892e-00661b0724-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:28:52.556 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.556 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.556 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.556 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.557 DEBUG:boto:Method: GET 2024-04-13T22:28:52.557 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.557 DEBUG:boto:Data: 2024-04-13T22:28:52.557 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.557 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.557 DEBUG:boto:Params: {} 2024-04-13T22:28:52.557 DEBUG:boto:Token: None 2024-04-13T22:28:52.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/?versions 2024-04-13T22:28:52.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY= 2024-04-13T22:28:52.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY=', 'Content-Length': '0'} 2024-04-13T22:28:52.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c16ed459a434b9-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.560 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:28:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:28:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:28:52.565 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:28:52.565 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.566 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.566 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.566 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.566 DEBUG:boto:Method: GET 2024-04-13T22:28:52.566 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.566 DEBUG:boto:Data: 2024-04-13T22:28:52.566 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.566 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.566 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.566 DEBUG:boto:Params: {} 2024-04-13T22:28:52.566 DEBUG:boto:Token: None 2024-04-13T22:28:52.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/?versions 2024-04-13T22:28:52.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY= 2024-04-13T22:28:52.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY=', 'Content-Length': '0'} 2024-04-13T22:28:52.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a64af3f2705089-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.570 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:28:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:28:52.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-13T22:28:52.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:28:52.574 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.575 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.575 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.575 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.575 DEBUG:boto:Method: GET 2024-04-13T22:28:52.575 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.575 DEBUG:boto:Data: 2024-04-13T22:28:52.575 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.575 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.575 DEBUG:boto:Params: {} 2024-04-13T22:28:52.576 DEBUG:boto:Token: None 2024-04-13T22:28:52.576 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/?versions 2024-04-13T22:28:52.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY= 2024-04-13T22:28:52.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY=', 'Content-Length': '0'} 2024-04-13T22:28:52.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000821f60beab95c394-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.578 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.580 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.580 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.580 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.580 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.580 DEBUG:boto:Method: GET 2024-04-13T22:28:52.580 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:28:52.580 DEBUG:boto:Data: 2024-04-13T22:28:52.581 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.581 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.581 DEBUG:boto:Params: {} 2024-04-13T22:28:52.581 DEBUG:boto:Token: None 2024-04-13T22:28:52.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/?versions 2024-04-13T22:28:52.581 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY= 2024-04-13T22:28:52.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9pwAYHXE+8mUSRtQb2Xbw9zhVlY=', 'Content-Length': '0'} 2024-04-13T22:28:52.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f239aab50709c1ea-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.585 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:28:52.587 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.587 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.587 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.587 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.587 DEBUG:boto:Method: GET 2024-04-13T22:28:52.587 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.587 DEBUG:boto:Data: 2024-04-13T22:28:52.587 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.587 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.588 DEBUG:boto:Params: {} 2024-04-13T22:28:52.588 DEBUG:boto:Token: None 2024-04-13T22:28:52.588 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4= 2024-04-13T22:28:52.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4=', 'Content-Length': '0'} 2024-04-13T22:28:52.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed5406d499d6be6-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.590 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.590 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.590 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.590 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.590 DEBUG:boto:Method: GET 2024-04-13T22:28:52.590 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.591 DEBUG:boto:Data: 2024-04-13T22:28:52.591 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.591 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.591 DEBUG:boto:Params: {} 2024-04-13T22:28:52.591 DEBUG:boto:Token: None 2024-04-13T22:28:52.591 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.591 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4= 2024-04-13T22:28:52.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4=', 'Content-Length': '0'} 2024-04-13T22:28:52.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f35b6bec20b5458-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.638 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.638 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.639 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.639 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.639 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.639 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.639 DEBUG:boto:Data: 2024-04-13T22:28:52.639 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.639 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.639 DEBUG:boto:Params: {} 2024-04-13T22:28:52.639 DEBUG:boto:Token: None 2024-04-13T22:28:52.639 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8f9n7gp++6T9EvdT/sSqpklzpBs= 2024-04-13T22:28:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8f9n7gp++6T9EvdT/sSqpklzpBs=', 'Content-Length': '0'} 2024-04-13T22:28:52.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000061b2c6e2b9769b5-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.642 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.642 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.642 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.642 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.642 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.642 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.642 DEBUG:boto:Data: 2024-04-13T22:28:52.642 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.642 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.642 DEBUG:boto:Params: {} 2024-04-13T22:28:52.643 DEBUG:boto:Token: None 2024-04-13T22:28:52.643 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8f9n7gp++6T9EvdT/sSqpklzpBs= 2024-04-13T22:28:52.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8f9n7gp++6T9EvdT/sSqpklzpBs=', 'Content-Length': '0'} 2024-04-13T22:28:52.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc059c5ab8a597f3-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:28:52.645 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.645 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.645 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.645 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.645 DEBUG:boto:Method: GET 2024-04-13T22:28:52.646 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.646 DEBUG:boto:Data: 2024-04-13T22:28:52.646 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.646 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.646 DEBUG:boto:Params: {} 2024-04-13T22:28:52.646 DEBUG:boto:Token: None 2024-04-13T22:28:52.646 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.646 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4= 2024-04-13T22:28:52.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4=', 'Content-Length': '0'} 2024-04-13T22:28:52.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaddc620e081eb96-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.648 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.649 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.649 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.649 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.649 DEBUG:boto:Method: GET 2024-04-13T22:28:52.649 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.649 DEBUG:boto:Data: 2024-04-13T22:28:52.649 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.649 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.649 DEBUG:boto:Params: {} 2024-04-13T22:28:52.650 DEBUG:boto:Token: None 2024-04-13T22:28:52.650 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:28:52.650 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4= 2024-04-13T22:28:52.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:63BmTbWd1V6XLnuAtYRdhOJD2J4=', 'Content-Length': '0'} 2024-04-13T22:28:52.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004230f0bcad57567a-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.652 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.652 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.652 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.652 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.652 DEBUG:boto:Method: GET 2024-04-13T22:28:52.652 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.652 DEBUG:boto:Data: 2024-04-13T22:28:52.653 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.653 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.653 DEBUG:boto:Params: {} 2024-04-13T22:28:52.653 DEBUG:boto:Token: None 2024-04-13T22:28:52.653 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aba69c7a603ea82b-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.656 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.656 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.656 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.656 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.656 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.656 DEBUG:boto:Method: GET 2024-04-13T22:28:52.657 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.657 DEBUG:boto:Data: 2024-04-13T22:28:52.657 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.657 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.657 DEBUG:boto:Params: {} 2024-04-13T22:28:52.657 DEBUG:boto:Token: None 2024-04-13T22:28:52.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa1baa4154213576-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.661 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.662 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.662 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.662 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.662 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:28:52.662 DEBUG:boto:Data: 2024-04-13T22:28:52.662 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.663 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.663 DEBUG:boto:Params: {} 2024-04-13T22:28:52.663 DEBUG:boto:Token: None 2024-04-13T22:28:52.663 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11 2024-04-13T22:28:52.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t+8NqC2vZ9VRWWBK/WOjcwWPwCg= 2024-04-13T22:28:52.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t+8NqC2vZ9VRWWBK/WOjcwWPwCg=', 'Content-Length': '0'} 2024-04-13T22:28:52.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000845337c15598340f-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.665 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.665 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.665 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.665 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:28:52.665 DEBUG:boto:Data: 2024-04-13T22:28:52.666 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.666 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.666 DEBUG:boto:Params: {} 2024-04-13T22:28:52.666 DEBUG:boto:Token: None 2024-04-13T22:28:52.666 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11 2024-04-13T22:28:52.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:t+8NqC2vZ9VRWWBK/WOjcwWPwCg= 2024-04-13T22:28:52.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t+8NqC2vZ9VRWWBK/WOjcwWPwCg=', 'Content-Length': '0'} 2024-04-13T22:28:52.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8471acae7784a8-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:28:52.668 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.668 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.668 DEBUG:boto:Method: GET 2024-04-13T22:28:52.669 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:28:52.669 DEBUG:boto:Data: 2024-04-13T22:28:52.669 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.669 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.669 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.669 DEBUG:boto:Params: {} 2024-04-13T22:28:52.669 DEBUG:boto:Token: None 2024-04-13T22:28:52.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11 2024-04-13T22:28:52.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o64J7LGRItLOa4M04aFOJvgIa8o= 2024-04-13T22:28:52.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o64J7LGRItLOa4M04aFOJvgIa8o=', 'Content-Length': '0'} 2024-04-13T22:28:52.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b991c35b3b2dc1a-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.714 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:28:52.714 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:28:52.715 DEBUG:boto:Method: GET 2024-04-13T22:28:52.715 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:28:52.715 DEBUG:boto:Data: 2024-04-13T22:28:52.715 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.715 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.715 DEBUG:boto:Params: {} 2024-04-13T22:28:52.715 DEBUG:boto:Token: None 2024-04-13T22:28:52.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj11 2024-04-13T22:28:52.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:o64J7LGRItLOa4M04aFOJvgIa8o= 2024-04-13T22:28:52.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o64J7LGRItLOa4M04aFOJvgIa8o=', 'Content-Length': '0'} 2024-04-13T22:28:52.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a79d5c7ff9e8736-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.717 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.718 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.718 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.718 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.718 DEBUG:boto:Method: GET 2024-04-13T22:28:52.718 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.718 DEBUG:boto:Data: 2024-04-13T22:28:52.718 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.718 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.718 DEBUG:boto:Params: {} 2024-04-13T22:28:52.718 DEBUG:boto:Token: None 2024-04-13T22:28:52.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a25ac26d618086-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.721 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.722 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.722 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.722 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.722 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.722 DEBUG:boto:Method: GET 2024-04-13T22:28:52.722 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:28:52.722 DEBUG:boto:Data: 2024-04-13T22:28:52.722 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.723 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.723 DEBUG:boto:Params: {} 2024-04-13T22:28:52.723 DEBUG:boto:Token: None 2024-04-13T22:28:52.723 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b1308c5d0ffa1b8-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.727 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:28:52.727 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.727 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.727 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.727 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.727 DEBUG:boto:Method: GET 2024-04-13T22:28:52.728 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.728 DEBUG:boto:Data: 2024-04-13T22:28:52.728 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.728 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.728 DEBUG:boto:Params: {} 2024-04-13T22:28:52.728 DEBUG:boto:Token: None 2024-04-13T22:28:52.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI= 2024-04-13T22:28:52.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI=', 'Content-Length': '0'} 2024-04-13T22:28:52.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f0ff0a322d2954-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.730 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.731 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.731 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.731 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.731 DEBUG:boto:Method: GET 2024-04-13T22:28:52.731 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.731 DEBUG:boto:Data: 2024-04-13T22:28:52.731 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.731 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.731 DEBUG:boto:Params: {} 2024-04-13T22:28:52.731 DEBUG:boto:Token: None 2024-04-13T22:28:52.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI= 2024-04-13T22:28:52.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI=', 'Content-Length': '0'} 2024-04-13T22:28:52.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071455d50bccf28d8-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.774 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.775 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.775 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.775 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.775 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.775 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.775 DEBUG:boto:Data: 2024-04-13T22:28:52.775 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.775 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.775 DEBUG:boto:Params: {} 2024-04-13T22:28:52.776 DEBUG:boto:Token: None 2024-04-13T22:28:52.776 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ol8dlHfDzIoZN1w464wm83Wi7V4= 2024-04-13T22:28:52.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ol8dlHfDzIoZN1w464wm83Wi7V4=', 'Content-Length': '0'} 2024-04-13T22:28:52.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ca4e775aa42dec-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.778 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.778 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.778 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.778 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.778 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.779 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.779 DEBUG:boto:Data: 2024-04-13T22:28:52.779 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.779 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.779 DEBUG:boto:Params: {} 2024-04-13T22:28:52.779 DEBUG:boto:Token: None 2024-04-13T22:28:52.779 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ol8dlHfDzIoZN1w464wm83Wi7V4= 2024-04-13T22:28:52.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ol8dlHfDzIoZN1w464wm83Wi7V4=', 'Content-Length': '0'} 2024-04-13T22:28:52.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d05d9ae2a1541330-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:28:52.782 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.782 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.782 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.782 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.782 DEBUG:boto:Method: GET 2024-04-13T22:28:52.782 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.782 DEBUG:boto:Data: 2024-04-13T22:28:52.783 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.783 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.783 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.783 DEBUG:boto:Params: {} 2024-04-13T22:28:52.783 DEBUG:boto:Token: None 2024-04-13T22:28:52.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI= 2024-04-13T22:28:52.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI=', 'Content-Length': '0'} 2024-04-13T22:28:52.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d498fd05c0134f-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.785 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.785 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.786 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.786 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.786 DEBUG:boto:Method: GET 2024-04-13T22:28:52.786 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.786 DEBUG:boto:Data: 2024-04-13T22:28:52.786 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.786 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.786 DEBUG:boto:Params: {} 2024-04-13T22:28:52.786 DEBUG:boto:Token: None 2024-04-13T22:28:52.787 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:28:52.787 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI= 2024-04-13T22:28:52.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ODa0AF0N5hN27JX+i2eYa6tZxFI=', 'Content-Length': '0'} 2024-04-13T22:28:52.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000308f503a5f7bd584-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.789 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.789 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.789 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.790 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.790 DEBUG:boto:Method: GET 2024-04-13T22:28:52.790 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.790 DEBUG:boto:Data: 2024-04-13T22:28:52.790 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.790 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.790 DEBUG:boto:Params: {} 2024-04-13T22:28:52.790 DEBUG:boto:Token: None 2024-04-13T22:28:52.790 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639af3ff95a7f4c4-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.793 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.793 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.794 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.794 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.794 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.794 DEBUG:boto:Method: GET 2024-04-13T22:28:52.794 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.794 DEBUG:boto:Data: 2024-04-13T22:28:52.794 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.794 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.794 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.794 DEBUG:boto:Params: {} 2024-04-13T22:28:52.795 DEBUG:boto:Token: None 2024-04-13T22:28:52.795 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.795 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b04a3708efc2a28-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.798 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.799 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.799 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.799 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.799 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:28:52.799 DEBUG:boto:Data: 2024-04-13T22:28:52.799 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.800 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.800 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.800 DEBUG:boto:Params: {} 2024-04-13T22:28:52.800 DEBUG:boto:Token: None 2024-04-13T22:28:52.800 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115 2024-04-13T22:28:52.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xYZXJC1XtZmvjcTQ5ZrxbjeU8A8= 2024-04-13T22:28:52.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xYZXJC1XtZmvjcTQ5ZrxbjeU8A8=', 'Content-Length': '0'} 2024-04-13T22:28:52.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a646079de63a42b-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.802 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.802 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.802 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.802 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:28:52.802 DEBUG:boto:Data: 2024-04-13T22:28:52.802 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.803 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.803 DEBUG:boto:Params: {} 2024-04-13T22:28:52.803 DEBUG:boto:Token: None 2024-04-13T22:28:52.803 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115 2024-04-13T22:28:52.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xYZXJC1XtZmvjcTQ5ZrxbjeU8A8= 2024-04-13T22:28:52.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xYZXJC1XtZmvjcTQ5ZrxbjeU8A8=', 'Content-Length': '0'} 2024-04-13T22:28:52.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006929b7a1bf086d4b-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:28:52.805 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.805 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.805 DEBUG:boto:Method: GET 2024-04-13T22:28:52.805 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:28:52.806 DEBUG:boto:Data: 2024-04-13T22:28:52.806 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.806 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.806 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.806 DEBUG:boto:Params: {} 2024-04-13T22:28:52.806 DEBUG:boto:Token: None 2024-04-13T22:28:52.806 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115 2024-04-13T22:28:52.806 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W8tWLLUI+jWWPW1o9NOwi5Z2p+Y= 2024-04-13T22:28:52.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W8tWLLUI+jWWPW1o9NOwi5Z2p+Y=', 'Content-Length': '0'} 2024-04-13T22:28:52.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011bf553909eb2154-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.808 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:28:52.808 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:28:52.809 DEBUG:boto:Method: GET 2024-04-13T22:28:52.809 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:28:52.809 DEBUG:boto:Data: 2024-04-13T22:28:52.809 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.809 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.809 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.809 DEBUG:boto:Params: {} 2024-04-13T22:28:52.809 DEBUG:boto:Token: None 2024-04-13T22:28:52.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj115 2024-04-13T22:28:52.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W8tWLLUI+jWWPW1o9NOwi5Z2p+Y= 2024-04-13T22:28:52.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W8tWLLUI+jWWPW1o9NOwi5Z2p+Y=', 'Content-Length': '0'} 2024-04-13T22:28:52.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d6941f53d346a1-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.811 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.811 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.812 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.812 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.812 DEBUG:boto:Method: GET 2024-04-13T22:28:52.812 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.812 DEBUG:boto:Data: 2024-04-13T22:28:52.812 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.812 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.812 DEBUG:boto:Params: {} 2024-04-13T22:28:52.812 DEBUG:boto:Token: None 2024-04-13T22:28:52.813 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa631251133892d-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.815 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.815 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.815 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.815 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.816 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.816 DEBUG:boto:Method: GET 2024-04-13T22:28:52.816 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:28:52.816 DEBUG:boto:Data: 2024-04-13T22:28:52.816 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.816 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.816 DEBUG:boto:Params: {} 2024-04-13T22:28:52.816 DEBUG:boto:Token: None 2024-04-13T22:28:52.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0883b473de3067-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.863 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:28:52.863 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.863 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.863 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.863 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.863 DEBUG:boto:Method: GET 2024-04-13T22:28:52.864 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.864 DEBUG:boto:Data: 2024-04-13T22:28:52.864 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.864 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.864 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.864 DEBUG:boto:Params: {} 2024-04-13T22:28:52.864 DEBUG:boto:Token: None 2024-04-13T22:28:52.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk= 2024-04-13T22:28:52.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk=', 'Content-Length': '0'} 2024-04-13T22:28:52.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b1231f962d6e51a-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.866 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.867 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.867 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.867 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.867 DEBUG:boto:Method: GET 2024-04-13T22:28:52.867 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.867 DEBUG:boto:Data: 2024-04-13T22:28:52.867 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.867 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.867 DEBUG:boto:Params: {} 2024-04-13T22:28:52.867 DEBUG:boto:Token: None 2024-04-13T22:28:52.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk= 2024-04-13T22:28:52.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk=', 'Content-Length': '0'} 2024-04-13T22:28:52.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002970377666d2598e-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.870 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.870 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.870 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.870 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.870 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.870 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.870 DEBUG:boto:Data: 2024-04-13T22:28:52.870 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.870 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.871 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.871 DEBUG:boto:Params: {} 2024-04-13T22:28:52.871 DEBUG:boto:Token: None 2024-04-13T22:28:52.871 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m9GHOxhNDHdUf2/10liSCS3WdBY= 2024-04-13T22:28:52.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m9GHOxhNDHdUf2/10liSCS3WdBY=', 'Content-Length': '0'} 2024-04-13T22:28:52.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af8356e2f180f4a-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.873 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.873 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.873 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.873 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.874 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.874 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.874 DEBUG:boto:Data: 2024-04-13T22:28:52.874 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.874 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.874 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.874 DEBUG:boto:Params: {} 2024-04-13T22:28:52.874 DEBUG:boto:Token: None 2024-04-13T22:28:52.874 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.874 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m9GHOxhNDHdUf2/10liSCS3WdBY= 2024-04-13T22:28:52.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m9GHOxhNDHdUf2/10liSCS3WdBY=', 'Content-Length': '0'} 2024-04-13T22:28:52.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6e1b9765f2b0f20-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:28:52.877 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.877 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.877 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.877 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.877 DEBUG:boto:Method: GET 2024-04-13T22:28:52.878 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.878 DEBUG:boto:Data: 2024-04-13T22:28:52.878 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.878 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.878 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.878 DEBUG:boto:Params: {} 2024-04-13T22:28:52.878 DEBUG:boto:Token: None 2024-04-13T22:28:52.878 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.878 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk= 2024-04-13T22:28:52.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk=', 'Content-Length': '0'} 2024-04-13T22:28:52.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd45938ff0b5b4a-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.880 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.880 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.881 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.881 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.881 DEBUG:boto:Method: GET 2024-04-13T22:28:52.881 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.881 DEBUG:boto:Data: 2024-04-13T22:28:52.881 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.881 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.881 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.881 DEBUG:boto:Params: {} 2024-04-13T22:28:52.881 DEBUG:boto:Token: None 2024-04-13T22:28:52.882 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:28:52.882 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk= 2024-04-13T22:28:52.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7hw1Bz8mvE+L3fWWHq4oWo8QVYk=', 'Content-Length': '0'} 2024-04-13T22:28:52.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b86fc6b68335ad-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.884 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.884 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.884 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.884 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.884 DEBUG:boto:Method: GET 2024-04-13T22:28:52.884 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.884 DEBUG:boto:Data: 2024-04-13T22:28:52.884 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.884 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.885 DEBUG:boto:Params: {} 2024-04-13T22:28:52.885 DEBUG:boto:Token: None 2024-04-13T22:28:52.885 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.885 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ce1a1518ef12cb-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.887 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.888 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.888 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.888 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.888 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.888 DEBUG:boto:Method: GET 2024-04-13T22:28:52.888 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.888 DEBUG:boto:Data: 2024-04-13T22:28:52.888 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.888 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.888 DEBUG:boto:Params: {} 2024-04-13T22:28:52.889 DEBUG:boto:Token: None 2024-04-13T22:28:52.889 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579bc52908ec594f-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.892 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.892 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.893 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.893 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.893 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:28:52.893 DEBUG:boto:Data: 2024-04-13T22:28:52.893 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.893 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.893 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.893 DEBUG:boto:Params: {} 2024-04-13T22:28:52.893 DEBUG:boto:Token: None 2024-04-13T22:28:52.893 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116 2024-04-13T22:28:52.894 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:APrPxL1bFxu+uNXMaEVe++BVB3w= 2024-04-13T22:28:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:APrPxL1bFxu+uNXMaEVe++BVB3w=', 'Content-Length': '0'} 2024-04-13T22:28:52.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b94c1cc982e5763-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.895 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.895 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.896 DEBUG:boto:Method: HEAD 2024-04-13T22:28:52.896 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:28:52.896 DEBUG:boto:Data: 2024-04-13T22:28:52.896 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.896 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.896 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.896 DEBUG:boto:Params: {} 2024-04-13T22:28:52.896 DEBUG:boto:Token: None 2024-04-13T22:28:52.896 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116 2024-04-13T22:28:52.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:APrPxL1bFxu+uNXMaEVe++BVB3w= 2024-04-13T22:28:52.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:APrPxL1bFxu+uNXMaEVe++BVB3w=', 'Content-Length': '0'} 2024-04-13T22:28:52.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49bfc60073eda32-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:28:52.898 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.899 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.899 DEBUG:boto:Method: GET 2024-04-13T22:28:52.899 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:28:52.899 DEBUG:boto:Data: 2024-04-13T22:28:52.899 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.899 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.899 DEBUG:boto:Params: {} 2024-04-13T22:28:52.899 DEBUG:boto:Token: None 2024-04-13T22:28:52.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116 2024-04-13T22:28:52.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:csSZXPjY7eAwGipmib+WzDQANZw= 2024-04-13T22:28:52.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:csSZXPjY7eAwGipmib+WzDQANZw=', 'Content-Length': '0'} 2024-04-13T22:28:52.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a9473f5665254f-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.942 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:28:52.942 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:28:52.943 DEBUG:boto:Method: GET 2024-04-13T22:28:52.943 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:28:52.943 DEBUG:boto:Data: 2024-04-13T22:28:52.943 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.943 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.943 DEBUG:boto:Params: {} 2024-04-13T22:28:52.943 DEBUG:boto:Token: None 2024-04-13T22:28:52.943 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj116 2024-04-13T22:28:52.943 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:csSZXPjY7eAwGipmib+WzDQANZw= 2024-04-13T22:28:52.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:csSZXPjY7eAwGipmib+WzDQANZw=', 'Content-Length': '0'} 2024-04-13T22:28:52.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c69cf32fba5e35-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.946 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.946 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.946 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.946 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.946 DEBUG:boto:Method: GET 2024-04-13T22:28:52.946 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.946 DEBUG:boto:Data: 2024-04-13T22:28:52.946 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.946 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.947 DEBUG:boto:Params: {} 2024-04-13T22:28:52.947 DEBUG:boto:Token: None 2024-04-13T22:28:52.947 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041113d51a15db765-00661b0724-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.949 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.950 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:52.950 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:52.950 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.950 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.950 DEBUG:boto:Method: GET 2024-04-13T22:28:52.950 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:28:52.950 DEBUG:boto:Data: 2024-04-13T22:28:52.950 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.950 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.950 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.950 DEBUG:boto:Params: {} 2024-04-13T22:28:52.951 DEBUG:boto:Token: None 2024-04-13T22:28:52.951 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/ 2024-04-13T22:28:52.951 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg= 2024-04-13T22:28:52.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3S4SM6MJQGHTwL1yTBcbaI8hJdg=', 'Content-Length': '0'} 2024-04-13T22:28:52.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a973444f7749287b-00661b0724-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.955 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:52.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:28:52.955 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:52.955 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:52.955 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.955 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.955 DEBUG:boto:Method: GET 2024-04-13T22:28:52.955 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.956 DEBUG:boto:Data: 2024-04-13T22:28:52.956 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:52.956 DEBUG:boto:Port: 8001 2024-04-13T22:28:52.956 DEBUG:boto:Params: {} 2024-04-13T22:28:52.956 DEBUG:boto:Token: None 2024-04-13T22:28:52.956 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jmPCuN09zADKFZD26wMoES1ycF4= 2024-04-13T22:28:52.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jmPCuN09zADKFZD26wMoES1ycF4=', 'Content-Length': '0'} 2024-04-13T22:28:52.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a801538abe24bf0d-00661b0724-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:52.958 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:52.958 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:52.958 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.959 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.959 DEBUG:boto:Method: GET 2024-04-13T22:28:52.959 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.959 DEBUG:boto:Data: 2024-04-13T22:28:52.959 DEBUG:boto:Headers: {} 2024-04-13T22:28:52.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:52.959 DEBUG:boto:Port: 8000 2024-04-13T22:28:52.959 DEBUG:boto:Params: {} 2024-04-13T22:28:52.959 DEBUG:boto:Token: None 2024-04-13T22:28:52.959 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:52 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:52.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jmPCuN09zADKFZD26wMoES1ycF4= 2024-04-13T22:28:52.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jmPCuN09zADKFZD26wMoES1ycF4=', 'Content-Length': '0'} 2024-04-13T22:28:52.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007049522ffa420d4b-00661b0724-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.002 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.002 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.003 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.003 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.003 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.003 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.003 DEBUG:boto:Data: 2024-04-13T22:28:53.003 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.003 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.003 DEBUG:boto:Params: {} 2024-04-13T22:28:53.003 DEBUG:boto:Token: None 2024-04-13T22:28:53.004 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2DKd8113eCdI4u2RT+8pTHBKjlg= 2024-04-13T22:28:53.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2DKd8113eCdI4u2RT+8pTHBKjlg=', 'Content-Length': '0'} 2024-04-13T22:28:53.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096da3065c1d895c3-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.006 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.006 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.006 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.006 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.006 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.006 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.006 DEBUG:boto:Data: 2024-04-13T22:28:53.007 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.007 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.007 DEBUG:boto:Params: {} 2024-04-13T22:28:53.007 DEBUG:boto:Token: None 2024-04-13T22:28:53.007 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2DKd8113eCdI4u2RT+8pTHBKjlg= 2024-04-13T22:28:53.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2DKd8113eCdI4u2RT+8pTHBKjlg=', 'Content-Length': '0'} 2024-04-13T22:28:53.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3b1b53061471fb-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:28:53.009 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.009 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.009 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.009 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.010 DEBUG:boto:Method: GET 2024-04-13T22:28:53.010 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.010 DEBUG:boto:Data: 2024-04-13T22:28:53.010 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.010 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.010 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.010 DEBUG:boto:Params: {} 2024-04-13T22:28:53.010 DEBUG:boto:Token: None 2024-04-13T22:28:53.010 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ajJ/1UKiuEl2O+0N3baD2JMugD8= 2024-04-13T22:28:53.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ajJ/1UKiuEl2O+0N3baD2JMugD8=', 'Content-Length': '0'} 2024-04-13T22:28:53.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f833a4742c3de54-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.013 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.013 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.013 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.013 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.013 DEBUG:boto:Method: GET 2024-04-13T22:28:53.013 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.013 DEBUG:boto:Data: 2024-04-13T22:28:53.014 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.014 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.014 DEBUG:boto:Params: {} 2024-04-13T22:28:53.014 DEBUG:boto:Token: None 2024-04-13T22:28:53.014 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:28:53.014 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ajJ/1UKiuEl2O+0N3baD2JMugD8= 2024-04-13T22:28:53.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ajJ/1UKiuEl2O+0N3baD2JMugD8=', 'Content-Length': '0'} 2024-04-13T22:28:53.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec6098ec4230bcd0-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.016 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.016 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.016 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.016 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.017 DEBUG:boto:Method: GET 2024-04-13T22:28:53.017 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.017 DEBUG:boto:Data: 2024-04-13T22:28:53.017 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.017 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.017 DEBUG:boto:Params: {} 2024-04-13T22:28:53.017 DEBUG:boto:Token: None 2024-04-13T22:28:53.017 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.018 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7cea9bee29c97f0-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.020 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.020 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.020 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.020 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.020 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.021 DEBUG:boto:Method: GET 2024-04-13T22:28:53.021 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.021 DEBUG:boto:Data: 2024-04-13T22:28:53.021 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.021 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.021 DEBUG:boto:Params: {} 2024-04-13T22:28:53.021 DEBUG:boto:Token: None 2024-04-13T22:28:53.021 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc5c2c2b1d42dc11-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.025 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.025 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.025 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.025 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.026 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:28:53.026 DEBUG:boto:Data: 2024-04-13T22:28:53.026 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.026 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.026 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.026 DEBUG:boto:Params: {} 2024-04-13T22:28:53.026 DEBUG:boto:Token: None 2024-04-13T22:28:53.026 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124 2024-04-13T22:28:53.026 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:efPxZRgLk1nMogMegpJhuGeEQbQ= 2024-04-13T22:28:53.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:efPxZRgLk1nMogMegpJhuGeEQbQ=', 'Content-Length': '0'} 2024-04-13T22:28:53.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092bcba0a7dc38897-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.028 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.028 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.029 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.029 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:28:53.029 DEBUG:boto:Data: 2024-04-13T22:28:53.029 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.029 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.029 DEBUG:boto:Params: {} 2024-04-13T22:28:53.029 DEBUG:boto:Token: None 2024-04-13T22:28:53.029 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124 2024-04-13T22:28:53.029 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:efPxZRgLk1nMogMegpJhuGeEQbQ= 2024-04-13T22:28:53.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:efPxZRgLk1nMogMegpJhuGeEQbQ=', 'Content-Length': '0'} 2024-04-13T22:28:53.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9bbb49b3db1167-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:28:53.031 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.031 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.031 DEBUG:boto:Method: GET 2024-04-13T22:28:53.032 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:28:53.032 DEBUG:boto:Data: 2024-04-13T22:28:53.032 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.032 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.032 DEBUG:boto:Params: {} 2024-04-13T22:28:53.032 DEBUG:boto:Token: None 2024-04-13T22:28:53.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124 2024-04-13T22:28:53.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mqX89i6hTo0ExY4pp/EVtGYh8lw= 2024-04-13T22:28:53.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mqX89i6hTo0ExY4pp/EVtGYh8lw=', 'Content-Length': '0'} 2024-04-13T22:28:53.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe3755fdcd0d294-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.034 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:28:53.034 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:28:53.034 DEBUG:boto:Method: GET 2024-04-13T22:28:53.034 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:28:53.035 DEBUG:boto:Data: 2024-04-13T22:28:53.035 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.035 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.035 DEBUG:boto:Params: {} 2024-04-13T22:28:53.035 DEBUG:boto:Token: None 2024-04-13T22:28:53.035 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj124 2024-04-13T22:28:53.035 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mqX89i6hTo0ExY4pp/EVtGYh8lw= 2024-04-13T22:28:53.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mqX89i6hTo0ExY4pp/EVtGYh8lw=', 'Content-Length': '0'} 2024-04-13T22:28:53.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000039f2d1c8018e51a-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.078 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.078 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.079 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.079 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.079 DEBUG:boto:Method: GET 2024-04-13T22:28:53.079 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.079 DEBUG:boto:Data: 2024-04-13T22:28:53.079 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.079 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.079 DEBUG:boto:Params: {} 2024-04-13T22:28:53.079 DEBUG:boto:Token: None 2024-04-13T22:28:53.080 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a073d301a678bcf-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.082 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.082 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.082 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.082 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.083 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.083 DEBUG:boto:Method: GET 2024-04-13T22:28:53.083 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:28:53.083 DEBUG:boto:Data: 2024-04-13T22:28:53.083 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.083 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.083 DEBUG:boto:Params: {} 2024-04-13T22:28:53.083 DEBUG:boto:Token: None 2024-04-13T22:28:53.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d21fee2317cdf3-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.087 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:28:53.088 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.088 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.088 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.088 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.088 DEBUG:boto:Method: GET 2024-04-13T22:28:53.088 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.088 DEBUG:boto:Data: 2024-04-13T22:28:53.088 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.088 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.088 DEBUG:boto:Params: {} 2024-04-13T22:28:53.089 DEBUG:boto:Token: None 2024-04-13T22:28:53.089 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0= 2024-04-13T22:28:53.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0=', 'Content-Length': '0'} 2024-04-13T22:28:53.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ef73d5df7f5c9a-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.091 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.091 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.091 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.091 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.091 DEBUG:boto:Method: GET 2024-04-13T22:28:53.091 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.092 DEBUG:boto:Data: 2024-04-13T22:28:53.092 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.092 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.092 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.092 DEBUG:boto:Params: {} 2024-04-13T22:28:53.092 DEBUG:boto:Token: None 2024-04-13T22:28:53.092 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0= 2024-04-13T22:28:53.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0=', 'Content-Length': '0'} 2024-04-13T22:28:53.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e02365ceeada61-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.094 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.094 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.094 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.095 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.095 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.095 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.095 DEBUG:boto:Data: 2024-04-13T22:28:53.095 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.095 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.095 DEBUG:boto:Params: {} 2024-04-13T22:28:53.095 DEBUG:boto:Token: None 2024-04-13T22:28:53.095 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hU4KXAkvilQIi9EALBt41pK5vPw= 2024-04-13T22:28:53.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hU4KXAkvilQIi9EALBt41pK5vPw=', 'Content-Length': '0'} 2024-04-13T22:28:53.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bff795652f1bb77-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.097 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.097 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.097 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.097 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.097 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.098 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.098 DEBUG:boto:Data: 2024-04-13T22:28:53.098 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.098 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.098 DEBUG:boto:Params: {} 2024-04-13T22:28:53.098 DEBUG:boto:Token: None 2024-04-13T22:28:53.098 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hU4KXAkvilQIi9EALBt41pK5vPw= 2024-04-13T22:28:53.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hU4KXAkvilQIi9EALBt41pK5vPw=', 'Content-Length': '0'} 2024-04-13T22:28:53.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c097b9824fcc7b9d-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:28:53.100 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.101 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.101 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.101 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.101 DEBUG:boto:Method: GET 2024-04-13T22:28:53.101 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.101 DEBUG:boto:Data: 2024-04-13T22:28:53.101 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.101 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.101 DEBUG:boto:Params: {} 2024-04-13T22:28:53.101 DEBUG:boto:Token: None 2024-04-13T22:28:53.102 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0= 2024-04-13T22:28:53.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0=', 'Content-Length': '0'} 2024-04-13T22:28:53.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffc04d99557129a-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.147 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.147 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.147 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.147 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.147 DEBUG:boto:Method: GET 2024-04-13T22:28:53.147 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.147 DEBUG:boto:Data: 2024-04-13T22:28:53.147 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.147 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.148 DEBUG:boto:Params: {} 2024-04-13T22:28:53.148 DEBUG:boto:Token: None 2024-04-13T22:28:53.148 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:28:53.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0= 2024-04-13T22:28:53.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsROMXzjk0/y+oXzNUKu/QJ6mt0=', 'Content-Length': '0'} 2024-04-13T22:28:53.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b09ab6fc385e9b2e-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.150 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.150 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.150 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.150 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.151 DEBUG:boto:Method: GET 2024-04-13T22:28:53.151 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.151 DEBUG:boto:Data: 2024-04-13T22:28:53.151 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.151 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.151 DEBUG:boto:Params: {} 2024-04-13T22:28:53.151 DEBUG:boto:Token: None 2024-04-13T22:28:53.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89bcc03b67221c5-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.153 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.154 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.154 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.154 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.154 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.154 DEBUG:boto:Method: GET 2024-04-13T22:28:53.154 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.154 DEBUG:boto:Data: 2024-04-13T22:28:53.155 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.155 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.155 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.155 DEBUG:boto:Params: {} 2024-04-13T22:28:53.155 DEBUG:boto:Token: None 2024-04-13T22:28:53.155 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa8ec009598b1b62-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.159 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.159 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.159 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.159 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.159 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:28:53.159 DEBUG:boto:Data: 2024-04-13T22:28:53.159 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.160 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.160 DEBUG:boto:Params: {} 2024-04-13T22:28:53.160 DEBUG:boto:Token: None 2024-04-13T22:28:53.160 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145 2024-04-13T22:28:53.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QczsMBE3AsQFUSJX4XFIeLsVGKU= 2024-04-13T22:28:53.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QczsMBE3AsQFUSJX4XFIeLsVGKU=', 'Content-Length': '0'} 2024-04-13T22:28:53.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267e46453e7649b9-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.162 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.162 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.162 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.162 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:28:53.163 DEBUG:boto:Data: 2024-04-13T22:28:53.163 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.163 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.163 DEBUG:boto:Params: {} 2024-04-13T22:28:53.163 DEBUG:boto:Token: None 2024-04-13T22:28:53.163 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145 2024-04-13T22:28:53.163 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QczsMBE3AsQFUSJX4XFIeLsVGKU= 2024-04-13T22:28:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QczsMBE3AsQFUSJX4XFIeLsVGKU=', 'Content-Length': '0'} 2024-04-13T22:28:53.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abdc4bd0e87c9ae3-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:28:53.166 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.166 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.166 DEBUG:boto:Method: GET 2024-04-13T22:28:53.166 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:28:53.166 DEBUG:boto:Data: 2024-04-13T22:28:53.166 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.166 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.166 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.166 DEBUG:boto:Params: {} 2024-04-13T22:28:53.166 DEBUG:boto:Token: None 2024-04-13T22:28:53.167 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145 2024-04-13T22:28:53.167 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NHZ1AtPUBZGGIccRxhpXo429Dk8= 2024-04-13T22:28:53.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NHZ1AtPUBZGGIccRxhpXo429Dk8=', 'Content-Length': '0'} 2024-04-13T22:28:53.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f87bfcd2463a9d5d-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.169 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:28:53.169 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:28:53.169 DEBUG:boto:Method: GET 2024-04-13T22:28:53.169 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:28:53.169 DEBUG:boto:Data: 2024-04-13T22:28:53.169 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.169 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.169 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.169 DEBUG:boto:Params: {} 2024-04-13T22:28:53.170 DEBUG:boto:Token: None 2024-04-13T22:28:53.170 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj145 2024-04-13T22:28:53.170 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NHZ1AtPUBZGGIccRxhpXo429Dk8= 2024-04-13T22:28:53.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NHZ1AtPUBZGGIccRxhpXo429Dk8=', 'Content-Length': '0'} 2024-04-13T22:28:53.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067267ee1840fc3bb-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.219 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.219 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.219 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.219 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.219 DEBUG:boto:Method: GET 2024-04-13T22:28:53.219 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.219 DEBUG:boto:Data: 2024-04-13T22:28:53.219 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.219 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.220 DEBUG:boto:Params: {} 2024-04-13T22:28:53.220 DEBUG:boto:Token: None 2024-04-13T22:28:53.220 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e82046b01d843f1c-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.222 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.223 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.223 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.223 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.223 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.223 DEBUG:boto:Method: GET 2024-04-13T22:28:53.223 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:28:53.223 DEBUG:boto:Data: 2024-04-13T22:28:53.223 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.223 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.224 DEBUG:boto:Params: {} 2024-04-13T22:28:53.224 DEBUG:boto:Token: None 2024-04-13T22:28:53.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a1f070cadb1284-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.228 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:28:53.228 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.228 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.228 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.228 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.229 DEBUG:boto:Method: GET 2024-04-13T22:28:53.229 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.229 DEBUG:boto:Data: 2024-04-13T22:28:53.229 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.229 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.229 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.229 DEBUG:boto:Params: {} 2024-04-13T22:28:53.229 DEBUG:boto:Token: None 2024-04-13T22:28:53.229 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw= 2024-04-13T22:28:53.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw=', 'Content-Length': '0'} 2024-04-13T22:28:53.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf26c9d41a97359-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.231 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.232 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.232 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.232 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.232 DEBUG:boto:Method: GET 2024-04-13T22:28:53.232 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.232 DEBUG:boto:Data: 2024-04-13T22:28:53.232 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.232 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.232 DEBUG:boto:Params: {} 2024-04-13T22:28:53.232 DEBUG:boto:Token: None 2024-04-13T22:28:53.233 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.233 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw= 2024-04-13T22:28:53.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw=', 'Content-Length': '0'} 2024-04-13T22:28:53.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010467d43c67a5400-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.235 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.235 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.235 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.235 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.235 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.235 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.235 DEBUG:boto:Data: 2024-04-13T22:28:53.235 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.235 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.235 DEBUG:boto:Params: {} 2024-04-13T22:28:53.236 DEBUG:boto:Token: None 2024-04-13T22:28:53.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s5IGytdQoy1SyLQCgSngnkMC7bE= 2024-04-13T22:28:53.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s5IGytdQoy1SyLQCgSngnkMC7bE=', 'Content-Length': '0'} 2024-04-13T22:28:53.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815eb3c6db60db1e-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.237 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.237 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.237 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.238 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.238 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.238 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.238 DEBUG:boto:Data: 2024-04-13T22:28:53.238 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.238 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.238 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.238 DEBUG:boto:Params: {} 2024-04-13T22:28:53.238 DEBUG:boto:Token: None 2024-04-13T22:28:53.238 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:s5IGytdQoy1SyLQCgSngnkMC7bE= 2024-04-13T22:28:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s5IGytdQoy1SyLQCgSngnkMC7bE=', 'Content-Length': '0'} 2024-04-13T22:28:53.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005065e2e3608f5953-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:28:53.241 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.241 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.241 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.241 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.241 DEBUG:boto:Method: GET 2024-04-13T22:28:53.241 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.241 DEBUG:boto:Data: 2024-04-13T22:28:53.241 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.241 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.241 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.241 DEBUG:boto:Params: {} 2024-04-13T22:28:53.242 DEBUG:boto:Token: None 2024-04-13T22:28:53.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw= 2024-04-13T22:28:53.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw=', 'Content-Length': '0'} 2024-04-13T22:28:53.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee422209b68c79c5-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.290 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.291 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.291 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.291 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.291 DEBUG:boto:Method: GET 2024-04-13T22:28:53.291 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.291 DEBUG:boto:Data: 2024-04-13T22:28:53.291 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.291 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.291 DEBUG:boto:Params: {} 2024-04-13T22:28:53.292 DEBUG:boto:Token: None 2024-04-13T22:28:53.292 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:28:53.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw= 2024-04-13T22:28:53.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lAL3Fc3WxwZrTMjH1YTKvgsuXOw=', 'Content-Length': '0'} 2024-04-13T22:28:53.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe0138300c26145-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.294 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.294 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.294 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.294 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.294 DEBUG:boto:Method: GET 2024-04-13T22:28:53.294 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.295 DEBUG:boto:Data: 2024-04-13T22:28:53.295 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.295 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.295 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.295 DEBUG:boto:Params: {} 2024-04-13T22:28:53.295 DEBUG:boto:Token: None 2024-04-13T22:28:53.295 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d49e96501cc5cdb4-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.298 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.298 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.298 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.298 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.298 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.298 DEBUG:boto:Method: GET 2024-04-13T22:28:53.298 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.298 DEBUG:boto:Data: 2024-04-13T22:28:53.298 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.299 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.299 DEBUG:boto:Params: {} 2024-04-13T22:28:53.299 DEBUG:boto:Token: None 2024-04-13T22:28:53.299 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.299 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dca9aadc9ba2db1-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.303 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.303 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.303 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.303 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.303 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:28:53.303 DEBUG:boto:Data: 2024-04-13T22:28:53.303 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.304 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.304 DEBUG:boto:Params: {} 2024-04-13T22:28:53.304 DEBUG:boto:Token: None 2024-04-13T22:28:53.304 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155 2024-04-13T22:28:53.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zCdlAU5izEs7EIh0dEb0uUcCxeA= 2024-04-13T22:28:53.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zCdlAU5izEs7EIh0dEb0uUcCxeA=', 'Content-Length': '0'} 2024-04-13T22:28:53.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000941fde84a0168347-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.306 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.306 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.306 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.306 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:28:53.306 DEBUG:boto:Data: 2024-04-13T22:28:53.307 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.307 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.307 DEBUG:boto:Params: {} 2024-04-13T22:28:53.307 DEBUG:boto:Token: None 2024-04-13T22:28:53.307 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155 2024-04-13T22:28:53.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zCdlAU5izEs7EIh0dEb0uUcCxeA= 2024-04-13T22:28:53.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zCdlAU5izEs7EIh0dEb0uUcCxeA=', 'Content-Length': '0'} 2024-04-13T22:28:53.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bcf95222353dcc7-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:28:53.309 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.309 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.309 DEBUG:boto:Method: GET 2024-04-13T22:28:53.309 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:28:53.309 DEBUG:boto:Data: 2024-04-13T22:28:53.309 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.310 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.310 DEBUG:boto:Params: {} 2024-04-13T22:28:53.310 DEBUG:boto:Token: None 2024-04-13T22:28:53.310 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155 2024-04-13T22:28:53.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GdwGcEsDWpjQmg7W2InyCNjoQ2M= 2024-04-13T22:28:53.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GdwGcEsDWpjQmg7W2InyCNjoQ2M=', 'Content-Length': '0'} 2024-04-13T22:28:53.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c6e9ef29389b8d-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.312 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:28:53.312 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:28:53.312 DEBUG:boto:Method: GET 2024-04-13T22:28:53.312 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:28:53.312 DEBUG:boto:Data: 2024-04-13T22:28:53.312 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.312 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.313 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.313 DEBUG:boto:Params: {} 2024-04-13T22:28:53.313 DEBUG:boto:Token: None 2024-04-13T22:28:53.313 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj155 2024-04-13T22:28:53.313 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GdwGcEsDWpjQmg7W2InyCNjoQ2M= 2024-04-13T22:28:53.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GdwGcEsDWpjQmg7W2InyCNjoQ2M=', 'Content-Length': '0'} 2024-04-13T22:28:53.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bfc907394edde5c-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.315 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.315 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.315 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.315 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.315 DEBUG:boto:Method: GET 2024-04-13T22:28:53.315 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.315 DEBUG:boto:Data: 2024-04-13T22:28:53.316 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.316 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.316 DEBUG:boto:Params: {} 2024-04-13T22:28:53.316 DEBUG:boto:Token: None 2024-04-13T22:28:53.316 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d725cc2f886cc167-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.319 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.320 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.320 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.320 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.320 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.320 DEBUG:boto:Method: GET 2024-04-13T22:28:53.320 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:28:53.320 DEBUG:boto:Data: 2024-04-13T22:28:53.320 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.320 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.320 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.321 DEBUG:boto:Params: {} 2024-04-13T22:28:53.321 DEBUG:boto:Token: None 2024-04-13T22:28:53.321 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bee1f4464d5380dd-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.324 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:28:53.325 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.325 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.325 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.325 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.325 DEBUG:boto:Method: GET 2024-04-13T22:28:53.325 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.325 DEBUG:boto:Data: 2024-04-13T22:28:53.325 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.325 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.325 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.325 DEBUG:boto:Params: {} 2024-04-13T22:28:53.325 DEBUG:boto:Token: None 2024-04-13T22:28:53.326 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.326 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U= 2024-04-13T22:28:53.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U=', 'Content-Length': '0'} 2024-04-13T22:28:53.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af19f50fe5eac76c-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.327 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.327 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.328 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.328 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.328 DEBUG:boto:Method: GET 2024-04-13T22:28:53.328 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.328 DEBUG:boto:Data: 2024-04-13T22:28:53.328 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.328 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.328 DEBUG:boto:Params: {} 2024-04-13T22:28:53.328 DEBUG:boto:Token: None 2024-04-13T22:28:53.328 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U= 2024-04-13T22:28:53.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U=', 'Content-Length': '0'} 2024-04-13T22:28:53.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b49c345e2410af0-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.370 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.370 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.371 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.371 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.371 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.371 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.371 DEBUG:boto:Data: 2024-04-13T22:28:53.371 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.371 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.371 DEBUG:boto:Params: {} 2024-04-13T22:28:53.371 DEBUG:boto:Token: None 2024-04-13T22:28:53.371 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M9weZp+FFDXGKe482uzfbjygDRU= 2024-04-13T22:28:53.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M9weZp+FFDXGKe482uzfbjygDRU=', 'Content-Length': '0'} 2024-04-13T22:28:53.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d578c4ef01b2a58a-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.373 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.373 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.373 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.374 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.374 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.374 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.374 DEBUG:boto:Data: 2024-04-13T22:28:53.374 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.374 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.374 DEBUG:boto:Params: {} 2024-04-13T22:28:53.374 DEBUG:boto:Token: None 2024-04-13T22:28:53.374 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.374 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M9weZp+FFDXGKe482uzfbjygDRU= 2024-04-13T22:28:53.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M9weZp+FFDXGKe482uzfbjygDRU=', 'Content-Length': '0'} 2024-04-13T22:28:53.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0b4f8398f80639-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:28:53.376 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.376 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.376 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.377 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.377 DEBUG:boto:Method: GET 2024-04-13T22:28:53.377 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.377 DEBUG:boto:Data: 2024-04-13T22:28:53.377 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.377 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.377 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.377 DEBUG:boto:Params: {} 2024-04-13T22:28:53.377 DEBUG:boto:Token: None 2024-04-13T22:28:53.377 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.377 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U= 2024-04-13T22:28:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U=', 'Content-Length': '0'} 2024-04-13T22:28:53.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388575b721d1aad1-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.379 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.380 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.380 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.380 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.380 DEBUG:boto:Method: GET 2024-04-13T22:28:53.380 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.380 DEBUG:boto:Data: 2024-04-13T22:28:53.380 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.380 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.380 DEBUG:boto:Params: {} 2024-04-13T22:28:53.380 DEBUG:boto:Token: None 2024-04-13T22:28:53.381 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:28:53.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U= 2024-04-13T22:28:53.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NewFkVPD2TMpTBjO3Jv3jVZ9V3U=', 'Content-Length': '0'} 2024-04-13T22:28:53.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc50b1785a390a2-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.383 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.383 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.383 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.383 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.383 DEBUG:boto:Method: GET 2024-04-13T22:28:53.383 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.383 DEBUG:boto:Data: 2024-04-13T22:28:53.383 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.384 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.384 DEBUG:boto:Params: {} 2024-04-13T22:28:53.384 DEBUG:boto:Token: None 2024-04-13T22:28:53.384 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4fa57320aff24fd-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.426 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.427 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.427 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.427 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.427 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.427 DEBUG:boto:Method: GET 2024-04-13T22:28:53.427 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.427 DEBUG:boto:Data: 2024-04-13T22:28:53.427 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.427 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.427 DEBUG:boto:Params: {} 2024-04-13T22:28:53.428 DEBUG:boto:Token: None 2024-04-13T22:28:53.428 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.428 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000935a8f2f6f7514eb-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.431 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.431 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.431 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.431 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.431 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:28:53.432 DEBUG:boto:Data: 2024-04-13T22:28:53.432 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.432 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.432 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.432 DEBUG:boto:Params: {} 2024-04-13T22:28:53.432 DEBUG:boto:Token: None 2024-04-13T22:28:53.432 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17 2024-04-13T22:28:53.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/IKGJnSo/0LsaitGPJQPInSxQac= 2024-04-13T22:28:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/IKGJnSo/0LsaitGPJQPInSxQac=', 'Content-Length': '0'} 2024-04-13T22:28:53.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb1b569d133c2d7e-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.434 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.434 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.434 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.434 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:28:53.434 DEBUG:boto:Data: 2024-04-13T22:28:53.435 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.435 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.435 DEBUG:boto:Params: {} 2024-04-13T22:28:53.435 DEBUG:boto:Token: None 2024-04-13T22:28:53.435 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17 2024-04-13T22:28:53.435 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/IKGJnSo/0LsaitGPJQPInSxQac= 2024-04-13T22:28:53.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/IKGJnSo/0LsaitGPJQPInSxQac=', 'Content-Length': '0'} 2024-04-13T22:28:53.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec5f0c8ef724c91-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:28:53.437 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.437 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.437 DEBUG:boto:Method: GET 2024-04-13T22:28:53.437 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:28:53.437 DEBUG:boto:Data: 2024-04-13T22:28:53.437 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.438 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.438 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.438 DEBUG:boto:Params: {} 2024-04-13T22:28:53.438 DEBUG:boto:Token: None 2024-04-13T22:28:53.438 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17 2024-04-13T22:28:53.438 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K/jUWDGMFsT+2m9cNnTxuDeZp/Q= 2024-04-13T22:28:53.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K/jUWDGMFsT+2m9cNnTxuDeZp/Q=', 'Content-Length': '0'} 2024-04-13T22:28:53.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187e11eb979d0b58-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.440 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:28:53.440 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:28:53.440 DEBUG:boto:Method: GET 2024-04-13T22:28:53.440 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:28:53.441 DEBUG:boto:Data: 2024-04-13T22:28:53.441 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.441 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.441 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.441 DEBUG:boto:Params: {} 2024-04-13T22:28:53.441 DEBUG:boto:Token: None 2024-04-13T22:28:53.441 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj17 2024-04-13T22:28:53.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K/jUWDGMFsT+2m9cNnTxuDeZp/Q= 2024-04-13T22:28:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K/jUWDGMFsT+2m9cNnTxuDeZp/Q=', 'Content-Length': '0'} 2024-04-13T22:28:53.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919b98b0361e93c5-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.443 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.443 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.443 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.443 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.443 DEBUG:boto:Method: GET 2024-04-13T22:28:53.443 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.443 DEBUG:boto:Data: 2024-04-13T22:28:53.444 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.444 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.444 DEBUG:boto:Params: {} 2024-04-13T22:28:53.444 DEBUG:boto:Token: None 2024-04-13T22:28:53.444 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000166110caa3a6d2a3-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.447 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.447 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.447 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.447 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.447 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.447 DEBUG:boto:Method: GET 2024-04-13T22:28:53.447 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:28:53.447 DEBUG:boto:Data: 2024-04-13T22:28:53.448 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.448 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.448 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.448 DEBUG:boto:Params: {} 2024-04-13T22:28:53.448 DEBUG:boto:Token: None 2024-04-13T22:28:53.448 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a3a16c7ce786b5-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.494 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:28:53.495 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.495 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.495 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.495 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.495 DEBUG:boto:Method: GET 2024-04-13T22:28:53.495 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.495 DEBUG:boto:Data: 2024-04-13T22:28:53.495 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.495 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.495 DEBUG:boto:Params: {} 2024-04-13T22:28:53.496 DEBUG:boto:Token: None 2024-04-13T22:28:53.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA= 2024-04-13T22:28:53.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA=', 'Content-Length': '0'} 2024-04-13T22:28:53.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c778808052a1b38-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.498 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.498 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.498 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.498 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.498 DEBUG:boto:Method: GET 2024-04-13T22:28:53.498 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.498 DEBUG:boto:Data: 2024-04-13T22:28:53.498 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.498 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.499 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.499 DEBUG:boto:Params: {} 2024-04-13T22:28:53.499 DEBUG:boto:Token: None 2024-04-13T22:28:53.499 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.499 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA= 2024-04-13T22:28:53.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA=', 'Content-Length': '0'} 2024-04-13T22:28:53.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000798446845053f4b0-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.501 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.501 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.501 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.501 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.501 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.501 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.501 DEBUG:boto:Data: 2024-04-13T22:28:53.501 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.501 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.501 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.502 DEBUG:boto:Params: {} 2024-04-13T22:28:53.502 DEBUG:boto:Token: None 2024-04-13T22:28:53.502 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.502 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DCmEsY5+rjIRLcib6EuKamUNmIw= 2024-04-13T22:28:53.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DCmEsY5+rjIRLcib6EuKamUNmIw=', 'Content-Length': '0'} 2024-04-13T22:28:53.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111bffecc60e94b5-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.504 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.504 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.504 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.504 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.504 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.504 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.504 DEBUG:boto:Data: 2024-04-13T22:28:53.504 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.504 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.504 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.504 DEBUG:boto:Params: {} 2024-04-13T22:28:53.505 DEBUG:boto:Token: None 2024-04-13T22:28:53.505 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.505 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DCmEsY5+rjIRLcib6EuKamUNmIw= 2024-04-13T22:28:53.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DCmEsY5+rjIRLcib6EuKamUNmIw=', 'Content-Length': '0'} 2024-04-13T22:28:53.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e7e5260d4d0f29-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:28:53.507 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.507 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.507 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.507 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.507 DEBUG:boto:Method: GET 2024-04-13T22:28:53.507 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.507 DEBUG:boto:Data: 2024-04-13T22:28:53.507 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.507 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.507 DEBUG:boto:Params: {} 2024-04-13T22:28:53.507 DEBUG:boto:Token: None 2024-04-13T22:28:53.508 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA= 2024-04-13T22:28:53.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA=', 'Content-Length': '0'} 2024-04-13T22:28:53.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060eac4dc269bec3-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.509 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.509 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.510 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.510 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.510 DEBUG:boto:Method: GET 2024-04-13T22:28:53.510 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.510 DEBUG:boto:Data: 2024-04-13T22:28:53.510 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.510 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.510 DEBUG:boto:Params: {} 2024-04-13T22:28:53.510 DEBUG:boto:Token: None 2024-04-13T22:28:53.510 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:28:53.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA= 2024-04-13T22:28:53.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qryATm+vPi62673XXGxsMOq/LZA=', 'Content-Length': '0'} 2024-04-13T22:28:53.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000415336f977309893-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.512 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.513 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.513 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.513 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.513 DEBUG:boto:Method: GET 2024-04-13T22:28:53.513 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.513 DEBUG:boto:Data: 2024-04-13T22:28:53.513 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.513 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.513 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.513 DEBUG:boto:Params: {} 2024-04-13T22:28:53.513 DEBUG:boto:Token: None 2024-04-13T22:28:53.513 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.514 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db419e66f509f029-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.516 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.516 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.516 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.516 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.516 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.517 DEBUG:boto:Method: GET 2024-04-13T22:28:53.517 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.517 DEBUG:boto:Data: 2024-04-13T22:28:53.517 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.517 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.517 DEBUG:boto:Params: {} 2024-04-13T22:28:53.517 DEBUG:boto:Token: None 2024-04-13T22:28:53.517 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.517 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006054b6f7321ef373-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.521 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.521 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.521 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.521 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.522 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:28:53.522 DEBUG:boto:Data: 2024-04-13T22:28:53.522 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.522 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.522 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.522 DEBUG:boto:Params: {} 2024-04-13T22:28:53.522 DEBUG:boto:Token: None 2024-04-13T22:28:53.522 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19 2024-04-13T22:28:53.522 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2IcIiOEN+BRtIdja189ZhFvA+d0= 2024-04-13T22:28:53.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2IcIiOEN+BRtIdja189ZhFvA+d0=', 'Content-Length': '0'} 2024-04-13T22:28:53.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f1f194748ed3654-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.524 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.524 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.524 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.524 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:28:53.524 DEBUG:boto:Data: 2024-04-13T22:28:53.524 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.525 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.525 DEBUG:boto:Params: {} 2024-04-13T22:28:53.525 DEBUG:boto:Token: None 2024-04-13T22:28:53.525 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19 2024-04-13T22:28:53.525 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2IcIiOEN+BRtIdja189ZhFvA+d0= 2024-04-13T22:28:53.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2IcIiOEN+BRtIdja189ZhFvA+d0=', 'Content-Length': '0'} 2024-04-13T22:28:53.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3e93845b9399080-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:28:53.527 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.527 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.527 DEBUG:boto:Method: GET 2024-04-13T22:28:53.527 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:28:53.527 DEBUG:boto:Data: 2024-04-13T22:28:53.527 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.527 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.527 DEBUG:boto:Params: {} 2024-04-13T22:28:53.527 DEBUG:boto:Token: None 2024-04-13T22:28:53.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19 2024-04-13T22:28:53.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MN+eHOSM+XcJNeUsvB+SAiCOmAA= 2024-04-13T22:28:53.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MN+eHOSM+XcJNeUsvB+SAiCOmAA=', 'Content-Length': '0'} 2024-04-13T22:28:53.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007858d47b3e6a61e3-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.530 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:28:53.530 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:28:53.530 DEBUG:boto:Method: GET 2024-04-13T22:28:53.530 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:28:53.530 DEBUG:boto:Data: 2024-04-13T22:28:53.530 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.530 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.530 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.530 DEBUG:boto:Params: {} 2024-04-13T22:28:53.530 DEBUG:boto:Token: None 2024-04-13T22:28:53.530 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj19 2024-04-13T22:28:53.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MN+eHOSM+XcJNeUsvB+SAiCOmAA= 2024-04-13T22:28:53.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MN+eHOSM+XcJNeUsvB+SAiCOmAA=', 'Content-Length': '0'} 2024-04-13T22:28:53.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338b414aa8aab733-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.575 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.575 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.575 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.575 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.575 DEBUG:boto:Method: GET 2024-04-13T22:28:53.575 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.575 DEBUG:boto:Data: 2024-04-13T22:28:53.576 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.576 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.576 DEBUG:boto:Params: {} 2024-04-13T22:28:53.576 DEBUG:boto:Token: None 2024-04-13T22:28:53.576 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000457b4265a89c6fb4-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.579 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.580 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.580 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.580 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.580 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.580 DEBUG:boto:Method: GET 2024-04-13T22:28:53.580 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:28:53.580 DEBUG:boto:Data: 2024-04-13T22:28:53.580 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.580 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.580 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.581 DEBUG:boto:Params: {} 2024-04-13T22:28:53.581 DEBUG:boto:Token: None 2024-04-13T22:28:53.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.581 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba3fd24a7f4eff3-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.586 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:28:53.586 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.586 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.587 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.587 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.587 DEBUG:boto:Method: GET 2024-04-13T22:28:53.587 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.587 DEBUG:boto:Data: 2024-04-13T22:28:53.587 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.587 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.587 DEBUG:boto:Params: {} 2024-04-13T22:28:53.587 DEBUG:boto:Token: None 2024-04-13T22:28:53.587 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w= 2024-04-13T22:28:53.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w=', 'Content-Length': '0'} 2024-04-13T22:28:53.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba98de4429f019e4-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.631 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.631 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.631 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.631 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.631 DEBUG:boto:Method: GET 2024-04-13T22:28:53.631 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.631 DEBUG:boto:Data: 2024-04-13T22:28:53.631 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.632 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.632 DEBUG:boto:Params: {} 2024-04-13T22:28:53.632 DEBUG:boto:Token: None 2024-04-13T22:28:53.632 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w= 2024-04-13T22:28:53.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w=', 'Content-Length': '0'} 2024-04-13T22:28:53.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004807e5215c896f04-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.634 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.634 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.634 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.635 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.635 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.635 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.635 DEBUG:boto:Data: 2024-04-13T22:28:53.635 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.635 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.635 DEBUG:boto:Params: {} 2024-04-13T22:28:53.635 DEBUG:boto:Token: None 2024-04-13T22:28:53.635 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U3ESAwKDMrJL7JRaXJPC2rHmHCQ= 2024-04-13T22:28:53.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U3ESAwKDMrJL7JRaXJPC2rHmHCQ=', 'Content-Length': '0'} 2024-04-13T22:28:53.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8379514715e03b2-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.637 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.637 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.638 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.638 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.638 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.638 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.638 DEBUG:boto:Data: 2024-04-13T22:28:53.638 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.638 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.638 DEBUG:boto:Params: {} 2024-04-13T22:28:53.638 DEBUG:boto:Token: None 2024-04-13T22:28:53.638 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U3ESAwKDMrJL7JRaXJPC2rHmHCQ= 2024-04-13T22:28:53.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U3ESAwKDMrJL7JRaXJPC2rHmHCQ=', 'Content-Length': '0'} 2024-04-13T22:28:53.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf5d5e1a9827d649-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:28:53.641 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.641 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.641 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.641 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.641 DEBUG:boto:Method: GET 2024-04-13T22:28:53.641 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.641 DEBUG:boto:Data: 2024-04-13T22:28:53.641 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.641 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.641 DEBUG:boto:Params: {} 2024-04-13T22:28:53.642 DEBUG:boto:Token: None 2024-04-13T22:28:53.642 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w= 2024-04-13T22:28:53.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w=', 'Content-Length': '0'} 2024-04-13T22:28:53.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058233f080372b150-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.644 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.644 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.644 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.644 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.644 DEBUG:boto:Method: GET 2024-04-13T22:28:53.644 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.644 DEBUG:boto:Data: 2024-04-13T22:28:53.644 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.645 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.645 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.645 DEBUG:boto:Params: {} 2024-04-13T22:28:53.645 DEBUG:boto:Token: None 2024-04-13T22:28:53.645 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:28:53.645 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w= 2024-04-13T22:28:53.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7v0lKD31WN+PexVq+En3fINFx/w=', 'Content-Length': '0'} 2024-04-13T22:28:53.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d7724aafc3d325-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.647 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.647 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.647 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.647 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.647 DEBUG:boto:Method: GET 2024-04-13T22:28:53.647 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.648 DEBUG:boto:Data: 2024-04-13T22:28:53.648 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.648 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.648 DEBUG:boto:Params: {} 2024-04-13T22:28:53.648 DEBUG:boto:Token: None 2024-04-13T22:28:53.648 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f8c4963681d9075-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.651 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.651 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.651 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.651 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.651 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.651 DEBUG:boto:Method: GET 2024-04-13T22:28:53.652 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.652 DEBUG:boto:Data: 2024-04-13T22:28:53.652 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.652 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.652 DEBUG:boto:Params: {} 2024-04-13T22:28:53.652 DEBUG:boto:Token: None 2024-04-13T22:28:53.652 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64a0246709173a3-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.656 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.656 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.656 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.656 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.656 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:28:53.656 DEBUG:boto:Data: 2024-04-13T22:28:53.656 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.656 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.657 DEBUG:boto:Params: {} 2024-04-13T22:28:53.657 DEBUG:boto:Token: None 2024-04-13T22:28:53.657 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22 2024-04-13T22:28:53.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G3pm4aoBBh61YQuWbEOshoFhXSw= 2024-04-13T22:28:53.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G3pm4aoBBh61YQuWbEOshoFhXSw=', 'Content-Length': '0'} 2024-04-13T22:28:53.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2cd11e09f78caa-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.659 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.659 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.659 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.659 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:28:53.659 DEBUG:boto:Data: 2024-04-13T22:28:53.659 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.659 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.659 DEBUG:boto:Params: {} 2024-04-13T22:28:53.659 DEBUG:boto:Token: None 2024-04-13T22:28:53.659 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22 2024-04-13T22:28:53.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G3pm4aoBBh61YQuWbEOshoFhXSw= 2024-04-13T22:28:53.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G3pm4aoBBh61YQuWbEOshoFhXSw=', 'Content-Length': '0'} 2024-04-13T22:28:53.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b027e93357e7ee7-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:28:53.662 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.662 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.662 DEBUG:boto:Method: GET 2024-04-13T22:28:53.662 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:28:53.662 DEBUG:boto:Data: 2024-04-13T22:28:53.662 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.662 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.662 DEBUG:boto:Params: {} 2024-04-13T22:28:53.662 DEBUG:boto:Token: None 2024-04-13T22:28:53.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22 2024-04-13T22:28:53.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J0mr6t1ttQ1RI57V18aLPqnCMyo= 2024-04-13T22:28:53.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J0mr6t1ttQ1RI57V18aLPqnCMyo=', 'Content-Length': '0'} 2024-04-13T22:28:53.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c187409da72c9d06-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.706 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:28:53.706 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:28:53.707 DEBUG:boto:Method: GET 2024-04-13T22:28:53.707 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:28:53.707 DEBUG:boto:Data: 2024-04-13T22:28:53.707 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.707 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.707 DEBUG:boto:Params: {} 2024-04-13T22:28:53.707 DEBUG:boto:Token: None 2024-04-13T22:28:53.707 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj22 2024-04-13T22:28:53.707 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J0mr6t1ttQ1RI57V18aLPqnCMyo= 2024-04-13T22:28:53.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J0mr6t1ttQ1RI57V18aLPqnCMyo=', 'Content-Length': '0'} 2024-04-13T22:28:53.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1c8320ed5a352f-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.709 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.710 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.710 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.710 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.710 DEBUG:boto:Method: GET 2024-04-13T22:28:53.710 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.710 DEBUG:boto:Data: 2024-04-13T22:28:53.710 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.710 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.710 DEBUG:boto:Params: {} 2024-04-13T22:28:53.711 DEBUG:boto:Token: None 2024-04-13T22:28:53.711 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b27514d27f828096-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.713 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.713 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.713 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.714 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.714 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.714 DEBUG:boto:Method: GET 2024-04-13T22:28:53.714 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:28:53.714 DEBUG:boto:Data: 2024-04-13T22:28:53.714 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.714 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.714 DEBUG:boto:Params: {} 2024-04-13T22:28:53.714 DEBUG:boto:Token: None 2024-04-13T22:28:53.714 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000200cdb1057d126cb-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.718 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:28:53.718 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.718 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.718 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.718 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.718 DEBUG:boto:Method: GET 2024-04-13T22:28:53.718 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.719 DEBUG:boto:Data: 2024-04-13T22:28:53.719 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.719 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.719 DEBUG:boto:Params: {} 2024-04-13T22:28:53.719 DEBUG:boto:Token: None 2024-04-13T22:28:53.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE= 2024-04-13T22:28:53.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE=', 'Content-Length': '0'} 2024-04-13T22:28:53.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f384695fa6362006-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.721 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.721 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.721 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.721 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.721 DEBUG:boto:Method: GET 2024-04-13T22:28:53.722 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.722 DEBUG:boto:Data: 2024-04-13T22:28:53.722 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.722 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.722 DEBUG:boto:Params: {} 2024-04-13T22:28:53.722 DEBUG:boto:Token: None 2024-04-13T22:28:53.722 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.722 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE= 2024-04-13T22:28:53.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE=', 'Content-Length': '0'} 2024-04-13T22:28:53.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93ed0cc9104c1ff-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.724 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.724 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.725 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.725 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.725 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.725 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.725 DEBUG:boto:Data: 2024-04-13T22:28:53.725 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.725 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.725 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.725 DEBUG:boto:Params: {} 2024-04-13T22:28:53.725 DEBUG:boto:Token: None 2024-04-13T22:28:53.725 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2Z7BcwXf9DJEoqY1vBEEouSX43I= 2024-04-13T22:28:53.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2Z7BcwXf9DJEoqY1vBEEouSX43I=', 'Content-Length': '0'} 2024-04-13T22:28:53.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195854a986db9124-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.727 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.727 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.727 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.728 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.728 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.728 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.728 DEBUG:boto:Data: 2024-04-13T22:28:53.728 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.728 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.728 DEBUG:boto:Params: {} 2024-04-13T22:28:53.728 DEBUG:boto:Token: None 2024-04-13T22:28:53.728 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2Z7BcwXf9DJEoqY1vBEEouSX43I= 2024-04-13T22:28:53.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2Z7BcwXf9DJEoqY1vBEEouSX43I=', 'Content-Length': '0'} 2024-04-13T22:28:53.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c182583f22d91630-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:28:53.731 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.731 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.731 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.731 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.731 DEBUG:boto:Method: GET 2024-04-13T22:28:53.731 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.731 DEBUG:boto:Data: 2024-04-13T22:28:53.731 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.731 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.731 DEBUG:boto:Params: {} 2024-04-13T22:28:53.731 DEBUG:boto:Token: None 2024-04-13T22:28:53.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE= 2024-04-13T22:28:53.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE=', 'Content-Length': '0'} 2024-04-13T22:28:53.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2a3c55811566b93-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.733 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.733 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.734 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.734 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.734 DEBUG:boto:Method: GET 2024-04-13T22:28:53.734 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.734 DEBUG:boto:Data: 2024-04-13T22:28:53.734 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.734 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.734 DEBUG:boto:Params: {} 2024-04-13T22:28:53.734 DEBUG:boto:Token: None 2024-04-13T22:28:53.734 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:28:53.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE= 2024-04-13T22:28:53.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6Q+YONMvwkCAfURwszMe3TQNzqE=', 'Content-Length': '0'} 2024-04-13T22:28:53.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011215e7e07001da4-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.736 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.737 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.737 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.737 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.737 DEBUG:boto:Method: GET 2024-04-13T22:28:53.737 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.737 DEBUG:boto:Data: 2024-04-13T22:28:53.737 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.737 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.737 DEBUG:boto:Params: {} 2024-04-13T22:28:53.737 DEBUG:boto:Token: None 2024-04-13T22:28:53.738 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.738 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020af30f918b437cc-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.740 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.741 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.741 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.741 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.741 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.741 DEBUG:boto:Method: GET 2024-04-13T22:28:53.741 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.741 DEBUG:boto:Data: 2024-04-13T22:28:53.741 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.741 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.741 DEBUG:boto:Params: {} 2024-04-13T22:28:53.742 DEBUG:boto:Token: None 2024-04-13T22:28:53.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e50e080e644b219-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.747 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.747 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.747 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.747 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.747 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:28:53.747 DEBUG:boto:Data: 2024-04-13T22:28:53.748 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.748 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.748 DEBUG:boto:Params: {} 2024-04-13T22:28:53.748 DEBUG:boto:Token: None 2024-04-13T22:28:53.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23 2024-04-13T22:28:53.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9eK2OEoM0uLCnaZmcuzI7REdxIw= 2024-04-13T22:28:53.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9eK2OEoM0uLCnaZmcuzI7REdxIw=', 'Content-Length': '0'} 2024-04-13T22:28:53.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000413e2241277e734e-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.750 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.750 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.750 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.750 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:28:53.750 DEBUG:boto:Data: 2024-04-13T22:28:53.750 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.750 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.750 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.751 DEBUG:boto:Params: {} 2024-04-13T22:28:53.751 DEBUG:boto:Token: None 2024-04-13T22:28:53.751 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23 2024-04-13T22:28:53.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9eK2OEoM0uLCnaZmcuzI7REdxIw= 2024-04-13T22:28:53.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9eK2OEoM0uLCnaZmcuzI7REdxIw=', 'Content-Length': '0'} 2024-04-13T22:28:53.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb7fc4e714cd507-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:28:53.753 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.753 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.753 DEBUG:boto:Method: GET 2024-04-13T22:28:53.753 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:28:53.753 DEBUG:boto:Data: 2024-04-13T22:28:53.753 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.753 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.753 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.753 DEBUG:boto:Params: {} 2024-04-13T22:28:53.754 DEBUG:boto:Token: None 2024-04-13T22:28:53.754 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23 2024-04-13T22:28:53.754 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1fuexnCpdCv247MrLVl/rHJOgvk= 2024-04-13T22:28:53.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1fuexnCpdCv247MrLVl/rHJOgvk=', 'Content-Length': '0'} 2024-04-13T22:28:53.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9b3e279fae7550f-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.756 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:28:53.756 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:28:53.756 DEBUG:boto:Method: GET 2024-04-13T22:28:53.756 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:28:53.756 DEBUG:boto:Data: 2024-04-13T22:28:53.756 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.756 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.756 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.756 DEBUG:boto:Params: {} 2024-04-13T22:28:53.756 DEBUG:boto:Token: None 2024-04-13T22:28:53.756 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj23 2024-04-13T22:28:53.757 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1fuexnCpdCv247MrLVl/rHJOgvk= 2024-04-13T22:28:53.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1fuexnCpdCv247MrLVl/rHJOgvk=', 'Content-Length': '0'} 2024-04-13T22:28:53.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e5635b8c26a47c7-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.758 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.759 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.759 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.759 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.759 DEBUG:boto:Method: GET 2024-04-13T22:28:53.759 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.759 DEBUG:boto:Data: 2024-04-13T22:28:53.759 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.759 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.759 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.759 DEBUG:boto:Params: {} 2024-04-13T22:28:53.759 DEBUG:boto:Token: None 2024-04-13T22:28:53.760 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.760 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eab4561ca0353c9e-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.762 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.763 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.763 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.763 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.763 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.763 DEBUG:boto:Method: GET 2024-04-13T22:28:53.763 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:28:53.763 DEBUG:boto:Data: 2024-04-13T22:28:53.763 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.763 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.763 DEBUG:boto:Params: {} 2024-04-13T22:28:53.763 DEBUG:boto:Token: None 2024-04-13T22:28:53.764 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b7d04572b5dceb4-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.810 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:28:53.811 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.811 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.811 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.811 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.811 DEBUG:boto:Method: GET 2024-04-13T22:28:53.811 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.811 DEBUG:boto:Data: 2024-04-13T22:28:53.812 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.812 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.812 DEBUG:boto:Params: {} 2024-04-13T22:28:53.812 DEBUG:boto:Token: None 2024-04-13T22:28:53.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE= 2024-04-13T22:28:53.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE=', 'Content-Length': '0'} 2024-04-13T22:28:53.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033973defce6a6f97-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.814 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.814 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.815 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.815 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.815 DEBUG:boto:Method: GET 2024-04-13T22:28:53.815 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.815 DEBUG:boto:Data: 2024-04-13T22:28:53.815 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.815 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.815 DEBUG:boto:Params: {} 2024-04-13T22:28:53.815 DEBUG:boto:Token: None 2024-04-13T22:28:53.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE= 2024-04-13T22:28:53.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE=', 'Content-Length': '0'} 2024-04-13T22:28:53.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008091e35651830847-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.817 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.817 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.817 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.818 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.818 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.818 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.818 DEBUG:boto:Data: 2024-04-13T22:28:53.818 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.818 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.818 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.818 DEBUG:boto:Params: {} 2024-04-13T22:28:53.818 DEBUG:boto:Token: None 2024-04-13T22:28:53.818 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.818 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lExTyfxidxFYMjA1zdu5n3D5PHQ= 2024-04-13T22:28:53.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lExTyfxidxFYMjA1zdu5n3D5PHQ=', 'Content-Length': '0'} 2024-04-13T22:28:53.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400dff87adcf7bf7-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.820 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.820 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.820 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.820 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.821 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.821 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.821 DEBUG:boto:Data: 2024-04-13T22:28:53.821 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.821 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.821 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.821 DEBUG:boto:Params: {} 2024-04-13T22:28:53.821 DEBUG:boto:Token: None 2024-04-13T22:28:53.821 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.821 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lExTyfxidxFYMjA1zdu5n3D5PHQ= 2024-04-13T22:28:53.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lExTyfxidxFYMjA1zdu5n3D5PHQ=', 'Content-Length': '0'} 2024-04-13T22:28:53.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67d4825fc25a912-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:28:53.823 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.823 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.823 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.823 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.823 DEBUG:boto:Method: GET 2024-04-13T22:28:53.824 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.824 DEBUG:boto:Data: 2024-04-13T22:28:53.824 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.824 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.824 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.824 DEBUG:boto:Params: {} 2024-04-13T22:28:53.824 DEBUG:boto:Token: None 2024-04-13T22:28:53.824 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE= 2024-04-13T22:28:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE=', 'Content-Length': '0'} 2024-04-13T22:28:53.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef6a0fd87afa68e-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.826 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.826 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.826 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.826 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.826 DEBUG:boto:Method: GET 2024-04-13T22:28:53.827 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.827 DEBUG:boto:Data: 2024-04-13T22:28:53.827 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.827 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.827 DEBUG:boto:Params: {} 2024-04-13T22:28:53.827 DEBUG:boto:Token: None 2024-04-13T22:28:53.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:28:53.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE= 2024-04-13T22:28:53.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yqFlv7R+fMxigV5BRLfQMoUzqsE=', 'Content-Length': '0'} 2024-04-13T22:28:53.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899c2423ada2f574-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.829 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.829 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.829 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.829 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.830 DEBUG:boto:Method: GET 2024-04-13T22:28:53.830 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.830 DEBUG:boto:Data: 2024-04-13T22:28:53.830 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.830 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.830 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.830 DEBUG:boto:Params: {} 2024-04-13T22:28:53.830 DEBUG:boto:Token: None 2024-04-13T22:28:53.830 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.830 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000630ec0a9aedc1b58-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.832 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.833 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.833 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.833 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.833 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.833 DEBUG:boto:Method: GET 2024-04-13T22:28:53.833 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.833 DEBUG:boto:Data: 2024-04-13T22:28:53.833 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.833 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.833 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.834 DEBUG:boto:Params: {} 2024-04-13T22:28:53.834 DEBUG:boto:Token: None 2024-04-13T22:28:53.834 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.834 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad6b0906e691f018-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.838 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.838 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.838 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.838 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.838 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:28:53.838 DEBUG:boto:Data: 2024-04-13T22:28:53.838 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.838 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.838 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.839 DEBUG:boto:Params: {} 2024-04-13T22:28:53.839 DEBUG:boto:Token: None 2024-04-13T22:28:53.839 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29 2024-04-13T22:28:53.839 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9eoz5Hs5CZqYDbqDZqk9bErn8Fc= 2024-04-13T22:28:53.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9eoz5Hs5CZqYDbqDZqk9bErn8Fc=', 'Content-Length': '0'} 2024-04-13T22:28:53.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061771db7a0a529c2-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.841 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.841 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.841 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.841 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:28:53.841 DEBUG:boto:Data: 2024-04-13T22:28:53.841 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.841 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.841 DEBUG:boto:Params: {} 2024-04-13T22:28:53.841 DEBUG:boto:Token: None 2024-04-13T22:28:53.842 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29 2024-04-13T22:28:53.842 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9eoz5Hs5CZqYDbqDZqk9bErn8Fc= 2024-04-13T22:28:53.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9eoz5Hs5CZqYDbqDZqk9bErn8Fc=', 'Content-Length': '0'} 2024-04-13T22:28:53.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459df17ef23d19b7-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:28:53.844 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.844 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.844 DEBUG:boto:Method: GET 2024-04-13T22:28:53.844 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:28:53.844 DEBUG:boto:Data: 2024-04-13T22:28:53.844 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.844 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.845 DEBUG:boto:Params: {} 2024-04-13T22:28:53.845 DEBUG:boto:Token: None 2024-04-13T22:28:53.845 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29 2024-04-13T22:28:53.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vovYNocCLKy7wRKt5V58ksr9r3I= 2024-04-13T22:28:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vovYNocCLKy7wRKt5V58ksr9r3I=', 'Content-Length': '0'} 2024-04-13T22:28:53.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ec7e3b08f5531f-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.847 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:28:53.847 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:28:53.847 DEBUG:boto:Method: GET 2024-04-13T22:28:53.847 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:28:53.847 DEBUG:boto:Data: 2024-04-13T22:28:53.847 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.847 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.847 DEBUG:boto:Params: {} 2024-04-13T22:28:53.848 DEBUG:boto:Token: None 2024-04-13T22:28:53.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj29 2024-04-13T22:28:53.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vovYNocCLKy7wRKt5V58ksr9r3I= 2024-04-13T22:28:53.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vovYNocCLKy7wRKt5V58ksr9r3I=', 'Content-Length': '0'} 2024-04-13T22:28:53.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002008dfd1cd198b36-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.850 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.850 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.850 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.850 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.850 DEBUG:boto:Method: GET 2024-04-13T22:28:53.850 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.851 DEBUG:boto:Data: 2024-04-13T22:28:53.851 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.851 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.851 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.851 DEBUG:boto:Params: {} 2024-04-13T22:28:53.851 DEBUG:boto:Token: None 2024-04-13T22:28:53.851 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f9ea307317ee77-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.894 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.895 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.895 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.895 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.895 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.895 DEBUG:boto:Method: GET 2024-04-13T22:28:53.895 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:28:53.895 DEBUG:boto:Data: 2024-04-13T22:28:53.895 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.895 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.895 DEBUG:boto:Params: {} 2024-04-13T22:28:53.896 DEBUG:boto:Token: None 2024-04-13T22:28:53.896 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a93918a7224ffc-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.899 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:28:53.900 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.900 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.900 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.900 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.900 DEBUG:boto:Method: GET 2024-04-13T22:28:53.900 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.900 DEBUG:boto:Data: 2024-04-13T22:28:53.900 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.900 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.901 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.901 DEBUG:boto:Params: {} 2024-04-13T22:28:53.901 DEBUG:boto:Token: None 2024-04-13T22:28:53.901 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.901 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g= 2024-04-13T22:28:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g=', 'Content-Length': '0'} 2024-04-13T22:28:53.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0cf4da070cc1a6-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.903 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.903 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.904 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.904 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.904 DEBUG:boto:Method: GET 2024-04-13T22:28:53.904 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.904 DEBUG:boto:Data: 2024-04-13T22:28:53.904 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.904 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.904 DEBUG:boto:Params: {} 2024-04-13T22:28:53.904 DEBUG:boto:Token: None 2024-04-13T22:28:53.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g= 2024-04-13T22:28:53.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g=', 'Content-Length': '0'} 2024-04-13T22:28:53.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000257a28a5581cd86e-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.950 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.951 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.951 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.951 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.951 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.951 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.951 DEBUG:boto:Data: 2024-04-13T22:28:53.951 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.951 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.951 DEBUG:boto:Params: {} 2024-04-13T22:28:53.952 DEBUG:boto:Token: None 2024-04-13T22:28:53.952 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8T1BZDjodv+zU6S+M5P9QU4b1ik= 2024-04-13T22:28:53.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8T1BZDjodv+zU6S+M5P9QU4b1ik=', 'Content-Length': '0'} 2024-04-13T22:28:53.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d45ca95ccd909b0-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.954 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.954 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.954 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.954 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.954 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.954 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.954 DEBUG:boto:Data: 2024-04-13T22:28:53.954 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.955 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.955 DEBUG:boto:Params: {} 2024-04-13T22:28:53.955 DEBUG:boto:Token: None 2024-04-13T22:28:53.955 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.955 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8T1BZDjodv+zU6S+M5P9QU4b1ik= 2024-04-13T22:28:53.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8T1BZDjodv+zU6S+M5P9QU4b1ik=', 'Content-Length': '0'} 2024-04-13T22:28:53.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4d2d3c387889cf-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:28:53.957 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.957 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.957 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.957 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.957 DEBUG:boto:Method: GET 2024-04-13T22:28:53.958 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.958 DEBUG:boto:Data: 2024-04-13T22:28:53.958 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.958 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.958 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.958 DEBUG:boto:Params: {} 2024-04-13T22:28:53.958 DEBUG:boto:Token: None 2024-04-13T22:28:53.958 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.958 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g= 2024-04-13T22:28:53.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g=', 'Content-Length': '0'} 2024-04-13T22:28:53.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac5edbd05e1e086-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.960 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.960 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.961 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.961 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.961 DEBUG:boto:Method: GET 2024-04-13T22:28:53.961 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.961 DEBUG:boto:Data: 2024-04-13T22:28:53.961 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.961 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.961 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.961 DEBUG:boto:Params: {} 2024-04-13T22:28:53.961 DEBUG:boto:Token: None 2024-04-13T22:28:53.961 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:28:53.962 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g= 2024-04-13T22:28:53.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6uijNUqorjY8ZjnIb7BKkCJ+93g=', 'Content-Length': '0'} 2024-04-13T22:28:53.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b0ead78151db65-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.964 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.964 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.964 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:53.964 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:53.964 DEBUG:boto:Method: GET 2024-04-13T22:28:53.964 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:28:53.964 DEBUG:boto:Data: 2024-04-13T22:28:53.964 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.964 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.964 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.964 DEBUG:boto:Params: {} 2024-04-13T22:28:53.965 DEBUG:boto:Token: None 2024-04-13T22:28:53.965 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b6732aedafb828-00661b0725-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.967 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.967 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:53.967 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:53.967 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:53.968 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:53.968 DEBUG:boto:Method: GET 2024-04-13T22:28:53.968 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:28:53.968 DEBUG:boto:Data: 2024-04-13T22:28:53.968 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.968 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.968 DEBUG:boto:Params: {} 2024-04-13T22:28:53.968 DEBUG:boto:Token: None 2024-04-13T22:28:53.968 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/ 2024-04-13T22:28:53.969 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8= 2024-04-13T22:28:53.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WgwhHVLnlNCTQgog75kmTiZTjn8=', 'Content-Length': '0'} 2024-04-13T22:28:53.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752eeba7e228d701-00661b0725-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.972 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:53.972 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.973 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.973 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.973 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:28:53.973 DEBUG:boto:Data: 2024-04-13T22:28:53.973 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.973 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.973 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.973 DEBUG:boto:Params: {} 2024-04-13T22:28:53.973 DEBUG:boto:Token: None 2024-04-13T22:28:53.974 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31 2024-04-13T22:28:53.974 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3iTbPyT4il43/aZ2Sxr2Q1+KFpg= 2024-04-13T22:28:53.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3iTbPyT4il43/aZ2Sxr2Q1+KFpg=', 'Content-Length': '0'} 2024-04-13T22:28:53.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef84af9422629b2-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.975 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.976 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.976 DEBUG:boto:Method: HEAD 2024-04-13T22:28:53.976 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:28:53.976 DEBUG:boto:Data: 2024-04-13T22:28:53.976 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.976 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.976 DEBUG:boto:Params: {} 2024-04-13T22:28:53.976 DEBUG:boto:Token: None 2024-04-13T22:28:53.976 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31 2024-04-13T22:28:53.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3iTbPyT4il43/aZ2Sxr2Q1+KFpg= 2024-04-13T22:28:53.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3iTbPyT4il43/aZ2Sxr2Q1+KFpg=', 'Content-Length': '0'} 2024-04-13T22:28:53.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0fae5080bbd15a1-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:28:53.979 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.979 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.979 DEBUG:boto:Method: GET 2024-04-13T22:28:53.979 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:28:53.979 DEBUG:boto:Data: 2024-04-13T22:28:53.979 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:53.979 DEBUG:boto:Port: 8001 2024-04-13T22:28:53.979 DEBUG:boto:Params: {} 2024-04-13T22:28:53.979 DEBUG:boto:Token: None 2024-04-13T22:28:53.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31 2024-04-13T22:28:53.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jnkXoL9Sp+DSyDa/ZCjVHD3KQPw= 2024-04-13T22:28:53.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jnkXoL9Sp+DSyDa/ZCjVHD3KQPw=', 'Content-Length': '0'} 2024-04-13T22:28:53.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b70a73e1427be8d-00661b0725-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:53.982 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:28:53.982 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:28:53.982 DEBUG:boto:Method: GET 2024-04-13T22:28:53.982 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:28:53.982 DEBUG:boto:Data: 2024-04-13T22:28:53.982 DEBUG:boto:Headers: {} 2024-04-13T22:28:53.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:53.983 DEBUG:boto:Port: 8000 2024-04-13T22:28:53.983 DEBUG:boto:Params: {} 2024-04-13T22:28:53.983 DEBUG:boto:Token: None 2024-04-13T22:28:53.983 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:53 GMT /vwnfns-61/obj31 2024-04-13T22:28:53.983 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jnkXoL9Sp+DSyDa/ZCjVHD3KQPw= 2024-04-13T22:28:53.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jnkXoL9Sp+DSyDa/ZCjVHD3KQPw=', 'Content-Length': '0'} 2024-04-13T22:28:53.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000079a65d6b4be7cf3-00661b0725-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.026 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.027 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.027 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:54.027 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:54.027 DEBUG:boto:Method: GET 2024-04-13T22:28:54.027 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:28:54.027 DEBUG:boto:Data: 2024-04-13T22:28:54.027 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.027 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.027 DEBUG:boto:Params: {} 2024-04-13T22:28:54.027 DEBUG:boto:Token: None 2024-04-13T22:28:54.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b42eed9e992b9153-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.030 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.030 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.031 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.031 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:54.031 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:28:54.031 DEBUG:boto:Method: GET 2024-04-13T22:28:54.031 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:28:54.031 DEBUG:boto:Data: 2024-04-13T22:28:54.031 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.031 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.031 DEBUG:boto:Params: {} 2024-04-13T22:28:54.031 DEBUG:boto:Token: None 2024-04-13T22:28:54.032 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.032 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f6706542c3308c-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.035 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:28:54.036 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.036 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.036 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.036 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.036 DEBUG:boto:Method: GET 2024-04-13T22:28:54.036 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.036 DEBUG:boto:Data: 2024-04-13T22:28:54.036 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.036 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.036 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.036 DEBUG:boto:Params: {} 2024-04-13T22:28:54.036 DEBUG:boto:Token: None 2024-04-13T22:28:54.037 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.037 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8= 2024-04-13T22:28:54.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8=', 'Content-Length': '0'} 2024-04-13T22:28:54.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27d4649b331b89c-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.039 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.039 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.039 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.039 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.039 DEBUG:boto:Method: GET 2024-04-13T22:28:54.039 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.039 DEBUG:boto:Data: 2024-04-13T22:28:54.039 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.039 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.039 DEBUG:boto:Params: {} 2024-04-13T22:28:54.040 DEBUG:boto:Token: None 2024-04-13T22:28:54.040 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8= 2024-04-13T22:28:54.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8=', 'Content-Length': '0'} 2024-04-13T22:28:54.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4912a801ab9231d-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.042 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.042 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.042 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.042 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.042 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.042 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.042 DEBUG:boto:Data: 2024-04-13T22:28:54.042 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.042 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.042 DEBUG:boto:Params: {} 2024-04-13T22:28:54.043 DEBUG:boto:Token: None 2024-04-13T22:28:54.043 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LweXxaZmrn5VMEmEp8Zbg6S9zrQ= 2024-04-13T22:28:54.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LweXxaZmrn5VMEmEp8Zbg6S9zrQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cead59dbf31af214-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.045 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.045 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.045 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.045 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.045 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.045 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.046 DEBUG:boto:Data: 2024-04-13T22:28:54.046 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.046 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.046 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.046 DEBUG:boto:Params: {} 2024-04-13T22:28:54.046 DEBUG:boto:Token: None 2024-04-13T22:28:54.046 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LweXxaZmrn5VMEmEp8Zbg6S9zrQ= 2024-04-13T22:28:54.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LweXxaZmrn5VMEmEp8Zbg6S9zrQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdefa235077eee2-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:28:54.048 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.048 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.049 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.049 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.049 DEBUG:boto:Method: GET 2024-04-13T22:28:54.049 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.049 DEBUG:boto:Data: 2024-04-13T22:28:54.049 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.049 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.049 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.049 DEBUG:boto:Params: {} 2024-04-13T22:28:54.049 DEBUG:boto:Token: None 2024-04-13T22:28:54.050 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.050 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8= 2024-04-13T22:28:54.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8=', 'Content-Length': '0'} 2024-04-13T22:28:54.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac48ce590ecf79a-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.094 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.095 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.095 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.095 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.095 DEBUG:boto:Method: GET 2024-04-13T22:28:54.095 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.095 DEBUG:boto:Data: 2024-04-13T22:28:54.095 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.095 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.095 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.095 DEBUG:boto:Params: {} 2024-04-13T22:28:54.096 DEBUG:boto:Token: None 2024-04-13T22:28:54.096 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:28:54.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8= 2024-04-13T22:28:54.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SLqfc3m2kqbt/oXRU1J9kC109Z8=', 'Content-Length': '0'} 2024-04-13T22:28:54.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9243a848e6e897d-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.098 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.098 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.098 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.098 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.098 DEBUG:boto:Method: GET 2024-04-13T22:28:54.098 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.098 DEBUG:boto:Data: 2024-04-13T22:28:54.099 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.099 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.099 DEBUG:boto:Params: {} 2024-04-13T22:28:54.099 DEBUG:boto:Token: None 2024-04-13T22:28:54.099 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4283db52b64152-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.102 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.102 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.102 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.102 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.102 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.103 DEBUG:boto:Method: GET 2024-04-13T22:28:54.103 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.103 DEBUG:boto:Data: 2024-04-13T22:28:54.103 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.103 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.103 DEBUG:boto:Params: {} 2024-04-13T22:28:54.103 DEBUG:boto:Token: None 2024-04-13T22:28:54.103 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c68db313e7bcaba-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.107 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.107 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.107 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.107 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.107 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:28:54.107 DEBUG:boto:Data: 2024-04-13T22:28:54.107 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.108 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.108 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.108 DEBUG:boto:Params: {} 2024-04-13T22:28:54.108 DEBUG:boto:Token: None 2024-04-13T22:28:54.108 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33 2024-04-13T22:28:54.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LYZjfSJj5P7wV6Kzh4x0qy6/cxM= 2024-04-13T22:28:54.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LYZjfSJj5P7wV6Kzh4x0qy6/cxM=', 'Content-Length': '0'} 2024-04-13T22:28:54.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e9f5697839abfd-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.110 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.110 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.110 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.110 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:28:54.111 DEBUG:boto:Data: 2024-04-13T22:28:54.111 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.111 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.111 DEBUG:boto:Params: {} 2024-04-13T22:28:54.111 DEBUG:boto:Token: None 2024-04-13T22:28:54.111 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33 2024-04-13T22:28:54.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LYZjfSJj5P7wV6Kzh4x0qy6/cxM= 2024-04-13T22:28:54.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LYZjfSJj5P7wV6Kzh4x0qy6/cxM=', 'Content-Length': '0'} 2024-04-13T22:28:54.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e6dd9d1824108e-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:28:54.113 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.113 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.113 DEBUG:boto:Method: GET 2024-04-13T22:28:54.113 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:28:54.113 DEBUG:boto:Data: 2024-04-13T22:28:54.113 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.114 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.114 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.114 DEBUG:boto:Params: {} 2024-04-13T22:28:54.114 DEBUG:boto:Token: None 2024-04-13T22:28:54.114 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33 2024-04-13T22:28:54.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2tjiwZRiXwJzgkDg+gAjfpdF8iE= 2024-04-13T22:28:54.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2tjiwZRiXwJzgkDg+gAjfpdF8iE=', 'Content-Length': '0'} 2024-04-13T22:28:54.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1359eb709ce390-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.116 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:28:54.116 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:28:54.116 DEBUG:boto:Method: GET 2024-04-13T22:28:54.116 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:28:54.116 DEBUG:boto:Data: 2024-04-13T22:28:54.116 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.117 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.117 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.117 DEBUG:boto:Params: {} 2024-04-13T22:28:54.117 DEBUG:boto:Token: None 2024-04-13T22:28:54.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj33 2024-04-13T22:28:54.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2tjiwZRiXwJzgkDg+gAjfpdF8iE= 2024-04-13T22:28:54.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2tjiwZRiXwJzgkDg+gAjfpdF8iE=', 'Content-Length': '0'} 2024-04-13T22:28:54.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389fca5122b8ed70-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.162 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.163 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.163 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.163 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.163 DEBUG:boto:Method: GET 2024-04-13T22:28:54.163 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.163 DEBUG:boto:Data: 2024-04-13T22:28:54.163 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.163 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.163 DEBUG:boto:Params: {} 2024-04-13T22:28:54.164 DEBUG:boto:Token: None 2024-04-13T22:28:54.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be5c399c38268d30-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.166 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.167 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.167 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.167 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.167 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.167 DEBUG:boto:Method: GET 2024-04-13T22:28:54.167 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:28:54.167 DEBUG:boto:Data: 2024-04-13T22:28:54.167 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.167 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.168 DEBUG:boto:Params: {} 2024-04-13T22:28:54.168 DEBUG:boto:Token: None 2024-04-13T22:28:54.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd3bbd6805346b7a-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.171 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:28:54.171 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.172 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.172 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.172 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.172 DEBUG:boto:Method: GET 2024-04-13T22:28:54.172 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.172 DEBUG:boto:Data: 2024-04-13T22:28:54.172 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.172 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.172 DEBUG:boto:Params: {} 2024-04-13T22:28:54.173 DEBUG:boto:Token: None 2024-04-13T22:28:54.173 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM= 2024-04-13T22:28:54.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM=', 'Content-Length': '0'} 2024-04-13T22:28:54.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86dcb9fdef27634-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.175 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.175 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.175 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.175 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.175 DEBUG:boto:Method: GET 2024-04-13T22:28:54.175 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.175 DEBUG:boto:Data: 2024-04-13T22:28:54.175 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.176 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.176 DEBUG:boto:Params: {} 2024-04-13T22:28:54.176 DEBUG:boto:Token: None 2024-04-13T22:28:54.176 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.176 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM= 2024-04-13T22:28:54.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM=', 'Content-Length': '0'} 2024-04-13T22:28:54.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050db6dd628a39b64-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.178 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.178 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.178 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.179 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.179 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.179 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.179 DEBUG:boto:Data: 2024-04-13T22:28:54.179 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.179 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.179 DEBUG:boto:Params: {} 2024-04-13T22:28:54.179 DEBUG:boto:Token: None 2024-04-13T22:28:54.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mNZ6xuXA3mD1Ps38bNPc6i+9clw= 2024-04-13T22:28:54.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mNZ6xuXA3mD1Ps38bNPc6i+9clw=', 'Content-Length': '0'} 2024-04-13T22:28:54.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088850f800d675293-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.181 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.181 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.182 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.182 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.182 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.182 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.182 DEBUG:boto:Data: 2024-04-13T22:28:54.182 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.182 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.182 DEBUG:boto:Params: {} 2024-04-13T22:28:54.182 DEBUG:boto:Token: None 2024-04-13T22:28:54.183 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mNZ6xuXA3mD1Ps38bNPc6i+9clw= 2024-04-13T22:28:54.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mNZ6xuXA3mD1Ps38bNPc6i+9clw=', 'Content-Length': '0'} 2024-04-13T22:28:54.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e717e34a09d09f-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:28:54.185 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.185 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.185 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.185 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.185 DEBUG:boto:Method: GET 2024-04-13T22:28:54.185 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.185 DEBUG:boto:Data: 2024-04-13T22:28:54.185 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.185 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.186 DEBUG:boto:Params: {} 2024-04-13T22:28:54.186 DEBUG:boto:Token: None 2024-04-13T22:28:54.186 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM= 2024-04-13T22:28:54.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM=', 'Content-Length': '0'} 2024-04-13T22:28:54.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de8c9668c59eaa1e-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.188 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.188 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.188 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.188 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.188 DEBUG:boto:Method: GET 2024-04-13T22:28:54.188 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.188 DEBUG:boto:Data: 2024-04-13T22:28:54.188 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.189 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.189 DEBUG:boto:Params: {} 2024-04-13T22:28:54.189 DEBUG:boto:Token: None 2024-04-13T22:28:54.189 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:28:54.189 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM= 2024-04-13T22:28:54.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TKmEyohhImDABk8ADoE1hlA0RIM=', 'Content-Length': '0'} 2024-04-13T22:28:54.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046b3fc110a04df08-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.191 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.191 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.191 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.191 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.191 DEBUG:boto:Method: GET 2024-04-13T22:28:54.191 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.192 DEBUG:boto:Data: 2024-04-13T22:28:54.192 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.192 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.192 DEBUG:boto:Params: {} 2024-04-13T22:28:54.192 DEBUG:boto:Token: None 2024-04-13T22:28:54.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f9d0166fb57872-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.194 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.195 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.195 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.195 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.195 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.195 DEBUG:boto:Method: GET 2024-04-13T22:28:54.195 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.195 DEBUG:boto:Data: 2024-04-13T22:28:54.195 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.196 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.196 DEBUG:boto:Params: {} 2024-04-13T22:28:54.196 DEBUG:boto:Token: None 2024-04-13T22:28:54.196 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009924853cf4bf2342-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.242 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.243 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.243 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.243 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.243 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:28:54.243 DEBUG:boto:Data: 2024-04-13T22:28:54.243 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.243 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.243 DEBUG:boto:Params: {} 2024-04-13T22:28:54.244 DEBUG:boto:Token: None 2024-04-13T22:28:54.244 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34 2024-04-13T22:28:54.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Zgq5Os9+jvGR431mM4AlDAYZiSQ= 2024-04-13T22:28:54.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Zgq5Os9+jvGR431mM4AlDAYZiSQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b5b1d82c884ce5-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.246 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.246 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.246 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.246 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:28:54.246 DEBUG:boto:Data: 2024-04-13T22:28:54.246 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.246 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.246 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.246 DEBUG:boto:Params: {} 2024-04-13T22:28:54.246 DEBUG:boto:Token: None 2024-04-13T22:28:54.246 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34 2024-04-13T22:28:54.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Zgq5Os9+jvGR431mM4AlDAYZiSQ= 2024-04-13T22:28:54.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Zgq5Os9+jvGR431mM4AlDAYZiSQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776d3f606d047623-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:28:54.248 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.249 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.249 DEBUG:boto:Method: GET 2024-04-13T22:28:54.249 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:28:54.249 DEBUG:boto:Data: 2024-04-13T22:28:54.249 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.249 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.249 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.249 DEBUG:boto:Params: {} 2024-04-13T22:28:54.249 DEBUG:boto:Token: None 2024-04-13T22:28:54.249 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34 2024-04-13T22:28:54.250 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/hGVIZ8EcXT7okippBvHKXKktJs= 2024-04-13T22:28:54.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/hGVIZ8EcXT7okippBvHKXKktJs=', 'Content-Length': '0'} 2024-04-13T22:28:54.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd8f168843f9827-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.252 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:28:54.252 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:28:54.252 DEBUG:boto:Method: GET 2024-04-13T22:28:54.252 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:28:54.252 DEBUG:boto:Data: 2024-04-13T22:28:54.252 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.252 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.252 DEBUG:boto:Params: {} 2024-04-13T22:28:54.252 DEBUG:boto:Token: None 2024-04-13T22:28:54.253 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj34 2024-04-13T22:28:54.253 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/hGVIZ8EcXT7okippBvHKXKktJs= 2024-04-13T22:28:54.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/hGVIZ8EcXT7okippBvHKXKktJs=', 'Content-Length': '0'} 2024-04-13T22:28:54.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ee40f7501ec7a7-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.255 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.255 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.255 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.255 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.255 DEBUG:boto:Method: GET 2024-04-13T22:28:54.255 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.255 DEBUG:boto:Data: 2024-04-13T22:28:54.255 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.255 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.255 DEBUG:boto:Params: {} 2024-04-13T22:28:54.256 DEBUG:boto:Token: None 2024-04-13T22:28:54.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009004733c6843a441-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.258 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.259 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.259 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.259 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.259 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.259 DEBUG:boto:Method: GET 2024-04-13T22:28:54.259 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:28:54.259 DEBUG:boto:Data: 2024-04-13T22:28:54.259 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.259 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.260 DEBUG:boto:Params: {} 2024-04-13T22:28:54.260 DEBUG:boto:Token: None 2024-04-13T22:28:54.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f783aee56e93cedc-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.263 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:28:54.264 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.264 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.264 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.264 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.264 DEBUG:boto:Method: GET 2024-04-13T22:28:54.264 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.264 DEBUG:boto:Data: 2024-04-13T22:28:54.265 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.265 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.265 DEBUG:boto:Params: {} 2024-04-13T22:28:54.265 DEBUG:boto:Token: None 2024-04-13T22:28:54.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw= 2024-04-13T22:28:54.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw=', 'Content-Length': '0'} 2024-04-13T22:28:54.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d14abe71bfd217-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.310 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.310 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.311 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.311 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.311 DEBUG:boto:Method: GET 2024-04-13T22:28:54.311 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.311 DEBUG:boto:Data: 2024-04-13T22:28:54.311 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.311 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.311 DEBUG:boto:Params: {} 2024-04-13T22:28:54.311 DEBUG:boto:Token: None 2024-04-13T22:28:54.311 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw= 2024-04-13T22:28:54.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw=', 'Content-Length': '0'} 2024-04-13T22:28:54.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e530ed3a4b7167-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.314 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.314 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.314 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.314 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.314 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.314 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.314 DEBUG:boto:Data: 2024-04-13T22:28:54.314 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.314 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.314 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.315 DEBUG:boto:Params: {} 2024-04-13T22:28:54.315 DEBUG:boto:Token: None 2024-04-13T22:28:54.315 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:My3M4ow6DoYp0dRX7Hui/QXKRsI= 2024-04-13T22:28:54.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:My3M4ow6DoYp0dRX7Hui/QXKRsI=', 'Content-Length': '0'} 2024-04-13T22:28:54.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f55724dc8697f5-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.317 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.317 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.317 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.317 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.317 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.317 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.317 DEBUG:boto:Data: 2024-04-13T22:28:54.317 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.317 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.317 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.317 DEBUG:boto:Params: {} 2024-04-13T22:28:54.318 DEBUG:boto:Token: None 2024-04-13T22:28:54.318 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.318 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:My3M4ow6DoYp0dRX7Hui/QXKRsI= 2024-04-13T22:28:54.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:My3M4ow6DoYp0dRX7Hui/QXKRsI=', 'Content-Length': '0'} 2024-04-13T22:28:54.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500762a2699d0328-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:28:54.320 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.320 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.320 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.320 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.321 DEBUG:boto:Method: GET 2024-04-13T22:28:54.321 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.321 DEBUG:boto:Data: 2024-04-13T22:28:54.321 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.321 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.321 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.321 DEBUG:boto:Params: {} 2024-04-13T22:28:54.321 DEBUG:boto:Token: None 2024-04-13T22:28:54.321 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.321 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw= 2024-04-13T22:28:54.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw=', 'Content-Length': '0'} 2024-04-13T22:28:54.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca598e7051dcd66-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.323 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.323 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.323 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.324 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.324 DEBUG:boto:Method: GET 2024-04-13T22:28:54.324 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.324 DEBUG:boto:Data: 2024-04-13T22:28:54.324 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.324 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.324 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.324 DEBUG:boto:Params: {} 2024-04-13T22:28:54.324 DEBUG:boto:Token: None 2024-04-13T22:28:54.324 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:28:54.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw= 2024-04-13T22:28:54.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qe/WERNIlCsZ7+XczRzNVUxdZVw=', 'Content-Length': '0'} 2024-04-13T22:28:54.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac39987941d846d-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.326 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.326 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.327 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.327 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.327 DEBUG:boto:Method: GET 2024-04-13T22:28:54.327 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.327 DEBUG:boto:Data: 2024-04-13T22:28:54.327 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.327 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.327 DEBUG:boto:Params: {} 2024-04-13T22:28:54.327 DEBUG:boto:Token: None 2024-04-13T22:28:54.328 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.328 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa0fd53fc2b5de7b-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.330 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.330 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.331 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.331 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.331 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.331 DEBUG:boto:Method: GET 2024-04-13T22:28:54.331 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.331 DEBUG:boto:Data: 2024-04-13T22:28:54.331 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.331 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.331 DEBUG:boto:Params: {} 2024-04-13T22:28:54.331 DEBUG:boto:Token: None 2024-04-13T22:28:54.332 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013cbe8bc1109d04c-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.335 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.336 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.336 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.336 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.336 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:28:54.336 DEBUG:boto:Data: 2024-04-13T22:28:54.336 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.336 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.336 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.336 DEBUG:boto:Params: {} 2024-04-13T22:28:54.336 DEBUG:boto:Token: None 2024-04-13T22:28:54.336 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38 2024-04-13T22:28:54.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFjL+O4tbaXlvDnQHhoFMvVE2jg= 2024-04-13T22:28:54.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFjL+O4tbaXlvDnQHhoFMvVE2jg=', 'Content-Length': '0'} 2024-04-13T22:28:54.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000276321b39ce74415-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.338 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.338 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.339 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.339 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:28:54.339 DEBUG:boto:Data: 2024-04-13T22:28:54.339 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.339 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.339 DEBUG:boto:Params: {} 2024-04-13T22:28:54.339 DEBUG:boto:Token: None 2024-04-13T22:28:54.339 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38 2024-04-13T22:28:54.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFjL+O4tbaXlvDnQHhoFMvVE2jg= 2024-04-13T22:28:54.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFjL+O4tbaXlvDnQHhoFMvVE2jg=', 'Content-Length': '0'} 2024-04-13T22:28:54.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784313dd968fda20-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:28:54.341 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.341 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.341 DEBUG:boto:Method: GET 2024-04-13T22:28:54.342 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:28:54.342 DEBUG:boto:Data: 2024-04-13T22:28:54.342 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.342 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.342 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.342 DEBUG:boto:Params: {} 2024-04-13T22:28:54.342 DEBUG:boto:Token: None 2024-04-13T22:28:54.342 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38 2024-04-13T22:28:54.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J7JW9gzmWnvAe/26BnA4YpP24TU= 2024-04-13T22:28:54.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J7JW9gzmWnvAe/26BnA4YpP24TU=', 'Content-Length': '0'} 2024-04-13T22:28:54.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b44bedf6f8cdee8-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.344 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:28:54.345 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:28:54.345 DEBUG:boto:Method: GET 2024-04-13T22:28:54.345 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:28:54.345 DEBUG:boto:Data: 2024-04-13T22:28:54.345 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.345 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.345 DEBUG:boto:Params: {} 2024-04-13T22:28:54.345 DEBUG:boto:Token: None 2024-04-13T22:28:54.345 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj38 2024-04-13T22:28:54.346 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J7JW9gzmWnvAe/26BnA4YpP24TU= 2024-04-13T22:28:54.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J7JW9gzmWnvAe/26BnA4YpP24TU=', 'Content-Length': '0'} 2024-04-13T22:28:54.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003993a61532abc112-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.394 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.395 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.395 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.395 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.395 DEBUG:boto:Method: GET 2024-04-13T22:28:54.395 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.395 DEBUG:boto:Data: 2024-04-13T22:28:54.395 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.395 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.395 DEBUG:boto:Params: {} 2024-04-13T22:28:54.395 DEBUG:boto:Token: None 2024-04-13T22:28:54.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000042b0a04ba9178b9-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.398 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.399 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.399 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.399 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.399 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.399 DEBUG:boto:Method: GET 2024-04-13T22:28:54.399 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:28:54.399 DEBUG:boto:Data: 2024-04-13T22:28:54.399 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.400 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.400 DEBUG:boto:Params: {} 2024-04-13T22:28:54.400 DEBUG:boto:Token: None 2024-04-13T22:28:54.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7ba3dc3497a491-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.403 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:28:54.404 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.404 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.404 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.404 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.404 DEBUG:boto:Method: GET 2024-04-13T22:28:54.404 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.404 DEBUG:boto:Data: 2024-04-13T22:28:54.404 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.405 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.405 DEBUG:boto:Params: {} 2024-04-13T22:28:54.405 DEBUG:boto:Token: None 2024-04-13T22:28:54.405 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.405 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI= 2024-04-13T22:28:54.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI=', 'Content-Length': '0'} 2024-04-13T22:28:54.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb4711a220bad71-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.450 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.451 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.451 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.451 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.451 DEBUG:boto:Method: GET 2024-04-13T22:28:54.451 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.451 DEBUG:boto:Data: 2024-04-13T22:28:54.451 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.451 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.451 DEBUG:boto:Params: {} 2024-04-13T22:28:54.451 DEBUG:boto:Token: None 2024-04-13T22:28:54.452 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.452 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI= 2024-04-13T22:28:54.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI=', 'Content-Length': '0'} 2024-04-13T22:28:54.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b5da270935bd7d-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.454 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.454 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.454 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.454 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.454 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.454 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.455 DEBUG:boto:Data: 2024-04-13T22:28:54.455 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.455 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.455 DEBUG:boto:Params: {} 2024-04-13T22:28:54.455 DEBUG:boto:Token: None 2024-04-13T22:28:54.455 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.455 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGHuEcsZyqhGHMI5aTd5c4wZtBI= 2024-04-13T22:28:54.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGHuEcsZyqhGHMI5aTd5c4wZtBI=', 'Content-Length': '0'} 2024-04-13T22:28:54.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e7c7988022c1a3b-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.457 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.457 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.457 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.457 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.457 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.457 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.458 DEBUG:boto:Data: 2024-04-13T22:28:54.458 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.458 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.458 DEBUG:boto:Params: {} 2024-04-13T22:28:54.458 DEBUG:boto:Token: None 2024-04-13T22:28:54.458 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.458 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGHuEcsZyqhGHMI5aTd5c4wZtBI= 2024-04-13T22:28:54.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGHuEcsZyqhGHMI5aTd5c4wZtBI=', 'Content-Length': '0'} 2024-04-13T22:28:54.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f729051f6f978ac-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:28:54.460 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.460 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.460 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.460 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.461 DEBUG:boto:Method: GET 2024-04-13T22:28:54.461 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.461 DEBUG:boto:Data: 2024-04-13T22:28:54.461 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.461 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.461 DEBUG:boto:Params: {} 2024-04-13T22:28:54.461 DEBUG:boto:Token: None 2024-04-13T22:28:54.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI= 2024-04-13T22:28:54.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI=', 'Content-Length': '0'} 2024-04-13T22:28:54.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001140f1cf3883262e-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.463 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.464 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.464 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.464 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.464 DEBUG:boto:Method: GET 2024-04-13T22:28:54.464 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.464 DEBUG:boto:Data: 2024-04-13T22:28:54.464 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.464 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.464 DEBUG:boto:Params: {} 2024-04-13T22:28:54.465 DEBUG:boto:Token: None 2024-04-13T22:28:54.465 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:28:54.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI= 2024-04-13T22:28:54.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0cfjjcmN8Ei1sN0Z3+96IL+WbtI=', 'Content-Length': '0'} 2024-04-13T22:28:54.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0dde4688c09387f-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.467 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.467 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.467 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.467 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.467 DEBUG:boto:Method: GET 2024-04-13T22:28:54.467 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.467 DEBUG:boto:Data: 2024-04-13T22:28:54.467 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.467 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.468 DEBUG:boto:Params: {} 2024-04-13T22:28:54.468 DEBUG:boto:Token: None 2024-04-13T22:28:54.468 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff5902d3dc312556-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.470 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.470 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.471 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.471 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.471 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.471 DEBUG:boto:Method: GET 2024-04-13T22:28:54.471 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.471 DEBUG:boto:Data: 2024-04-13T22:28:54.471 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.471 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.471 DEBUG:boto:Params: {} 2024-04-13T22:28:54.472 DEBUG:boto:Token: None 2024-04-13T22:28:54.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be745202939a0e6a-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.475 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.475 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.476 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.476 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.476 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:28:54.476 DEBUG:boto:Data: 2024-04-13T22:28:54.476 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.476 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.476 DEBUG:boto:Params: {} 2024-04-13T22:28:54.476 DEBUG:boto:Token: None 2024-04-13T22:28:54.476 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44 2024-04-13T22:28:54.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OdiDY1a8uzfc/4/HaOxXgF2AQeM= 2024-04-13T22:28:54.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OdiDY1a8uzfc/4/HaOxXgF2AQeM=', 'Content-Length': '0'} 2024-04-13T22:28:54.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012fc4edf994b803a-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.478 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.478 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.478 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.479 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:28:54.479 DEBUG:boto:Data: 2024-04-13T22:28:54.479 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.479 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.479 DEBUG:boto:Params: {} 2024-04-13T22:28:54.479 DEBUG:boto:Token: None 2024-04-13T22:28:54.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44 2024-04-13T22:28:54.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OdiDY1a8uzfc/4/HaOxXgF2AQeM= 2024-04-13T22:28:54.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OdiDY1a8uzfc/4/HaOxXgF2AQeM=', 'Content-Length': '0'} 2024-04-13T22:28:54.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374217a64b35c59b-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:28:54.481 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.481 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.481 DEBUG:boto:Method: GET 2024-04-13T22:28:54.481 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:28:54.481 DEBUG:boto:Data: 2024-04-13T22:28:54.482 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.482 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.482 DEBUG:boto:Params: {} 2024-04-13T22:28:54.482 DEBUG:boto:Token: None 2024-04-13T22:28:54.482 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44 2024-04-13T22:28:54.482 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D69HfxRangHBRZVsECe1i21oSFE= 2024-04-13T22:28:54.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D69HfxRangHBRZVsECe1i21oSFE=', 'Content-Length': '0'} 2024-04-13T22:28:54.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d0c7403a4f9f16-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.484 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:28:54.485 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:28:54.485 DEBUG:boto:Method: GET 2024-04-13T22:28:54.485 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:28:54.485 DEBUG:boto:Data: 2024-04-13T22:28:54.485 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.485 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.485 DEBUG:boto:Params: {} 2024-04-13T22:28:54.485 DEBUG:boto:Token: None 2024-04-13T22:28:54.485 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj44 2024-04-13T22:28:54.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D69HfxRangHBRZVsECe1i21oSFE= 2024-04-13T22:28:54.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D69HfxRangHBRZVsECe1i21oSFE=', 'Content-Length': '0'} 2024-04-13T22:28:54.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f78f5dd3efa306-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.530 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.531 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.531 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.531 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.531 DEBUG:boto:Method: GET 2024-04-13T22:28:54.531 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.531 DEBUG:boto:Data: 2024-04-13T22:28:54.531 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.531 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.531 DEBUG:boto:Params: {} 2024-04-13T22:28:54.531 DEBUG:boto:Token: None 2024-04-13T22:28:54.532 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007eaa70c5d4fb73c-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.534 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.535 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.535 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.535 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.535 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.535 DEBUG:boto:Method: GET 2024-04-13T22:28:54.535 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:28:54.535 DEBUG:boto:Data: 2024-04-13T22:28:54.535 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.536 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.536 DEBUG:boto:Params: {} 2024-04-13T22:28:54.536 DEBUG:boto:Token: None 2024-04-13T22:28:54.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c406c70721d17695-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.539 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:28:54.540 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.540 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.540 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.540 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.540 DEBUG:boto:Method: GET 2024-04-13T22:28:54.540 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.540 DEBUG:boto:Data: 2024-04-13T22:28:54.540 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.540 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.541 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.541 DEBUG:boto:Params: {} 2024-04-13T22:28:54.541 DEBUG:boto:Token: None 2024-04-13T22:28:54.541 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg= 2024-04-13T22:28:54.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg=', 'Content-Length': '0'} 2024-04-13T22:28:54.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be14767d6ab477f-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.586 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.586 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.587 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.587 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.587 DEBUG:boto:Method: GET 2024-04-13T22:28:54.587 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.587 DEBUG:boto:Data: 2024-04-13T22:28:54.587 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.587 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.587 DEBUG:boto:Params: {} 2024-04-13T22:28:54.587 DEBUG:boto:Token: None 2024-04-13T22:28:54.587 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.588 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg= 2024-04-13T22:28:54.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg=', 'Content-Length': '0'} 2024-04-13T22:28:54.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002103c9077af7562e-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.590 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.590 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.590 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.590 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.590 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.590 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.590 DEBUG:boto:Data: 2024-04-13T22:28:54.590 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.590 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.591 DEBUG:boto:Params: {} 2024-04-13T22:28:54.591 DEBUG:boto:Token: None 2024-04-13T22:28:54.591 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.591 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PkCIBQBYvGITaRFzzaDIyZQC/Hk= 2024-04-13T22:28:54.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PkCIBQBYvGITaRFzzaDIyZQC/Hk=', 'Content-Length': '0'} 2024-04-13T22:28:54.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d143390b63a2c6f6-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.593 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.593 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.593 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.593 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.593 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.593 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.593 DEBUG:boto:Data: 2024-04-13T22:28:54.593 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.594 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.594 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.594 DEBUG:boto:Params: {} 2024-04-13T22:28:54.594 DEBUG:boto:Token: None 2024-04-13T22:28:54.594 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PkCIBQBYvGITaRFzzaDIyZQC/Hk= 2024-04-13T22:28:54.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PkCIBQBYvGITaRFzzaDIyZQC/Hk=', 'Content-Length': '0'} 2024-04-13T22:28:54.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026648c190cc1f4e7-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:28:54.596 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.596 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.596 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.596 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.596 DEBUG:boto:Method: GET 2024-04-13T22:28:54.596 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.597 DEBUG:boto:Data: 2024-04-13T22:28:54.597 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.597 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.597 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.597 DEBUG:boto:Params: {} 2024-04-13T22:28:54.597 DEBUG:boto:Token: None 2024-04-13T22:28:54.597 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg= 2024-04-13T22:28:54.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg=', 'Content-Length': '0'} 2024-04-13T22:28:54.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1947fc99270eb2-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.599 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.599 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.599 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.600 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.600 DEBUG:boto:Method: GET 2024-04-13T22:28:54.600 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.600 DEBUG:boto:Data: 2024-04-13T22:28:54.600 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.600 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.600 DEBUG:boto:Params: {} 2024-04-13T22:28:54.600 DEBUG:boto:Token: None 2024-04-13T22:28:54.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:28:54.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg= 2024-04-13T22:28:54.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PsxsNXcCp4OKhESAklLmwFUProg=', 'Content-Length': '0'} 2024-04-13T22:28:54.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002023863877f41872-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.602 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.603 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.603 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.603 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.603 DEBUG:boto:Method: GET 2024-04-13T22:28:54.603 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.603 DEBUG:boto:Data: 2024-04-13T22:28:54.603 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.603 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.603 DEBUG:boto:Params: {} 2024-04-13T22:28:54.604 DEBUG:boto:Token: None 2024-04-13T22:28:54.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b383bdfcce502775-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.606 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.606 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.606 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.607 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.607 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.607 DEBUG:boto:Method: GET 2024-04-13T22:28:54.607 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.607 DEBUG:boto:Data: 2024-04-13T22:28:54.607 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.607 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.607 DEBUG:boto:Params: {} 2024-04-13T22:28:54.607 DEBUG:boto:Token: None 2024-04-13T22:28:54.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f97e02679b7266-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.611 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.611 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.612 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.612 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.612 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:28:54.612 DEBUG:boto:Data: 2024-04-13T22:28:54.612 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.612 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.612 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.612 DEBUG:boto:Params: {} 2024-04-13T22:28:54.612 DEBUG:boto:Token: None 2024-04-13T22:28:54.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46 2024-04-13T22:28:54.613 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xl4dxwPu2u+rxa1fCKdze95yeSk= 2024-04-13T22:28:54.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xl4dxwPu2u+rxa1fCKdze95yeSk=', 'Content-Length': '0'} 2024-04-13T22:28:54.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d6ea4aec05763f-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.614 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.614 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.614 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.615 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:28:54.615 DEBUG:boto:Data: 2024-04-13T22:28:54.615 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.615 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.615 DEBUG:boto:Params: {} 2024-04-13T22:28:54.615 DEBUG:boto:Token: None 2024-04-13T22:28:54.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46 2024-04-13T22:28:54.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xl4dxwPu2u+rxa1fCKdze95yeSk= 2024-04-13T22:28:54.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xl4dxwPu2u+rxa1fCKdze95yeSk=', 'Content-Length': '0'} 2024-04-13T22:28:54.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad68b882b6b9c8c-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:28:54.617 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.617 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.617 DEBUG:boto:Method: GET 2024-04-13T22:28:54.617 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:28:54.617 DEBUG:boto:Data: 2024-04-13T22:28:54.618 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.618 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.618 DEBUG:boto:Params: {} 2024-04-13T22:28:54.618 DEBUG:boto:Token: None 2024-04-13T22:28:54.618 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46 2024-04-13T22:28:54.618 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vVh0IWGJnKCj3HqDmjj75DHKdks= 2024-04-13T22:28:54.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vVh0IWGJnKCj3HqDmjj75DHKdks=', 'Content-Length': '0'} 2024-04-13T22:28:54.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b30c49350b2b7dd-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.620 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:28:54.620 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:28:54.620 DEBUG:boto:Method: GET 2024-04-13T22:28:54.620 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:28:54.620 DEBUG:boto:Data: 2024-04-13T22:28:54.620 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.621 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.621 DEBUG:boto:Params: {} 2024-04-13T22:28:54.621 DEBUG:boto:Token: None 2024-04-13T22:28:54.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj46 2024-04-13T22:28:54.621 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vVh0IWGJnKCj3HqDmjj75DHKdks= 2024-04-13T22:28:54.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vVh0IWGJnKCj3HqDmjj75DHKdks=', 'Content-Length': '0'} 2024-04-13T22:28:54.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2da9a0af007a15c-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.666 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.666 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.666 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.667 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.667 DEBUG:boto:Method: GET 2024-04-13T22:28:54.667 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.667 DEBUG:boto:Data: 2024-04-13T22:28:54.667 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.667 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.667 DEBUG:boto:Params: {} 2024-04-13T22:28:54.667 DEBUG:boto:Token: None 2024-04-13T22:28:54.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.668 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce50b1896e30125c-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.670 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.670 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.670 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.671 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.671 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.671 DEBUG:boto:Method: GET 2024-04-13T22:28:54.671 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:28:54.671 DEBUG:boto:Data: 2024-04-13T22:28:54.671 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.671 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.671 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.671 DEBUG:boto:Params: {} 2024-04-13T22:28:54.671 DEBUG:boto:Token: None 2024-04-13T22:28:54.671 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dca0b35e02535ff-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.675 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:28:54.675 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.676 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.676 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.676 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.676 DEBUG:boto:Method: GET 2024-04-13T22:28:54.676 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.676 DEBUG:boto:Data: 2024-04-13T22:28:54.676 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.676 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.676 DEBUG:boto:Params: {} 2024-04-13T22:28:54.677 DEBUG:boto:Token: None 2024-04-13T22:28:54.677 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI= 2024-04-13T22:28:54.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI=', 'Content-Length': '0'} 2024-04-13T22:28:54.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddc375c69fc8547c-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.679 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.679 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.679 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.679 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.679 DEBUG:boto:Method: GET 2024-04-13T22:28:54.679 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.679 DEBUG:boto:Data: 2024-04-13T22:28:54.679 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.679 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.680 DEBUG:boto:Params: {} 2024-04-13T22:28:54.680 DEBUG:boto:Token: None 2024-04-13T22:28:54.680 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI= 2024-04-13T22:28:54.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI=', 'Content-Length': '0'} 2024-04-13T22:28:54.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdae6213909c6c21-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.682 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.682 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.683 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.683 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.683 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.683 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.683 DEBUG:boto:Data: 2024-04-13T22:28:54.683 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.683 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.683 DEBUG:boto:Params: {} 2024-04-13T22:28:54.683 DEBUG:boto:Token: None 2024-04-13T22:28:54.684 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F6uoG7BLc8CajZU75IxceW1iLFg= 2024-04-13T22:28:54.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F6uoG7BLc8CajZU75IxceW1iLFg=', 'Content-Length': '0'} 2024-04-13T22:28:54.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd8318915d87308-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.685 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.685 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.686 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.686 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.686 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.686 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.686 DEBUG:boto:Data: 2024-04-13T22:28:54.686 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.686 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.686 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.686 DEBUG:boto:Params: {} 2024-04-13T22:28:54.686 DEBUG:boto:Token: None 2024-04-13T22:28:54.687 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F6uoG7BLc8CajZU75IxceW1iLFg= 2024-04-13T22:28:54.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F6uoG7BLc8CajZU75IxceW1iLFg=', 'Content-Length': '0'} 2024-04-13T22:28:54.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e4a1dd42eb2cad5-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:28:54.689 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.689 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.689 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.689 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.689 DEBUG:boto:Method: GET 2024-04-13T22:28:54.689 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.689 DEBUG:boto:Data: 2024-04-13T22:28:54.689 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.690 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.690 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.690 DEBUG:boto:Params: {} 2024-04-13T22:28:54.690 DEBUG:boto:Token: None 2024-04-13T22:28:54.690 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI= 2024-04-13T22:28:54.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI=', 'Content-Length': '0'} 2024-04-13T22:28:54.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5f05e3c77dbd7c-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.734 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.735 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.735 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.735 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.735 DEBUG:boto:Method: GET 2024-04-13T22:28:54.735 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.735 DEBUG:boto:Data: 2024-04-13T22:28:54.735 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.735 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.735 DEBUG:boto:Params: {} 2024-04-13T22:28:54.735 DEBUG:boto:Token: None 2024-04-13T22:28:54.736 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:28:54.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI= 2024-04-13T22:28:54.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wAc/Z+Wf2nigyItSR9jdHSBtaQI=', 'Content-Length': '0'} 2024-04-13T22:28:54.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099c001d9b276c88-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.738 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.738 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.738 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.738 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.738 DEBUG:boto:Method: GET 2024-04-13T22:28:54.738 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.738 DEBUG:boto:Data: 2024-04-13T22:28:54.739 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.739 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.739 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.739 DEBUG:boto:Params: {} 2024-04-13T22:28:54.739 DEBUG:boto:Token: None 2024-04-13T22:28:54.739 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.739 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe01ae7548c66525-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.741 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.742 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.742 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.742 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.742 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.742 DEBUG:boto:Method: GET 2024-04-13T22:28:54.742 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.743 DEBUG:boto:Data: 2024-04-13T22:28:54.743 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.743 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.743 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.743 DEBUG:boto:Params: {} 2024-04-13T22:28:54.743 DEBUG:boto:Token: None 2024-04-13T22:28:54.743 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.743 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f1e028cb9fbf9dc-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.747 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.747 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.747 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.747 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.747 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:28:54.747 DEBUG:boto:Data: 2024-04-13T22:28:54.747 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.748 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.748 DEBUG:boto:Params: {} 2024-04-13T22:28:54.748 DEBUG:boto:Token: None 2024-04-13T22:28:54.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51 2024-04-13T22:28:54.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1zNVngEKBk8/9JDoLc1SqKQSqYo= 2024-04-13T22:28:54.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1zNVngEKBk8/9JDoLc1SqKQSqYo=', 'Content-Length': '0'} 2024-04-13T22:28:54.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af5e695801581cb-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.750 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.750 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.750 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.751 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:28:54.751 DEBUG:boto:Data: 2024-04-13T22:28:54.751 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.751 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.751 DEBUG:boto:Params: {} 2024-04-13T22:28:54.751 DEBUG:boto:Token: None 2024-04-13T22:28:54.751 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51 2024-04-13T22:28:54.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1zNVngEKBk8/9JDoLc1SqKQSqYo= 2024-04-13T22:28:54.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1zNVngEKBk8/9JDoLc1SqKQSqYo=', 'Content-Length': '0'} 2024-04-13T22:28:54.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c4a35d14732c41-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:28:54.753 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.754 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.754 DEBUG:boto:Method: GET 2024-04-13T22:28:54.754 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:28:54.754 DEBUG:boto:Data: 2024-04-13T22:28:54.754 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.754 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.754 DEBUG:boto:Params: {} 2024-04-13T22:28:54.754 DEBUG:boto:Token: None 2024-04-13T22:28:54.755 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51 2024-04-13T22:28:54.755 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wlsIm4vf0z3cEupH21IJN+i051A= 2024-04-13T22:28:54.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wlsIm4vf0z3cEupH21IJN+i051A=', 'Content-Length': '0'} 2024-04-13T22:28:54.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b442430c039f3f8-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.756 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:28:54.757 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:28:54.757 DEBUG:boto:Method: GET 2024-04-13T22:28:54.757 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:28:54.757 DEBUG:boto:Data: 2024-04-13T22:28:54.757 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.757 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.757 DEBUG:boto:Params: {} 2024-04-13T22:28:54.757 DEBUG:boto:Token: None 2024-04-13T22:28:54.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj51 2024-04-13T22:28:54.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wlsIm4vf0z3cEupH21IJN+i051A= 2024-04-13T22:28:54.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wlsIm4vf0z3cEupH21IJN+i051A=', 'Content-Length': '0'} 2024-04-13T22:28:54.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc24de774096022-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.760 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.760 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.760 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.760 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.760 DEBUG:boto:Method: GET 2024-04-13T22:28:54.760 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.760 DEBUG:boto:Data: 2024-04-13T22:28:54.760 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.760 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.760 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.760 DEBUG:boto:Params: {} 2024-04-13T22:28:54.761 DEBUG:boto:Token: None 2024-04-13T22:28:54.761 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f98d4dab71bb57e-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.763 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.763 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.764 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.764 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.764 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.764 DEBUG:boto:Method: GET 2024-04-13T22:28:54.764 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:28:54.764 DEBUG:boto:Data: 2024-04-13T22:28:54.764 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.764 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.764 DEBUG:boto:Params: {} 2024-04-13T22:28:54.764 DEBUG:boto:Token: None 2024-04-13T22:28:54.765 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004127a4041e64be95-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.768 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:28:54.768 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.769 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.769 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.769 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.769 DEBUG:boto:Method: GET 2024-04-13T22:28:54.769 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.769 DEBUG:boto:Data: 2024-04-13T22:28:54.769 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.769 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.769 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.769 DEBUG:boto:Params: {} 2024-04-13T22:28:54.770 DEBUG:boto:Token: None 2024-04-13T22:28:54.770 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.770 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM= 2024-04-13T22:28:54.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM=', 'Content-Length': '0'} 2024-04-13T22:28:54.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b594b4c4957f95be-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.772 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.772 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.772 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.772 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.772 DEBUG:boto:Method: GET 2024-04-13T22:28:54.772 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.772 DEBUG:boto:Data: 2024-04-13T22:28:54.772 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.773 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.773 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.773 DEBUG:boto:Params: {} 2024-04-13T22:28:54.773 DEBUG:boto:Token: None 2024-04-13T22:28:54.773 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.773 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM= 2024-04-13T22:28:54.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM=', 'Content-Length': '0'} 2024-04-13T22:28:54.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc84d252160823f9-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.775 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.775 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.775 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.775 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.775 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.775 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.775 DEBUG:boto:Data: 2024-04-13T22:28:54.776 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.776 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.776 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.776 DEBUG:boto:Params: {} 2024-04-13T22:28:54.776 DEBUG:boto:Token: None 2024-04-13T22:28:54.776 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qp9MQr6yWiPIvCWrph7ImZe1VUs= 2024-04-13T22:28:54.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qp9MQr6yWiPIvCWrph7ImZe1VUs=', 'Content-Length': '0'} 2024-04-13T22:28:54.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead3b4c836b85cd1-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.778 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.778 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.778 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.778 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.778 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.779 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.779 DEBUG:boto:Data: 2024-04-13T22:28:54.779 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.779 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.779 DEBUG:boto:Params: {} 2024-04-13T22:28:54.779 DEBUG:boto:Token: None 2024-04-13T22:28:54.779 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qp9MQr6yWiPIvCWrph7ImZe1VUs= 2024-04-13T22:28:54.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qp9MQr6yWiPIvCWrph7ImZe1VUs=', 'Content-Length': '0'} 2024-04-13T22:28:54.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001e5537fe604a4f2-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:28:54.781 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.782 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.782 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.782 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.782 DEBUG:boto:Method: GET 2024-04-13T22:28:54.782 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.782 DEBUG:boto:Data: 2024-04-13T22:28:54.782 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.782 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.783 DEBUG:boto:Params: {} 2024-04-13T22:28:54.783 DEBUG:boto:Token: None 2024-04-13T22:28:54.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM= 2024-04-13T22:28:54.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM=', 'Content-Length': '0'} 2024-04-13T22:28:54.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006243f3c3ad632e0c-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.785 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.785 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.785 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.785 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.785 DEBUG:boto:Method: GET 2024-04-13T22:28:54.785 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.785 DEBUG:boto:Data: 2024-04-13T22:28:54.786 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.786 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.786 DEBUG:boto:Params: {} 2024-04-13T22:28:54.786 DEBUG:boto:Token: None 2024-04-13T22:28:54.786 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:28:54.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM= 2024-04-13T22:28:54.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IqEfHGsUykyiqxh9RfIZmhWx2KM=', 'Content-Length': '0'} 2024-04-13T22:28:54.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cadf95ad0e3c2b97-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.830 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.830 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.831 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.831 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.831 DEBUG:boto:Method: GET 2024-04-13T22:28:54.831 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.831 DEBUG:boto:Data: 2024-04-13T22:28:54.831 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.831 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.831 DEBUG:boto:Params: {} 2024-04-13T22:28:54.831 DEBUG:boto:Token: None 2024-04-13T22:28:54.832 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.832 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e32fea0c30a186a3-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.834 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.834 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.834 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.834 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.834 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.835 DEBUG:boto:Method: GET 2024-04-13T22:28:54.835 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.835 DEBUG:boto:Data: 2024-04-13T22:28:54.835 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.835 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.835 DEBUG:boto:Params: {} 2024-04-13T22:28:54.835 DEBUG:boto:Token: None 2024-04-13T22:28:54.835 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.835 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005153ac34d3504056-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.839 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.839 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.839 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.839 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.840 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:28:54.840 DEBUG:boto:Data: 2024-04-13T22:28:54.840 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.840 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.840 DEBUG:boto:Params: {} 2024-04-13T22:28:54.840 DEBUG:boto:Token: None 2024-04-13T22:28:54.840 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55 2024-04-13T22:28:54.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nxenQhq/s6Z8OIoPy60pscs5ilQ= 2024-04-13T22:28:54.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nxenQhq/s6Z8OIoPy60pscs5ilQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a83e1e996ba01ea2-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.842 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.842 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.842 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.843 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:28:54.843 DEBUG:boto:Data: 2024-04-13T22:28:54.843 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.843 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.843 DEBUG:boto:Params: {} 2024-04-13T22:28:54.843 DEBUG:boto:Token: None 2024-04-13T22:28:54.843 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55 2024-04-13T22:28:54.843 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nxenQhq/s6Z8OIoPy60pscs5ilQ= 2024-04-13T22:28:54.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nxenQhq/s6Z8OIoPy60pscs5ilQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037b1645fdfa8774-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:28:54.845 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.845 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.845 DEBUG:boto:Method: GET 2024-04-13T22:28:54.846 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:28:54.846 DEBUG:boto:Data: 2024-04-13T22:28:54.846 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.846 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.846 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.846 DEBUG:boto:Params: {} 2024-04-13T22:28:54.846 DEBUG:boto:Token: None 2024-04-13T22:28:54.846 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55 2024-04-13T22:28:54.846 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PYCDehgKiNHuths7N9PRvZUZbhc= 2024-04-13T22:28:54.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PYCDehgKiNHuths7N9PRvZUZbhc=', 'Content-Length': '0'} 2024-04-13T22:28:54.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d66fac6121a2bd1-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.848 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:28:54.848 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:28:54.848 DEBUG:boto:Method: GET 2024-04-13T22:28:54.849 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:28:54.849 DEBUG:boto:Data: 2024-04-13T22:28:54.849 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.849 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.849 DEBUG:boto:Params: {} 2024-04-13T22:28:54.849 DEBUG:boto:Token: None 2024-04-13T22:28:54.849 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj55 2024-04-13T22:28:54.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PYCDehgKiNHuths7N9PRvZUZbhc= 2024-04-13T22:28:54.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PYCDehgKiNHuths7N9PRvZUZbhc=', 'Content-Length': '0'} 2024-04-13T22:28:54.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005afe119d507613f5-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.851 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.851 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.851 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.851 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.851 DEBUG:boto:Method: GET 2024-04-13T22:28:54.852 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.852 DEBUG:boto:Data: 2024-04-13T22:28:54.852 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.852 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.852 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.852 DEBUG:boto:Params: {} 2024-04-13T22:28:54.852 DEBUG:boto:Token: None 2024-04-13T22:28:54.852 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d465217327b70d-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.855 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.855 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.855 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.855 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.855 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.855 DEBUG:boto:Method: GET 2024-04-13T22:28:54.855 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:28:54.855 DEBUG:boto:Data: 2024-04-13T22:28:54.856 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.856 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.856 DEBUG:boto:Params: {} 2024-04-13T22:28:54.856 DEBUG:boto:Token: None 2024-04-13T22:28:54.856 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6af6a66f22a2ef0-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.902 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:28:54.903 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.903 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.903 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.903 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.903 DEBUG:boto:Method: GET 2024-04-13T22:28:54.903 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.903 DEBUG:boto:Data: 2024-04-13T22:28:54.903 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.904 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.904 DEBUG:boto:Params: {} 2024-04-13T22:28:54.904 DEBUG:boto:Token: None 2024-04-13T22:28:54.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ= 2024-04-13T22:28:54.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0b71752a606a88-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.906 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.906 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.907 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.907 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.907 DEBUG:boto:Method: GET 2024-04-13T22:28:54.907 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.907 DEBUG:boto:Data: 2024-04-13T22:28:54.907 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.907 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.907 DEBUG:boto:Params: {} 2024-04-13T22:28:54.907 DEBUG:boto:Token: None 2024-04-13T22:28:54.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ= 2024-04-13T22:28:54.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000472a5d473fff323-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.910 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.910 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.910 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.910 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.910 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.910 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.910 DEBUG:boto:Data: 2024-04-13T22:28:54.910 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.910 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.911 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.911 DEBUG:boto:Params: {} 2024-04-13T22:28:54.911 DEBUG:boto:Token: None 2024-04-13T22:28:54.911 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tMeHGas2LVN4hn9RwCeNLZTEqI0= 2024-04-13T22:28:54.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tMeHGas2LVN4hn9RwCeNLZTEqI0=', 'Content-Length': '0'} 2024-04-13T22:28:54.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000335086cc0a4212e6-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.913 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.913 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.913 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.914 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.914 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.914 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.914 DEBUG:boto:Data: 2024-04-13T22:28:54.914 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.914 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.914 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.914 DEBUG:boto:Params: {} 2024-04-13T22:28:54.914 DEBUG:boto:Token: None 2024-04-13T22:28:54.914 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.915 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tMeHGas2LVN4hn9RwCeNLZTEqI0= 2024-04-13T22:28:54.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tMeHGas2LVN4hn9RwCeNLZTEqI0=', 'Content-Length': '0'} 2024-04-13T22:28:54.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc4f0690df70dbc-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:28:54.916 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.917 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.917 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.917 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.917 DEBUG:boto:Method: GET 2024-04-13T22:28:54.917 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.917 DEBUG:boto:Data: 2024-04-13T22:28:54.917 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.917 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.917 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.917 DEBUG:boto:Params: {} 2024-04-13T22:28:54.918 DEBUG:boto:Token: None 2024-04-13T22:28:54.918 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.918 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ= 2024-04-13T22:28:54.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008576170a01ef1b54-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.920 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.920 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.920 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.920 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.920 DEBUG:boto:Method: GET 2024-04-13T22:28:54.920 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.920 DEBUG:boto:Data: 2024-04-13T22:28:54.921 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.921 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.921 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.921 DEBUG:boto:Params: {} 2024-04-13T22:28:54.921 DEBUG:boto:Token: None 2024-04-13T22:28:54.921 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:28:54.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ= 2024-04-13T22:28:54.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VLA3kqS6mKTGlKvV8yBnm3NrRxQ=', 'Content-Length': '0'} 2024-04-13T22:28:54.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbfbdce338d91246-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.923 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.923 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.923 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.923 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.924 DEBUG:boto:Method: GET 2024-04-13T22:28:54.924 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.924 DEBUG:boto:Data: 2024-04-13T22:28:54.924 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.924 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.924 DEBUG:boto:Params: {} 2024-04-13T22:28:54.924 DEBUG:boto:Token: None 2024-04-13T22:28:54.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee43a6c77e24073-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.927 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.927 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.927 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.927 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.927 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.928 DEBUG:boto:Method: GET 2024-04-13T22:28:54.928 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.928 DEBUG:boto:Data: 2024-04-13T22:28:54.928 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.928 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.928 DEBUG:boto:Params: {} 2024-04-13T22:28:54.928 DEBUG:boto:Token: None 2024-04-13T22:28:54.928 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e387c5b63f78811-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.932 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.932 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.932 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.932 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.933 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:28:54.933 DEBUG:boto:Data: 2024-04-13T22:28:54.933 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.933 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.933 DEBUG:boto:Params: {} 2024-04-13T22:28:54.933 DEBUG:boto:Token: None 2024-04-13T22:28:54.933 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57 2024-04-13T22:28:54.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3EhapsyGkFFwURBdmwtyGJc+wKA= 2024-04-13T22:28:54.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3EhapsyGkFFwURBdmwtyGJc+wKA=', 'Content-Length': '0'} 2024-04-13T22:28:54.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd69a4b5cfd694ae-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.935 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.935 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.935 DEBUG:boto:Method: HEAD 2024-04-13T22:28:54.935 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:28:54.935 DEBUG:boto:Data: 2024-04-13T22:28:54.935 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.935 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.936 DEBUG:boto:Params: {} 2024-04-13T22:28:54.936 DEBUG:boto:Token: None 2024-04-13T22:28:54.936 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57 2024-04-13T22:28:54.936 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3EhapsyGkFFwURBdmwtyGJc+wKA= 2024-04-13T22:28:54.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3EhapsyGkFFwURBdmwtyGJc+wKA=', 'Content-Length': '0'} 2024-04-13T22:28:54.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a144434a8c2d45a4-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:28:54.938 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.938 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.938 DEBUG:boto:Method: GET 2024-04-13T22:28:54.938 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:28:54.938 DEBUG:boto:Data: 2024-04-13T22:28:54.938 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.939 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.939 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.939 DEBUG:boto:Params: {} 2024-04-13T22:28:54.939 DEBUG:boto:Token: None 2024-04-13T22:28:54.939 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57 2024-04-13T22:28:54.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tKfY6ZYv8KC/woCNZkkbxciv77w= 2024-04-13T22:28:54.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tKfY6ZYv8KC/woCNZkkbxciv77w=', 'Content-Length': '0'} 2024-04-13T22:28:54.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07088059625eada-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.982 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:28:54.982 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:28:54.983 DEBUG:boto:Method: GET 2024-04-13T22:28:54.983 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:28:54.983 DEBUG:boto:Data: 2024-04-13T22:28:54.983 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.983 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.983 DEBUG:boto:Params: {} 2024-04-13T22:28:54.983 DEBUG:boto:Token: None 2024-04-13T22:28:54.983 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj57 2024-04-13T22:28:54.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tKfY6ZYv8KC/woCNZkkbxciv77w= 2024-04-13T22:28:54.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tKfY6ZYv8KC/woCNZkkbxciv77w=', 'Content-Length': '0'} 2024-04-13T22:28:54.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad31fa6fa39c3192-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.986 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.987 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.987 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.987 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.987 DEBUG:boto:Method: GET 2024-04-13T22:28:54.987 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.987 DEBUG:boto:Data: 2024-04-13T22:28:54.987 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.987 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.987 DEBUG:boto:Params: {} 2024-04-13T22:28:54.988 DEBUG:boto:Token: None 2024-04-13T22:28:54.988 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065111bbd7825ad30-00661b0726-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.991 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.991 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:54.991 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:54.991 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.991 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.991 DEBUG:boto:Method: GET 2024-04-13T22:28:54.991 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:28:54.992 DEBUG:boto:Data: 2024-04-13T22:28:54.992 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.992 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:54.992 DEBUG:boto:Port: 8000 2024-04-13T22:28:54.992 DEBUG:boto:Params: {} 2024-04-13T22:28:54.992 DEBUG:boto:Token: None 2024-04-13T22:28:54.992 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/ 2024-04-13T22:28:54.992 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA= 2024-04-13T22:28:54.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:txLuv/3d1xk7ZPKs8tklW57BqUA=', 'Content-Length': '0'} 2024-04-13T22:28:54.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad2818e012548f8d-00661b0726-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.995 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:54.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:28:54.996 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:54.996 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:54.996 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:54.996 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:54.996 DEBUG:boto:Method: GET 2024-04-13T22:28:54.996 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:54.996 DEBUG:boto:Data: 2024-04-13T22:28:54.997 DEBUG:boto:Headers: {} 2024-04-13T22:28:54.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:54.997 DEBUG:boto:Port: 8001 2024-04-13T22:28:54.997 DEBUG:boto:Params: {} 2024-04-13T22:28:54.997 DEBUG:boto:Token: None 2024-04-13T22:28:54.997 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:54 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:54.997 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jFJArpNW4rSYPlp0Ei0e29XQxN0= 2024-04-13T22:28:54.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jFJArpNW4rSYPlp0Ei0e29XQxN0=', 'Content-Length': '0'} 2024-04-13T22:28:54.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab5a98f2e48d59f-00661b0726-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:54.999 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:54.999 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:54.999 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:54.999 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:54.999 DEBUG:boto:Method: GET 2024-04-13T22:28:54.999 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.000 DEBUG:boto:Data: 2024-04-13T22:28:55.000 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.000 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.000 DEBUG:boto:Params: {} 2024-04-13T22:28:55.000 DEBUG:boto:Token: None 2024-04-13T22:28:55.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kFd6M7y1jkIleJ3gPKR58WCSvMI= 2024-04-13T22:28:55.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kFd6M7y1jkIleJ3gPKR58WCSvMI=', 'Content-Length': '0'} 2024-04-13T22:28:55.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d3faa31e8b98f8-00661b0726-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.002 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.002 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.002 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.002 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.002 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.002 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.003 DEBUG:boto:Data: 2024-04-13T22:28:55.003 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.003 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.003 DEBUG:boto:Params: {} 2024-04-13T22:28:55.003 DEBUG:boto:Token: None 2024-04-13T22:28:55.003 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vEyEL9zXX0m4I+bmPrs+Z8ZQ1lk= 2024-04-13T22:28:55.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vEyEL9zXX0m4I+bmPrs+Z8ZQ1lk=', 'Content-Length': '0'} 2024-04-13T22:28:55.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058f4c166f5bc95b-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.005 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.005 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.005 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.005 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.005 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.005 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.005 DEBUG:boto:Data: 2024-04-13T22:28:55.006 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.006 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.006 DEBUG:boto:Params: {} 2024-04-13T22:28:55.006 DEBUG:boto:Token: None 2024-04-13T22:28:55.006 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vEyEL9zXX0m4I+bmPrs+Z8ZQ1lk= 2024-04-13T22:28:55.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vEyEL9zXX0m4I+bmPrs+Z8ZQ1lk=', 'Content-Length': '0'} 2024-04-13T22:28:55.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ebc4911bc0c2e3-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:28:55.008 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.008 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.008 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.008 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.009 DEBUG:boto:Method: GET 2024-04-13T22:28:55.009 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.009 DEBUG:boto:Data: 2024-04-13T22:28:55.009 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.009 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.009 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.009 DEBUG:boto:Params: {} 2024-04-13T22:28:55.009 DEBUG:boto:Token: None 2024-04-13T22:28:55.009 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kFd6M7y1jkIleJ3gPKR58WCSvMI= 2024-04-13T22:28:55.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kFd6M7y1jkIleJ3gPKR58WCSvMI=', 'Content-Length': '0'} 2024-04-13T22:28:55.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008acdadbbdf5dc643-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.054 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.054 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.055 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.055 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.055 DEBUG:boto:Method: GET 2024-04-13T22:28:55.055 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.055 DEBUG:boto:Data: 2024-04-13T22:28:55.055 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.055 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.055 DEBUG:boto:Params: {} 2024-04-13T22:28:55.055 DEBUG:boto:Token: None 2024-04-13T22:28:55.055 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:28:55.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kFd6M7y1jkIleJ3gPKR58WCSvMI= 2024-04-13T22:28:55.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kFd6M7y1jkIleJ3gPKR58WCSvMI=', 'Content-Length': '0'} 2024-04-13T22:28:55.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000723787aaee7b75d4-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.057 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.058 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.058 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.058 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.058 DEBUG:boto:Method: GET 2024-04-13T22:28:55.058 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.058 DEBUG:boto:Data: 2024-04-13T22:28:55.058 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.058 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.058 DEBUG:boto:Params: {} 2024-04-13T22:28:55.058 DEBUG:boto:Token: None 2024-04-13T22:28:55.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ff0e44e208789d-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.061 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.061 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.061 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.061 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.062 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.062 DEBUG:boto:Method: GET 2024-04-13T22:28:55.062 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.062 DEBUG:boto:Data: 2024-04-13T22:28:55.062 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.062 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.062 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.062 DEBUG:boto:Params: {} 2024-04-13T22:28:55.062 DEBUG:boto:Token: None 2024-04-13T22:28:55.062 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.062 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d430ca5c4dab3e61-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.066 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.066 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.067 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.067 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.067 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:28:55.067 DEBUG:boto:Data: 2024-04-13T22:28:55.067 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.067 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.067 DEBUG:boto:Params: {} 2024-04-13T22:28:55.067 DEBUG:boto:Token: None 2024-04-13T22:28:55.067 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58 2024-04-13T22:28:55.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P+Z4/RXsXpDtFY14g0dFUtI3vJU= 2024-04-13T22:28:55.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P+Z4/RXsXpDtFY14g0dFUtI3vJU=', 'Content-Length': '0'} 2024-04-13T22:28:55.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001987c3cb5a701729-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.069 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.069 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.070 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.070 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:28:55.070 DEBUG:boto:Data: 2024-04-13T22:28:55.070 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.070 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.070 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.070 DEBUG:boto:Params: {} 2024-04-13T22:28:55.070 DEBUG:boto:Token: None 2024-04-13T22:28:55.071 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58 2024-04-13T22:28:55.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P+Z4/RXsXpDtFY14g0dFUtI3vJU= 2024-04-13T22:28:55.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P+Z4/RXsXpDtFY14g0dFUtI3vJU=', 'Content-Length': '0'} 2024-04-13T22:28:55.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a01705240e50bac-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:28:55.073 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.073 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.073 DEBUG:boto:Method: GET 2024-04-13T22:28:55.073 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:28:55.073 DEBUG:boto:Data: 2024-04-13T22:28:55.073 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.074 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.074 DEBUG:boto:Params: {} 2024-04-13T22:28:55.074 DEBUG:boto:Token: None 2024-04-13T22:28:55.074 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58 2024-04-13T22:28:55.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HO6vkrcV7JtU1YPIuxUn2YxczD4= 2024-04-13T22:28:55.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HO6vkrcV7JtU1YPIuxUn2YxczD4=', 'Content-Length': '0'} 2024-04-13T22:28:55.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffab4e049074f4d5-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.076 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:28:55.076 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:28:55.076 DEBUG:boto:Method: GET 2024-04-13T22:28:55.076 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:28:55.076 DEBUG:boto:Data: 2024-04-13T22:28:55.076 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.076 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.076 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.076 DEBUG:boto:Params: {} 2024-04-13T22:28:55.076 DEBUG:boto:Token: None 2024-04-13T22:28:55.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj58 2024-04-13T22:28:55.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HO6vkrcV7JtU1YPIuxUn2YxczD4= 2024-04-13T22:28:55.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HO6vkrcV7JtU1YPIuxUn2YxczD4=', 'Content-Length': '0'} 2024-04-13T22:28:55.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d838d53795a4d36-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.078 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.078 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.078 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.079 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.079 DEBUG:boto:Method: GET 2024-04-13T22:28:55.079 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.079 DEBUG:boto:Data: 2024-04-13T22:28:55.079 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.079 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.079 DEBUG:boto:Params: {} 2024-04-13T22:28:55.079 DEBUG:boto:Token: None 2024-04-13T22:28:55.079 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fa10275a8ea433-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.082 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.082 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.083 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.083 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.083 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.083 DEBUG:boto:Method: GET 2024-04-13T22:28:55.083 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:28:55.083 DEBUG:boto:Data: 2024-04-13T22:28:55.083 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.083 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.083 DEBUG:boto:Params: {} 2024-04-13T22:28:55.083 DEBUG:boto:Token: None 2024-04-13T22:28:55.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002775f437441168ab-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.134 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:28:55.135 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.135 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.135 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.135 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.135 DEBUG:boto:Method: GET 2024-04-13T22:28:55.135 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.135 DEBUG:boto:Data: 2024-04-13T22:28:55.135 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.135 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.136 DEBUG:boto:Params: {} 2024-04-13T22:28:55.136 DEBUG:boto:Token: None 2024-04-13T22:28:55.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA= 2024-04-13T22:28:55.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA=', 'Content-Length': '0'} 2024-04-13T22:28:55.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c02481c038fecf1-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.138 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.138 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.138 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.138 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.138 DEBUG:boto:Method: GET 2024-04-13T22:28:55.138 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.138 DEBUG:boto:Data: 2024-04-13T22:28:55.139 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.139 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.139 DEBUG:boto:Params: {} 2024-04-13T22:28:55.139 DEBUG:boto:Token: None 2024-04-13T22:28:55.139 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA= 2024-04-13T22:28:55.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA=', 'Content-Length': '0'} 2024-04-13T22:28:55.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac745e1f65fa61a-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.141 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.141 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.141 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.141 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.141 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.141 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.142 DEBUG:boto:Data: 2024-04-13T22:28:55.142 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.142 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.142 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.142 DEBUG:boto:Params: {} 2024-04-13T22:28:55.142 DEBUG:boto:Token: None 2024-04-13T22:28:55.142 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AiFY/X4dtHSZs9NU/DqVF8U+JIU= 2024-04-13T22:28:55.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AiFY/X4dtHSZs9NU/DqVF8U+JIU=', 'Content-Length': '0'} 2024-04-13T22:28:55.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241e73d60614a634-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.144 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.144 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.144 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.144 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.145 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.145 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.145 DEBUG:boto:Data: 2024-04-13T22:28:55.145 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.145 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.145 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.145 DEBUG:boto:Params: {} 2024-04-13T22:28:55.145 DEBUG:boto:Token: None 2024-04-13T22:28:55.145 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.145 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AiFY/X4dtHSZs9NU/DqVF8U+JIU= 2024-04-13T22:28:55.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AiFY/X4dtHSZs9NU/DqVF8U+JIU=', 'Content-Length': '0'} 2024-04-13T22:28:55.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd07adb4121e62c3-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:28:55.147 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.147 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.147 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.148 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.148 DEBUG:boto:Method: GET 2024-04-13T22:28:55.148 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.148 DEBUG:boto:Data: 2024-04-13T22:28:55.148 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.148 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.148 DEBUG:boto:Params: {} 2024-04-13T22:28:55.148 DEBUG:boto:Token: None 2024-04-13T22:28:55.148 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA= 2024-04-13T22:28:55.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA=', 'Content-Length': '0'} 2024-04-13T22:28:55.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f70f821c5a17151-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.150 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.150 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.150 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.150 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.150 DEBUG:boto:Method: GET 2024-04-13T22:28:55.151 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.151 DEBUG:boto:Data: 2024-04-13T22:28:55.151 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.151 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.151 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.151 DEBUG:boto:Params: {} 2024-04-13T22:28:55.151 DEBUG:boto:Token: None 2024-04-13T22:28:55.151 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:28:55.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA= 2024-04-13T22:28:55.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4fdJ+BRDMa0jf3i4tsyk1oQLATA=', 'Content-Length': '0'} 2024-04-13T22:28:55.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec598b95a4539e48-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.153 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.153 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.154 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.154 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.154 DEBUG:boto:Method: GET 2024-04-13T22:28:55.154 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.154 DEBUG:boto:Data: 2024-04-13T22:28:55.154 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.154 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.154 DEBUG:boto:Params: {} 2024-04-13T22:28:55.154 DEBUG:boto:Token: None 2024-04-13T22:28:55.155 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000328ac5171ef874c9-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.157 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.157 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.157 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.157 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.157 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.158 DEBUG:boto:Method: GET 2024-04-13T22:28:55.158 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.158 DEBUG:boto:Data: 2024-04-13T22:28:55.158 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.158 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.158 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.158 DEBUG:boto:Params: {} 2024-04-13T22:28:55.158 DEBUG:boto:Token: None 2024-04-13T22:28:55.158 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.158 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea2f6dcff6eaf079-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.162 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.162 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.162 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.162 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.162 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:28:55.162 DEBUG:boto:Data: 2024-04-13T22:28:55.163 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.163 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.163 DEBUG:boto:Params: {} 2024-04-13T22:28:55.163 DEBUG:boto:Token: None 2024-04-13T22:28:55.163 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62 2024-04-13T22:28:55.163 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eOlo8mhCdpTxr+xEptQWd8a0MqQ= 2024-04-13T22:28:55.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eOlo8mhCdpTxr+xEptQWd8a0MqQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8d9eb785291a9c-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.165 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.165 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.165 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.165 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:28:55.165 DEBUG:boto:Data: 2024-04-13T22:28:55.165 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.165 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.165 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.166 DEBUG:boto:Params: {} 2024-04-13T22:28:55.166 DEBUG:boto:Token: None 2024-04-13T22:28:55.166 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62 2024-04-13T22:28:55.166 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eOlo8mhCdpTxr+xEptQWd8a0MqQ= 2024-04-13T22:28:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eOlo8mhCdpTxr+xEptQWd8a0MqQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b8acfeb1e9eeac-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:28:55.168 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.168 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.168 DEBUG:boto:Method: GET 2024-04-13T22:28:55.168 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:28:55.168 DEBUG:boto:Data: 2024-04-13T22:28:55.168 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.169 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.169 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.169 DEBUG:boto:Params: {} 2024-04-13T22:28:55.169 DEBUG:boto:Token: None 2024-04-13T22:28:55.169 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62 2024-04-13T22:28:55.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XZ31goZXG7oVfIv88xaERsIN6Gc= 2024-04-13T22:28:55.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XZ31goZXG7oVfIv88xaERsIN6Gc=', 'Content-Length': '0'} 2024-04-13T22:28:55.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098083e42aa40afdb-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.171 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:28:55.171 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:28:55.171 DEBUG:boto:Method: GET 2024-04-13T22:28:55.171 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:28:55.171 DEBUG:boto:Data: 2024-04-13T22:28:55.171 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.171 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.171 DEBUG:boto:Params: {} 2024-04-13T22:28:55.172 DEBUG:boto:Token: None 2024-04-13T22:28:55.172 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj62 2024-04-13T22:28:55.172 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XZ31goZXG7oVfIv88xaERsIN6Gc= 2024-04-13T22:28:55.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XZ31goZXG7oVfIv88xaERsIN6Gc=', 'Content-Length': '0'} 2024-04-13T22:28:55.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095eebec302a352b3-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.174 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.174 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.174 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.174 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.174 DEBUG:boto:Method: GET 2024-04-13T22:28:55.174 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.175 DEBUG:boto:Data: 2024-04-13T22:28:55.175 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.175 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.175 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.175 DEBUG:boto:Params: {} 2024-04-13T22:28:55.175 DEBUG:boto:Token: None 2024-04-13T22:28:55.175 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9860d40398ba83a-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.177 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.178 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.178 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.178 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.178 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.178 DEBUG:boto:Method: GET 2024-04-13T22:28:55.178 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:28:55.178 DEBUG:boto:Data: 2024-04-13T22:28:55.178 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.178 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.179 DEBUG:boto:Params: {} 2024-04-13T22:28:55.179 DEBUG:boto:Token: None 2024-04-13T22:28:55.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001212eae467ab27bd-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.182 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:28:55.182 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.183 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.183 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.183 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.183 DEBUG:boto:Method: GET 2024-04-13T22:28:55.183 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.183 DEBUG:boto:Data: 2024-04-13T22:28:55.183 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.183 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.183 DEBUG:boto:Params: {} 2024-04-13T22:28:55.183 DEBUG:boto:Token: None 2024-04-13T22:28:55.184 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k= 2024-04-13T22:28:55.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k=', 'Content-Length': '0'} 2024-04-13T22:28:55.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c6917e69d9ccea-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.186 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.186 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.186 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.186 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.186 DEBUG:boto:Method: GET 2024-04-13T22:28:55.186 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.186 DEBUG:boto:Data: 2024-04-13T22:28:55.186 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.186 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.186 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.186 DEBUG:boto:Params: {} 2024-04-13T22:28:55.187 DEBUG:boto:Token: None 2024-04-13T22:28:55.187 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.187 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k= 2024-04-13T22:28:55.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k=', 'Content-Length': '0'} 2024-04-13T22:28:55.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0f496d58fe64109-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.188 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.189 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.189 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.189 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.189 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.189 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.189 DEBUG:boto:Data: 2024-04-13T22:28:55.189 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.189 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.189 DEBUG:boto:Params: {} 2024-04-13T22:28:55.189 DEBUG:boto:Token: None 2024-04-13T22:28:55.189 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWb1gKn/tbHv3jfAUmck6c3J/Kg= 2024-04-13T22:28:55.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWb1gKn/tbHv3jfAUmck6c3J/Kg=', 'Content-Length': '0'} 2024-04-13T22:28:55.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000548d1dc0270536d2-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.191 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.191 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.191 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.191 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.192 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.192 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.192 DEBUG:boto:Data: 2024-04-13T22:28:55.192 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.192 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.192 DEBUG:boto:Params: {} 2024-04-13T22:28:55.192 DEBUG:boto:Token: None 2024-04-13T22:28:55.192 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dWb1gKn/tbHv3jfAUmck6c3J/Kg= 2024-04-13T22:28:55.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWb1gKn/tbHv3jfAUmck6c3J/Kg=', 'Content-Length': '0'} 2024-04-13T22:28:55.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc7209e57413a8b-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:28:55.194 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.194 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.194 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.194 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.194 DEBUG:boto:Method: GET 2024-04-13T22:28:55.194 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.195 DEBUG:boto:Data: 2024-04-13T22:28:55.195 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.195 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.195 DEBUG:boto:Params: {} 2024-04-13T22:28:55.195 DEBUG:boto:Token: None 2024-04-13T22:28:55.195 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k= 2024-04-13T22:28:55.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k=', 'Content-Length': '0'} 2024-04-13T22:28:55.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004488a7a1bd496613-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.238 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.239 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.239 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.239 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.239 DEBUG:boto:Method: GET 2024-04-13T22:28:55.239 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.239 DEBUG:boto:Data: 2024-04-13T22:28:55.239 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.239 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.239 DEBUG:boto:Params: {} 2024-04-13T22:28:55.239 DEBUG:boto:Token: None 2024-04-13T22:28:55.240 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:28:55.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k= 2024-04-13T22:28:55.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xK6Eg6veLXHfCSDCmG0PHmVF3+k=', 'Content-Length': '0'} 2024-04-13T22:28:55.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a1371f89d173a7-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.242 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.242 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.242 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.242 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.242 DEBUG:boto:Method: GET 2024-04-13T22:28:55.242 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.242 DEBUG:boto:Data: 2024-04-13T22:28:55.242 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.243 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.243 DEBUG:boto:Params: {} 2024-04-13T22:28:55.243 DEBUG:boto:Token: None 2024-04-13T22:28:55.243 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738522fe0565d628-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.245 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.245 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.245 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.246 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.246 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.246 DEBUG:boto:Method: GET 2024-04-13T22:28:55.246 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.246 DEBUG:boto:Data: 2024-04-13T22:28:55.246 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.246 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.246 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.246 DEBUG:boto:Params: {} 2024-04-13T22:28:55.246 DEBUG:boto:Token: None 2024-04-13T22:28:55.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c048efd6eb5538-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.250 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.250 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.250 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.251 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.251 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:28:55.251 DEBUG:boto:Data: 2024-04-13T22:28:55.251 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.251 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.251 DEBUG:boto:Params: {} 2024-04-13T22:28:55.251 DEBUG:boto:Token: None 2024-04-13T22:28:55.251 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68 2024-04-13T22:28:55.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7zdqw2nJ6W8u0xv+mNphe4wadZo= 2024-04-13T22:28:55.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7zdqw2nJ6W8u0xv+mNphe4wadZo=', 'Content-Length': '0'} 2024-04-13T22:28:55.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfd4c4cdd4391f81-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.253 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.253 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.253 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.253 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:28:55.253 DEBUG:boto:Data: 2024-04-13T22:28:55.253 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.253 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.254 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.254 DEBUG:boto:Params: {} 2024-04-13T22:28:55.254 DEBUG:boto:Token: None 2024-04-13T22:28:55.254 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68 2024-04-13T22:28:55.254 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7zdqw2nJ6W8u0xv+mNphe4wadZo= 2024-04-13T22:28:55.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7zdqw2nJ6W8u0xv+mNphe4wadZo=', 'Content-Length': '0'} 2024-04-13T22:28:55.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a1af2791614596-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:28:55.256 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.256 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.256 DEBUG:boto:Method: GET 2024-04-13T22:28:55.256 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:28:55.256 DEBUG:boto:Data: 2024-04-13T22:28:55.256 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.256 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.257 DEBUG:boto:Params: {} 2024-04-13T22:28:55.257 DEBUG:boto:Token: None 2024-04-13T22:28:55.257 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68 2024-04-13T22:28:55.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+3zKO73N06ByjHxfu+ki82kMkKE= 2024-04-13T22:28:55.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+3zKO73N06ByjHxfu+ki82kMkKE=', 'Content-Length': '0'} 2024-04-13T22:28:55.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000653ca877cdbf8920-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.259 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:28:55.259 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:28:55.259 DEBUG:boto:Method: GET 2024-04-13T22:28:55.259 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:28:55.259 DEBUG:boto:Data: 2024-04-13T22:28:55.259 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.259 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.259 DEBUG:boto:Params: {} 2024-04-13T22:28:55.260 DEBUG:boto:Token: None 2024-04-13T22:28:55.260 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj68 2024-04-13T22:28:55.260 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+3zKO73N06ByjHxfu+ki82kMkKE= 2024-04-13T22:28:55.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+3zKO73N06ByjHxfu+ki82kMkKE=', 'Content-Length': '0'} 2024-04-13T22:28:55.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c07efcb2a00d1ba-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.262 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.262 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.262 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.262 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.262 DEBUG:boto:Method: GET 2024-04-13T22:28:55.262 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.262 DEBUG:boto:Data: 2024-04-13T22:28:55.262 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.263 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.263 DEBUG:boto:Params: {} 2024-04-13T22:28:55.263 DEBUG:boto:Token: None 2024-04-13T22:28:55.263 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040f926c746de53c3-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.265 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.265 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.266 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.266 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.266 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.266 DEBUG:boto:Method: GET 2024-04-13T22:28:55.266 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:28:55.266 DEBUG:boto:Data: 2024-04-13T22:28:55.266 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.266 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.266 DEBUG:boto:Params: {} 2024-04-13T22:28:55.266 DEBUG:boto:Token: None 2024-04-13T22:28:55.267 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002167f3d4d9d2de7c-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.270 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:28:55.270 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.270 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.270 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.270 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.271 DEBUG:boto:Method: GET 2024-04-13T22:28:55.271 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.271 DEBUG:boto:Data: 2024-04-13T22:28:55.271 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.271 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.271 DEBUG:boto:Params: {} 2024-04-13T22:28:55.271 DEBUG:boto:Token: None 2024-04-13T22:28:55.271 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw= 2024-04-13T22:28:55.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw=', 'Content-Length': '0'} 2024-04-13T22:28:55.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073fd1ef80e059ae7-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.273 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.273 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.273 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.273 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.273 DEBUG:boto:Method: GET 2024-04-13T22:28:55.274 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.274 DEBUG:boto:Data: 2024-04-13T22:28:55.274 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.274 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.274 DEBUG:boto:Params: {} 2024-04-13T22:28:55.274 DEBUG:boto:Token: None 2024-04-13T22:28:55.274 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw= 2024-04-13T22:28:55.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw=', 'Content-Length': '0'} 2024-04-13T22:28:55.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f73bf5dd262cd15-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.276 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.276 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.276 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.277 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.277 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.277 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.277 DEBUG:boto:Data: 2024-04-13T22:28:55.277 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.277 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.277 DEBUG:boto:Params: {} 2024-04-13T22:28:55.277 DEBUG:boto:Token: None 2024-04-13T22:28:55.277 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.277 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sevPgNl6G/XNwS9rNTGC5MVl+ak= 2024-04-13T22:28:55.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sevPgNl6G/XNwS9rNTGC5MVl+ak=', 'Content-Length': '0'} 2024-04-13T22:28:55.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eeb1b5a3ef69f9e-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.279 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.279 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.279 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.279 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.279 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.279 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.280 DEBUG:boto:Data: 2024-04-13T22:28:55.280 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.280 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.280 DEBUG:boto:Params: {} 2024-04-13T22:28:55.280 DEBUG:boto:Token: None 2024-04-13T22:28:55.280 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sevPgNl6G/XNwS9rNTGC5MVl+ak= 2024-04-13T22:28:55.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sevPgNl6G/XNwS9rNTGC5MVl+ak=', 'Content-Length': '0'} 2024-04-13T22:28:55.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762b07582d342a89-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:28:55.282 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.282 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.282 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.282 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.282 DEBUG:boto:Method: GET 2024-04-13T22:28:55.282 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.282 DEBUG:boto:Data: 2024-04-13T22:28:55.283 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.283 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.283 DEBUG:boto:Params: {} 2024-04-13T22:28:55.283 DEBUG:boto:Token: None 2024-04-13T22:28:55.283 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw= 2024-04-13T22:28:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw=', 'Content-Length': '0'} 2024-04-13T22:28:55.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073438be4e21e9f5-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.326 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.326 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.326 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.327 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.327 DEBUG:boto:Method: GET 2024-04-13T22:28:55.327 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.327 DEBUG:boto:Data: 2024-04-13T22:28:55.327 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.327 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.327 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.327 DEBUG:boto:Params: {} 2024-04-13T22:28:55.327 DEBUG:boto:Token: None 2024-04-13T22:28:55.327 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:28:55.327 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw= 2024-04-13T22:28:55.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f0xlBwlOQzIeOj24wNKE8TU9Ezw=', 'Content-Length': '0'} 2024-04-13T22:28:55.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6248f4ddbe91e91-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.329 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.329 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.329 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.330 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.330 DEBUG:boto:Method: GET 2024-04-13T22:28:55.330 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.330 DEBUG:boto:Data: 2024-04-13T22:28:55.330 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.330 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.330 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.330 DEBUG:boto:Params: {} 2024-04-13T22:28:55.330 DEBUG:boto:Token: None 2024-04-13T22:28:55.330 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.330 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005acddfdc1911eefa-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.333 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.333 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.333 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.333 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.334 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.334 DEBUG:boto:Method: GET 2024-04-13T22:28:55.334 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.334 DEBUG:boto:Data: 2024-04-13T22:28:55.334 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.334 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.334 DEBUG:boto:Params: {} 2024-04-13T22:28:55.334 DEBUG:boto:Token: None 2024-04-13T22:28:55.334 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd5e877a9caaf095-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.338 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.338 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.338 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.338 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.339 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:28:55.339 DEBUG:boto:Data: 2024-04-13T22:28:55.339 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.339 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.339 DEBUG:boto:Params: {} 2024-04-13T22:28:55.339 DEBUG:boto:Token: None 2024-04-13T22:28:55.339 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69 2024-04-13T22:28:55.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HBY4V+acrbXex3HWDc4SQB/5oY4= 2024-04-13T22:28:55.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HBY4V+acrbXex3HWDc4SQB/5oY4=', 'Content-Length': '0'} 2024-04-13T22:28:55.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d27452e7dc10176c-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.341 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.341 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.341 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.341 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:28:55.341 DEBUG:boto:Data: 2024-04-13T22:28:55.341 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.341 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.342 DEBUG:boto:Params: {} 2024-04-13T22:28:55.342 DEBUG:boto:Token: None 2024-04-13T22:28:55.342 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69 2024-04-13T22:28:55.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HBY4V+acrbXex3HWDc4SQB/5oY4= 2024-04-13T22:28:55.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HBY4V+acrbXex3HWDc4SQB/5oY4=', 'Content-Length': '0'} 2024-04-13T22:28:55.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b2d4e148a24c2f-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:28:55.344 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.344 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.344 DEBUG:boto:Method: GET 2024-04-13T22:28:55.344 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:28:55.344 DEBUG:boto:Data: 2024-04-13T22:28:55.344 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.344 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.344 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.344 DEBUG:boto:Params: {} 2024-04-13T22:28:55.344 DEBUG:boto:Token: None 2024-04-13T22:28:55.345 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69 2024-04-13T22:28:55.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ed5Jr7QRx5c5UJlnRaElLKMU8dY= 2024-04-13T22:28:55.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ed5Jr7QRx5c5UJlnRaElLKMU8dY=', 'Content-Length': '0'} 2024-04-13T22:28:55.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b88bb620d93ccf-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.347 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:28:55.347 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:28:55.347 DEBUG:boto:Method: GET 2024-04-13T22:28:55.347 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:28:55.347 DEBUG:boto:Data: 2024-04-13T22:28:55.347 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.347 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.347 DEBUG:boto:Params: {} 2024-04-13T22:28:55.347 DEBUG:boto:Token: None 2024-04-13T22:28:55.347 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj69 2024-04-13T22:28:55.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ed5Jr7QRx5c5UJlnRaElLKMU8dY= 2024-04-13T22:28:55.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ed5Jr7QRx5c5UJlnRaElLKMU8dY=', 'Content-Length': '0'} 2024-04-13T22:28:55.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098b42fe5589ab104-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.349 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.350 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.350 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.350 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.350 DEBUG:boto:Method: GET 2024-04-13T22:28:55.350 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.350 DEBUG:boto:Data: 2024-04-13T22:28:55.350 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.350 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.350 DEBUG:boto:Params: {} 2024-04-13T22:28:55.350 DEBUG:boto:Token: None 2024-04-13T22:28:55.351 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca824a82b5cdb16e-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.353 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.353 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.353 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.353 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.353 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.354 DEBUG:boto:Method: GET 2024-04-13T22:28:55.354 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:28:55.354 DEBUG:boto:Data: 2024-04-13T22:28:55.354 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.354 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.354 DEBUG:boto:Params: {} 2024-04-13T22:28:55.354 DEBUG:boto:Token: None 2024-04-13T22:28:55.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd840811d84f6161-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.358 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:28:55.358 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.358 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.358 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.358 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.358 DEBUG:boto:Method: GET 2024-04-13T22:28:55.358 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.359 DEBUG:boto:Data: 2024-04-13T22:28:55.359 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.359 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.359 DEBUG:boto:Params: {} 2024-04-13T22:28:55.359 DEBUG:boto:Token: None 2024-04-13T22:28:55.359 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.359 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8= 2024-04-13T22:28:55.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8=', 'Content-Length': '0'} 2024-04-13T22:28:55.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070c6d3d657e1bb6d-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.361 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.361 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.361 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.362 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.362 DEBUG:boto:Method: GET 2024-04-13T22:28:55.362 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.362 DEBUG:boto:Data: 2024-04-13T22:28:55.362 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.362 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.362 DEBUG:boto:Params: {} 2024-04-13T22:28:55.362 DEBUG:boto:Token: None 2024-04-13T22:28:55.362 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.362 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8= 2024-04-13T22:28:55.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8=', 'Content-Length': '0'} 2024-04-13T22:28:55.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db6ee5a22d93fa0b-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.364 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.364 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.364 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.364 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.364 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.364 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.365 DEBUG:boto:Data: 2024-04-13T22:28:55.365 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.365 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.365 DEBUG:boto:Params: {} 2024-04-13T22:28:55.365 DEBUG:boto:Token: None 2024-04-13T22:28:55.365 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CNqVkdjgKNfK0uPSs3PpQ9GtNpU= 2024-04-13T22:28:55.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CNqVkdjgKNfK0uPSs3PpQ9GtNpU=', 'Content-Length': '0'} 2024-04-13T22:28:55.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b7c73c0014f6cb-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.367 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.367 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.367 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.367 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.367 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.367 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.367 DEBUG:boto:Data: 2024-04-13T22:28:55.367 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.368 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.368 DEBUG:boto:Params: {} 2024-04-13T22:28:55.368 DEBUG:boto:Token: None 2024-04-13T22:28:55.368 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CNqVkdjgKNfK0uPSs3PpQ9GtNpU= 2024-04-13T22:28:55.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CNqVkdjgKNfK0uPSs3PpQ9GtNpU=', 'Content-Length': '0'} 2024-04-13T22:28:55.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002984ab60e40cdaec-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:28:55.370 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.370 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.370 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.370 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.370 DEBUG:boto:Method: GET 2024-04-13T22:28:55.370 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.370 DEBUG:boto:Data: 2024-04-13T22:28:55.370 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.371 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.371 DEBUG:boto:Params: {} 2024-04-13T22:28:55.371 DEBUG:boto:Token: None 2024-04-13T22:28:55.371 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8= 2024-04-13T22:28:55.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8=', 'Content-Length': '0'} 2024-04-13T22:28:55.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad085badcb570753-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.414 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.414 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.414 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.415 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.415 DEBUG:boto:Method: GET 2024-04-13T22:28:55.415 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.415 DEBUG:boto:Data: 2024-04-13T22:28:55.415 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.415 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.415 DEBUG:boto:Params: {} 2024-04-13T22:28:55.415 DEBUG:boto:Token: None 2024-04-13T22:28:55.415 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:28:55.415 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8= 2024-04-13T22:28:55.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IBtP+XWquoS1iegyjz+olbr37M8=', 'Content-Length': '0'} 2024-04-13T22:28:55.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007deb1624324add74-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.418 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.418 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.418 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.418 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.418 DEBUG:boto:Method: GET 2024-04-13T22:28:55.418 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.418 DEBUG:boto:Data: 2024-04-13T22:28:55.418 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.418 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.418 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.418 DEBUG:boto:Params: {} 2024-04-13T22:28:55.418 DEBUG:boto:Token: None 2024-04-13T22:28:55.419 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.419 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000447b18c2c941fc8f-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.421 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.421 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.421 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.422 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.422 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.422 DEBUG:boto:Method: GET 2024-04-13T22:28:55.422 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.422 DEBUG:boto:Data: 2024-04-13T22:28:55.422 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.422 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.422 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.422 DEBUG:boto:Params: {} 2024-04-13T22:28:55.422 DEBUG:boto:Token: None 2024-04-13T22:28:55.423 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.423 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b08515a920b3e39-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.426 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.426 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.426 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.427 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.427 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:28:55.427 DEBUG:boto:Data: 2024-04-13T22:28:55.427 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.427 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.427 DEBUG:boto:Params: {} 2024-04-13T22:28:55.427 DEBUG:boto:Token: None 2024-04-13T22:28:55.427 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76 2024-04-13T22:28:55.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rhb2QwCy87A7FpxLU5hVvyO2JVY= 2024-04-13T22:28:55.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rhb2QwCy87A7FpxLU5hVvyO2JVY=', 'Content-Length': '0'} 2024-04-13T22:28:55.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cb17869fbed687f-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.429 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.429 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.429 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.429 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:28:55.429 DEBUG:boto:Data: 2024-04-13T22:28:55.429 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.430 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.430 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.430 DEBUG:boto:Params: {} 2024-04-13T22:28:55.430 DEBUG:boto:Token: None 2024-04-13T22:28:55.430 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76 2024-04-13T22:28:55.430 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rhb2QwCy87A7FpxLU5hVvyO2JVY= 2024-04-13T22:28:55.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rhb2QwCy87A7FpxLU5hVvyO2JVY=', 'Content-Length': '0'} 2024-04-13T22:28:55.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2be657958f9ea47-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:28:55.432 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.432 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.432 DEBUG:boto:Method: GET 2024-04-13T22:28:55.432 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:28:55.433 DEBUG:boto:Data: 2024-04-13T22:28:55.433 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.433 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.433 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.433 DEBUG:boto:Params: {} 2024-04-13T22:28:55.433 DEBUG:boto:Token: None 2024-04-13T22:28:55.433 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76 2024-04-13T22:28:55.433 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tEZ6YngvL63LRBvVv9s6W2vtT0I= 2024-04-13T22:28:55.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tEZ6YngvL63LRBvVv9s6W2vtT0I=', 'Content-Length': '0'} 2024-04-13T22:28:55.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b44938f59577060-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.435 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:28:55.435 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:28:55.435 DEBUG:boto:Method: GET 2024-04-13T22:28:55.435 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:28:55.435 DEBUG:boto:Data: 2024-04-13T22:28:55.435 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.436 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.436 DEBUG:boto:Params: {} 2024-04-13T22:28:55.436 DEBUG:boto:Token: None 2024-04-13T22:28:55.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj76 2024-04-13T22:28:55.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tEZ6YngvL63LRBvVv9s6W2vtT0I= 2024-04-13T22:28:55.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tEZ6YngvL63LRBvVv9s6W2vtT0I=', 'Content-Length': '0'} 2024-04-13T22:28:55.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d01b7ceab8e1813c-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.438 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.438 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.438 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.438 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.438 DEBUG:boto:Method: GET 2024-04-13T22:28:55.438 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.438 DEBUG:boto:Data: 2024-04-13T22:28:55.439 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.439 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.439 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.439 DEBUG:boto:Params: {} 2024-04-13T22:28:55.439 DEBUG:boto:Token: None 2024-04-13T22:28:55.439 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.439 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd0a81acc568763a-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.441 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.441 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.442 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.442 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.442 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.442 DEBUG:boto:Method: GET 2024-04-13T22:28:55.442 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:28:55.442 DEBUG:boto:Data: 2024-04-13T22:28:55.442 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.442 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.442 DEBUG:boto:Params: {} 2024-04-13T22:28:55.443 DEBUG:boto:Token: None 2024-04-13T22:28:55.443 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090aff72ede5ce08a-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.446 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:28:55.447 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.447 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.447 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.447 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.447 DEBUG:boto:Method: GET 2024-04-13T22:28:55.447 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.447 DEBUG:boto:Data: 2024-04-13T22:28:55.447 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.448 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.448 DEBUG:boto:Params: {} 2024-04-13T22:28:55.448 DEBUG:boto:Token: None 2024-04-13T22:28:55.448 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM= 2024-04-13T22:28:55.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM=', 'Content-Length': '0'} 2024-04-13T22:28:55.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d2b2b3965993a47-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.450 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.450 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.450 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.450 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.450 DEBUG:boto:Method: GET 2024-04-13T22:28:55.450 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.450 DEBUG:boto:Data: 2024-04-13T22:28:55.451 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.451 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.451 DEBUG:boto:Params: {} 2024-04-13T22:28:55.451 DEBUG:boto:Token: None 2024-04-13T22:28:55.451 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM= 2024-04-13T22:28:55.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM=', 'Content-Length': '0'} 2024-04-13T22:28:55.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d799b189f84ae0d-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.494 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.495 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.495 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.495 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.495 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.495 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.495 DEBUG:boto:Data: 2024-04-13T22:28:55.495 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.495 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.495 DEBUG:boto:Params: {} 2024-04-13T22:28:55.495 DEBUG:boto:Token: None 2024-04-13T22:28:55.496 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W9NsOhmJgIQ0d8sFTPSJSWr7DBc= 2024-04-13T22:28:55.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W9NsOhmJgIQ0d8sFTPSJSWr7DBc=', 'Content-Length': '0'} 2024-04-13T22:28:55.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a335fbb5cd6b5790-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.498 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.498 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.498 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.498 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.498 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.498 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.498 DEBUG:boto:Data: 2024-04-13T22:28:55.498 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.498 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.499 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.499 DEBUG:boto:Params: {} 2024-04-13T22:28:55.499 DEBUG:boto:Token: None 2024-04-13T22:28:55.499 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.499 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:W9NsOhmJgIQ0d8sFTPSJSWr7DBc= 2024-04-13T22:28:55.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W9NsOhmJgIQ0d8sFTPSJSWr7DBc=', 'Content-Length': '0'} 2024-04-13T22:28:55.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d977a497187d6f-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:28:55.501 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.501 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.501 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.501 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.501 DEBUG:boto:Method: GET 2024-04-13T22:28:55.501 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.501 DEBUG:boto:Data: 2024-04-13T22:28:55.501 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.502 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.502 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.502 DEBUG:boto:Params: {} 2024-04-13T22:28:55.502 DEBUG:boto:Token: None 2024-04-13T22:28:55.502 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.502 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM= 2024-04-13T22:28:55.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM=', 'Content-Length': '0'} 2024-04-13T22:28:55.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d172c9fc4bf833-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.504 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.504 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.504 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.505 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.505 DEBUG:boto:Method: GET 2024-04-13T22:28:55.505 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.505 DEBUG:boto:Data: 2024-04-13T22:28:55.505 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.505 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.505 DEBUG:boto:Params: {} 2024-04-13T22:28:55.505 DEBUG:boto:Token: None 2024-04-13T22:28:55.505 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:28:55.505 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM= 2024-04-13T22:28:55.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iKky34mjIEjC+XbGF9V47SpzQcM=', 'Content-Length': '0'} 2024-04-13T22:28:55.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee97d050a7a730c-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.507 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.507 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.508 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.508 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.508 DEBUG:boto:Method: GET 2024-04-13T22:28:55.508 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.508 DEBUG:boto:Data: 2024-04-13T22:28:55.508 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.508 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.508 DEBUG:boto:Params: {} 2024-04-13T22:28:55.508 DEBUG:boto:Token: None 2024-04-13T22:28:55.508 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.509 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba4fa35e9ba1a2c2-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.511 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.511 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.511 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.511 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.511 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.512 DEBUG:boto:Method: GET 2024-04-13T22:28:55.512 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.512 DEBUG:boto:Data: 2024-04-13T22:28:55.512 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.512 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.512 DEBUG:boto:Params: {} 2024-04-13T22:28:55.512 DEBUG:boto:Token: None 2024-04-13T22:28:55.512 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d5c86f495d02aa7-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.516 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.516 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.516 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.516 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.516 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:28:55.516 DEBUG:boto:Data: 2024-04-13T22:28:55.517 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.517 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.517 DEBUG:boto:Params: {} 2024-04-13T22:28:55.517 DEBUG:boto:Token: None 2024-04-13T22:28:55.517 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85 2024-04-13T22:28:55.517 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aol4KumWB7Vohn4qW9BkTuw1MNc= 2024-04-13T22:28:55.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aol4KumWB7Vohn4qW9BkTuw1MNc=', 'Content-Length': '0'} 2024-04-13T22:28:55.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f4c4155517b6e08-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.519 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.519 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.519 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.519 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:28:55.519 DEBUG:boto:Data: 2024-04-13T22:28:55.519 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.520 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.520 DEBUG:boto:Params: {} 2024-04-13T22:28:55.520 DEBUG:boto:Token: None 2024-04-13T22:28:55.520 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85 2024-04-13T22:28:55.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aol4KumWB7Vohn4qW9BkTuw1MNc= 2024-04-13T22:28:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aol4KumWB7Vohn4qW9BkTuw1MNc=', 'Content-Length': '0'} 2024-04-13T22:28:55.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1b8f9bf074b48c-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:28:55.523 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.523 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.523 DEBUG:boto:Method: GET 2024-04-13T22:28:55.523 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:28:55.523 DEBUG:boto:Data: 2024-04-13T22:28:55.523 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.523 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.523 DEBUG:boto:Params: {} 2024-04-13T22:28:55.523 DEBUG:boto:Token: None 2024-04-13T22:28:55.523 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85 2024-04-13T22:28:55.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJNw2kJj3QMbZ4PH78kdXJAWIFc= 2024-04-13T22:28:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJNw2kJj3QMbZ4PH78kdXJAWIFc=', 'Content-Length': '0'} 2024-04-13T22:28:55.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681530c6ee17a187-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.566 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:28:55.566 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:28:55.567 DEBUG:boto:Method: GET 2024-04-13T22:28:55.567 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:28:55.567 DEBUG:boto:Data: 2024-04-13T22:28:55.567 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.567 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.567 DEBUG:boto:Params: {} 2024-04-13T22:28:55.567 DEBUG:boto:Token: None 2024-04-13T22:28:55.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj85 2024-04-13T22:28:55.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NJNw2kJj3QMbZ4PH78kdXJAWIFc= 2024-04-13T22:28:55.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NJNw2kJj3QMbZ4PH78kdXJAWIFc=', 'Content-Length': '0'} 2024-04-13T22:28:55.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2fd0d009acfb804-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.569 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.570 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.570 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.570 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.570 DEBUG:boto:Method: GET 2024-04-13T22:28:55.570 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.570 DEBUG:boto:Data: 2024-04-13T22:28:55.570 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.570 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.570 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.570 DEBUG:boto:Params: {} 2024-04-13T22:28:55.570 DEBUG:boto:Token: None 2024-04-13T22:28:55.571 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b63d72981f6cc3d-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.573 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.573 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.573 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.573 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.573 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.573 DEBUG:boto:Method: GET 2024-04-13T22:28:55.573 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:28:55.574 DEBUG:boto:Data: 2024-04-13T22:28:55.574 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.574 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.574 DEBUG:boto:Params: {} 2024-04-13T22:28:55.574 DEBUG:boto:Token: None 2024-04-13T22:28:55.574 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.574 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be9e7d17b3d641a7-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.578 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:28:55.579 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.579 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.579 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.579 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.579 DEBUG:boto:Method: GET 2024-04-13T22:28:55.579 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.579 DEBUG:boto:Data: 2024-04-13T22:28:55.579 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.579 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.579 DEBUG:boto:Params: {} 2024-04-13T22:28:55.580 DEBUG:boto:Token: None 2024-04-13T22:28:55.580 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw= 2024-04-13T22:28:55.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw=', 'Content-Length': '0'} 2024-04-13T22:28:55.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004b20a1876a5522-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.582 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.582 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.582 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.582 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.582 DEBUG:boto:Method: GET 2024-04-13T22:28:55.582 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.582 DEBUG:boto:Data: 2024-04-13T22:28:55.582 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.583 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.583 DEBUG:boto:Params: {} 2024-04-13T22:28:55.583 DEBUG:boto:Token: None 2024-04-13T22:28:55.583 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.583 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw= 2024-04-13T22:28:55.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw=', 'Content-Length': '0'} 2024-04-13T22:28:55.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76f7a9c91fa5015-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.586 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.586 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.586 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.586 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.586 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.586 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.586 DEBUG:boto:Data: 2024-04-13T22:28:55.586 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.586 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.587 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.587 DEBUG:boto:Params: {} 2024-04-13T22:28:55.587 DEBUG:boto:Token: None 2024-04-13T22:28:55.587 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1Q+ULfmng+ZqFoT5NCXCxAse/rA= 2024-04-13T22:28:55.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Q+ULfmng+ZqFoT5NCXCxAse/rA=', 'Content-Length': '0'} 2024-04-13T22:28:55.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed4ee407f7aa0aa-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.589 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.589 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.589 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.589 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.590 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.590 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.590 DEBUG:boto:Data: 2024-04-13T22:28:55.590 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.590 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.590 DEBUG:boto:Params: {} 2024-04-13T22:28:55.590 DEBUG:boto:Token: None 2024-04-13T22:28:55.590 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.590 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1Q+ULfmng+ZqFoT5NCXCxAse/rA= 2024-04-13T22:28:55.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Q+ULfmng+ZqFoT5NCXCxAse/rA=', 'Content-Length': '0'} 2024-04-13T22:28:55.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049deff5c7723789b-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:28:55.593 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.593 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.593 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.593 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.593 DEBUG:boto:Method: GET 2024-04-13T22:28:55.593 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.593 DEBUG:boto:Data: 2024-04-13T22:28:55.593 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.593 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.593 DEBUG:boto:Params: {} 2024-04-13T22:28:55.594 DEBUG:boto:Token: None 2024-04-13T22:28:55.594 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw= 2024-04-13T22:28:55.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw=', 'Content-Length': '0'} 2024-04-13T22:28:55.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e3f0976004e4c0-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.596 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.596 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.596 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.596 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.596 DEBUG:boto:Method: GET 2024-04-13T22:28:55.597 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.597 DEBUG:boto:Data: 2024-04-13T22:28:55.597 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.597 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.597 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.597 DEBUG:boto:Params: {} 2024-04-13T22:28:55.597 DEBUG:boto:Token: None 2024-04-13T22:28:55.597 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:28:55.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw= 2024-04-13T22:28:55.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1kmlyHnkwxEo1uHjW0Qj83DlTqw=', 'Content-Length': '0'} 2024-04-13T22:28:55.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdbf1d1aa5271138-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.599 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.600 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.600 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.600 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.600 DEBUG:boto:Method: GET 2024-04-13T22:28:55.600 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.600 DEBUG:boto:Data: 2024-04-13T22:28:55.600 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.600 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.600 DEBUG:boto:Params: {} 2024-04-13T22:28:55.600 DEBUG:boto:Token: None 2024-04-13T22:28:55.601 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9c17c083480975-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.603 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.603 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.603 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.604 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.604 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.604 DEBUG:boto:Method: GET 2024-04-13T22:28:55.604 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.604 DEBUG:boto:Data: 2024-04-13T22:28:55.604 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.604 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.604 DEBUG:boto:Params: {} 2024-04-13T22:28:55.604 DEBUG:boto:Token: None 2024-04-13T22:28:55.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000471221eed29c29ef-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.608 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.608 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.608 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.608 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.608 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:28:55.609 DEBUG:boto:Data: 2024-04-13T22:28:55.609 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.609 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.609 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.609 DEBUG:boto:Params: {} 2024-04-13T22:28:55.609 DEBUG:boto:Token: None 2024-04-13T22:28:55.609 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87 2024-04-13T22:28:55.609 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bfN+WH9GdHkqgsg59jADC6aMggs= 2024-04-13T22:28:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bfN+WH9GdHkqgsg59jADC6aMggs=', 'Content-Length': '0'} 2024-04-13T22:28:55.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045dae11fb2797fda-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.611 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.611 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.611 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.611 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:28:55.612 DEBUG:boto:Data: 2024-04-13T22:28:55.612 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.612 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.612 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.612 DEBUG:boto:Params: {} 2024-04-13T22:28:55.612 DEBUG:boto:Token: None 2024-04-13T22:28:55.612 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87 2024-04-13T22:28:55.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bfN+WH9GdHkqgsg59jADC6aMggs= 2024-04-13T22:28:55.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bfN+WH9GdHkqgsg59jADC6aMggs=', 'Content-Length': '0'} 2024-04-13T22:28:55.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cbc93a0ee0fc751-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:28:55.614 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.614 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.614 DEBUG:boto:Method: GET 2024-04-13T22:28:55.614 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:28:55.614 DEBUG:boto:Data: 2024-04-13T22:28:55.615 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.615 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.615 DEBUG:boto:Params: {} 2024-04-13T22:28:55.615 DEBUG:boto:Token: None 2024-04-13T22:28:55.615 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87 2024-04-13T22:28:55.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZH89XM0BjtG1DLKXnB6vjq9YESw= 2024-04-13T22:28:55.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZH89XM0BjtG1DLKXnB6vjq9YESw=', 'Content-Length': '0'} 2024-04-13T22:28:55.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087318db7efda7567-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.658 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:28:55.658 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:28:55.659 DEBUG:boto:Method: GET 2024-04-13T22:28:55.659 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:28:55.659 DEBUG:boto:Data: 2024-04-13T22:28:55.659 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.659 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.659 DEBUG:boto:Params: {} 2024-04-13T22:28:55.659 DEBUG:boto:Token: None 2024-04-13T22:28:55.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj87 2024-04-13T22:28:55.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZH89XM0BjtG1DLKXnB6vjq9YESw= 2024-04-13T22:28:55.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZH89XM0BjtG1DLKXnB6vjq9YESw=', 'Content-Length': '0'} 2024-04-13T22:28:55.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8017a4c5204f77c-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.662 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.662 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.662 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.662 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.662 DEBUG:boto:Method: GET 2024-04-13T22:28:55.662 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.662 DEBUG:boto:Data: 2024-04-13T22:28:55.662 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.662 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.663 DEBUG:boto:Params: {} 2024-04-13T22:28:55.663 DEBUG:boto:Token: None 2024-04-13T22:28:55.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb88187fd9e46924-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.665 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.666 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.666 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.666 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.666 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.666 DEBUG:boto:Method: GET 2024-04-13T22:28:55.666 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:28:55.666 DEBUG:boto:Data: 2024-04-13T22:28:55.667 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.667 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.667 DEBUG:boto:Params: {} 2024-04-13T22:28:55.667 DEBUG:boto:Token: None 2024-04-13T22:28:55.667 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.667 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c411645594d218-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.670 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:28:55.671 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.671 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.671 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.671 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.671 DEBUG:boto:Method: GET 2024-04-13T22:28:55.671 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.671 DEBUG:boto:Data: 2024-04-13T22:28:55.671 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.671 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.671 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.671 DEBUG:boto:Params: {} 2024-04-13T22:28:55.672 DEBUG:boto:Token: None 2024-04-13T22:28:55.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ= 2024-04-13T22:28:55.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58634d125d60564-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.674 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.674 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.674 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.674 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.674 DEBUG:boto:Method: GET 2024-04-13T22:28:55.674 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.674 DEBUG:boto:Data: 2024-04-13T22:28:55.674 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.675 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.675 DEBUG:boto:Params: {} 2024-04-13T22:28:55.675 DEBUG:boto:Token: None 2024-04-13T22:28:55.675 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.675 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ= 2024-04-13T22:28:55.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b48783658ad4df-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.677 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.677 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.677 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.678 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.678 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.678 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.678 DEBUG:boto:Data: 2024-04-13T22:28:55.678 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.678 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.678 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.678 DEBUG:boto:Params: {} 2024-04-13T22:28:55.678 DEBUG:boto:Token: None 2024-04-13T22:28:55.678 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.679 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D/+/KuGuEV3yymMC6JAjoSBmk0E= 2024-04-13T22:28:55.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D/+/KuGuEV3yymMC6JAjoSBmk0E=', 'Content-Length': '0'} 2024-04-13T22:28:55.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bedfdc73ba87e68-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.681 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.681 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.681 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.681 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.681 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.681 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.681 DEBUG:boto:Data: 2024-04-13T22:28:55.681 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.681 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.681 DEBUG:boto:Params: {} 2024-04-13T22:28:55.682 DEBUG:boto:Token: None 2024-04-13T22:28:55.682 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D/+/KuGuEV3yymMC6JAjoSBmk0E= 2024-04-13T22:28:55.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D/+/KuGuEV3yymMC6JAjoSBmk0E=', 'Content-Length': '0'} 2024-04-13T22:28:55.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002acfabbca4e9c932-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:28:55.684 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.684 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.684 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.684 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.684 DEBUG:boto:Method: GET 2024-04-13T22:28:55.684 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.684 DEBUG:boto:Data: 2024-04-13T22:28:55.684 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.685 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.685 DEBUG:boto:Params: {} 2024-04-13T22:28:55.685 DEBUG:boto:Token: None 2024-04-13T22:28:55.685 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ= 2024-04-13T22:28:55.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023773631413164d6-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.687 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.687 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.687 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.687 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.687 DEBUG:boto:Method: GET 2024-04-13T22:28:55.688 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.688 DEBUG:boto:Data: 2024-04-13T22:28:55.688 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.688 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.688 DEBUG:boto:Params: {} 2024-04-13T22:28:55.688 DEBUG:boto:Token: None 2024-04-13T22:28:55.688 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:28:55.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ= 2024-04-13T22:28:55.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+m0if16ODANgjSoOhpvo9zOEcOQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c2c6f2541e01d5-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.690 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.690 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.690 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.690 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.690 DEBUG:boto:Method: GET 2024-04-13T22:28:55.691 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.691 DEBUG:boto:Data: 2024-04-13T22:28:55.691 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.691 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.691 DEBUG:boto:Params: {} 2024-04-13T22:28:55.691 DEBUG:boto:Token: None 2024-04-13T22:28:55.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f5d9e1558f71150-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.693 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.694 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.694 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.694 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.694 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.694 DEBUG:boto:Method: GET 2024-04-13T22:28:55.694 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.694 DEBUG:boto:Data: 2024-04-13T22:28:55.694 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.695 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.695 DEBUG:boto:Params: {} 2024-04-13T22:28:55.695 DEBUG:boto:Token: None 2024-04-13T22:28:55.695 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.695 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7fc4ca9ea52fc0-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.698 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.698 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.698 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.698 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.699 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:28:55.699 DEBUG:boto:Data: 2024-04-13T22:28:55.699 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.699 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.699 DEBUG:boto:Params: {} 2024-04-13T22:28:55.699 DEBUG:boto:Token: None 2024-04-13T22:28:55.699 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92 2024-04-13T22:28:55.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1hShOdY16Twk2s4l9XPGLA0C2w= 2024-04-13T22:28:55.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1hShOdY16Twk2s4l9XPGLA0C2w=', 'Content-Length': '0'} 2024-04-13T22:28:55.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300701f7f8092e3f-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.701 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.701 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.701 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.701 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:28:55.701 DEBUG:boto:Data: 2024-04-13T22:28:55.701 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.702 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.702 DEBUG:boto:Params: {} 2024-04-13T22:28:55.702 DEBUG:boto:Token: None 2024-04-13T22:28:55.702 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92 2024-04-13T22:28:55.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1hShOdY16Twk2s4l9XPGLA0C2w= 2024-04-13T22:28:55.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1hShOdY16Twk2s4l9XPGLA0C2w=', 'Content-Length': '0'} 2024-04-13T22:28:55.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b77441e78828a9-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:28:55.704 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.704 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.704 DEBUG:boto:Method: GET 2024-04-13T22:28:55.704 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:28:55.704 DEBUG:boto:Data: 2024-04-13T22:28:55.704 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.705 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.705 DEBUG:boto:Params: {} 2024-04-13T22:28:55.705 DEBUG:boto:Token: None 2024-04-13T22:28:55.705 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92 2024-04-13T22:28:55.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vStUoAV3By932iflLBbUNsQ/lvk= 2024-04-13T22:28:55.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vStUoAV3By932iflLBbUNsQ/lvk=', 'Content-Length': '0'} 2024-04-13T22:28:55.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8bafb20f4923b65-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.707 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:28:55.707 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:28:55.707 DEBUG:boto:Method: GET 2024-04-13T22:28:55.707 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:28:55.707 DEBUG:boto:Data: 2024-04-13T22:28:55.707 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.708 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.708 DEBUG:boto:Params: {} 2024-04-13T22:28:55.708 DEBUG:boto:Token: None 2024-04-13T22:28:55.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj92 2024-04-13T22:28:55.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vStUoAV3By932iflLBbUNsQ/lvk= 2024-04-13T22:28:55.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vStUoAV3By932iflLBbUNsQ/lvk=', 'Content-Length': '0'} 2024-04-13T22:28:55.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000889e18d16468b8bc-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.710 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.710 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.710 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.710 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.710 DEBUG:boto:Method: GET 2024-04-13T22:28:55.710 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.710 DEBUG:boto:Data: 2024-04-13T22:28:55.710 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.711 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.711 DEBUG:boto:Params: {} 2024-04-13T22:28:55.711 DEBUG:boto:Token: None 2024-04-13T22:28:55.711 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff8a131985ca7e7-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.762 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.763 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.763 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.763 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.763 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.763 DEBUG:boto:Method: GET 2024-04-13T22:28:55.763 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:28:55.763 DEBUG:boto:Data: 2024-04-13T22:28:55.763 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.764 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.764 DEBUG:boto:Params: {} 2024-04-13T22:28:55.764 DEBUG:boto:Token: None 2024-04-13T22:28:55.764 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088df9e417ea19b5f-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.768 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:28:55.768 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.768 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.769 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.769 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.769 DEBUG:boto:Method: GET 2024-04-13T22:28:55.769 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.769 DEBUG:boto:Data: 2024-04-13T22:28:55.769 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.769 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.769 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.769 DEBUG:boto:Params: {} 2024-04-13T22:28:55.769 DEBUG:boto:Token: None 2024-04-13T22:28:55.769 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.770 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4= 2024-04-13T22:28:55.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4=', 'Content-Length': '0'} 2024-04-13T22:28:55.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7885392902f8d6a-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.772 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.772 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.772 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.772 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.772 DEBUG:boto:Method: GET 2024-04-13T22:28:55.772 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.773 DEBUG:boto:Data: 2024-04-13T22:28:55.773 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.773 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.773 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.773 DEBUG:boto:Params: {} 2024-04-13T22:28:55.773 DEBUG:boto:Token: None 2024-04-13T22:28:55.773 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.773 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4= 2024-04-13T22:28:55.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4=', 'Content-Length': '0'} 2024-04-13T22:28:55.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094b57c595b1a2273-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.822 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.823 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.823 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.823 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.823 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.823 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.823 DEBUG:boto:Data: 2024-04-13T22:28:55.823 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.823 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.823 DEBUG:boto:Params: {} 2024-04-13T22:28:55.823 DEBUG:boto:Token: None 2024-04-13T22:28:55.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TdCp/6+osGfZVFZtDrQle6SOPBg= 2024-04-13T22:28:55.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TdCp/6+osGfZVFZtDrQle6SOPBg=', 'Content-Length': '0'} 2024-04-13T22:28:55.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c65ce5aae085136-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.826 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.826 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.826 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.826 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.826 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.826 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.826 DEBUG:boto:Data: 2024-04-13T22:28:55.826 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.827 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.827 DEBUG:boto:Params: {} 2024-04-13T22:28:55.827 DEBUG:boto:Token: None 2024-04-13T22:28:55.827 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TdCp/6+osGfZVFZtDrQle6SOPBg= 2024-04-13T22:28:55.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TdCp/6+osGfZVFZtDrQle6SOPBg=', 'Content-Length': '0'} 2024-04-13T22:28:55.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440445ad58e4f75e-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:28:55.829 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.829 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.829 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.829 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.829 DEBUG:boto:Method: GET 2024-04-13T22:28:55.829 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.829 DEBUG:boto:Data: 2024-04-13T22:28:55.829 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.829 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.830 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.830 DEBUG:boto:Params: {} 2024-04-13T22:28:55.830 DEBUG:boto:Token: None 2024-04-13T22:28:55.830 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.830 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4= 2024-04-13T22:28:55.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4=', 'Content-Length': '0'} 2024-04-13T22:28:55.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d674163564f3112-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.832 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.832 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.832 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.832 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.832 DEBUG:boto:Method: GET 2024-04-13T22:28:55.832 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.833 DEBUG:boto:Data: 2024-04-13T22:28:55.833 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.833 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.833 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.833 DEBUG:boto:Params: {} 2024-04-13T22:28:55.833 DEBUG:boto:Token: None 2024-04-13T22:28:55.833 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:28:55.833 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4= 2024-04-13T22:28:55.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IUiqFMR9HqIrjQ9D3UrOd1hwsm4=', 'Content-Length': '0'} 2024-04-13T22:28:55.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046fb0d387c570b3c-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.835 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.835 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.835 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.835 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.835 DEBUG:boto:Method: GET 2024-04-13T22:28:55.835 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.836 DEBUG:boto:Data: 2024-04-13T22:28:55.836 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.836 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.836 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.836 DEBUG:boto:Params: {} 2024-04-13T22:28:55.836 DEBUG:boto:Token: None 2024-04-13T22:28:55.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d84ab470c00901a-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.839 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.839 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.839 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.839 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.839 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.839 DEBUG:boto:Method: GET 2024-04-13T22:28:55.839 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.839 DEBUG:boto:Data: 2024-04-13T22:28:55.840 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.840 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.840 DEBUG:boto:Params: {} 2024-04-13T22:28:55.840 DEBUG:boto:Token: None 2024-04-13T22:28:55.840 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a1854547e5298e-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.844 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.844 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.844 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.844 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.844 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:28:55.844 DEBUG:boto:Data: 2024-04-13T22:28:55.844 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.844 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.845 DEBUG:boto:Params: {} 2024-04-13T22:28:55.845 DEBUG:boto:Token: None 2024-04-13T22:28:55.845 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93 2024-04-13T22:28:55.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y+5socgnYFLpBP3nGuodjmUhiUs= 2024-04-13T22:28:55.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y+5socgnYFLpBP3nGuodjmUhiUs=', 'Content-Length': '0'} 2024-04-13T22:28:55.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8ae73df27f6a2d-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.847 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.847 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.847 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.847 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:28:55.847 DEBUG:boto:Data: 2024-04-13T22:28:55.847 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.847 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.848 DEBUG:boto:Params: {} 2024-04-13T22:28:55.848 DEBUG:boto:Token: None 2024-04-13T22:28:55.848 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93 2024-04-13T22:28:55.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y+5socgnYFLpBP3nGuodjmUhiUs= 2024-04-13T22:28:55.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y+5socgnYFLpBP3nGuodjmUhiUs=', 'Content-Length': '0'} 2024-04-13T22:28:55.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078807d6415502ec0-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:28:55.850 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.850 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.850 DEBUG:boto:Method: GET 2024-04-13T22:28:55.850 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:28:55.850 DEBUG:boto:Data: 2024-04-13T22:28:55.850 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.850 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.850 DEBUG:boto:Params: {} 2024-04-13T22:28:55.851 DEBUG:boto:Token: None 2024-04-13T22:28:55.851 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93 2024-04-13T22:28:55.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P31jUje1afABrsjyu+PkDko4ZWo= 2024-04-13T22:28:55.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P31jUje1afABrsjyu+PkDko4ZWo=', 'Content-Length': '0'} 2024-04-13T22:28:55.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178b35cf0ac6d4a2-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.894 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:28:55.895 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:28:55.895 DEBUG:boto:Method: GET 2024-04-13T22:28:55.895 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:28:55.895 DEBUG:boto:Data: 2024-04-13T22:28:55.895 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.895 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.895 DEBUG:boto:Params: {} 2024-04-13T22:28:55.895 DEBUG:boto:Token: None 2024-04-13T22:28:55.895 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj93 2024-04-13T22:28:55.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:P31jUje1afABrsjyu+PkDko4ZWo= 2024-04-13T22:28:55.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P31jUje1afABrsjyu+PkDko4ZWo=', 'Content-Length': '0'} 2024-04-13T22:28:55.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25980c6dac8d93f-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.897 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.898 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.898 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.898 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.898 DEBUG:boto:Method: GET 2024-04-13T22:28:55.898 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.898 DEBUG:boto:Data: 2024-04-13T22:28:55.898 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.898 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.898 DEBUG:boto:Params: {} 2024-04-13T22:28:55.899 DEBUG:boto:Token: None 2024-04-13T22:28:55.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c042002d5937a50-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.901 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.901 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.901 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.901 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.901 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.902 DEBUG:boto:Method: GET 2024-04-13T22:28:55.902 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:28:55.902 DEBUG:boto:Data: 2024-04-13T22:28:55.902 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.902 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.902 DEBUG:boto:Params: {} 2024-04-13T22:28:55.902 DEBUG:boto:Token: None 2024-04-13T22:28:55.902 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552d28d93af67956-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.906 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:28:55.906 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.906 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.906 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.906 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.906 DEBUG:boto:Method: GET 2024-04-13T22:28:55.907 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.907 DEBUG:boto:Data: 2024-04-13T22:28:55.907 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.907 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.907 DEBUG:boto:Params: {} 2024-04-13T22:28:55.907 DEBUG:boto:Token: None 2024-04-13T22:28:55.907 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ= 2024-04-13T22:28:55.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014dc7081554b20fb-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.909 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.909 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.909 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.909 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.909 DEBUG:boto:Method: GET 2024-04-13T22:28:55.910 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.910 DEBUG:boto:Data: 2024-04-13T22:28:55.910 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.910 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.910 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.910 DEBUG:boto:Params: {} 2024-04-13T22:28:55.910 DEBUG:boto:Token: None 2024-04-13T22:28:55.910 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.910 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ= 2024-04-13T22:28:55.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8532e80b8daf315-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.912 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.912 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.912 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.912 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.913 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.913 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.913 DEBUG:boto:Data: 2024-04-13T22:28:55.913 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.913 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.913 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.913 DEBUG:boto:Params: {} 2024-04-13T22:28:55.913 DEBUG:boto:Token: None 2024-04-13T22:28:55.913 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eioDXD7p7xBURbtCFLDLvrNi8Hk= 2024-04-13T22:28:55.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eioDXD7p7xBURbtCFLDLvrNi8Hk=', 'Content-Length': '0'} 2024-04-13T22:28:55.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6b4901695f2b45-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.915 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.915 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.915 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.916 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.916 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.916 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.916 DEBUG:boto:Data: 2024-04-13T22:28:55.916 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.916 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.916 DEBUG:boto:Params: {} 2024-04-13T22:28:55.916 DEBUG:boto:Token: None 2024-04-13T22:28:55.916 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.917 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eioDXD7p7xBURbtCFLDLvrNi8Hk= 2024-04-13T22:28:55.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eioDXD7p7xBURbtCFLDLvrNi8Hk=', 'Content-Length': '0'} 2024-04-13T22:28:55.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004752d52981072a40-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:28:55.918 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.919 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.919 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.919 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.919 DEBUG:boto:Method: GET 2024-04-13T22:28:55.919 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.919 DEBUG:boto:Data: 2024-04-13T22:28:55.919 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.919 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.919 DEBUG:boto:Params: {} 2024-04-13T22:28:55.919 DEBUG:boto:Token: None 2024-04-13T22:28:55.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ= 2024-04-13T22:28:55.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da05cb9bfce6c37-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.922 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.922 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.922 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.922 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.922 DEBUG:boto:Method: GET 2024-04-13T22:28:55.922 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.922 DEBUG:boto:Data: 2024-04-13T22:28:55.922 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.922 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.923 DEBUG:boto:Params: {} 2024-04-13T22:28:55.923 DEBUG:boto:Token: None 2024-04-13T22:28:55.923 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:28:55.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ= 2024-04-13T22:28:55.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hKfVEu211S/oCpTBNpoY8UG/VTQ=', 'Content-Length': '0'} 2024-04-13T22:28:55.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6bec972743fa1d4-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.925 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.925 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.925 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.925 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.925 DEBUG:boto:Method: GET 2024-04-13T22:28:55.925 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.925 DEBUG:boto:Data: 2024-04-13T22:28:55.925 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.925 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.926 DEBUG:boto:Params: {} 2024-04-13T22:28:55.926 DEBUG:boto:Token: None 2024-04-13T22:28:55.926 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30bee163c4ac8ca-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.928 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.929 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.929 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.929 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.929 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.929 DEBUG:boto:Method: GET 2024-04-13T22:28:55.929 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.929 DEBUG:boto:Data: 2024-04-13T22:28:55.929 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.929 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.929 DEBUG:boto:Params: {} 2024-04-13T22:28:55.929 DEBUG:boto:Token: None 2024-04-13T22:28:55.930 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8774c9cdeb71f8-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.933 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.933 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.933 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.934 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.934 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:28:55.934 DEBUG:boto:Data: 2024-04-13T22:28:55.934 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.934 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.934 DEBUG:boto:Params: {} 2024-04-13T22:28:55.934 DEBUG:boto:Token: None 2024-04-13T22:28:55.934 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95 2024-04-13T22:28:55.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Ma5owaLaUdrRHgJ1yflsE4ls3k= 2024-04-13T22:28:55.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Ma5owaLaUdrRHgJ1yflsE4ls3k=', 'Content-Length': '0'} 2024-04-13T22:28:55.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b86ae1660064113-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.936 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.936 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.936 DEBUG:boto:Method: HEAD 2024-04-13T22:28:55.936 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:28:55.937 DEBUG:boto:Data: 2024-04-13T22:28:55.937 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.937 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.937 DEBUG:boto:Params: {} 2024-04-13T22:28:55.937 DEBUG:boto:Token: None 2024-04-13T22:28:55.937 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95 2024-04-13T22:28:55.937 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7Ma5owaLaUdrRHgJ1yflsE4ls3k= 2024-04-13T22:28:55.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7Ma5owaLaUdrRHgJ1yflsE4ls3k=', 'Content-Length': '0'} 2024-04-13T22:28:55.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd807e9107ec9f13-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:28:55.939 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.939 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.939 DEBUG:boto:Method: GET 2024-04-13T22:28:55.939 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:28:55.939 DEBUG:boto:Data: 2024-04-13T22:28:55.939 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.940 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.940 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.940 DEBUG:boto:Params: {} 2024-04-13T22:28:55.940 DEBUG:boto:Token: None 2024-04-13T22:28:55.940 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95 2024-04-13T22:28:55.940 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YPuT5vbZav3YNaYyQcjf8CIlPJI= 2024-04-13T22:28:55.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YPuT5vbZav3YNaYyQcjf8CIlPJI=', 'Content-Length': '0'} 2024-04-13T22:28:55.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022350f950a506674-00661b0727-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.942 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:28:55.942 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:28:55.942 DEBUG:boto:Method: GET 2024-04-13T22:28:55.942 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:28:55.942 DEBUG:boto:Data: 2024-04-13T22:28:55.943 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.943 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.943 DEBUG:boto:Params: {} 2024-04-13T22:28:55.943 DEBUG:boto:Token: None 2024-04-13T22:28:55.943 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/obj95 2024-04-13T22:28:55.943 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YPuT5vbZav3YNaYyQcjf8CIlPJI= 2024-04-13T22:28:55.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YPuT5vbZav3YNaYyQcjf8CIlPJI=', 'Content-Length': '0'} 2024-04-13T22:28:55.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002124f1da7c715df4-00661b0727-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.945 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.945 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.945 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.946 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.946 DEBUG:boto:Method: GET 2024-04-13T22:28:55.946 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.946 DEBUG:boto:Data: 2024-04-13T22:28:55.946 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:28:55.946 DEBUG:boto:Port: 8001 2024-04-13T22:28:55.946 DEBUG:boto:Params: {} 2024-04-13T22:28:55.946 DEBUG:boto:Token: None 2024-04-13T22:28:55.946 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.946 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5cb08cf923615da-00661b0727-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.949 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.949 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:55.949 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:55.950 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.950 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.950 DEBUG:boto:Method: GET 2024-04-13T22:28:55.950 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:28:55.950 DEBUG:boto:Data: 2024-04-13T22:28:55.950 DEBUG:boto:Headers: {} 2024-04-13T22:28:55.950 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:55.950 DEBUG:boto:Port: 8000 2024-04-13T22:28:55.950 DEBUG:boto:Params: {} 2024-04-13T22:28:55.950 DEBUG:boto:Token: None 2024-04-13T22:28:55.950 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:28:55 GMT /vwnfns-61/ 2024-04-13T22:28:55.951 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po= 2024-04-13T22:28:55.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EKYo3no0bavUowJBM4GB6PQU1po=', 'Content-Length': '0'} 2024-04-13T22:28:55.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d7c216c4b04610-00661b0727-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:28:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:55.954 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:28:55.954 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:28:55.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:28:55.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:28:55.955 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:28:56.007 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:28:56.007 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:28:56.058 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:28:56.058 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-61 2024-04-13T22:28:56.070 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 30 2024-04-13T22:28:56.131 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:56.131 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:56.131 DEBUG:boto:Method: HEAD 2024-04-13T22:28:56.131 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:56.131 DEBUG:boto:Data: 2024-04-13T22:28:56.131 DEBUG:boto:Headers: {} 2024-04-13T22:28:56.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.131 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.131 DEBUG:boto:Params: {} 2024-04-13T22:28:56.131 DEBUG:boto:Token: None 2024-04-13T22:28:56.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/ 2024-04-13T22:28:56.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU= 2024-04-13T22:28:56.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU=', 'Content-Length': '0'} 2024-04-13T22:28:56.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092050b81ca5b3ae6-00661b0728-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.136 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:28:56.136 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:28:56.136 DEBUG:boto:Method: PUT 2024-04-13T22:28:56.136 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:28:56.136 DEBUG:boto:Data: 2024-04-13T22:28:56.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:28:56.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.137 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.137 DEBUG:boto:Params: {} 2024-04-13T22:28:56.137 DEBUG:boto:Token: None 2024-04-13T22:28:56.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/obj37 2024-04-13T22:28:56.137 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ifRmXUdUPfUuF9tpALA+un68Wco= 2024-04-13T22:28:56.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ifRmXUdUPfUuF9tpALA+un68Wco='} 2024-04-13T22:28:56.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3591c3dd1395d9c-00661b0728-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.142 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:56.143 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:56.143 DEBUG:boto:Method: HEAD 2024-04-13T22:28:56.143 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:56.143 DEBUG:boto:Data: 2024-04-13T22:28:56.143 DEBUG:boto:Headers: {} 2024-04-13T22:28:56.143 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.143 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.143 DEBUG:boto:Params: {} 2024-04-13T22:28:56.143 DEBUG:boto:Token: None 2024-04-13T22:28:56.143 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/ 2024-04-13T22:28:56.143 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU= 2024-04-13T22:28:56.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU=', 'Content-Length': '0'} 2024-04-13T22:28:56.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da892992dbefae24-00661b0728-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.147 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:28:56.147 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:28:56.147 DEBUG:boto:Method: PUT 2024-04-13T22:28:56.147 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:28:56.147 DEBUG:boto:Data: 2024-04-13T22:28:56.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:28:56.147 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.147 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.147 DEBUG:boto:Params: {} 2024-04-13T22:28:56.147 DEBUG:boto:Token: None 2024-04-13T22:28:56.148 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/obj74 2024-04-13T22:28:56.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FPEfHegbbEc5DIaYTSH+38gFYSU= 2024-04-13T22:28:56.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FPEfHegbbEc5DIaYTSH+38gFYSU='} 2024-04-13T22:28:56.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f995fc81e59555b9-00661b0728-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.190 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:56.191 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:56.191 DEBUG:boto:Method: HEAD 2024-04-13T22:28:56.191 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:56.191 DEBUG:boto:Data: 2024-04-13T22:28:56.191 DEBUG:boto:Headers: {} 2024-04-13T22:28:56.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.191 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.191 DEBUG:boto:Params: {} 2024-04-13T22:28:56.191 DEBUG:boto:Token: None 2024-04-13T22:28:56.191 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/ 2024-04-13T22:28:56.191 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU= 2024-04-13T22:28:56.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU=', 'Content-Length': '0'} 2024-04-13T22:28:56.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005160a8ae7c436802-00661b0728-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.195 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:28:56.195 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:28:56.195 DEBUG:boto:Method: PUT 2024-04-13T22:28:56.195 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:28:56.195 DEBUG:boto:Data: 2024-04-13T22:28:56.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:28:56.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.196 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.196 DEBUG:boto:Params: {} 2024-04-13T22:28:56.196 DEBUG:boto:Token: None 2024-04-13T22:28:56.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/obj111 2024-04-13T22:28:56.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XvLimQeAbERP027igz+E9J0CUqs= 2024-04-13T22:28:56.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XvLimQeAbERP027igz+E9J0CUqs='} 2024-04-13T22:28:56.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043ef5bab9f1a8e5a-00661b0728-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.201 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:56.201 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:56.201 DEBUG:boto:Method: HEAD 2024-04-13T22:28:56.201 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:56.201 DEBUG:boto:Data: 2024-04-13T22:28:56.202 DEBUG:boto:Headers: {} 2024-04-13T22:28:56.202 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.202 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.202 DEBUG:boto:Params: {} 2024-04-13T22:28:56.202 DEBUG:boto:Token: None 2024-04-13T22:28:56.202 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/ 2024-04-13T22:28:56.202 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU= 2024-04-13T22:28:56.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU=', 'Content-Length': '0'} 2024-04-13T22:28:56.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019dffdff19b9868c-00661b0728-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.206 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:28:56.206 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:28:56.206 DEBUG:boto:Method: PUT 2024-04-13T22:28:56.206 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:28:56.206 DEBUG:boto:Data: 2024-04-13T22:28:56.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:28:56.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.207 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.207 DEBUG:boto:Params: {} 2024-04-13T22:28:56.207 DEBUG:boto:Token: None 2024-04-13T22:28:56.207 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/obj148 2024-04-13T22:28:56.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kjSrn7B5wR//J1LKsx4FuVEI80M= 2024-04-13T22:28:56.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kjSrn7B5wR//J1LKsx4FuVEI80M='} 2024-04-13T22:28:56.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afb1a0aa2e92308b-00661b0728-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.251 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:28:56.251 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:28:56.251 DEBUG:boto:Method: HEAD 2024-04-13T22:28:56.251 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:28:56.251 DEBUG:boto:Data: 2024-04-13T22:28:56.251 DEBUG:boto:Headers: {} 2024-04-13T22:28:56.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.251 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.251 DEBUG:boto:Params: {} 2024-04-13T22:28:56.251 DEBUG:boto:Token: None 2024-04-13T22:28:56.251 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/ 2024-04-13T22:28:56.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU= 2024-04-13T22:28:56.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WIbq86kqQ74HCV553UuIxhJ02EU=', 'Content-Length': '0'} 2024-04-13T22:28:56.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052eb838f7c67d1a1-00661b0728-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.255 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:28:56.255 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:28:56.256 DEBUG:boto:Method: PUT 2024-04-13T22:28:56.256 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:28:56.256 DEBUG:boto:Data: 2024-04-13T22:28:56.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:28:56.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:28:56.256 DEBUG:boto:Port: 8000 2024-04-13T22:28:56.256 DEBUG:boto:Params: {} 2024-04-13T22:28:56.256 DEBUG:boto:Token: None 2024-04-13T22:28:56.256 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:28:56 GMT /vwnfns-61/obj185 2024-04-13T22:28:56.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:60YU96DpFWc3PWyT32W/5sdNzhI= 2024-04-13T22:28:56.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:28:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:60YU96DpFWc3PWyT32W/5sdNzhI='} 2024-04-13T22:28:56.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf27012836e0e078-00661b0728-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:28:56.262 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:28:56.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:28:56.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:28:56.262 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:28:56.317 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:28:56.317 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:28:56.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:28:56.351+0000 7f4244452780 1 waiting to reach latest gen 6, on 5.. 2024-04-13T22:29:01.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:01.353+0000 7f4244452780 1 waiting to reach latest gen 6, on 5.. 2024-04-13T22:29:06.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:06.353+0000 7f4244452780 1 waiting to reach latest gen 6, on 5.. 2024-04-13T22:29:11.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:11.357+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:11.360 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:11.360 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:16.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:16.358+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:16.363 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:16.363 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:21.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:21.361+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:21.365 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:21.365 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:26.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:26.364+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:26.368 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:26.368 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:31.370 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:31.366+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:31.370 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:31.371 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:36.373 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:36.369+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:36.373 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:36.373 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:41.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:41.372+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:41.375 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:41.375 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:46.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:46.375+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:46.378 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:46.378 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:51.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:51.379+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:51.383 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:51.383 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:56.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:29:56.381+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:29:56.385 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:29:56.385 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:01.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:01.384+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:01.388 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:01.388 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:06.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:06.386+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:06.390 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:06.390 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:11.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:11.389+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:11.393 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:11.393 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:16.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:16.393+0000 7f4244452780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:16.395 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , , , , , 00000000002.87.6, , , , , , , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:16.395 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:21.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:21.395+0000 7f4244452780 1 bucket sync caught up with source: 2024-04-13T22:30:21.398 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:21.398 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:21.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:21.395+0000 7f4244452780 0 bucket checkpoint complete 2024-04-13T22:30:21.416 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:30:21.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:30:21.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:30:21.417 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:30:21.477 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:21.478 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:21.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:21.514+0000 7f91021df780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:21.517 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000005.1221.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000004.639.6, 00000000006.1910.6, 00000000005.1766.6] 2024-04-13T22:30:21.517 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000005.1221.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000004.639.6, 00000000007.1914.6, 00000000006.1768.6] 2024-04-13T22:30:26.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:26.516+0000 7f91021df780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:26.518 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000005.1221.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000004.639.6, 00000000006.1910.6, 00000000005.1766.6] 2024-04-13T22:30:26.519 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000005.1221.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000004.639.6, 00000000007.1914.6, 00000000006.1768.6] 2024-04-13T22:30:31.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:31.517+0000 7f91021df780 1 bucket sync caught up with source: 2024-04-13T22:30:31.520 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000005.1221.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000004.639.6, 00000000007.1914.6, 00000000006.1768.6] 2024-04-13T22:30:31.520 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000003.2072.6, 00000000002.2294.6, 00000000002.3339.6, 00000000005.1221.6, 00000000004.1155.6, 00000000001.1756.6, 00000000001.2066.6, 00000000004.639.6, 00000000007.1914.6, 00000000006.1768.6] 2024-04-13T22:30:31.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:31.517+0000 7f91021df780 0 bucket checkpoint complete 2024-04-13T22:30:31.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:30:31.538 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.539 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.539 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.539 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:30:31.539 DEBUG:boto:Data: 2024-04-13T22:30:31.539 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.539 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.539 DEBUG:boto:Params: {} 2024-04-13T22:30:31.540 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:30:31.540 DEBUG:boto:Token: None 2024-04-13T22:30:31.540 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AhDKCcEaqnGwi8bYS+Cy39CLuLQ= 2024-04-13T22:30:31.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AhDKCcEaqnGwi8bYS+Cy39CLuLQ=', 'Content-Length': '0'} 2024-04-13T22:30:31.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea586c13fce13c4b-00661b0787-1f77-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.547 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.547 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.547 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.547 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:30:31.547 DEBUG:boto:Data: 2024-04-13T22:30:31.548 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.548 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.548 DEBUG:boto:Params: {} 2024-04-13T22:30:31.548 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:30:31.548 DEBUG:boto:Token: None 2024-04-13T22:30:31.548 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AhDKCcEaqnGwi8bYS+Cy39CLuLQ= 2024-04-13T22:30:31.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AhDKCcEaqnGwi8bYS+Cy39CLuLQ=', 'Content-Length': '0'} 2024-04-13T22:30:31.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1d384ace40c48b9-00661b0787-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:30:31.555 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.556 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.556 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.556 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.556 DEBUG:boto:Method: GET 2024-04-13T22:30:31.556 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.556 DEBUG:boto:Data: 2024-04-13T22:30:31.556 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.556 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.556 DEBUG:boto:Params: {} 2024-04-13T22:30:31.556 DEBUG:boto:Token: None 2024-04-13T22:30:31.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/?versions 2024-04-13T22:30:31.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50= 2024-04-13T22:30:31.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50=', 'Content-Length': '0'} 2024-04-13T22:30:31.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051eda85ccbdb9c84-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.560 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:30:31.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-13T22:30:31.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:30:31.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:30:31.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-13T22:30:31.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-13T22:30:31.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:30:31.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:30:31.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:30:31.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-13T22:30:31.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:30:31.565 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:30:31.565 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.565 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.565 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.565 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.565 DEBUG:boto:Method: GET 2024-04-13T22:30:31.565 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.565 DEBUG:boto:Data: 2024-04-13T22:30:31.566 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.566 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.566 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.566 DEBUG:boto:Params: {} 2024-04-13T22:30:31.566 DEBUG:boto:Token: None 2024-04-13T22:30:31.566 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/?versions 2024-04-13T22:30:31.566 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50= 2024-04-13T22:30:31.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50=', 'Content-Length': '0'} 2024-04-13T22:30:31.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ca44ae5b9f8600-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.571 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:30:31.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-13T22:30:31.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:30:31.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:30:31.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:30:31.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:30:31.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-13T22:30:31.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:30:31.575 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.575 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.575 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.575 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.575 DEBUG:boto:Method: GET 2024-04-13T22:30:31.576 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.576 DEBUG:boto:Data: 2024-04-13T22:30:31.576 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.576 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.576 DEBUG:boto:Params: {} 2024-04-13T22:30:31.576 DEBUG:boto:Token: None 2024-04-13T22:30:31.576 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/?versions 2024-04-13T22:30:31.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50= 2024-04-13T22:30:31.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50=', 'Content-Length': '0'} 2024-04-13T22:30:31.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2e2914985f68be-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.579 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.581 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.581 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.581 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.581 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.581 DEBUG:boto:Method: GET 2024-04-13T22:30:31.581 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:31.581 DEBUG:boto:Data: 2024-04-13T22:30:31.581 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.582 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.582 DEBUG:boto:Params: {} 2024-04-13T22:30:31.582 DEBUG:boto:Token: None 2024-04-13T22:30:31.582 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/?versions 2024-04-13T22:30:31.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50= 2024-04-13T22:30:31.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BcTL+8pAVlFFzHP2Io8sUvMAW50=', 'Content-Length': '0'} 2024-04-13T22:30:31.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051993fa781e07736-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.586 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:30:31.588 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.588 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.588 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.588 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.588 DEBUG:boto:Method: GET 2024-04-13T22:30:31.588 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.588 DEBUG:boto:Data: 2024-04-13T22:30:31.588 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.589 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.589 DEBUG:boto:Params: {} 2024-04-13T22:30:31.589 DEBUG:boto:Token: None 2024-04-13T22:30:31.589 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk= 2024-04-13T22:30:31.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk=', 'Content-Length': '0'} 2024-04-13T22:30:31.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7da0f6f9af3ce9-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.634 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.634 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.635 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.635 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.635 DEBUG:boto:Method: GET 2024-04-13T22:30:31.635 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.635 DEBUG:boto:Data: 2024-04-13T22:30:31.635 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.635 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.635 DEBUG:boto:Params: {} 2024-04-13T22:30:31.635 DEBUG:boto:Token: None 2024-04-13T22:30:31.635 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk= 2024-04-13T22:30:31.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk=', 'Content-Length': '0'} 2024-04-13T22:30:31.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045faeb6120fbfb6a-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.638 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.638 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.638 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.638 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.638 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.638 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.638 DEBUG:boto:Data: 2024-04-13T22:30:31.638 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.639 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.639 DEBUG:boto:Params: {} 2024-04-13T22:30:31.639 DEBUG:boto:Token: None 2024-04-13T22:30:31.639 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TXuQ7fp7UkVAkteciP1LCK82e6I= 2024-04-13T22:30:31.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TXuQ7fp7UkVAkteciP1LCK82e6I=', 'Content-Length': '0'} 2024-04-13T22:30:31.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a200565501c638-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.641 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.641 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.641 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.641 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.641 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.641 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.641 DEBUG:boto:Data: 2024-04-13T22:30:31.641 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.642 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.642 DEBUG:boto:Params: {} 2024-04-13T22:30:31.642 DEBUG:boto:Token: None 2024-04-13T22:30:31.642 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TXuQ7fp7UkVAkteciP1LCK82e6I= 2024-04-13T22:30:31.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TXuQ7fp7UkVAkteciP1LCK82e6I=', 'Content-Length': '0'} 2024-04-13T22:30:31.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0365f9e22a935ca-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:30:31.644 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.644 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.644 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.644 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.645 DEBUG:boto:Method: GET 2024-04-13T22:30:31.645 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.645 DEBUG:boto:Data: 2024-04-13T22:30:31.645 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.645 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.645 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.645 DEBUG:boto:Params: {} 2024-04-13T22:30:31.645 DEBUG:boto:Token: None 2024-04-13T22:30:31.645 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.645 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk= 2024-04-13T22:30:31.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk=', 'Content-Length': '0'} 2024-04-13T22:30:31.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d20e0e19e96c1fc-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.648 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.648 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.648 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.648 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.648 DEBUG:boto:Method: GET 2024-04-13T22:30:31.648 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.648 DEBUG:boto:Data: 2024-04-13T22:30:31.648 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.649 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.649 DEBUG:boto:Params: {} 2024-04-13T22:30:31.649 DEBUG:boto:Token: None 2024-04-13T22:30:31.649 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:31.649 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk= 2024-04-13T22:30:31.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z20FYkvsSPee2WAYy/nxTzCmsZk=', 'Content-Length': '0'} 2024-04-13T22:30:31.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32553196253dd69-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.651 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.651 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.652 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.652 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.652 DEBUG:boto:Method: GET 2024-04-13T22:30:31.652 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.652 DEBUG:boto:Data: 2024-04-13T22:30:31.652 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.652 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.652 DEBUG:boto:Params: {} 2024-04-13T22:30:31.653 DEBUG:boto:Token: None 2024-04-13T22:30:31.653 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5907061c54517b2-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.655 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.656 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.656 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.656 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.656 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.656 DEBUG:boto:Method: GET 2024-04-13T22:30:31.656 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.656 DEBUG:boto:Data: 2024-04-13T22:30:31.656 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.657 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.657 DEBUG:boto:Params: {} 2024-04-13T22:30:31.657 DEBUG:boto:Token: None 2024-04-13T22:30:31.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033df4847c0887551-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.661 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.661 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.662 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.662 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.662 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:31.662 DEBUG:boto:Data: 2024-04-13T22:30:31.662 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.662 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.662 DEBUG:boto:Params: {} 2024-04-13T22:30:31.662 DEBUG:boto:Token: None 2024-04-13T22:30:31.663 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11 2024-04-13T22:30:31.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R+jb2z/s8vahEgp+Px9B+rVAfZg= 2024-04-13T22:30:31.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R+jb2z/s8vahEgp+Px9B+rVAfZg=', 'Content-Length': '0'} 2024-04-13T22:30:31.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002129402231be9523-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.665 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.665 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.665 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.665 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:31.665 DEBUG:boto:Data: 2024-04-13T22:30:31.665 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.665 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.665 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.665 DEBUG:boto:Params: {} 2024-04-13T22:30:31.666 DEBUG:boto:Token: None 2024-04-13T22:30:31.666 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11 2024-04-13T22:30:31.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R+jb2z/s8vahEgp+Px9B+rVAfZg= 2024-04-13T22:30:31.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R+jb2z/s8vahEgp+Px9B+rVAfZg=', 'Content-Length': '0'} 2024-04-13T22:30:31.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba895c1a35d8fbb0-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:30:31.668 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.668 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.668 DEBUG:boto:Method: GET 2024-04-13T22:30:31.668 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:31.668 DEBUG:boto:Data: 2024-04-13T22:30:31.668 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.668 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.668 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.668 DEBUG:boto:Params: {} 2024-04-13T22:30:31.669 DEBUG:boto:Token: None 2024-04-13T22:30:31.669 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11 2024-04-13T22:30:31.669 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y53xQFGJj73d1kCD1c+DgMvPgsA= 2024-04-13T22:30:31.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y53xQFGJj73d1kCD1c+DgMvPgsA=', 'Content-Length': '0'} 2024-04-13T22:30:31.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052548b970fdff9b1-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.714 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:31.714 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:31.715 DEBUG:boto:Method: GET 2024-04-13T22:30:31.715 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:31.715 DEBUG:boto:Data: 2024-04-13T22:30:31.715 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.715 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.715 DEBUG:boto:Params: {} 2024-04-13T22:30:31.715 DEBUG:boto:Token: None 2024-04-13T22:30:31.715 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj11 2024-04-13T22:30:31.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y53xQFGJj73d1kCD1c+DgMvPgsA= 2024-04-13T22:30:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y53xQFGJj73d1kCD1c+DgMvPgsA=', 'Content-Length': '0'} 2024-04-13T22:30:31.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e496e4984a9b988b-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.718 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.718 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.718 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.718 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.718 DEBUG:boto:Method: GET 2024-04-13T22:30:31.718 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.718 DEBUG:boto:Data: 2024-04-13T22:30:31.718 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.719 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.719 DEBUG:boto:Params: {} 2024-04-13T22:30:31.719 DEBUG:boto:Token: None 2024-04-13T22:30:31.719 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054e0c9ca903e51b1-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.722 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.722 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.722 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.722 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.723 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.723 DEBUG:boto:Method: GET 2024-04-13T22:30:31.723 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:31.723 DEBUG:boto:Data: 2024-04-13T22:30:31.723 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.723 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.723 DEBUG:boto:Params: {} 2024-04-13T22:30:31.723 DEBUG:boto:Token: None 2024-04-13T22:30:31.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1523b45561d7c42-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.728 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-13T22:30:31.728 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.729 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.729 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.729 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.729 DEBUG:boto:Method: GET 2024-04-13T22:30:31.729 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.729 DEBUG:boto:Data: 2024-04-13T22:30:31.729 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.729 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.729 DEBUG:boto:Params: {} 2024-04-13T22:30:31.730 DEBUG:boto:Token: None 2024-04-13T22:30:31.730 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.730 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA= 2024-04-13T22:30:31.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA=', 'Content-Length': '0'} 2024-04-13T22:30:31.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001192e843415201b0-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.732 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.732 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.732 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.732 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.732 DEBUG:boto:Method: GET 2024-04-13T22:30:31.733 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.733 DEBUG:boto:Data: 2024-04-13T22:30:31.733 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.733 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.733 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.733 DEBUG:boto:Params: {} 2024-04-13T22:30:31.733 DEBUG:boto:Token: None 2024-04-13T22:30:31.733 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA= 2024-04-13T22:30:31.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA=', 'Content-Length': '0'} 2024-04-13T22:30:31.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f1cfdf6d48dcf1-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.736 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.736 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.736 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.736 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.736 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.736 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.736 DEBUG:boto:Data: 2024-04-13T22:30:31.737 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.737 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.737 DEBUG:boto:Params: {} 2024-04-13T22:30:31.737 DEBUG:boto:Token: None 2024-04-13T22:30:31.737 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uPYzuFe4bVsNzaOq1gYZozQ5fPc= 2024-04-13T22:30:31.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uPYzuFe4bVsNzaOq1gYZozQ5fPc=', 'Content-Length': '0'} 2024-04-13T22:30:31.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a25395cfb1d77f0-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.741 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.741 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.741 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.741 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.741 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.742 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.742 DEBUG:boto:Data: 2024-04-13T22:30:31.742 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.742 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.742 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.742 DEBUG:boto:Params: {} 2024-04-13T22:30:31.742 DEBUG:boto:Token: None 2024-04-13T22:30:31.742 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uPYzuFe4bVsNzaOq1gYZozQ5fPc= 2024-04-13T22:30:31.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uPYzuFe4bVsNzaOq1gYZozQ5fPc=', 'Content-Length': '0'} 2024-04-13T22:30:31.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b09d614c964b07f6-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-13T22:30:31.745 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.745 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.745 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.745 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.745 DEBUG:boto:Method: GET 2024-04-13T22:30:31.745 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.745 DEBUG:boto:Data: 2024-04-13T22:30:31.745 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.746 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.746 DEBUG:boto:Params: {} 2024-04-13T22:30:31.746 DEBUG:boto:Token: None 2024-04-13T22:30:31.746 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.746 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA= 2024-04-13T22:30:31.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA=', 'Content-Length': '0'} 2024-04-13T22:30:31.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000723f9e77947fc64-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.748 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.748 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.748 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.748 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.748 DEBUG:boto:Method: GET 2024-04-13T22:30:31.749 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.749 DEBUG:boto:Data: 2024-04-13T22:30:31.749 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.749 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.749 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.749 DEBUG:boto:Params: {} 2024-04-13T22:30:31.749 DEBUG:boto:Token: None 2024-04-13T22:30:31.749 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:31.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA= 2024-04-13T22:30:31.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U1PpKCB/q+nGPuko5WpVV+aioEA=', 'Content-Length': '0'} 2024-04-13T22:30:31.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a267b0400135a19-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.751 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.752 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.752 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.752 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.752 DEBUG:boto:Method: GET 2024-04-13T22:30:31.752 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.752 DEBUG:boto:Data: 2024-04-13T22:30:31.752 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.752 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.752 DEBUG:boto:Params: {} 2024-04-13T22:30:31.753 DEBUG:boto:Token: None 2024-04-13T22:30:31.753 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.753 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b93b6619f06d9cb4-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.756 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.756 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.756 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.756 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.757 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.757 DEBUG:boto:Method: GET 2024-04-13T22:30:31.757 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.757 DEBUG:boto:Data: 2024-04-13T22:30:31.757 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.757 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.757 DEBUG:boto:Params: {} 2024-04-13T22:30:31.757 DEBUG:boto:Token: None 2024-04-13T22:30:31.757 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031f4c28030486f89-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.762 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.762 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.762 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.763 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.763 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:31.763 DEBUG:boto:Data: 2024-04-13T22:30:31.763 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.763 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.763 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.763 DEBUG:boto:Params: {} 2024-04-13T22:30:31.763 DEBUG:boto:Token: None 2024-04-13T22:30:31.763 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111 2024-04-13T22:30:31.764 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kxdJ6SVydUe7/NrjCRuOlGVnLkw= 2024-04-13T22:30:31.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kxdJ6SVydUe7/NrjCRuOlGVnLkw=', 'Content-Length': '0'} 2024-04-13T22:30:31.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb9d23de25047f4-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.766 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.766 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.766 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.766 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:31.766 DEBUG:boto:Data: 2024-04-13T22:30:31.766 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.766 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.766 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.767 DEBUG:boto:Params: {} 2024-04-13T22:30:31.767 DEBUG:boto:Token: None 2024-04-13T22:30:31.767 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111 2024-04-13T22:30:31.767 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kxdJ6SVydUe7/NrjCRuOlGVnLkw= 2024-04-13T22:30:31.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kxdJ6SVydUe7/NrjCRuOlGVnLkw=', 'Content-Length': '0'} 2024-04-13T22:30:31.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000852a8ad7435a7b48-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-13T22:30:31.769 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.769 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.769 DEBUG:boto:Method: GET 2024-04-13T22:30:31.769 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:31.769 DEBUG:boto:Data: 2024-04-13T22:30:31.769 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.769 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.770 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.770 DEBUG:boto:Params: {} 2024-04-13T22:30:31.770 DEBUG:boto:Token: None 2024-04-13T22:30:31.770 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111 2024-04-13T22:30:31.770 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BUaRrsiR/NlkuIzlVpipqiA6MxE= 2024-04-13T22:30:31.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BUaRrsiR/NlkuIzlVpipqiA6MxE=', 'Content-Length': '0'} 2024-04-13T22:30:31.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029240189d1327bbd-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.818 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:31.818 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:31.819 DEBUG:boto:Method: GET 2024-04-13T22:30:31.819 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:31.819 DEBUG:boto:Data: 2024-04-13T22:30:31.819 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.819 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.819 DEBUG:boto:Params: {} 2024-04-13T22:30:31.819 DEBUG:boto:Token: None 2024-04-13T22:30:31.819 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj111 2024-04-13T22:30:31.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BUaRrsiR/NlkuIzlVpipqiA6MxE= 2024-04-13T22:30:31.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BUaRrsiR/NlkuIzlVpipqiA6MxE=', 'Content-Length': '0'} 2024-04-13T22:30:31.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b73c6ec6e444cbc-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.822 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.822 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.822 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.822 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.822 DEBUG:boto:Method: GET 2024-04-13T22:30:31.822 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.822 DEBUG:boto:Data: 2024-04-13T22:30:31.822 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.823 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.823 DEBUG:boto:Params: {} 2024-04-13T22:30:31.823 DEBUG:boto:Token: None 2024-04-13T22:30:31.823 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007043c264a01159f2-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.825 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.826 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.826 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.826 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.826 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.826 DEBUG:boto:Method: GET 2024-04-13T22:30:31.826 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:31.826 DEBUG:boto:Data: 2024-04-13T22:30:31.826 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.827 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.827 DEBUG:boto:Params: {} 2024-04-13T22:30:31.827 DEBUG:boto:Token: None 2024-04-13T22:30:31.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d701f35c262bbfd-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.831 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:30:31.831 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.831 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.832 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.832 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.832 DEBUG:boto:Method: GET 2024-04-13T22:30:31.832 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.832 DEBUG:boto:Data: 2024-04-13T22:30:31.832 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.832 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.832 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.832 DEBUG:boto:Params: {} 2024-04-13T22:30:31.832 DEBUG:boto:Token: None 2024-04-13T22:30:31.833 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.833 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU= 2024-04-13T22:30:31.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU=', 'Content-Length': '0'} 2024-04-13T22:30:31.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061c731ec3221dec1-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.835 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.835 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.835 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.835 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.835 DEBUG:boto:Method: GET 2024-04-13T22:30:31.835 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.835 DEBUG:boto:Data: 2024-04-13T22:30:31.835 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.836 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.836 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.836 DEBUG:boto:Params: {} 2024-04-13T22:30:31.836 DEBUG:boto:Token: None 2024-04-13T22:30:31.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU= 2024-04-13T22:30:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU=', 'Content-Length': '0'} 2024-04-13T22:30:31.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41b5c1f10104a7c-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.838 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.838 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.839 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.839 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.839 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.839 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.839 DEBUG:boto:Data: 2024-04-13T22:30:31.839 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.839 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.839 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.839 DEBUG:boto:Params: {} 2024-04-13T22:30:31.839 DEBUG:boto:Token: None 2024-04-13T22:30:31.839 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:84+iixge8ilzYLKSx57er7jNE9k= 2024-04-13T22:30:31.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:84+iixge8ilzYLKSx57er7jNE9k=', 'Content-Length': '0'} 2024-04-13T22:30:31.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099dbd6897856c99b-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.841 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.841 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.842 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.842 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.842 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.842 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.842 DEBUG:boto:Data: 2024-04-13T22:30:31.842 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.842 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.842 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.842 DEBUG:boto:Params: {} 2024-04-13T22:30:31.843 DEBUG:boto:Token: None 2024-04-13T22:30:31.843 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.843 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:84+iixge8ilzYLKSx57er7jNE9k= 2024-04-13T22:30:31.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:84+iixge8ilzYLKSx57er7jNE9k=', 'Content-Length': '0'} 2024-04-13T22:30:31.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f011ee5e922d0b33-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:30:31.845 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.845 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.845 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.845 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.845 DEBUG:boto:Method: GET 2024-04-13T22:30:31.845 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.845 DEBUG:boto:Data: 2024-04-13T22:30:31.846 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.846 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.846 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.846 DEBUG:boto:Params: {} 2024-04-13T22:30:31.846 DEBUG:boto:Token: None 2024-04-13T22:30:31.846 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.846 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU= 2024-04-13T22:30:31.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU=', 'Content-Length': '0'} 2024-04-13T22:30:31.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b793f9dfcc1d7b-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.848 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.848 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.849 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.849 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.849 DEBUG:boto:Method: GET 2024-04-13T22:30:31.849 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.849 DEBUG:boto:Data: 2024-04-13T22:30:31.849 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.849 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.849 DEBUG:boto:Params: {} 2024-04-13T22:30:31.849 DEBUG:boto:Token: None 2024-04-13T22:30:31.850 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:31.850 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU= 2024-04-13T22:30:31.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mzEV8TjL7kofGpVUV5STq+qoyBU=', 'Content-Length': '0'} 2024-04-13T22:30:31.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d98261da97e27be-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.852 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.852 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.852 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.852 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.852 DEBUG:boto:Method: GET 2024-04-13T22:30:31.852 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.852 DEBUG:boto:Data: 2024-04-13T22:30:31.853 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.853 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.853 DEBUG:boto:Params: {} 2024-04-13T22:30:31.853 DEBUG:boto:Token: None 2024-04-13T22:30:31.853 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.853 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca76b5534538f52-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.855 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.856 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.856 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.856 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.856 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.856 DEBUG:boto:Method: GET 2024-04-13T22:30:31.856 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.856 DEBUG:boto:Data: 2024-04-13T22:30:31.856 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.856 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.857 DEBUG:boto:Params: {} 2024-04-13T22:30:31.857 DEBUG:boto:Token: None 2024-04-13T22:30:31.857 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9c6b692e2f71aa-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.861 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.861 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.861 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.861 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.862 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:31.862 DEBUG:boto:Data: 2024-04-13T22:30:31.862 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.862 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.862 DEBUG:boto:Params: {} 2024-04-13T22:30:31.862 DEBUG:boto:Token: None 2024-04-13T22:30:31.862 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115 2024-04-13T22:30:31.862 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HHwxF2uPwBCpLhHOv2E2F0/kXHg= 2024-04-13T22:30:31.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HHwxF2uPwBCpLhHOv2E2F0/kXHg=', 'Content-Length': '0'} 2024-04-13T22:30:31.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007322a89c044f146-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.864 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.864 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.864 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.865 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:31.865 DEBUG:boto:Data: 2024-04-13T22:30:31.865 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.865 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.865 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.865 DEBUG:boto:Params: {} 2024-04-13T22:30:31.865 DEBUG:boto:Token: None 2024-04-13T22:30:31.865 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115 2024-04-13T22:30:31.865 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HHwxF2uPwBCpLhHOv2E2F0/kXHg= 2024-04-13T22:30:31.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HHwxF2uPwBCpLhHOv2E2F0/kXHg=', 'Content-Length': '0'} 2024-04-13T22:30:31.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d622d130371fb5a8-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:30:31.868 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.868 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.868 DEBUG:boto:Method: GET 2024-04-13T22:30:31.868 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:31.868 DEBUG:boto:Data: 2024-04-13T22:30:31.868 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.868 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.868 DEBUG:boto:Params: {} 2024-04-13T22:30:31.868 DEBUG:boto:Token: None 2024-04-13T22:30:31.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115 2024-04-13T22:30:31.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wy+PR1coo5JhWeeAnPO1I1psGd0= 2024-04-13T22:30:31.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wy+PR1coo5JhWeeAnPO1I1psGd0=', 'Content-Length': '0'} 2024-04-13T22:30:31.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3b88b0802b367f-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.910 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:31.910 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:31.911 DEBUG:boto:Method: GET 2024-04-13T22:30:31.911 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:31.911 DEBUG:boto:Data: 2024-04-13T22:30:31.911 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.911 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.911 DEBUG:boto:Params: {} 2024-04-13T22:30:31.911 DEBUG:boto:Token: None 2024-04-13T22:30:31.912 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj115 2024-04-13T22:30:31.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wy+PR1coo5JhWeeAnPO1I1psGd0= 2024-04-13T22:30:31.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wy+PR1coo5JhWeeAnPO1I1psGd0=', 'Content-Length': '0'} 2024-04-13T22:30:31.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006217c816864506dd-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.914 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.914 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.914 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.914 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.914 DEBUG:boto:Method: GET 2024-04-13T22:30:31.914 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.914 DEBUG:boto:Data: 2024-04-13T22:30:31.915 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.915 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.915 DEBUG:boto:Params: {} 2024-04-13T22:30:31.915 DEBUG:boto:Token: None 2024-04-13T22:30:31.915 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.915 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1e66f73b7a70524-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.917 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.918 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.918 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.918 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.918 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.918 DEBUG:boto:Method: GET 2024-04-13T22:30:31.918 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:31.918 DEBUG:boto:Data: 2024-04-13T22:30:31.918 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.918 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.918 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.919 DEBUG:boto:Params: {} 2024-04-13T22:30:31.919 DEBUG:boto:Token: None 2024-04-13T22:30:31.919 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be4a704498fa9353-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.923 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:30:31.924 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.924 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.924 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.924 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.924 DEBUG:boto:Method: GET 2024-04-13T22:30:31.924 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.924 DEBUG:boto:Data: 2024-04-13T22:30:31.924 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.924 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.924 DEBUG:boto:Params: {} 2024-04-13T22:30:31.924 DEBUG:boto:Token: None 2024-04-13T22:30:31.925 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI= 2024-04-13T22:30:31.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI=', 'Content-Length': '0'} 2024-04-13T22:30:31.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930d9ba978991de0-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.927 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.927 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.927 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.927 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.927 DEBUG:boto:Method: GET 2024-04-13T22:30:31.927 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.927 DEBUG:boto:Data: 2024-04-13T22:30:31.928 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.928 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.928 DEBUG:boto:Params: {} 2024-04-13T22:30:31.928 DEBUG:boto:Token: None 2024-04-13T22:30:31.928 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI= 2024-04-13T22:30:31.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI=', 'Content-Length': '0'} 2024-04-13T22:30:31.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1b4143be486f26-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.970 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.970 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.971 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.971 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.971 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.971 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.971 DEBUG:boto:Data: 2024-04-13T22:30:31.971 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.971 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.971 DEBUG:boto:Params: {} 2024-04-13T22:30:31.971 DEBUG:boto:Token: None 2024-04-13T22:30:31.971 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/eL7hprcbpbwQsNPt/vMQphwYaQ= 2024-04-13T22:30:31.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/eL7hprcbpbwQsNPt/vMQphwYaQ=', 'Content-Length': '0'} 2024-04-13T22:30:31.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d344e94d55518219-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.973 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.974 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.974 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.974 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.974 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.974 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.974 DEBUG:boto:Data: 2024-04-13T22:30:31.974 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.974 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.974 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.974 DEBUG:boto:Params: {} 2024-04-13T22:30:31.975 DEBUG:boto:Token: None 2024-04-13T22:30:31.975 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.975 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/eL7hprcbpbwQsNPt/vMQphwYaQ= 2024-04-13T22:30:31.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/eL7hprcbpbwQsNPt/vMQphwYaQ=', 'Content-Length': '0'} 2024-04-13T22:30:31.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd41b33b53efb6b-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:30:31.977 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.977 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.977 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.977 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.977 DEBUG:boto:Method: GET 2024-04-13T22:30:31.977 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.978 DEBUG:boto:Data: 2024-04-13T22:30:31.978 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.978 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.978 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.978 DEBUG:boto:Params: {} 2024-04-13T22:30:31.978 DEBUG:boto:Token: None 2024-04-13T22:30:31.978 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.978 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI= 2024-04-13T22:30:31.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI=', 'Content-Length': '0'} 2024-04-13T22:30:31.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be8cd57d26013c6-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.980 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.980 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.981 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.981 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.981 DEBUG:boto:Method: GET 2024-04-13T22:30:31.981 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.981 DEBUG:boto:Data: 2024-04-13T22:30:31.981 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.981 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.981 DEBUG:boto:Params: {} 2024-04-13T22:30:31.981 DEBUG:boto:Token: None 2024-04-13T22:30:31.981 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:31.982 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI= 2024-04-13T22:30:31.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mgJ0oDVrlPUWQvlDKs6cslD+qCI=', 'Content-Length': '0'} 2024-04-13T22:30:31.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4b567e8f0ac0e8-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.983 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.984 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.984 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:31.984 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:31.984 DEBUG:boto:Method: GET 2024-04-13T22:30:31.984 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:31.984 DEBUG:boto:Data: 2024-04-13T22:30:31.984 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.984 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.984 DEBUG:boto:Params: {} 2024-04-13T22:30:31.985 DEBUG:boto:Token: None 2024-04-13T22:30:31.985 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000685cc23ba2364afd-00661b0787-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.987 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.988 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:31.988 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:31.988 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:31.988 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:31.988 DEBUG:boto:Method: GET 2024-04-13T22:30:31.988 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:31.988 DEBUG:boto:Data: 2024-04-13T22:30:31.988 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.989 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.989 DEBUG:boto:Params: {} 2024-04-13T22:30:31.989 DEBUG:boto:Token: None 2024-04-13T22:30:31.989 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/ 2024-04-13T22:30:31.989 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU= 2024-04-13T22:30:31.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ysJE3UF3lEOkOptRXisQ2LlSlHU=', 'Content-Length': '0'} 2024-04-13T22:30:31.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003769c5cba4cb4d1b-00661b0787-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.993 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:31.993 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.993 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.993 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.994 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:31.994 DEBUG:boto:Data: 2024-04-13T22:30:31.994 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:31.994 DEBUG:boto:Port: 8001 2024-04-13T22:30:31.994 DEBUG:boto:Params: {} 2024-04-13T22:30:31.994 DEBUG:boto:Token: None 2024-04-13T22:30:31.994 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116 2024-04-13T22:30:31.994 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ROp59HfzLhZmkBXYDDkiWzr1bVc= 2024-04-13T22:30:31.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ROp59HfzLhZmkBXYDDkiWzr1bVc=', 'Content-Length': '0'} 2024-04-13T22:30:31.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db56a2453c17ea10-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.996 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.996 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.996 DEBUG:boto:Method: HEAD 2024-04-13T22:30:31.997 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:31.997 DEBUG:boto:Data: 2024-04-13T22:30:31.997 DEBUG:boto:Headers: {} 2024-04-13T22:30:31.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:31.997 DEBUG:boto:Port: 8000 2024-04-13T22:30:31.997 DEBUG:boto:Params: {} 2024-04-13T22:30:31.997 DEBUG:boto:Token: None 2024-04-13T22:30:31.997 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:31 GMT /vwnfns-61/obj116 2024-04-13T22:30:31.997 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ROp59HfzLhZmkBXYDDkiWzr1bVc= 2024-04-13T22:30:31.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ROp59HfzLhZmkBXYDDkiWzr1bVc=', 'Content-Length': '0'} 2024-04-13T22:30:31.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de323b4feefaa47-00661b0787-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:31.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:30:31.999 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:31.999 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:31.999 DEBUG:boto:Method: GET 2024-04-13T22:30:31.999 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:31.999 DEBUG:boto:Data: 2024-04-13T22:30:32.000 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.000 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.000 DEBUG:boto:Params: {} 2024-04-13T22:30:32.000 DEBUG:boto:Token: None 2024-04-13T22:30:32.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj116 2024-04-13T22:30:32.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LbjJobrKqH5grADyfh57TJgytRI= 2024-04-13T22:30:32.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LbjJobrKqH5grADyfh57TJgytRI=', 'Content-Length': '0'} 2024-04-13T22:30:32.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a100a5705edad4b5-00661b0787-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.002 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:32.002 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:32.002 DEBUG:boto:Method: GET 2024-04-13T22:30:32.002 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:32.003 DEBUG:boto:Data: 2024-04-13T22:30:32.003 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.003 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.003 DEBUG:boto:Params: {} 2024-04-13T22:30:32.003 DEBUG:boto:Token: None 2024-04-13T22:30:32.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj116 2024-04-13T22:30:32.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LbjJobrKqH5grADyfh57TJgytRI= 2024-04-13T22:30:32.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LbjJobrKqH5grADyfh57TJgytRI=', 'Content-Length': '0'} 2024-04-13T22:30:32.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e78bfb3110046ece-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.050 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.051 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.051 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:32.051 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:32.051 DEBUG:boto:Method: GET 2024-04-13T22:30:32.051 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:32.051 DEBUG:boto:Data: 2024-04-13T22:30:32.051 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.051 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.051 DEBUG:boto:Params: {} 2024-04-13T22:30:32.052 DEBUG:boto:Token: None 2024-04-13T22:30:32.052 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000910d7dadd538bea0-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.054 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.055 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.055 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.055 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:32.055 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:32.055 DEBUG:boto:Method: GET 2024-04-13T22:30:32.055 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:32.055 DEBUG:boto:Data: 2024-04-13T22:30:32.055 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.055 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.055 DEBUG:boto:Params: {} 2024-04-13T22:30:32.056 DEBUG:boto:Token: None 2024-04-13T22:30:32.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f42f8ed6ebf3785-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.060 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:30:32.060 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.060 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.060 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.060 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.060 DEBUG:boto:Method: GET 2024-04-13T22:30:32.061 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.061 DEBUG:boto:Data: 2024-04-13T22:30:32.061 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.061 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.061 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.061 DEBUG:boto:Params: {} 2024-04-13T22:30:32.061 DEBUG:boto:Token: None 2024-04-13T22:30:32.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0= 2024-04-13T22:30:32.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0=', 'Content-Length': '0'} 2024-04-13T22:30:32.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227a79fba3d39bdc-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.106 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.106 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.106 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.107 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.107 DEBUG:boto:Method: GET 2024-04-13T22:30:32.107 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.107 DEBUG:boto:Data: 2024-04-13T22:30:32.107 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.107 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.107 DEBUG:boto:Params: {} 2024-04-13T22:30:32.107 DEBUG:boto:Token: None 2024-04-13T22:30:32.107 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0= 2024-04-13T22:30:32.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0=', 'Content-Length': '0'} 2024-04-13T22:30:32.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3176ee462881281-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.109 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.110 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.110 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.110 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.110 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.110 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.110 DEBUG:boto:Data: 2024-04-13T22:30:32.110 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.110 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.110 DEBUG:boto:Params: {} 2024-04-13T22:30:32.110 DEBUG:boto:Token: None 2024-04-13T22:30:32.111 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8WJv8jPk5MqlFSYsKJK9/XDy5co= 2024-04-13T22:30:32.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8WJv8jPk5MqlFSYsKJK9/XDy5co=', 'Content-Length': '0'} 2024-04-13T22:30:32.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016748b2220e2f682-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.113 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.113 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.113 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.113 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.113 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.113 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.114 DEBUG:boto:Data: 2024-04-13T22:30:32.114 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.114 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.114 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.114 DEBUG:boto:Params: {} 2024-04-13T22:30:32.114 DEBUG:boto:Token: None 2024-04-13T22:30:32.114 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8WJv8jPk5MqlFSYsKJK9/XDy5co= 2024-04-13T22:30:32.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8WJv8jPk5MqlFSYsKJK9/XDy5co=', 'Content-Length': '0'} 2024-04-13T22:30:32.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009649ae2d256fc5b4-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:30:32.117 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.117 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.117 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.117 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.117 DEBUG:boto:Method: GET 2024-04-13T22:30:32.117 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.117 DEBUG:boto:Data: 2024-04-13T22:30:32.117 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.117 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.118 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.118 DEBUG:boto:Params: {} 2024-04-13T22:30:32.118 DEBUG:boto:Token: None 2024-04-13T22:30:32.118 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.118 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0= 2024-04-13T22:30:32.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0=', 'Content-Length': '0'} 2024-04-13T22:30:32.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ee66b0c9795e15-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.120 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.120 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.120 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.120 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.121 DEBUG:boto:Method: GET 2024-04-13T22:30:32.121 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.121 DEBUG:boto:Data: 2024-04-13T22:30:32.121 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.121 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.121 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.121 DEBUG:boto:Params: {} 2024-04-13T22:30:32.121 DEBUG:boto:Token: None 2024-04-13T22:30:32.121 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:32.121 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0= 2024-04-13T22:30:32.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5LFN1YPVYVXvhSHOi5mpWrxXJC0=', 'Content-Length': '0'} 2024-04-13T22:30:32.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4d42f7a19ab3ce-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.123 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.123 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.123 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.124 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.124 DEBUG:boto:Method: GET 2024-04-13T22:30:32.124 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.124 DEBUG:boto:Data: 2024-04-13T22:30:32.124 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.124 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.124 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.124 DEBUG:boto:Params: {} 2024-04-13T22:30:32.124 DEBUG:boto:Token: None 2024-04-13T22:30:32.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681d5b559e3ada40-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.127 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.127 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.127 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.128 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.128 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.128 DEBUG:boto:Method: GET 2024-04-13T22:30:32.128 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.128 DEBUG:boto:Data: 2024-04-13T22:30:32.128 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.128 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.128 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.128 DEBUG:boto:Params: {} 2024-04-13T22:30:32.128 DEBUG:boto:Token: None 2024-04-13T22:30:32.128 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23e244bb28cda86-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.132 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.133 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.133 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.133 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.133 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:32.133 DEBUG:boto:Data: 2024-04-13T22:30:32.133 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.133 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.133 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.133 DEBUG:boto:Params: {} 2024-04-13T22:30:32.133 DEBUG:boto:Token: None 2024-04-13T22:30:32.133 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124 2024-04-13T22:30:32.134 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bNeEgN0UmgAKiG+RFoaGhjRU++M= 2024-04-13T22:30:32.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bNeEgN0UmgAKiG+RFoaGhjRU++M=', 'Content-Length': '0'} 2024-04-13T22:30:32.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8e7fe82aa66bbb-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.135 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.135 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.136 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.136 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:32.136 DEBUG:boto:Data: 2024-04-13T22:30:32.136 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.136 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.136 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.136 DEBUG:boto:Params: {} 2024-04-13T22:30:32.136 DEBUG:boto:Token: None 2024-04-13T22:30:32.136 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124 2024-04-13T22:30:32.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bNeEgN0UmgAKiG+RFoaGhjRU++M= 2024-04-13T22:30:32.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bNeEgN0UmgAKiG+RFoaGhjRU++M=', 'Content-Length': '0'} 2024-04-13T22:30:32.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c14a4e9ae9ba28-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:30:32.138 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.138 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.139 DEBUG:boto:Method: GET 2024-04-13T22:30:32.139 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:32.139 DEBUG:boto:Data: 2024-04-13T22:30:32.139 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.139 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.139 DEBUG:boto:Params: {} 2024-04-13T22:30:32.139 DEBUG:boto:Token: None 2024-04-13T22:30:32.139 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124 2024-04-13T22:30:32.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MmcEk8pUFH7zkn7J4qWLVtNcGZM= 2024-04-13T22:30:32.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MmcEk8pUFH7zkn7J4qWLVtNcGZM=', 'Content-Length': '0'} 2024-04-13T22:30:32.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca6b30d35d019da-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.142 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:32.142 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:32.142 DEBUG:boto:Method: GET 2024-04-13T22:30:32.142 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:32.142 DEBUG:boto:Data: 2024-04-13T22:30:32.142 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.142 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.143 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.143 DEBUG:boto:Params: {} 2024-04-13T22:30:32.143 DEBUG:boto:Token: None 2024-04-13T22:30:32.143 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj124 2024-04-13T22:30:32.143 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MmcEk8pUFH7zkn7J4qWLVtNcGZM= 2024-04-13T22:30:32.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MmcEk8pUFH7zkn7J4qWLVtNcGZM=', 'Content-Length': '0'} 2024-04-13T22:30:32.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d9bb5cbdd0aa30c-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.186 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.186 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.187 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.187 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.187 DEBUG:boto:Method: GET 2024-04-13T22:30:32.187 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.187 DEBUG:boto:Data: 2024-04-13T22:30:32.187 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.187 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.187 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.187 DEBUG:boto:Params: {} 2024-04-13T22:30:32.187 DEBUG:boto:Token: None 2024-04-13T22:30:32.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a1f8ea1731da4b1-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.190 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.190 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.191 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.191 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.191 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.191 DEBUG:boto:Method: GET 2024-04-13T22:30:32.191 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:32.191 DEBUG:boto:Data: 2024-04-13T22:30:32.191 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.191 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.191 DEBUG:boto:Params: {} 2024-04-13T22:30:32.191 DEBUG:boto:Token: None 2024-04-13T22:30:32.192 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c120970900b89a30-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.195 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:30:32.196 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.196 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.196 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.196 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.196 DEBUG:boto:Method: GET 2024-04-13T22:30:32.196 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.197 DEBUG:boto:Data: 2024-04-13T22:30:32.197 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.197 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.197 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.197 DEBUG:boto:Params: {} 2024-04-13T22:30:32.197 DEBUG:boto:Token: None 2024-04-13T22:30:32.197 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.197 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg= 2024-04-13T22:30:32.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg=', 'Content-Length': '0'} 2024-04-13T22:30:32.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000835e45a1b6e41779-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.199 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.199 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.199 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.199 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.199 DEBUG:boto:Method: GET 2024-04-13T22:30:32.199 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.200 DEBUG:boto:Data: 2024-04-13T22:30:32.200 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.200 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.200 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.200 DEBUG:boto:Params: {} 2024-04-13T22:30:32.200 DEBUG:boto:Token: None 2024-04-13T22:30:32.200 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg= 2024-04-13T22:30:32.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg=', 'Content-Length': '0'} 2024-04-13T22:30:32.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c010a399888a2c5e-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.202 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.203 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.203 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.203 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.203 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.203 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.203 DEBUG:boto:Data: 2024-04-13T22:30:32.203 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.203 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.203 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.203 DEBUG:boto:Params: {} 2024-04-13T22:30:32.204 DEBUG:boto:Token: None 2024-04-13T22:30:32.204 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EK6x7/RfCJd7L/39qHun26z48XA= 2024-04-13T22:30:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EK6x7/RfCJd7L/39qHun26z48XA=', 'Content-Length': '0'} 2024-04-13T22:30:32.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b8e4c2e081e4fae-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.206 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.206 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.206 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.206 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.206 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.206 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.206 DEBUG:boto:Data: 2024-04-13T22:30:32.206 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.206 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.206 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.206 DEBUG:boto:Params: {} 2024-04-13T22:30:32.207 DEBUG:boto:Token: None 2024-04-13T22:30:32.207 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EK6x7/RfCJd7L/39qHun26z48XA= 2024-04-13T22:30:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EK6x7/RfCJd7L/39qHun26z48XA=', 'Content-Length': '0'} 2024-04-13T22:30:32.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ce2cb93c7de823-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:30:32.209 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.209 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.209 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.209 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.209 DEBUG:boto:Method: GET 2024-04-13T22:30:32.209 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.209 DEBUG:boto:Data: 2024-04-13T22:30:32.209 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.209 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.210 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.210 DEBUG:boto:Params: {} 2024-04-13T22:30:32.210 DEBUG:boto:Token: None 2024-04-13T22:30:32.210 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg= 2024-04-13T22:30:32.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg=', 'Content-Length': '0'} 2024-04-13T22:30:32.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2bb5a55696f568-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.254 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.255 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.255 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.255 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.255 DEBUG:boto:Method: GET 2024-04-13T22:30:32.255 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.255 DEBUG:boto:Data: 2024-04-13T22:30:32.255 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.255 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.255 DEBUG:boto:Params: {} 2024-04-13T22:30:32.256 DEBUG:boto:Token: None 2024-04-13T22:30:32.256 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:32.256 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg= 2024-04-13T22:30:32.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NAzwEnv6X0iV9CfP7SxSasuVUsg=', 'Content-Length': '0'} 2024-04-13T22:30:32.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc2c7079c291d10-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.258 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.258 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.258 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.258 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.258 DEBUG:boto:Method: GET 2024-04-13T22:30:32.259 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.259 DEBUG:boto:Data: 2024-04-13T22:30:32.259 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.259 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.259 DEBUG:boto:Params: {} 2024-04-13T22:30:32.259 DEBUG:boto:Token: None 2024-04-13T22:30:32.259 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093a8b2d382484ea3-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.262 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.262 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.262 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.262 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.263 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.263 DEBUG:boto:Method: GET 2024-04-13T22:30:32.263 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.263 DEBUG:boto:Data: 2024-04-13T22:30:32.263 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.263 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.263 DEBUG:boto:Params: {} 2024-04-13T22:30:32.263 DEBUG:boto:Token: None 2024-04-13T22:30:32.263 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602966037b2cb0be-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.267 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.268 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.268 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.268 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.268 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:32.268 DEBUG:boto:Data: 2024-04-13T22:30:32.268 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.268 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.268 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.268 DEBUG:boto:Params: {} 2024-04-13T22:30:32.269 DEBUG:boto:Token: None 2024-04-13T22:30:32.269 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145 2024-04-13T22:30:32.269 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4fhmMGSBWE5FoSv3vT8psV2nmzM= 2024-04-13T22:30:32.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4fhmMGSBWE5FoSv3vT8psV2nmzM=', 'Content-Length': '0'} 2024-04-13T22:30:32.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda050691980aed1-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.271 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.271 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.271 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.271 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:32.271 DEBUG:boto:Data: 2024-04-13T22:30:32.271 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.272 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.272 DEBUG:boto:Params: {} 2024-04-13T22:30:32.272 DEBUG:boto:Token: None 2024-04-13T22:30:32.272 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145 2024-04-13T22:30:32.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4fhmMGSBWE5FoSv3vT8psV2nmzM= 2024-04-13T22:30:32.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4fhmMGSBWE5FoSv3vT8psV2nmzM=', 'Content-Length': '0'} 2024-04-13T22:30:32.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db7723c067bf48bf-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:30:32.274 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.274 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.274 DEBUG:boto:Method: GET 2024-04-13T22:30:32.274 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:32.274 DEBUG:boto:Data: 2024-04-13T22:30:32.274 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.275 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.275 DEBUG:boto:Params: {} 2024-04-13T22:30:32.275 DEBUG:boto:Token: None 2024-04-13T22:30:32.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145 2024-04-13T22:30:32.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H8jG9Zmu8AJpyiH4gMJn2YNQzn4= 2024-04-13T22:30:32.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H8jG9Zmu8AJpyiH4gMJn2YNQzn4=', 'Content-Length': '0'} 2024-04-13T22:30:32.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0b0da2992fdbb3-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.277 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:32.277 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:32.277 DEBUG:boto:Method: GET 2024-04-13T22:30:32.277 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:32.277 DEBUG:boto:Data: 2024-04-13T22:30:32.277 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.277 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.278 DEBUG:boto:Params: {} 2024-04-13T22:30:32.278 DEBUG:boto:Token: None 2024-04-13T22:30:32.278 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj145 2024-04-13T22:30:32.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H8jG9Zmu8AJpyiH4gMJn2YNQzn4= 2024-04-13T22:30:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H8jG9Zmu8AJpyiH4gMJn2YNQzn4=', 'Content-Length': '0'} 2024-04-13T22:30:32.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db06543d8d17ef2-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.280 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.280 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.280 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.280 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.280 DEBUG:boto:Method: GET 2024-04-13T22:30:32.280 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.280 DEBUG:boto:Data: 2024-04-13T22:30:32.280 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.280 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.281 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.281 DEBUG:boto:Params: {} 2024-04-13T22:30:32.281 DEBUG:boto:Token: None 2024-04-13T22:30:32.281 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a36cfbc60259ce0e-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.283 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.283 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.283 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.284 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.284 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.284 DEBUG:boto:Method: GET 2024-04-13T22:30:32.284 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:32.284 DEBUG:boto:Data: 2024-04-13T22:30:32.284 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.284 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.284 DEBUG:boto:Params: {} 2024-04-13T22:30:32.284 DEBUG:boto:Token: None 2024-04-13T22:30:32.284 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7bc2a05a60d0f9-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.288 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-13T22:30:32.289 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.289 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.289 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.289 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.289 DEBUG:boto:Method: GET 2024-04-13T22:30:32.289 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.289 DEBUG:boto:Data: 2024-04-13T22:30:32.289 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.290 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.290 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.290 DEBUG:boto:Params: {} 2024-04-13T22:30:32.290 DEBUG:boto:Token: None 2024-04-13T22:30:32.290 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.290 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ= 2024-04-13T22:30:32.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ=', 'Content-Length': '0'} 2024-04-13T22:30:32.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eccd97d5a82ca14-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.334 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.334 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.334 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.335 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.335 DEBUG:boto:Method: GET 2024-04-13T22:30:32.335 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.335 DEBUG:boto:Data: 2024-04-13T22:30:32.335 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.335 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.335 DEBUG:boto:Params: {} 2024-04-13T22:30:32.335 DEBUG:boto:Token: None 2024-04-13T22:30:32.335 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ= 2024-04-13T22:30:32.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ=', 'Content-Length': '0'} 2024-04-13T22:30:32.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035046e8197d4ef9c-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.338 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.338 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.338 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.338 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.338 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.338 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.338 DEBUG:boto:Data: 2024-04-13T22:30:32.338 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.339 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.339 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.339 DEBUG:boto:Params: {} 2024-04-13T22:30:32.339 DEBUG:boto:Token: None 2024-04-13T22:30:32.339 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G/pMWQyUCDOkGIZdgp/gUx3ObZg= 2024-04-13T22:30:32.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G/pMWQyUCDOkGIZdgp/gUx3ObZg=', 'Content-Length': '0'} 2024-04-13T22:30:32.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c833fade788de61-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.341 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.341 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.341 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.341 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.341 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.341 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.341 DEBUG:boto:Data: 2024-04-13T22:30:32.341 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.342 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.342 DEBUG:boto:Params: {} 2024-04-13T22:30:32.342 DEBUG:boto:Token: None 2024-04-13T22:30:32.342 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:G/pMWQyUCDOkGIZdgp/gUx3ObZg= 2024-04-13T22:30:32.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G/pMWQyUCDOkGIZdgp/gUx3ObZg=', 'Content-Length': '0'} 2024-04-13T22:30:32.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000404f5633300dd63-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-13T22:30:32.344 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.344 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.344 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.344 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.344 DEBUG:boto:Method: GET 2024-04-13T22:30:32.344 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.344 DEBUG:boto:Data: 2024-04-13T22:30:32.345 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.345 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.345 DEBUG:boto:Params: {} 2024-04-13T22:30:32.345 DEBUG:boto:Token: None 2024-04-13T22:30:32.345 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ= 2024-04-13T22:30:32.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ=', 'Content-Length': '0'} 2024-04-13T22:30:32.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd608eb1e6f1c904-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.347 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.347 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.347 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.347 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.347 DEBUG:boto:Method: GET 2024-04-13T22:30:32.347 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.347 DEBUG:boto:Data: 2024-04-13T22:30:32.347 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.347 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.348 DEBUG:boto:Params: {} 2024-04-13T22:30:32.348 DEBUG:boto:Token: None 2024-04-13T22:30:32.348 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:32.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ= 2024-04-13T22:30:32.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2eVZo3F7PpQxNP+euNx2G3MJMKQ=', 'Content-Length': '0'} 2024-04-13T22:30:32.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a809ea4e25c65e19-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.350 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.350 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.350 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.350 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.350 DEBUG:boto:Method: GET 2024-04-13T22:30:32.353 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.353 DEBUG:boto:Data: 2024-04-13T22:30:32.353 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.353 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.353 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.353 DEBUG:boto:Params: {} 2024-04-13T22:30:32.354 DEBUG:boto:Token: None 2024-04-13T22:30:32.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006ad5c44a4558bb-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.356 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.356 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.356 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.357 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.357 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.357 DEBUG:boto:Method: GET 2024-04-13T22:30:32.357 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.357 DEBUG:boto:Data: 2024-04-13T22:30:32.357 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.357 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.357 DEBUG:boto:Params: {} 2024-04-13T22:30:32.357 DEBUG:boto:Token: None 2024-04-13T22:30:32.358 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94adef0ffbd23a0-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.361 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.362 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.362 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.362 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.362 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:32.362 DEBUG:boto:Data: 2024-04-13T22:30:32.362 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.362 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.362 DEBUG:boto:Params: {} 2024-04-13T22:30:32.363 DEBUG:boto:Token: None 2024-04-13T22:30:32.363 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148 2024-04-13T22:30:32.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Uqr4l8rIxTv2qmRkqt2j4B0nGaI= 2024-04-13T22:30:32.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Uqr4l8rIxTv2qmRkqt2j4B0nGaI=', 'Content-Length': '0'} 2024-04-13T22:30:32.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d83ae792258a5622-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.365 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.365 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.365 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.365 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:32.365 DEBUG:boto:Data: 2024-04-13T22:30:32.365 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.365 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.365 DEBUG:boto:Params: {} 2024-04-13T22:30:32.366 DEBUG:boto:Token: None 2024-04-13T22:30:32.366 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148 2024-04-13T22:30:32.366 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Uqr4l8rIxTv2qmRkqt2j4B0nGaI= 2024-04-13T22:30:32.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Uqr4l8rIxTv2qmRkqt2j4B0nGaI=', 'Content-Length': '0'} 2024-04-13T22:30:32.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018599e5ce5f1b882-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-13T22:30:32.367 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.368 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.368 DEBUG:boto:Method: GET 2024-04-13T22:30:32.368 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:32.368 DEBUG:boto:Data: 2024-04-13T22:30:32.368 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.368 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.368 DEBUG:boto:Params: {} 2024-04-13T22:30:32.368 DEBUG:boto:Token: None 2024-04-13T22:30:32.369 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148 2024-04-13T22:30:32.369 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JK9oNteeWkXX3f8RoRpkxyQIeko= 2024-04-13T22:30:32.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JK9oNteeWkXX3f8RoRpkxyQIeko=', 'Content-Length': '0'} 2024-04-13T22:30:32.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f3ed9c4b1d4119-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.371 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:32.371 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:32.371 DEBUG:boto:Method: GET 2024-04-13T22:30:32.371 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:32.371 DEBUG:boto:Data: 2024-04-13T22:30:32.371 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.371 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.371 DEBUG:boto:Params: {} 2024-04-13T22:30:32.371 DEBUG:boto:Token: None 2024-04-13T22:30:32.372 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj148 2024-04-13T22:30:32.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JK9oNteeWkXX3f8RoRpkxyQIeko= 2024-04-13T22:30:32.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JK9oNteeWkXX3f8RoRpkxyQIeko=', 'Content-Length': '0'} 2024-04-13T22:30:32.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f03637562e00c963-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.373 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.373 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.374 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.374 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.374 DEBUG:boto:Method: GET 2024-04-13T22:30:32.374 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.374 DEBUG:boto:Data: 2024-04-13T22:30:32.374 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.374 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.374 DEBUG:boto:Params: {} 2024-04-13T22:30:32.374 DEBUG:boto:Token: None 2024-04-13T22:30:32.374 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3b0a9969806c29-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.422 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.423 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.423 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.423 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.423 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.423 DEBUG:boto:Method: GET 2024-04-13T22:30:32.423 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:32.423 DEBUG:boto:Data: 2024-04-13T22:30:32.423 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.423 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.423 DEBUG:boto:Params: {} 2024-04-13T22:30:32.424 DEBUG:boto:Token: None 2024-04-13T22:30:32.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1fe778b0abe6920-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.428 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:30:32.428 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.428 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.428 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.428 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.428 DEBUG:boto:Method: GET 2024-04-13T22:30:32.428 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.429 DEBUG:boto:Data: 2024-04-13T22:30:32.429 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.429 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.429 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.429 DEBUG:boto:Params: {} 2024-04-13T22:30:32.429 DEBUG:boto:Token: None 2024-04-13T22:30:32.429 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.429 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q= 2024-04-13T22:30:32.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q=', 'Content-Length': '0'} 2024-04-13T22:30:32.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a130cc0a934155-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.432 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.432 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.432 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.432 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.432 DEBUG:boto:Method: GET 2024-04-13T22:30:32.432 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.432 DEBUG:boto:Data: 2024-04-13T22:30:32.432 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.432 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.432 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.432 DEBUG:boto:Params: {} 2024-04-13T22:30:32.433 DEBUG:boto:Token: None 2024-04-13T22:30:32.433 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.433 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q= 2024-04-13T22:30:32.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q=', 'Content-Length': '0'} 2024-04-13T22:30:32.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc777f16a0965028-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.435 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.435 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.435 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.435 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.435 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.435 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.435 DEBUG:boto:Data: 2024-04-13T22:30:32.435 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.435 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.435 DEBUG:boto:Params: {} 2024-04-13T22:30:32.436 DEBUG:boto:Token: None 2024-04-13T22:30:32.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6o04JV0KUij9FQBtNEjHgcbi6Uc= 2024-04-13T22:30:32.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6o04JV0KUij9FQBtNEjHgcbi6Uc=', 'Content-Length': '0'} 2024-04-13T22:30:32.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f52e65f41d992228-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.437 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.438 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.438 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.438 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.438 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.438 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.438 DEBUG:boto:Data: 2024-04-13T22:30:32.438 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.438 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.438 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.438 DEBUG:boto:Params: {} 2024-04-13T22:30:32.439 DEBUG:boto:Token: None 2024-04-13T22:30:32.439 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.439 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6o04JV0KUij9FQBtNEjHgcbi6Uc= 2024-04-13T22:30:32.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6o04JV0KUij9FQBtNEjHgcbi6Uc=', 'Content-Length': '0'} 2024-04-13T22:30:32.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f021f78fcb840813-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:30:32.441 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.441 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.441 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.441 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.441 DEBUG:boto:Method: GET 2024-04-13T22:30:32.441 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.441 DEBUG:boto:Data: 2024-04-13T22:30:32.441 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.441 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.441 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.441 DEBUG:boto:Params: {} 2024-04-13T22:30:32.441 DEBUG:boto:Token: None 2024-04-13T22:30:32.442 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.442 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q= 2024-04-13T22:30:32.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q=', 'Content-Length': '0'} 2024-04-13T22:30:32.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048679df373c893c0-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.444 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.444 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.444 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.444 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.444 DEBUG:boto:Method: GET 2024-04-13T22:30:32.444 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.444 DEBUG:boto:Data: 2024-04-13T22:30:32.444 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.444 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.444 DEBUG:boto:Params: {} 2024-04-13T22:30:32.444 DEBUG:boto:Token: None 2024-04-13T22:30:32.445 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:32.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q= 2024-04-13T22:30:32.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yJuGRIrv87hY94cADDf8VCjDA2Q=', 'Content-Length': '0'} 2024-04-13T22:30:32.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026fa7a9f7127bbe2-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.491 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.491 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.491 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.491 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.491 DEBUG:boto:Method: GET 2024-04-13T22:30:32.491 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.491 DEBUG:boto:Data: 2024-04-13T22:30:32.491 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.491 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.491 DEBUG:boto:Params: {} 2024-04-13T22:30:32.492 DEBUG:boto:Token: None 2024-04-13T22:30:32.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be3219b51da61f3f-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.494 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.494 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.494 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.494 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.495 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.495 DEBUG:boto:Method: GET 2024-04-13T22:30:32.495 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.495 DEBUG:boto:Data: 2024-04-13T22:30:32.495 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.495 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.495 DEBUG:boto:Params: {} 2024-04-13T22:30:32.495 DEBUG:boto:Token: None 2024-04-13T22:30:32.495 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d761bbb545d7ae-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.499 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.499 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.499 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.499 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.500 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:32.500 DEBUG:boto:Data: 2024-04-13T22:30:32.500 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.500 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.500 DEBUG:boto:Params: {} 2024-04-13T22:30:32.500 DEBUG:boto:Token: None 2024-04-13T22:30:32.500 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155 2024-04-13T22:30:32.500 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gL7X0DNVa9ie3kwS3tBGix2Mhjw= 2024-04-13T22:30:32.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gL7X0DNVa9ie3kwS3tBGix2Mhjw=', 'Content-Length': '0'} 2024-04-13T22:30:32.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3f1f1d772c7c3e-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.503 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.503 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.503 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.503 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:32.503 DEBUG:boto:Data: 2024-04-13T22:30:32.503 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.503 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.503 DEBUG:boto:Params: {} 2024-04-13T22:30:32.503 DEBUG:boto:Token: None 2024-04-13T22:30:32.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155 2024-04-13T22:30:32.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gL7X0DNVa9ie3kwS3tBGix2Mhjw= 2024-04-13T22:30:32.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gL7X0DNVa9ie3kwS3tBGix2Mhjw=', 'Content-Length': '0'} 2024-04-13T22:30:32.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008138db9572cec68-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:30:32.505 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.506 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.506 DEBUG:boto:Method: GET 2024-04-13T22:30:32.506 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:32.506 DEBUG:boto:Data: 2024-04-13T22:30:32.506 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.506 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.506 DEBUG:boto:Params: {} 2024-04-13T22:30:32.506 DEBUG:boto:Token: None 2024-04-13T22:30:32.506 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155 2024-04-13T22:30:32.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNm4IDSzJRpY4JFW23uF2T+Zo7Q= 2024-04-13T22:30:32.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNm4IDSzJRpY4JFW23uF2T+Zo7Q=', 'Content-Length': '0'} 2024-04-13T22:30:32.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051cc3098a1e57014-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.550 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:32.550 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:32.551 DEBUG:boto:Method: GET 2024-04-13T22:30:32.551 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:32.551 DEBUG:boto:Data: 2024-04-13T22:30:32.551 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.551 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.551 DEBUG:boto:Params: {} 2024-04-13T22:30:32.551 DEBUG:boto:Token: None 2024-04-13T22:30:32.551 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj155 2024-04-13T22:30:32.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNm4IDSzJRpY4JFW23uF2T+Zo7Q= 2024-04-13T22:30:32.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNm4IDSzJRpY4JFW23uF2T+Zo7Q=', 'Content-Length': '0'} 2024-04-13T22:30:32.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbadfacce7a4c7ab-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.553 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.553 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.553 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.554 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.554 DEBUG:boto:Method: GET 2024-04-13T22:30:32.554 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.554 DEBUG:boto:Data: 2024-04-13T22:30:32.554 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.554 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.554 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.554 DEBUG:boto:Params: {} 2024-04-13T22:30:32.554 DEBUG:boto:Token: None 2024-04-13T22:30:32.554 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aaad8eee265c18f-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.559 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.560 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.560 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.560 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.560 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.560 DEBUG:boto:Method: GET 2024-04-13T22:30:32.560 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:32.560 DEBUG:boto:Data: 2024-04-13T22:30:32.561 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.561 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.561 DEBUG:boto:Params: {} 2024-04-13T22:30:32.561 DEBUG:boto:Token: None 2024-04-13T22:30:32.561 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.561 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005910a959c57b63f8-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.565 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:30:32.566 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.566 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.566 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.566 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.566 DEBUG:boto:Method: GET 2024-04-13T22:30:32.566 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.566 DEBUG:boto:Data: 2024-04-13T22:30:32.566 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.566 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.566 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.566 DEBUG:boto:Params: {} 2024-04-13T22:30:32.567 DEBUG:boto:Token: None 2024-04-13T22:30:32.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA= 2024-04-13T22:30:32.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA=', 'Content-Length': '0'} 2024-04-13T22:30:32.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351bddb81c83f8f5-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.569 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.569 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.569 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.569 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.569 DEBUG:boto:Method: GET 2024-04-13T22:30:32.569 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.569 DEBUG:boto:Data: 2024-04-13T22:30:32.570 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.570 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.570 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.570 DEBUG:boto:Params: {} 2024-04-13T22:30:32.570 DEBUG:boto:Token: None 2024-04-13T22:30:32.570 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.570 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA= 2024-04-13T22:30:32.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA=', 'Content-Length': '0'} 2024-04-13T22:30:32.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec65101c0c77fbc-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.614 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.614 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.615 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.615 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.615 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.615 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.615 DEBUG:boto:Data: 2024-04-13T22:30:32.615 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.615 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.615 DEBUG:boto:Params: {} 2024-04-13T22:30:32.615 DEBUG:boto:Token: None 2024-04-13T22:30:32.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mlMEjS5gmdf/2fO+CJP18wHB7tk= 2024-04-13T22:30:32.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mlMEjS5gmdf/2fO+CJP18wHB7tk=', 'Content-Length': '0'} 2024-04-13T22:30:32.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be8de641b1fcff5-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.617 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.617 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.618 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.618 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.618 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.618 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.618 DEBUG:boto:Data: 2024-04-13T22:30:32.618 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.618 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.618 DEBUG:boto:Params: {} 2024-04-13T22:30:32.618 DEBUG:boto:Token: None 2024-04-13T22:30:32.619 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mlMEjS5gmdf/2fO+CJP18wHB7tk= 2024-04-13T22:30:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mlMEjS5gmdf/2fO+CJP18wHB7tk=', 'Content-Length': '0'} 2024-04-13T22:30:32.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d433764dac6d450d-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:30:32.620 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.621 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.621 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.621 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.621 DEBUG:boto:Method: GET 2024-04-13T22:30:32.621 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.621 DEBUG:boto:Data: 2024-04-13T22:30:32.621 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.621 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.621 DEBUG:boto:Params: {} 2024-04-13T22:30:32.621 DEBUG:boto:Token: None 2024-04-13T22:30:32.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA= 2024-04-13T22:30:32.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA=', 'Content-Length': '0'} 2024-04-13T22:30:32.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928c427c22930db4-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.624 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.624 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.624 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.624 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.624 DEBUG:boto:Method: GET 2024-04-13T22:30:32.624 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.624 DEBUG:boto:Data: 2024-04-13T22:30:32.624 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.625 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.625 DEBUG:boto:Params: {} 2024-04-13T22:30:32.625 DEBUG:boto:Token: None 2024-04-13T22:30:32.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:32.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA= 2024-04-13T22:30:32.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/I3k2uIkAGP1ZV83507DGYXWpA=', 'Content-Length': '0'} 2024-04-13T22:30:32.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a3eec710964419-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.628 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.628 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.628 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.628 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.628 DEBUG:boto:Method: GET 2024-04-13T22:30:32.628 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.628 DEBUG:boto:Data: 2024-04-13T22:30:32.628 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.628 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.628 DEBUG:boto:Params: {} 2024-04-13T22:30:32.629 DEBUG:boto:Token: None 2024-04-13T22:30:32.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a0badde81597a2e-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.674 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.675 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.675 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.675 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.675 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.675 DEBUG:boto:Method: GET 2024-04-13T22:30:32.675 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.675 DEBUG:boto:Data: 2024-04-13T22:30:32.675 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.676 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.676 DEBUG:boto:Params: {} 2024-04-13T22:30:32.676 DEBUG:boto:Token: None 2024-04-13T22:30:32.676 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af89310771c1c3e9-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.680 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.680 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.680 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.680 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.680 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:32.680 DEBUG:boto:Data: 2024-04-13T22:30:32.681 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.681 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.681 DEBUG:boto:Params: {} 2024-04-13T22:30:32.681 DEBUG:boto:Token: None 2024-04-13T22:30:32.681 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17 2024-04-13T22:30:32.681 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnU13O1XQTpbbt16GIEp1zMMWOs= 2024-04-13T22:30:32.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnU13O1XQTpbbt16GIEp1zMMWOs=', 'Content-Length': '0'} 2024-04-13T22:30:32.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f752d1e6df5db199-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.683 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.683 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.683 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.683 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:32.683 DEBUG:boto:Data: 2024-04-13T22:30:32.683 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.683 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.684 DEBUG:boto:Params: {} 2024-04-13T22:30:32.684 DEBUG:boto:Token: None 2024-04-13T22:30:32.684 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17 2024-04-13T22:30:32.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NnU13O1XQTpbbt16GIEp1zMMWOs= 2024-04-13T22:30:32.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnU13O1XQTpbbt16GIEp1zMMWOs=', 'Content-Length': '0'} 2024-04-13T22:30:32.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013728476b8e384ff-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:30:32.686 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.686 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.686 DEBUG:boto:Method: GET 2024-04-13T22:30:32.686 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:32.686 DEBUG:boto:Data: 2024-04-13T22:30:32.686 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.686 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.686 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.686 DEBUG:boto:Params: {} 2024-04-13T22:30:32.686 DEBUG:boto:Token: None 2024-04-13T22:30:32.687 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17 2024-04-13T22:30:32.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nZHvPWFSbrYPN33HfYBTyNE+cVs= 2024-04-13T22:30:32.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nZHvPWFSbrYPN33HfYBTyNE+cVs=', 'Content-Length': '0'} 2024-04-13T22:30:32.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2a8e1d9b066ed87-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.689 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:32.689 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:32.689 DEBUG:boto:Method: GET 2024-04-13T22:30:32.689 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:32.689 DEBUG:boto:Data: 2024-04-13T22:30:32.689 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.689 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.689 DEBUG:boto:Params: {} 2024-04-13T22:30:32.689 DEBUG:boto:Token: None 2024-04-13T22:30:32.690 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj17 2024-04-13T22:30:32.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nZHvPWFSbrYPN33HfYBTyNE+cVs= 2024-04-13T22:30:32.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nZHvPWFSbrYPN33HfYBTyNE+cVs=', 'Content-Length': '0'} 2024-04-13T22:30:32.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000911421893cfd6e9a-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.692 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.692 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.692 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.692 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.692 DEBUG:boto:Method: GET 2024-04-13T22:30:32.692 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.692 DEBUG:boto:Data: 2024-04-13T22:30:32.692 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.693 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.693 DEBUG:boto:Params: {} 2024-04-13T22:30:32.693 DEBUG:boto:Token: None 2024-04-13T22:30:32.693 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.693 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f33ef273864c9fc-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.695 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.696 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.696 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.696 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.696 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.696 DEBUG:boto:Method: GET 2024-04-13T22:30:32.696 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:32.696 DEBUG:boto:Data: 2024-04-13T22:30:32.696 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.696 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.696 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.697 DEBUG:boto:Params: {} 2024-04-13T22:30:32.697 DEBUG:boto:Token: None 2024-04-13T22:30:32.697 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.697 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001500e2310600cc5b-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.700 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-13T22:30:32.701 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.701 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.701 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.701 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.701 DEBUG:boto:Method: GET 2024-04-13T22:30:32.701 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.701 DEBUG:boto:Data: 2024-04-13T22:30:32.701 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.702 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.702 DEBUG:boto:Params: {} 2024-04-13T22:30:32.702 DEBUG:boto:Token: None 2024-04-13T22:30:32.702 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg= 2024-04-13T22:30:32.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg=', 'Content-Length': '0'} 2024-04-13T22:30:32.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890b8b5585a55f54-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.746 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.747 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.747 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.747 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.747 DEBUG:boto:Method: GET 2024-04-13T22:30:32.747 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.747 DEBUG:boto:Data: 2024-04-13T22:30:32.747 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.747 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.747 DEBUG:boto:Params: {} 2024-04-13T22:30:32.747 DEBUG:boto:Token: None 2024-04-13T22:30:32.748 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg= 2024-04-13T22:30:32.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg=', 'Content-Length': '0'} 2024-04-13T22:30:32.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005da4b6e65b35f72-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.750 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.750 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.750 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.750 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.750 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.750 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.750 DEBUG:boto:Data: 2024-04-13T22:30:32.751 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.751 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.751 DEBUG:boto:Params: {} 2024-04-13T22:30:32.751 DEBUG:boto:Token: None 2024-04-13T22:30:32.751 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q+JThiH5lPg++vCkzNkkcavkMGY= 2024-04-13T22:30:32.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q+JThiH5lPg++vCkzNkkcavkMGY=', 'Content-Length': '0'} 2024-04-13T22:30:32.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a555f20bc5b6909a-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.753 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.753 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.753 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.753 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.754 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.754 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.754 DEBUG:boto:Data: 2024-04-13T22:30:32.754 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.754 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.754 DEBUG:boto:Params: {} 2024-04-13T22:30:32.754 DEBUG:boto:Token: None 2024-04-13T22:30:32.754 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.755 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q+JThiH5lPg++vCkzNkkcavkMGY= 2024-04-13T22:30:32.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q+JThiH5lPg++vCkzNkkcavkMGY=', 'Content-Length': '0'} 2024-04-13T22:30:32.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a7f5b80d4a5e04e-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-13T22:30:32.757 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.757 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.757 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.757 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.757 DEBUG:boto:Method: GET 2024-04-13T22:30:32.757 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.757 DEBUG:boto:Data: 2024-04-13T22:30:32.757 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.757 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.757 DEBUG:boto:Params: {} 2024-04-13T22:30:32.758 DEBUG:boto:Token: None 2024-04-13T22:30:32.758 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg= 2024-04-13T22:30:32.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg=', 'Content-Length': '0'} 2024-04-13T22:30:32.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9bde66c7ddc03d-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.760 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.760 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.760 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.760 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.761 DEBUG:boto:Method: GET 2024-04-13T22:30:32.761 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.761 DEBUG:boto:Data: 2024-04-13T22:30:32.761 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.761 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.761 DEBUG:boto:Params: {} 2024-04-13T22:30:32.761 DEBUG:boto:Token: None 2024-04-13T22:30:32.761 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:32.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg= 2024-04-13T22:30:32.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xph6C3UfDFAxr4uWLc6nLUmuIDg=', 'Content-Length': '0'} 2024-04-13T22:30:32.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000513a5623d4f2ac52-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.764 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.764 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.764 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.764 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.764 DEBUG:boto:Method: GET 2024-04-13T22:30:32.764 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.764 DEBUG:boto:Data: 2024-04-13T22:30:32.764 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.765 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.765 DEBUG:boto:Params: {} 2024-04-13T22:30:32.765 DEBUG:boto:Token: None 2024-04-13T22:30:32.765 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272e07bfe145e756-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.767 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.768 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.768 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.768 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.768 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.768 DEBUG:boto:Method: GET 2024-04-13T22:30:32.768 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.768 DEBUG:boto:Data: 2024-04-13T22:30:32.768 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.768 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.769 DEBUG:boto:Params: {} 2024-04-13T22:30:32.769 DEBUG:boto:Token: None 2024-04-13T22:30:32.769 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.769 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e93608d190258268-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.814 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.815 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.815 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.815 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.815 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:32.815 DEBUG:boto:Data: 2024-04-13T22:30:32.815 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.816 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.816 DEBUG:boto:Params: {} 2024-04-13T22:30:32.816 DEBUG:boto:Token: None 2024-04-13T22:30:32.816 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185 2024-04-13T22:30:32.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DiBnBLnHzRBisa+UdrbCOLWvPmM= 2024-04-13T22:30:32.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DiBnBLnHzRBisa+UdrbCOLWvPmM=', 'Content-Length': '0'} 2024-04-13T22:30:32.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1426f743c457e1-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.818 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.818 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.819 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.819 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:32.819 DEBUG:boto:Data: 2024-04-13T22:30:32.819 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.819 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.819 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.819 DEBUG:boto:Params: {} 2024-04-13T22:30:32.819 DEBUG:boto:Token: None 2024-04-13T22:30:32.819 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185 2024-04-13T22:30:32.820 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DiBnBLnHzRBisa+UdrbCOLWvPmM= 2024-04-13T22:30:32.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DiBnBLnHzRBisa+UdrbCOLWvPmM=', 'Content-Length': '0'} 2024-04-13T22:30:32.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003973b45150b89249-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-13T22:30:32.822 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.822 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.822 DEBUG:boto:Method: GET 2024-04-13T22:30:32.822 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:32.822 DEBUG:boto:Data: 2024-04-13T22:30:32.822 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.822 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.822 DEBUG:boto:Params: {} 2024-04-13T22:30:32.823 DEBUG:boto:Token: None 2024-04-13T22:30:32.823 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185 2024-04-13T22:30:32.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bpewaHENIxLCj44p+bHnaygP/r4= 2024-04-13T22:30:32.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bpewaHENIxLCj44p+bHnaygP/r4=', 'Content-Length': '0'} 2024-04-13T22:30:32.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a36d19141161309a-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.825 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:32.825 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:32.825 DEBUG:boto:Method: GET 2024-04-13T22:30:32.825 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:32.825 DEBUG:boto:Data: 2024-04-13T22:30:32.825 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.825 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.825 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.825 DEBUG:boto:Params: {} 2024-04-13T22:30:32.826 DEBUG:boto:Token: None 2024-04-13T22:30:32.826 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj185 2024-04-13T22:30:32.826 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bpewaHENIxLCj44p+bHnaygP/r4= 2024-04-13T22:30:32.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bpewaHENIxLCj44p+bHnaygP/r4=', 'Content-Length': '0'} 2024-04-13T22:30:32.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003105db6faafc324f-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.828 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.828 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.828 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.828 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.828 DEBUG:boto:Method: GET 2024-04-13T22:30:32.828 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.828 DEBUG:boto:Data: 2024-04-13T22:30:32.828 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.829 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.829 DEBUG:boto:Params: {} 2024-04-13T22:30:32.829 DEBUG:boto:Token: None 2024-04-13T22:30:32.829 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.829 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158c62e69130e1e1-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.831 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.831 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.832 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.832 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.832 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.832 DEBUG:boto:Method: GET 2024-04-13T22:30:32.832 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:32.832 DEBUG:boto:Data: 2024-04-13T22:30:32.832 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.832 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.832 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.832 DEBUG:boto:Params: {} 2024-04-13T22:30:32.833 DEBUG:boto:Token: None 2024-04-13T22:30:32.833 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.833 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001329d93cf44357f0-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.837 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:30:32.837 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.837 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.837 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.837 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.837 DEBUG:boto:Method: GET 2024-04-13T22:30:32.838 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.838 DEBUG:boto:Data: 2024-04-13T22:30:32.838 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.838 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.838 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.838 DEBUG:boto:Params: {} 2024-04-13T22:30:32.838 DEBUG:boto:Token: None 2024-04-13T22:30:32.838 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.838 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0= 2024-04-13T22:30:32.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0=', 'Content-Length': '0'} 2024-04-13T22:30:32.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e30f2010616df3-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.840 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.841 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.841 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.841 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.841 DEBUG:boto:Method: GET 2024-04-13T22:30:32.841 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.841 DEBUG:boto:Data: 2024-04-13T22:30:32.841 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.841 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.841 DEBUG:boto:Params: {} 2024-04-13T22:30:32.841 DEBUG:boto:Token: None 2024-04-13T22:30:32.842 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.842 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0= 2024-04-13T22:30:32.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0=', 'Content-Length': '0'} 2024-04-13T22:30:32.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f645df74f19217b-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.844 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.844 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.844 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.844 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.844 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.844 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.844 DEBUG:boto:Data: 2024-04-13T22:30:32.844 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.845 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.845 DEBUG:boto:Params: {} 2024-04-13T22:30:32.845 DEBUG:boto:Token: None 2024-04-13T22:30:32.845 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HWqJfw1O0gbkZ2vyx7dy/PFgRQM= 2024-04-13T22:30:32.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HWqJfw1O0gbkZ2vyx7dy/PFgRQM=', 'Content-Length': '0'} 2024-04-13T22:30:32.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd1866d1c5ae4feb-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.847 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.847 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.847 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.847 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.847 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.847 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.848 DEBUG:boto:Data: 2024-04-13T22:30:32.848 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.848 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.848 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.848 DEBUG:boto:Params: {} 2024-04-13T22:30:32.848 DEBUG:boto:Token: None 2024-04-13T22:30:32.848 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HWqJfw1O0gbkZ2vyx7dy/PFgRQM= 2024-04-13T22:30:32.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HWqJfw1O0gbkZ2vyx7dy/PFgRQM=', 'Content-Length': '0'} 2024-04-13T22:30:32.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc2bca4f4d5b82b-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:30:32.850 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.850 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.851 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.851 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.851 DEBUG:boto:Method: GET 2024-04-13T22:30:32.851 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.851 DEBUG:boto:Data: 2024-04-13T22:30:32.851 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.851 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.851 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.851 DEBUG:boto:Params: {} 2024-04-13T22:30:32.851 DEBUG:boto:Token: None 2024-04-13T22:30:32.852 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0= 2024-04-13T22:30:32.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0=', 'Content-Length': '0'} 2024-04-13T22:30:32.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d56e7ccf3bf5530-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.894 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.894 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.894 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.895 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.895 DEBUG:boto:Method: GET 2024-04-13T22:30:32.895 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.895 DEBUG:boto:Data: 2024-04-13T22:30:32.895 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.895 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.895 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.895 DEBUG:boto:Params: {} 2024-04-13T22:30:32.895 DEBUG:boto:Token: None 2024-04-13T22:30:32.895 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:32.896 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0= 2024-04-13T22:30:32.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IH8Th27vDfh3LhHuT0bbQRFa3w0=', 'Content-Length': '0'} 2024-04-13T22:30:32.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000944a1d214a56f3d5-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.898 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.898 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.898 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.898 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.898 DEBUG:boto:Method: GET 2024-04-13T22:30:32.898 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.898 DEBUG:boto:Data: 2024-04-13T22:30:32.898 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.899 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.899 DEBUG:boto:Params: {} 2024-04-13T22:30:32.899 DEBUG:boto:Token: None 2024-04-13T22:30:32.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc5b4952c8ebcb73-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.901 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.902 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.902 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.902 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.902 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.902 DEBUG:boto:Method: GET 2024-04-13T22:30:32.902 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.903 DEBUG:boto:Data: 2024-04-13T22:30:32.903 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.903 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.903 DEBUG:boto:Params: {} 2024-04-13T22:30:32.903 DEBUG:boto:Token: None 2024-04-13T22:30:32.903 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f06ecb353206a513-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.907 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.908 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.908 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.908 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.908 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:32.908 DEBUG:boto:Data: 2024-04-13T22:30:32.908 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.908 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.908 DEBUG:boto:Params: {} 2024-04-13T22:30:32.908 DEBUG:boto:Token: None 2024-04-13T22:30:32.909 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19 2024-04-13T22:30:32.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mJW6H5OMKfdXh/Q/aLK2/URItQ4= 2024-04-13T22:30:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mJW6H5OMKfdXh/Q/aLK2/URItQ4=', 'Content-Length': '0'} 2024-04-13T22:30:32.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089a61143c4d8e0be-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.910 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.910 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.911 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.911 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:32.911 DEBUG:boto:Data: 2024-04-13T22:30:32.911 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.911 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.911 DEBUG:boto:Params: {} 2024-04-13T22:30:32.911 DEBUG:boto:Token: None 2024-04-13T22:30:32.911 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19 2024-04-13T22:30:32.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mJW6H5OMKfdXh/Q/aLK2/URItQ4= 2024-04-13T22:30:32.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mJW6H5OMKfdXh/Q/aLK2/URItQ4=', 'Content-Length': '0'} 2024-04-13T22:30:32.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d51c55a51a2c892-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:30:32.913 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.913 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.914 DEBUG:boto:Method: GET 2024-04-13T22:30:32.914 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:32.914 DEBUG:boto:Data: 2024-04-13T22:30:32.914 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.914 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.914 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.914 DEBUG:boto:Params: {} 2024-04-13T22:30:32.914 DEBUG:boto:Token: None 2024-04-13T22:30:32.914 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19 2024-04-13T22:30:32.914 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:14TxZamz/32ly5U1ybFShOnd320= 2024-04-13T22:30:32.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:14TxZamz/32ly5U1ybFShOnd320=', 'Content-Length': '0'} 2024-04-13T22:30:32.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974fdb499a4e667e-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.916 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:32.916 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:32.916 DEBUG:boto:Method: GET 2024-04-13T22:30:32.916 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:32.917 DEBUG:boto:Data: 2024-04-13T22:30:32.917 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.917 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.917 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.917 DEBUG:boto:Params: {} 2024-04-13T22:30:32.917 DEBUG:boto:Token: None 2024-04-13T22:30:32.917 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj19 2024-04-13T22:30:32.917 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:14TxZamz/32ly5U1ybFShOnd320= 2024-04-13T22:30:32.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:14TxZamz/32ly5U1ybFShOnd320=', 'Content-Length': '0'} 2024-04-13T22:30:32.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3473109c0ecdd3-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.962 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.963 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.963 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.963 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.963 DEBUG:boto:Method: GET 2024-04-13T22:30:32.963 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.963 DEBUG:boto:Data: 2024-04-13T22:30:32.963 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.963 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.963 DEBUG:boto:Params: {} 2024-04-13T22:30:32.964 DEBUG:boto:Token: None 2024-04-13T22:30:32.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055efb03ffe7b113e-00661b0788-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.970 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.970 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:32.970 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:32.970 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.970 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.971 DEBUG:boto:Method: GET 2024-04-13T22:30:32.971 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:32.971 DEBUG:boto:Data: 2024-04-13T22:30:32.971 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.971 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.971 DEBUG:boto:Params: {} 2024-04-13T22:30:32.972 DEBUG:boto:Token: None 2024-04-13T22:30:32.972 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/ 2024-04-13T22:30:32.972 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg= 2024-04-13T22:30:32.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TaLV0OorToN3dVLLTGd0TnDB4Dg=', 'Content-Length': '0'} 2024-04-13T22:30:32.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb7a805de13fe97-00661b0788-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.976 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:32.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:30:32.976 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:32.976 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:32.976 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.977 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.977 DEBUG:boto:Method: GET 2024-04-13T22:30:32.977 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.977 DEBUG:boto:Data: 2024-04-13T22:30:32.977 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.977 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.977 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.977 DEBUG:boto:Params: {} 2024-04-13T22:30:32.977 DEBUG:boto:Token: None 2024-04-13T22:30:32.978 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.978 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nNYNwVMqU9UW99gsTUrkCgQp8OE= 2024-04-13T22:30:32.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nNYNwVMqU9UW99gsTUrkCgQp8OE=', 'Content-Length': '0'} 2024-04-13T22:30:32.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9863d79d2470559-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.980 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:32.980 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:32.980 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.980 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.980 DEBUG:boto:Method: GET 2024-04-13T22:30:32.980 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.980 DEBUG:boto:Data: 2024-04-13T22:30:32.980 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.981 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.981 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.981 DEBUG:boto:Params: {} 2024-04-13T22:30:32.981 DEBUG:boto:Token: None 2024-04-13T22:30:32.981 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.981 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nNYNwVMqU9UW99gsTUrkCgQp8OE= 2024-04-13T22:30:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nNYNwVMqU9UW99gsTUrkCgQp8OE=', 'Content-Length': '0'} 2024-04-13T22:30:32.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dcb543a08e15d28-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.983 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:32.983 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:32.983 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.983 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.984 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.984 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.984 DEBUG:boto:Data: 2024-04-13T22:30:32.984 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.984 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.984 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.984 DEBUG:boto:Params: {} 2024-04-13T22:30:32.984 DEBUG:boto:Token: None 2024-04-13T22:30:32.984 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1MvLphQtcOmL+ARXoJagHUcTiDg= 2024-04-13T22:30:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1MvLphQtcOmL+ARXoJagHUcTiDg=', 'Content-Length': '0'} 2024-04-13T22:30:32.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e9aa8d7fd47915-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.986 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:32.987 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:32.987 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.987 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.987 DEBUG:boto:Method: HEAD 2024-04-13T22:30:32.987 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.987 DEBUG:boto:Data: 2024-04-13T22:30:32.987 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:32.987 DEBUG:boto:Port: 8000 2024-04-13T22:30:32.988 DEBUG:boto:Params: {} 2024-04-13T22:30:32.988 DEBUG:boto:Token: None 2024-04-13T22:30:32.988 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1MvLphQtcOmL+ARXoJagHUcTiDg= 2024-04-13T22:30:32.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1MvLphQtcOmL+ARXoJagHUcTiDg=', 'Content-Length': '0'} 2024-04-13T22:30:32.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e073de38ac38de32-00661b0788-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:32.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:30:32.990 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:32.990 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:32.990 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.990 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.990 DEBUG:boto:Method: GET 2024-04-13T22:30:32.991 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.991 DEBUG:boto:Data: 2024-04-13T22:30:32.991 DEBUG:boto:Headers: {} 2024-04-13T22:30:32.991 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:32.991 DEBUG:boto:Port: 8001 2024-04-13T22:30:32.991 DEBUG:boto:Params: {} 2024-04-13T22:30:32.991 DEBUG:boto:Token: None 2024-04-13T22:30:32.991 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:32 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:32.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nNYNwVMqU9UW99gsTUrkCgQp8OE= 2024-04-13T22:30:32.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nNYNwVMqU9UW99gsTUrkCgQp8OE=', 'Content-Length': '0'} 2024-04-13T22:30:32.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7988cc2a22407d-00661b0788-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.034 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:33.035 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:33.035 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:33.035 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:33.035 DEBUG:boto:Method: GET 2024-04-13T22:30:33.035 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:33.035 DEBUG:boto:Data: 2024-04-13T22:30:33.035 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.035 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.035 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.035 DEBUG:boto:Params: {} 2024-04-13T22:30:33.036 DEBUG:boto:Token: None 2024-04-13T22:30:33.036 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:33.036 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sU9s9phQxep85v+k0uo4pEhN0SI= 2024-04-13T22:30:33.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sU9s9phQxep85v+k0uo4pEhN0SI=', 'Content-Length': '0'} 2024-04-13T22:30:33.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001383fcc1ae7e6a19-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.038 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.038 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.038 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.038 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.039 DEBUG:boto:Method: GET 2024-04-13T22:30:33.039 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.039 DEBUG:boto:Data: 2024-04-13T22:30:33.039 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.039 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.039 DEBUG:boto:Params: {} 2024-04-13T22:30:33.039 DEBUG:boto:Token: None 2024-04-13T22:30:33.039 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d7e47dbe559510-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.042 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.042 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.043 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.043 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.043 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.043 DEBUG:boto:Method: GET 2024-04-13T22:30:33.043 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.043 DEBUG:boto:Data: 2024-04-13T22:30:33.043 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.043 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.043 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.044 DEBUG:boto:Params: {} 2024-04-13T22:30:33.044 DEBUG:boto:Token: None 2024-04-13T22:30:33.044 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3dde12a885e9b78-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.048 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.048 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:33.048 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:33.048 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.048 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:33.048 DEBUG:boto:Data: 2024-04-13T22:30:33.048 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.049 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.049 DEBUG:boto:Params: {} 2024-04-13T22:30:33.049 DEBUG:boto:Token: None 2024-04-13T22:30:33.049 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj22 2024-04-13T22:30:33.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ab4CI9O4ZurL4qBufI/N0hrSMxc= 2024-04-13T22:30:33.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ab4CI9O4ZurL4qBufI/N0hrSMxc=', 'Content-Length': '0'} 2024-04-13T22:30:33.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310a22695c8aa498-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.051 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:33.051 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:33.051 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.051 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:33.052 DEBUG:boto:Data: 2024-04-13T22:30:33.052 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.052 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.052 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.052 DEBUG:boto:Params: {} 2024-04-13T22:30:33.052 DEBUG:boto:Token: None 2024-04-13T22:30:33.052 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj22 2024-04-13T22:30:33.052 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ab4CI9O4ZurL4qBufI/N0hrSMxc= 2024-04-13T22:30:33.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ab4CI9O4ZurL4qBufI/N0hrSMxc=', 'Content-Length': '0'} 2024-04-13T22:30:33.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c5ca8003cc4d7e-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:30:33.055 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:33.055 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:33.055 DEBUG:boto:Method: GET 2024-04-13T22:30:33.055 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:33.055 DEBUG:boto:Data: 2024-04-13T22:30:33.055 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.055 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.055 DEBUG:boto:Params: {} 2024-04-13T22:30:33.056 DEBUG:boto:Token: None 2024-04-13T22:30:33.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj22 2024-04-13T22:30:33.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kq7Ue8Lr2W3nRIZ9ESf3+YxWELI= 2024-04-13T22:30:33.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kq7Ue8Lr2W3nRIZ9ESf3+YxWELI=', 'Content-Length': '0'} 2024-04-13T22:30:33.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be7eb807e8d9563-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.058 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:33.058 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:33.058 DEBUG:boto:Method: GET 2024-04-13T22:30:33.058 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:33.058 DEBUG:boto:Data: 2024-04-13T22:30:33.058 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.058 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.059 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.059 DEBUG:boto:Params: {} 2024-04-13T22:30:33.059 DEBUG:boto:Token: None 2024-04-13T22:30:33.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj22 2024-04-13T22:30:33.059 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Kq7Ue8Lr2W3nRIZ9ESf3+YxWELI= 2024-04-13T22:30:33.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Kq7Ue8Lr2W3nRIZ9ESf3+YxWELI=', 'Content-Length': '0'} 2024-04-13T22:30:33.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587266b21ba026c5-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.061 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.061 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.061 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.062 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.062 DEBUG:boto:Method: GET 2024-04-13T22:30:33.062 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.062 DEBUG:boto:Data: 2024-04-13T22:30:33.062 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.062 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.062 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.062 DEBUG:boto:Params: {} 2024-04-13T22:30:33.062 DEBUG:boto:Token: None 2024-04-13T22:30:33.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188d246ad26fa2c3-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.065 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.066 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.066 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.066 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.066 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.066 DEBUG:boto:Method: GET 2024-04-13T22:30:33.066 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:33.066 DEBUG:boto:Data: 2024-04-13T22:30:33.066 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.066 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.066 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.066 DEBUG:boto:Params: {} 2024-04-13T22:30:33.067 DEBUG:boto:Token: None 2024-04-13T22:30:33.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1ee12fa028ad216-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.070 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:30:33.071 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.071 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.071 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.071 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.071 DEBUG:boto:Method: GET 2024-04-13T22:30:33.071 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.071 DEBUG:boto:Data: 2024-04-13T22:30:33.071 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.072 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.072 DEBUG:boto:Params: {} 2024-04-13T22:30:33.072 DEBUG:boto:Token: None 2024-04-13T22:30:33.072 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg= 2024-04-13T22:30:33.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg=', 'Content-Length': '0'} 2024-04-13T22:30:33.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae25a61d9b00c9e-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.074 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.074 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.075 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.075 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.075 DEBUG:boto:Method: GET 2024-04-13T22:30:33.075 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.075 DEBUG:boto:Data: 2024-04-13T22:30:33.075 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.075 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.075 DEBUG:boto:Params: {} 2024-04-13T22:30:33.075 DEBUG:boto:Token: None 2024-04-13T22:30:33.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg= 2024-04-13T22:30:33.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg=', 'Content-Length': '0'} 2024-04-13T22:30:33.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2df5428bca2b7a4-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.118 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.118 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.119 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.119 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.119 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.119 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.119 DEBUG:boto:Data: 2024-04-13T22:30:33.119 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.119 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.119 DEBUG:boto:Params: {} 2024-04-13T22:30:33.119 DEBUG:boto:Token: None 2024-04-13T22:30:33.120 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jUlWwwA3sPPnZOQBxyvxC9hRQ/A= 2024-04-13T22:30:33.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jUlWwwA3sPPnZOQBxyvxC9hRQ/A=', 'Content-Length': '0'} 2024-04-13T22:30:33.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccda8a2e0be4dde9-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.121 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.121 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.122 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.122 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.122 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.122 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.122 DEBUG:boto:Data: 2024-04-13T22:30:33.122 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.122 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.122 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.122 DEBUG:boto:Params: {} 2024-04-13T22:30:33.122 DEBUG:boto:Token: None 2024-04-13T22:30:33.122 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jUlWwwA3sPPnZOQBxyvxC9hRQ/A= 2024-04-13T22:30:33.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jUlWwwA3sPPnZOQBxyvxC9hRQ/A=', 'Content-Length': '0'} 2024-04-13T22:30:33.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca3686a9daaf4b0c-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:30:33.125 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.125 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.125 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.125 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.125 DEBUG:boto:Method: GET 2024-04-13T22:30:33.125 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.125 DEBUG:boto:Data: 2024-04-13T22:30:33.125 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.125 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.126 DEBUG:boto:Params: {} 2024-04-13T22:30:33.126 DEBUG:boto:Token: None 2024-04-13T22:30:33.126 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg= 2024-04-13T22:30:33.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg=', 'Content-Length': '0'} 2024-04-13T22:30:33.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30917c4c7ce2750-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.128 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.128 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.128 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.128 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.128 DEBUG:boto:Method: GET 2024-04-13T22:30:33.128 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.128 DEBUG:boto:Data: 2024-04-13T22:30:33.128 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.129 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.129 DEBUG:boto:Params: {} 2024-04-13T22:30:33.129 DEBUG:boto:Token: None 2024-04-13T22:30:33.129 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:33.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg= 2024-04-13T22:30:33.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Zwo9noLhUdnqcwfe2BBda3JuyQg=', 'Content-Length': '0'} 2024-04-13T22:30:33.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000205fc2b3dc50c6e3-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.131 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.131 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.131 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.131 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.131 DEBUG:boto:Method: GET 2024-04-13T22:30:33.131 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.132 DEBUG:boto:Data: 2024-04-13T22:30:33.132 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.132 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.132 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.132 DEBUG:boto:Params: {} 2024-04-13T22:30:33.132 DEBUG:boto:Token: None 2024-04-13T22:30:33.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef33df76305bc19a-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.134 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.135 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.135 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.135 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.135 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.135 DEBUG:boto:Method: GET 2024-04-13T22:30:33.135 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.135 DEBUG:boto:Data: 2024-04-13T22:30:33.135 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.136 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.136 DEBUG:boto:Params: {} 2024-04-13T22:30:33.136 DEBUG:boto:Token: None 2024-04-13T22:30:33.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da1f0b12efc30eb-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.139 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.140 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.140 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.140 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.140 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:33.140 DEBUG:boto:Data: 2024-04-13T22:30:33.140 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.140 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.140 DEBUG:boto:Params: {} 2024-04-13T22:30:33.141 DEBUG:boto:Token: None 2024-04-13T22:30:33.141 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23 2024-04-13T22:30:33.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EyHIzuJiYmZMgoIJWrZrKO/ctys= 2024-04-13T22:30:33.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EyHIzuJiYmZMgoIJWrZrKO/ctys=', 'Content-Length': '0'} 2024-04-13T22:30:33.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04bf8c5b645ab33-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.143 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.143 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.143 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.143 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:33.143 DEBUG:boto:Data: 2024-04-13T22:30:33.143 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.143 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.143 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.143 DEBUG:boto:Params: {} 2024-04-13T22:30:33.143 DEBUG:boto:Token: None 2024-04-13T22:30:33.144 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23 2024-04-13T22:30:33.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EyHIzuJiYmZMgoIJWrZrKO/ctys= 2024-04-13T22:30:33.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EyHIzuJiYmZMgoIJWrZrKO/ctys=', 'Content-Length': '0'} 2024-04-13T22:30:33.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036045e265971a555-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:30:33.145 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.146 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.146 DEBUG:boto:Method: GET 2024-04-13T22:30:33.146 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:33.146 DEBUG:boto:Data: 2024-04-13T22:30:33.146 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.146 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.146 DEBUG:boto:Params: {} 2024-04-13T22:30:33.146 DEBUG:boto:Token: None 2024-04-13T22:30:33.147 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23 2024-04-13T22:30:33.147 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:COgq8XZ9C+SbFZWcRcKkOFl7MSs= 2024-04-13T22:30:33.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:COgq8XZ9C+SbFZWcRcKkOFl7MSs=', 'Content-Length': '0'} 2024-04-13T22:30:33.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a58b1e4b0ab9d9ac-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.190 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:33.191 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:33.191 DEBUG:boto:Method: GET 2024-04-13T22:30:33.191 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:33.191 DEBUG:boto:Data: 2024-04-13T22:30:33.191 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.191 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.191 DEBUG:boto:Params: {} 2024-04-13T22:30:33.191 DEBUG:boto:Token: None 2024-04-13T22:30:33.191 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj23 2024-04-13T22:30:33.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:COgq8XZ9C+SbFZWcRcKkOFl7MSs= 2024-04-13T22:30:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:COgq8XZ9C+SbFZWcRcKkOFl7MSs=', 'Content-Length': '0'} 2024-04-13T22:30:33.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c6f3da2c4b2a958-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.193 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.194 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.194 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.194 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.194 DEBUG:boto:Method: GET 2024-04-13T22:30:33.194 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.194 DEBUG:boto:Data: 2024-04-13T22:30:33.194 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.194 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.194 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.194 DEBUG:boto:Params: {} 2024-04-13T22:30:33.195 DEBUG:boto:Token: None 2024-04-13T22:30:33.195 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e370f5068c20601-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.197 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.197 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.198 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.198 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.198 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.198 DEBUG:boto:Method: GET 2024-04-13T22:30:33.198 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:33.198 DEBUG:boto:Data: 2024-04-13T22:30:33.198 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.198 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.198 DEBUG:boto:Params: {} 2024-04-13T22:30:33.199 DEBUG:boto:Token: None 2024-04-13T22:30:33.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981f63fe09404900-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.202 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:30:33.203 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.203 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.203 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.203 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.203 DEBUG:boto:Method: GET 2024-04-13T22:30:33.203 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.203 DEBUG:boto:Data: 2024-04-13T22:30:33.203 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.204 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.204 DEBUG:boto:Params: {} 2024-04-13T22:30:33.204 DEBUG:boto:Token: None 2024-04-13T22:30:33.204 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.204 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg= 2024-04-13T22:30:33.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg=', 'Content-Length': '0'} 2024-04-13T22:30:33.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f0a0ad9bc89652-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.206 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.206 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.207 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.207 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.207 DEBUG:boto:Method: GET 2024-04-13T22:30:33.207 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.207 DEBUG:boto:Data: 2024-04-13T22:30:33.207 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.207 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.207 DEBUG:boto:Params: {} 2024-04-13T22:30:33.207 DEBUG:boto:Token: None 2024-04-13T22:30:33.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg= 2024-04-13T22:30:33.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg=', 'Content-Length': '0'} 2024-04-13T22:30:33.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028815274e29461aa-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.210 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.210 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.211 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.211 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.211 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.211 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.211 DEBUG:boto:Data: 2024-04-13T22:30:33.211 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.211 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.211 DEBUG:boto:Params: {} 2024-04-13T22:30:33.211 DEBUG:boto:Token: None 2024-04-13T22:30:33.211 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.212 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6r+2DbafKtDpxy9qXkQ+0xZG/38= 2024-04-13T22:30:33.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6r+2DbafKtDpxy9qXkQ+0xZG/38=', 'Content-Length': '0'} 2024-04-13T22:30:33.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebaf9783fb4e1c0b-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.213 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.213 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.213 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.214 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.214 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.214 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.214 DEBUG:boto:Data: 2024-04-13T22:30:33.214 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.214 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.214 DEBUG:boto:Params: {} 2024-04-13T22:30:33.214 DEBUG:boto:Token: None 2024-04-13T22:30:33.215 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6r+2DbafKtDpxy9qXkQ+0xZG/38= 2024-04-13T22:30:33.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6r+2DbafKtDpxy9qXkQ+0xZG/38=', 'Content-Length': '0'} 2024-04-13T22:30:33.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028edbc03fbef126f-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:30:33.217 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.217 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.217 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.217 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.217 DEBUG:boto:Method: GET 2024-04-13T22:30:33.217 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.217 DEBUG:boto:Data: 2024-04-13T22:30:33.217 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.217 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.218 DEBUG:boto:Params: {} 2024-04-13T22:30:33.218 DEBUG:boto:Token: None 2024-04-13T22:30:33.218 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg= 2024-04-13T22:30:33.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg=', 'Content-Length': '0'} 2024-04-13T22:30:33.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d518b3193cde6b6-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.220 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.220 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.220 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.220 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.220 DEBUG:boto:Method: GET 2024-04-13T22:30:33.220 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.220 DEBUG:boto:Data: 2024-04-13T22:30:33.220 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.220 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.221 DEBUG:boto:Params: {} 2024-04-13T22:30:33.221 DEBUG:boto:Token: None 2024-04-13T22:30:33.221 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:33.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg= 2024-04-13T22:30:33.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+N6CtdWYIyPIxDe5Q2HiatlZzg=', 'Content-Length': '0'} 2024-04-13T22:30:33.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce03dbe186dc79e-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.223 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.223 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.223 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.223 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.223 DEBUG:boto:Method: GET 2024-04-13T22:30:33.223 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.223 DEBUG:boto:Data: 2024-04-13T22:30:33.223 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.223 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.223 DEBUG:boto:Params: {} 2024-04-13T22:30:33.224 DEBUG:boto:Token: None 2024-04-13T22:30:33.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008408056645171474-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.226 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.226 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.226 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.227 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.227 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.227 DEBUG:boto:Method: GET 2024-04-13T22:30:33.227 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.227 DEBUG:boto:Data: 2024-04-13T22:30:33.227 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.227 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.227 DEBUG:boto:Params: {} 2024-04-13T22:30:33.227 DEBUG:boto:Token: None 2024-04-13T22:30:33.227 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343ad542b423a73d-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.231 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.232 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.232 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.232 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.232 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:33.232 DEBUG:boto:Data: 2024-04-13T22:30:33.232 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.232 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.232 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.232 DEBUG:boto:Params: {} 2024-04-13T22:30:33.232 DEBUG:boto:Token: None 2024-04-13T22:30:33.233 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29 2024-04-13T22:30:33.233 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m4QBo89SqgfIkWS/Jrh3rNX5X70= 2024-04-13T22:30:33.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m4QBo89SqgfIkWS/Jrh3rNX5X70=', 'Content-Length': '0'} 2024-04-13T22:30:33.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082225c11b30de295-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.234 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.235 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.235 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.235 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:33.235 DEBUG:boto:Data: 2024-04-13T22:30:33.235 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.235 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.235 DEBUG:boto:Params: {} 2024-04-13T22:30:33.235 DEBUG:boto:Token: None 2024-04-13T22:30:33.235 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29 2024-04-13T22:30:33.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m4QBo89SqgfIkWS/Jrh3rNX5X70= 2024-04-13T22:30:33.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m4QBo89SqgfIkWS/Jrh3rNX5X70=', 'Content-Length': '0'} 2024-04-13T22:30:33.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079850bbbb5785d3c-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:30:33.237 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.237 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.237 DEBUG:boto:Method: GET 2024-04-13T22:30:33.237 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:33.237 DEBUG:boto:Data: 2024-04-13T22:30:33.238 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.238 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.238 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.238 DEBUG:boto:Params: {} 2024-04-13T22:30:33.238 DEBUG:boto:Token: None 2024-04-13T22:30:33.238 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29 2024-04-13T22:30:33.238 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BTog1e51vJLsOnGX+Zkr7NRND7g= 2024-04-13T22:30:33.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BTog1e51vJLsOnGX+Zkr7NRND7g=', 'Content-Length': '0'} 2024-04-13T22:30:33.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000082a367f76174961-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.240 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:33.240 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:33.240 DEBUG:boto:Method: GET 2024-04-13T22:30:33.240 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:33.241 DEBUG:boto:Data: 2024-04-13T22:30:33.241 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.241 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.241 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.241 DEBUG:boto:Params: {} 2024-04-13T22:30:33.241 DEBUG:boto:Token: None 2024-04-13T22:30:33.241 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj29 2024-04-13T22:30:33.241 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BTog1e51vJLsOnGX+Zkr7NRND7g= 2024-04-13T22:30:33.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BTog1e51vJLsOnGX+Zkr7NRND7g=', 'Content-Length': '0'} 2024-04-13T22:30:33.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000845228844a7d956c-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.286 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.287 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.287 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.287 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.287 DEBUG:boto:Method: GET 2024-04-13T22:30:33.287 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.287 DEBUG:boto:Data: 2024-04-13T22:30:33.287 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.287 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.287 DEBUG:boto:Params: {} 2024-04-13T22:30:33.288 DEBUG:boto:Token: None 2024-04-13T22:30:33.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa2c1d40e3f4b38-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.290 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.290 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.291 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.291 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.291 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.291 DEBUG:boto:Method: GET 2024-04-13T22:30:33.291 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:33.291 DEBUG:boto:Data: 2024-04-13T22:30:33.291 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.291 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.291 DEBUG:boto:Params: {} 2024-04-13T22:30:33.291 DEBUG:boto:Token: None 2024-04-13T22:30:33.292 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b142ecb27ca13f0-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.295 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:30:33.296 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.296 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.296 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.296 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.296 DEBUG:boto:Method: GET 2024-04-13T22:30:33.296 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.296 DEBUG:boto:Data: 2024-04-13T22:30:33.296 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.297 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.297 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.297 DEBUG:boto:Params: {} 2024-04-13T22:30:33.297 DEBUG:boto:Token: None 2024-04-13T22:30:33.297 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.297 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A= 2024-04-13T22:30:33.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A=', 'Content-Length': '0'} 2024-04-13T22:30:33.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76208ac2cca9740-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.299 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.299 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.299 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.299 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.299 DEBUG:boto:Method: GET 2024-04-13T22:30:33.300 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.300 DEBUG:boto:Data: 2024-04-13T22:30:33.300 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.300 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.300 DEBUG:boto:Params: {} 2024-04-13T22:30:33.300 DEBUG:boto:Token: None 2024-04-13T22:30:33.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A= 2024-04-13T22:30:33.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A=', 'Content-Length': '0'} 2024-04-13T22:30:33.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9916c4e6e783550-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.302 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.302 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.303 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.303 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.303 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.303 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.303 DEBUG:boto:Data: 2024-04-13T22:30:33.303 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.303 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.303 DEBUG:boto:Params: {} 2024-04-13T22:30:33.303 DEBUG:boto:Token: None 2024-04-13T22:30:33.303 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k+e/4D1hDG7fgWS98vKV98iUnCM= 2024-04-13T22:30:33.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k+e/4D1hDG7fgWS98vKV98iUnCM=', 'Content-Length': '0'} 2024-04-13T22:30:33.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e147792c31ae43ee-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.306 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.306 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.306 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.306 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.306 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.306 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.306 DEBUG:boto:Data: 2024-04-13T22:30:33.306 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.306 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.306 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.306 DEBUG:boto:Params: {} 2024-04-13T22:30:33.307 DEBUG:boto:Token: None 2024-04-13T22:30:33.307 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k+e/4D1hDG7fgWS98vKV98iUnCM= 2024-04-13T22:30:33.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k+e/4D1hDG7fgWS98vKV98iUnCM=', 'Content-Length': '0'} 2024-04-13T22:30:33.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f873e3b11f13429-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:30:33.309 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.309 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.309 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.309 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.309 DEBUG:boto:Method: GET 2024-04-13T22:30:33.309 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.309 DEBUG:boto:Data: 2024-04-13T22:30:33.310 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.310 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.310 DEBUG:boto:Params: {} 2024-04-13T22:30:33.310 DEBUG:boto:Token: None 2024-04-13T22:30:33.310 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A= 2024-04-13T22:30:33.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A=', 'Content-Length': '0'} 2024-04-13T22:30:33.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e581f5fb5874491a-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.312 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.312 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.312 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.313 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.313 DEBUG:boto:Method: GET 2024-04-13T22:30:33.313 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.313 DEBUG:boto:Data: 2024-04-13T22:30:33.313 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.313 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.313 DEBUG:boto:Params: {} 2024-04-13T22:30:33.313 DEBUG:boto:Token: None 2024-04-13T22:30:33.314 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:33.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A= 2024-04-13T22:30:33.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SRJroHSce+3lAgkThvx8XwRhs9A=', 'Content-Length': '0'} 2024-04-13T22:30:33.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6bd444dacac712e-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.358 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.358 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.358 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.359 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.359 DEBUG:boto:Method: GET 2024-04-13T22:30:33.359 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.359 DEBUG:boto:Data: 2024-04-13T22:30:33.359 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.359 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.359 DEBUG:boto:Params: {} 2024-04-13T22:30:33.359 DEBUG:boto:Token: None 2024-04-13T22:30:33.359 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000528431b17fa58d0b-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.362 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.362 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.362 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.362 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.362 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.363 DEBUG:boto:Method: GET 2024-04-13T22:30:33.363 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.363 DEBUG:boto:Data: 2024-04-13T22:30:33.363 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.363 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.363 DEBUG:boto:Params: {} 2024-04-13T22:30:33.363 DEBUG:boto:Token: None 2024-04-13T22:30:33.363 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.364 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e482353eaee61c17-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.367 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.368 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.368 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.368 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.368 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:33.368 DEBUG:boto:Data: 2024-04-13T22:30:33.368 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.368 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.368 DEBUG:boto:Params: {} 2024-04-13T22:30:33.368 DEBUG:boto:Token: None 2024-04-13T22:30:33.369 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31 2024-04-13T22:30:33.369 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RalAw3gfH4kWA4WbcDQVSkU/NJ0= 2024-04-13T22:30:33.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RalAw3gfH4kWA4WbcDQVSkU/NJ0=', 'Content-Length': '0'} 2024-04-13T22:30:33.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085d325013afa8a19-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.371 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.371 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.371 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.371 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:33.371 DEBUG:boto:Data: 2024-04-13T22:30:33.371 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.371 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.371 DEBUG:boto:Params: {} 2024-04-13T22:30:33.371 DEBUG:boto:Token: None 2024-04-13T22:30:33.372 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31 2024-04-13T22:30:33.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RalAw3gfH4kWA4WbcDQVSkU/NJ0= 2024-04-13T22:30:33.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RalAw3gfH4kWA4WbcDQVSkU/NJ0=', 'Content-Length': '0'} 2024-04-13T22:30:33.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29c8bc6bcca00ec-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:30:33.373 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.374 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.374 DEBUG:boto:Method: GET 2024-04-13T22:30:33.374 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:33.374 DEBUG:boto:Data: 2024-04-13T22:30:33.374 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.374 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.374 DEBUG:boto:Params: {} 2024-04-13T22:30:33.375 DEBUG:boto:Token: None 2024-04-13T22:30:33.375 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31 2024-04-13T22:30:33.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gnCHm1HVLqTqMGhiiwxgEbS+ZcY= 2024-04-13T22:30:33.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gnCHm1HVLqTqMGhiiwxgEbS+ZcY=', 'Content-Length': '0'} 2024-04-13T22:30:33.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad793181298374e-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.418 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:33.418 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:33.419 DEBUG:boto:Method: GET 2024-04-13T22:30:33.419 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:33.419 DEBUG:boto:Data: 2024-04-13T22:30:33.419 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.419 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.419 DEBUG:boto:Params: {} 2024-04-13T22:30:33.419 DEBUG:boto:Token: None 2024-04-13T22:30:33.419 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj31 2024-04-13T22:30:33.420 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gnCHm1HVLqTqMGhiiwxgEbS+ZcY= 2024-04-13T22:30:33.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gnCHm1HVLqTqMGhiiwxgEbS+ZcY=', 'Content-Length': '0'} 2024-04-13T22:30:33.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f17c9e8113259e20-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.421 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.422 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.422 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.422 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.422 DEBUG:boto:Method: GET 2024-04-13T22:30:33.422 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.422 DEBUG:boto:Data: 2024-04-13T22:30:33.422 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.422 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.422 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.422 DEBUG:boto:Params: {} 2024-04-13T22:30:33.423 DEBUG:boto:Token: None 2024-04-13T22:30:33.423 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.423 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc9ec2e7c3a3ddba-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.425 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.425 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.426 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.426 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.426 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.426 DEBUG:boto:Method: GET 2024-04-13T22:30:33.426 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:33.426 DEBUG:boto:Data: 2024-04-13T22:30:33.426 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.426 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.426 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.426 DEBUG:boto:Params: {} 2024-04-13T22:30:33.427 DEBUG:boto:Token: None 2024-04-13T22:30:33.427 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e25202bb6343e7a-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.431 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:30:33.431 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.431 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.432 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.432 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.432 DEBUG:boto:Method: GET 2024-04-13T22:30:33.432 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.432 DEBUG:boto:Data: 2024-04-13T22:30:33.432 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.432 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.432 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.432 DEBUG:boto:Params: {} 2024-04-13T22:30:33.432 DEBUG:boto:Token: None 2024-04-13T22:30:33.432 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.433 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI= 2024-04-13T22:30:33.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI=', 'Content-Length': '0'} 2024-04-13T22:30:33.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa443fb2a4557cef-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.435 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.435 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.435 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.435 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.435 DEBUG:boto:Method: GET 2024-04-13T22:30:33.435 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.435 DEBUG:boto:Data: 2024-04-13T22:30:33.435 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.435 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.435 DEBUG:boto:Params: {} 2024-04-13T22:30:33.436 DEBUG:boto:Token: None 2024-04-13T22:30:33.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI= 2024-04-13T22:30:33.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI=', 'Content-Length': '0'} 2024-04-13T22:30:33.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006262a61ceb86ca3-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.478 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.478 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.478 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.479 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.479 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.479 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.479 DEBUG:boto:Data: 2024-04-13T22:30:33.479 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.479 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.479 DEBUG:boto:Params: {} 2024-04-13T22:30:33.479 DEBUG:boto:Token: None 2024-04-13T22:30:33.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.480 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:siANMYz2OxpIjZDXEuc5VXKImVc= 2024-04-13T22:30:33.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:siANMYz2OxpIjZDXEuc5VXKImVc=', 'Content-Length': '0'} 2024-04-13T22:30:33.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d4ee0de5606fb0-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.481 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.482 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.482 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.482 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.482 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.482 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.482 DEBUG:boto:Data: 2024-04-13T22:30:33.482 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.482 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.482 DEBUG:boto:Params: {} 2024-04-13T22:30:33.483 DEBUG:boto:Token: None 2024-04-13T22:30:33.483 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:siANMYz2OxpIjZDXEuc5VXKImVc= 2024-04-13T22:30:33.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:siANMYz2OxpIjZDXEuc5VXKImVc=', 'Content-Length': '0'} 2024-04-13T22:30:33.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95ec394937d5859-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:30:33.485 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.485 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.485 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.485 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.485 DEBUG:boto:Method: GET 2024-04-13T22:30:33.485 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.485 DEBUG:boto:Data: 2024-04-13T22:30:33.485 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.485 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.485 DEBUG:boto:Params: {} 2024-04-13T22:30:33.486 DEBUG:boto:Token: None 2024-04-13T22:30:33.486 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI= 2024-04-13T22:30:33.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI=', 'Content-Length': '0'} 2024-04-13T22:30:33.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000961fc344261ca32f-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.488 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.488 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.488 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.488 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.488 DEBUG:boto:Method: GET 2024-04-13T22:30:33.488 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.488 DEBUG:boto:Data: 2024-04-13T22:30:33.488 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.488 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.488 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.488 DEBUG:boto:Params: {} 2024-04-13T22:30:33.488 DEBUG:boto:Token: None 2024-04-13T22:30:33.489 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:33.489 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI= 2024-04-13T22:30:33.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCMSFCC6BCrHp76MbxknO9P/fEI=', 'Content-Length': '0'} 2024-04-13T22:30:33.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b1fdf6f86a9271-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.491 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.491 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.491 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.491 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.491 DEBUG:boto:Method: GET 2024-04-13T22:30:33.491 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.492 DEBUG:boto:Data: 2024-04-13T22:30:33.492 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.492 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.492 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.492 DEBUG:boto:Params: {} 2024-04-13T22:30:33.492 DEBUG:boto:Token: None 2024-04-13T22:30:33.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed8b2be6e1ce082-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.494 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.495 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.495 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.495 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.495 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.495 DEBUG:boto:Method: GET 2024-04-13T22:30:33.495 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.495 DEBUG:boto:Data: 2024-04-13T22:30:33.495 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.496 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.496 DEBUG:boto:Params: {} 2024-04-13T22:30:33.496 DEBUG:boto:Token: None 2024-04-13T22:30:33.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000949701b5b9890491-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.500 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.500 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.500 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.500 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.500 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:33.500 DEBUG:boto:Data: 2024-04-13T22:30:33.500 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.501 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.501 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.501 DEBUG:boto:Params: {} 2024-04-13T22:30:33.501 DEBUG:boto:Token: None 2024-04-13T22:30:33.501 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33 2024-04-13T22:30:33.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E/ToUP7pE52niLpob7m7U5hhF+k= 2024-04-13T22:30:33.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E/ToUP7pE52niLpob7m7U5hhF+k=', 'Content-Length': '0'} 2024-04-13T22:30:33.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f04d2300ef40566-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.503 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.503 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.503 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.503 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:33.503 DEBUG:boto:Data: 2024-04-13T22:30:33.503 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.503 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.503 DEBUG:boto:Params: {} 2024-04-13T22:30:33.504 DEBUG:boto:Token: None 2024-04-13T22:30:33.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33 2024-04-13T22:30:33.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E/ToUP7pE52niLpob7m7U5hhF+k= 2024-04-13T22:30:33.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E/ToUP7pE52niLpob7m7U5hhF+k=', 'Content-Length': '0'} 2024-04-13T22:30:33.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058fed6eba1a6461a-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:30:33.506 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.506 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.506 DEBUG:boto:Method: GET 2024-04-13T22:30:33.506 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:33.506 DEBUG:boto:Data: 2024-04-13T22:30:33.506 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.507 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.507 DEBUG:boto:Params: {} 2024-04-13T22:30:33.507 DEBUG:boto:Token: None 2024-04-13T22:30:33.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33 2024-04-13T22:30:33.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D21zzzACk33NrFUnywJaz7b8VFI= 2024-04-13T22:30:33.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D21zzzACk33NrFUnywJaz7b8VFI=', 'Content-Length': '0'} 2024-04-13T22:30:33.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724fa1c4cace4f71-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.550 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:33.551 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:33.551 DEBUG:boto:Method: GET 2024-04-13T22:30:33.551 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:33.551 DEBUG:boto:Data: 2024-04-13T22:30:33.551 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.551 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.551 DEBUG:boto:Params: {} 2024-04-13T22:30:33.551 DEBUG:boto:Token: None 2024-04-13T22:30:33.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj33 2024-04-13T22:30:33.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D21zzzACk33NrFUnywJaz7b8VFI= 2024-04-13T22:30:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D21zzzACk33NrFUnywJaz7b8VFI=', 'Content-Length': '0'} 2024-04-13T22:30:33.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1addeefd594be4-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.554 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.554 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.554 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.554 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.554 DEBUG:boto:Method: GET 2024-04-13T22:30:33.555 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.555 DEBUG:boto:Data: 2024-04-13T22:30:33.555 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.555 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.555 DEBUG:boto:Params: {} 2024-04-13T22:30:33.555 DEBUG:boto:Token: None 2024-04-13T22:30:33.555 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000551ff57cf5dd4b0b-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.558 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.558 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.558 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.558 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.558 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.558 DEBUG:boto:Method: GET 2024-04-13T22:30:33.559 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:33.559 DEBUG:boto:Data: 2024-04-13T22:30:33.559 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.559 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.559 DEBUG:boto:Params: {} 2024-04-13T22:30:33.559 DEBUG:boto:Token: None 2024-04-13T22:30:33.559 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b59e4c1480c36ac-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.563 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:30:33.563 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.563 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.564 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.564 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.564 DEBUG:boto:Method: GET 2024-04-13T22:30:33.564 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.564 DEBUG:boto:Data: 2024-04-13T22:30:33.564 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.564 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.564 DEBUG:boto:Params: {} 2024-04-13T22:30:33.564 DEBUG:boto:Token: None 2024-04-13T22:30:33.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8= 2024-04-13T22:30:33.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8=', 'Content-Length': '0'} 2024-04-13T22:30:33.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991d73c8d4c5bbb3-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.566 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.567 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.567 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.567 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.567 DEBUG:boto:Method: GET 2024-04-13T22:30:33.567 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.567 DEBUG:boto:Data: 2024-04-13T22:30:33.567 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.567 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.567 DEBUG:boto:Params: {} 2024-04-13T22:30:33.568 DEBUG:boto:Token: None 2024-04-13T22:30:33.568 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8= 2024-04-13T22:30:33.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8=', 'Content-Length': '0'} 2024-04-13T22:30:33.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3d885a26932c0f-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.570 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.570 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.570 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.570 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.570 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.571 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.571 DEBUG:boto:Data: 2024-04-13T22:30:33.571 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.571 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.571 DEBUG:boto:Params: {} 2024-04-13T22:30:33.571 DEBUG:boto:Token: None 2024-04-13T22:30:33.571 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.571 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yZSVIX3J8/EgUDK91uJh98ZiFe4= 2024-04-13T22:30:33.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yZSVIX3J8/EgUDK91uJh98ZiFe4=', 'Content-Length': '0'} 2024-04-13T22:30:33.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c62318d93cfdc5-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.573 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.574 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.574 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.574 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.574 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.574 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.574 DEBUG:boto:Data: 2024-04-13T22:30:33.574 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.574 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.574 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.574 DEBUG:boto:Params: {} 2024-04-13T22:30:33.575 DEBUG:boto:Token: None 2024-04-13T22:30:33.575 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yZSVIX3J8/EgUDK91uJh98ZiFe4= 2024-04-13T22:30:33.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yZSVIX3J8/EgUDK91uJh98ZiFe4=', 'Content-Length': '0'} 2024-04-13T22:30:33.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc33a1095257377b-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:30:33.577 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.577 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.578 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.578 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.578 DEBUG:boto:Method: GET 2024-04-13T22:30:33.578 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.578 DEBUG:boto:Data: 2024-04-13T22:30:33.578 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.578 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.578 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.578 DEBUG:boto:Params: {} 2024-04-13T22:30:33.578 DEBUG:boto:Token: None 2024-04-13T22:30:33.579 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.579 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8= 2024-04-13T22:30:33.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8=', 'Content-Length': '0'} 2024-04-13T22:30:33.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1af4d7471527928-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.580 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.580 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.581 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.581 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.581 DEBUG:boto:Method: GET 2024-04-13T22:30:33.581 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.581 DEBUG:boto:Data: 2024-04-13T22:30:33.581 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.581 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.581 DEBUG:boto:Params: {} 2024-04-13T22:30:33.581 DEBUG:boto:Token: None 2024-04-13T22:30:33.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:33.582 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8= 2024-04-13T22:30:33.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:J/Aid/ocOakDiK4g06Tu5VHO1l8=', 'Content-Length': '0'} 2024-04-13T22:30:33.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b456e16496765ab8-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.584 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.584 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.584 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.584 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.584 DEBUG:boto:Method: GET 2024-04-13T22:30:33.584 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.584 DEBUG:boto:Data: 2024-04-13T22:30:33.584 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.585 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.585 DEBUG:boto:Params: {} 2024-04-13T22:30:33.585 DEBUG:boto:Token: None 2024-04-13T22:30:33.585 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efe9c9ac7bcb5996-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.587 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.588 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.588 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.588 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.588 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.588 DEBUG:boto:Method: GET 2024-04-13T22:30:33.588 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.588 DEBUG:boto:Data: 2024-04-13T22:30:33.588 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.588 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.589 DEBUG:boto:Params: {} 2024-04-13T22:30:33.589 DEBUG:boto:Token: None 2024-04-13T22:30:33.589 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057fc880b289e96ac-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.593 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.593 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.593 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.593 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.593 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:33.593 DEBUG:boto:Data: 2024-04-13T22:30:33.594 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.594 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.594 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.594 DEBUG:boto:Params: {} 2024-04-13T22:30:33.594 DEBUG:boto:Token: None 2024-04-13T22:30:33.594 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34 2024-04-13T22:30:33.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QXFUygcZDCpMx9FUGqhFEHGeMlM= 2024-04-13T22:30:33.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QXFUygcZDCpMx9FUGqhFEHGeMlM=', 'Content-Length': '0'} 2024-04-13T22:30:33.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfbdc24a27496cf7-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.596 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.596 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.596 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.596 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:33.596 DEBUG:boto:Data: 2024-04-13T22:30:33.597 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.597 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.597 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.597 DEBUG:boto:Params: {} 2024-04-13T22:30:33.597 DEBUG:boto:Token: None 2024-04-13T22:30:33.597 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34 2024-04-13T22:30:33.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QXFUygcZDCpMx9FUGqhFEHGeMlM= 2024-04-13T22:30:33.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QXFUygcZDCpMx9FUGqhFEHGeMlM=', 'Content-Length': '0'} 2024-04-13T22:30:33.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57a7a7c0218995d-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:30:33.599 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.600 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.600 DEBUG:boto:Method: GET 2024-04-13T22:30:33.600 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:33.600 DEBUG:boto:Data: 2024-04-13T22:30:33.600 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.600 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.600 DEBUG:boto:Params: {} 2024-04-13T22:30:33.600 DEBUG:boto:Token: None 2024-04-13T22:30:33.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34 2024-04-13T22:30:33.601 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXv+OqJ30eL5mnXa0+e/PrKo1ZE= 2024-04-13T22:30:33.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXv+OqJ30eL5mnXa0+e/PrKo1ZE=', 'Content-Length': '0'} 2024-04-13T22:30:33.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ab24986d32a25f-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.646 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:33.646 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:33.647 DEBUG:boto:Method: GET 2024-04-13T22:30:33.647 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:33.647 DEBUG:boto:Data: 2024-04-13T22:30:33.647 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.647 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.647 DEBUG:boto:Params: {} 2024-04-13T22:30:33.648 DEBUG:boto:Token: None 2024-04-13T22:30:33.648 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj34 2024-04-13T22:30:33.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXv+OqJ30eL5mnXa0+e/PrKo1ZE= 2024-04-13T22:30:33.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXv+OqJ30eL5mnXa0+e/PrKo1ZE=', 'Content-Length': '0'} 2024-04-13T22:30:33.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e76adbde943542-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.650 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.650 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.650 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.650 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.650 DEBUG:boto:Method: GET 2024-04-13T22:30:33.650 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.650 DEBUG:boto:Data: 2024-04-13T22:30:33.650 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.650 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.650 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.650 DEBUG:boto:Params: {} 2024-04-13T22:30:33.651 DEBUG:boto:Token: None 2024-04-13T22:30:33.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005def920f3eff8dee-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.653 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.653 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.654 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.654 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.654 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.654 DEBUG:boto:Method: GET 2024-04-13T22:30:33.654 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:33.654 DEBUG:boto:Data: 2024-04-13T22:30:33.654 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.654 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.654 DEBUG:boto:Params: {} 2024-04-13T22:30:33.654 DEBUG:boto:Token: None 2024-04-13T22:30:33.655 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb38f50be3df5a69-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.658 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-13T22:30:33.659 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.659 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.659 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.659 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.659 DEBUG:boto:Method: GET 2024-04-13T22:30:33.659 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.659 DEBUG:boto:Data: 2024-04-13T22:30:33.660 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.660 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.660 DEBUG:boto:Params: {} 2024-04-13T22:30:33.660 DEBUG:boto:Token: None 2024-04-13T22:30:33.660 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw= 2024-04-13T22:30:33.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw=', 'Content-Length': '0'} 2024-04-13T22:30:33.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43dd3d8fd4ab1eb-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.663 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.663 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.663 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.663 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.663 DEBUG:boto:Method: GET 2024-04-13T22:30:33.663 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.663 DEBUG:boto:Data: 2024-04-13T22:30:33.663 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.664 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.664 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.664 DEBUG:boto:Params: {} 2024-04-13T22:30:33.664 DEBUG:boto:Token: None 2024-04-13T22:30:33.664 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw= 2024-04-13T22:30:33.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw=', 'Content-Length': '0'} 2024-04-13T22:30:33.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d143273dda81790f-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.706 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.706 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.707 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.707 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.707 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.707 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.707 DEBUG:boto:Data: 2024-04-13T22:30:33.707 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.707 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.707 DEBUG:boto:Params: {} 2024-04-13T22:30:33.707 DEBUG:boto:Token: None 2024-04-13T22:30:33.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2UPxfu6kkcot/brlh5D/r1Q4OC8= 2024-04-13T22:30:33.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2UPxfu6kkcot/brlh5D/r1Q4OC8=', 'Content-Length': '0'} 2024-04-13T22:30:33.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e6a0a1e4ddbb4d-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.709 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.710 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.710 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.710 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.710 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.710 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.710 DEBUG:boto:Data: 2024-04-13T22:30:33.710 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.710 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.710 DEBUG:boto:Params: {} 2024-04-13T22:30:33.711 DEBUG:boto:Token: None 2024-04-13T22:30:33.711 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2UPxfu6kkcot/brlh5D/r1Q4OC8= 2024-04-13T22:30:33.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2UPxfu6kkcot/brlh5D/r1Q4OC8=', 'Content-Length': '0'} 2024-04-13T22:30:33.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eca53c7cf5bb3801-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-13T22:30:33.713 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.713 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.713 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.713 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.713 DEBUG:boto:Method: GET 2024-04-13T22:30:33.713 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.713 DEBUG:boto:Data: 2024-04-13T22:30:33.713 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.713 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.713 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.714 DEBUG:boto:Params: {} 2024-04-13T22:30:33.714 DEBUG:boto:Token: None 2024-04-13T22:30:33.714 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw= 2024-04-13T22:30:33.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw=', 'Content-Length': '0'} 2024-04-13T22:30:33.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056cd1a6e8bd18f12-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.716 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.716 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.716 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.716 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.716 DEBUG:boto:Method: GET 2024-04-13T22:30:33.716 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.716 DEBUG:boto:Data: 2024-04-13T22:30:33.716 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.716 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.717 DEBUG:boto:Params: {} 2024-04-13T22:30:33.717 DEBUG:boto:Token: None 2024-04-13T22:30:33.717 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:33.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw= 2024-04-13T22:30:33.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VN1j0f3PVf3+CMZomo3j5Et/9Yw=', 'Content-Length': '0'} 2024-04-13T22:30:33.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006893058376539605-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.719 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.719 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.719 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.719 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.719 DEBUG:boto:Method: GET 2024-04-13T22:30:33.719 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.719 DEBUG:boto:Data: 2024-04-13T22:30:33.719 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.719 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.720 DEBUG:boto:Params: {} 2024-04-13T22:30:33.720 DEBUG:boto:Token: None 2024-04-13T22:30:33.720 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7ae532decf7f29b-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.722 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.723 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.723 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.723 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.723 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.723 DEBUG:boto:Method: GET 2024-04-13T22:30:33.723 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.723 DEBUG:boto:Data: 2024-04-13T22:30:33.723 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.724 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.724 DEBUG:boto:Params: {} 2024-04-13T22:30:33.724 DEBUG:boto:Token: None 2024-04-13T22:30:33.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbfa800125ece501-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.728 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.728 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.728 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.728 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.728 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:33.729 DEBUG:boto:Data: 2024-04-13T22:30:33.729 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.729 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.729 DEBUG:boto:Params: {} 2024-04-13T22:30:33.729 DEBUG:boto:Token: None 2024-04-13T22:30:33.729 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37 2024-04-13T22:30:33.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g3g7lChMSdiQ9YcHmIWCRBuPl58= 2024-04-13T22:30:33.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g3g7lChMSdiQ9YcHmIWCRBuPl58=', 'Content-Length': '0'} 2024-04-13T22:30:33.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b99760361c007ed0-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.731 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.731 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.732 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.732 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:33.732 DEBUG:boto:Data: 2024-04-13T22:30:33.732 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.732 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.732 DEBUG:boto:Params: {} 2024-04-13T22:30:33.732 DEBUG:boto:Token: None 2024-04-13T22:30:33.732 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37 2024-04-13T22:30:33.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:g3g7lChMSdiQ9YcHmIWCRBuPl58= 2024-04-13T22:30:33.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g3g7lChMSdiQ9YcHmIWCRBuPl58=', 'Content-Length': '0'} 2024-04-13T22:30:33.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f65f0a99d9e9ef0-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-13T22:30:33.734 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.734 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.734 DEBUG:boto:Method: GET 2024-04-13T22:30:33.735 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:33.735 DEBUG:boto:Data: 2024-04-13T22:30:33.735 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.735 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.735 DEBUG:boto:Params: {} 2024-04-13T22:30:33.735 DEBUG:boto:Token: None 2024-04-13T22:30:33.735 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37 2024-04-13T22:30:33.735 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/XafrkgPs3yx6sXUn1CXitk/RU= 2024-04-13T22:30:33.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/XafrkgPs3yx6sXUn1CXitk/RU=', 'Content-Length': '0'} 2024-04-13T22:30:33.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906a955c493fe1ea-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.778 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:33.778 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:33.778 DEBUG:boto:Method: GET 2024-04-13T22:30:33.779 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:33.779 DEBUG:boto:Data: 2024-04-13T22:30:33.779 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.779 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.779 DEBUG:boto:Params: {} 2024-04-13T22:30:33.779 DEBUG:boto:Token: None 2024-04-13T22:30:33.779 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj37 2024-04-13T22:30:33.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C/XafrkgPs3yx6sXUn1CXitk/RU= 2024-04-13T22:30:33.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C/XafrkgPs3yx6sXUn1CXitk/RU=', 'Content-Length': '0'} 2024-04-13T22:30:33.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a3223e85af4107-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.781 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.781 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.782 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.782 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.782 DEBUG:boto:Method: GET 2024-04-13T22:30:33.782 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.782 DEBUG:boto:Data: 2024-04-13T22:30:33.782 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.782 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.782 DEBUG:boto:Params: {} 2024-04-13T22:30:33.782 DEBUG:boto:Token: None 2024-04-13T22:30:33.782 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f3dd1db0a55ca8-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.785 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.785 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.785 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.785 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.785 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.785 DEBUG:boto:Method: GET 2024-04-13T22:30:33.786 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:33.786 DEBUG:boto:Data: 2024-04-13T22:30:33.786 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.786 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.786 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.786 DEBUG:boto:Params: {} 2024-04-13T22:30:33.786 DEBUG:boto:Token: None 2024-04-13T22:30:33.786 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007092b7381e2c59e6-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.790 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:30:33.791 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.791 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.791 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.791 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.791 DEBUG:boto:Method: GET 2024-04-13T22:30:33.791 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.791 DEBUG:boto:Data: 2024-04-13T22:30:33.791 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.791 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.791 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.791 DEBUG:boto:Params: {} 2024-04-13T22:30:33.792 DEBUG:boto:Token: None 2024-04-13T22:30:33.792 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ= 2024-04-13T22:30:33.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ=', 'Content-Length': '0'} 2024-04-13T22:30:33.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe632c5e48964f9-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.794 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.794 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.794 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.794 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.794 DEBUG:boto:Method: GET 2024-04-13T22:30:33.794 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.794 DEBUG:boto:Data: 2024-04-13T22:30:33.794 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.795 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.795 DEBUG:boto:Params: {} 2024-04-13T22:30:33.795 DEBUG:boto:Token: None 2024-04-13T22:30:33.795 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.795 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ= 2024-04-13T22:30:33.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ=', 'Content-Length': '0'} 2024-04-13T22:30:33.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174505200b9334c4-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.797 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.797 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.797 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.797 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.797 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.798 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.798 DEBUG:boto:Data: 2024-04-13T22:30:33.798 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.798 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.798 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.798 DEBUG:boto:Params: {} 2024-04-13T22:30:33.798 DEBUG:boto:Token: None 2024-04-13T22:30:33.798 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.798 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yeRoAgSjB5vI1uGNhesszxezcII= 2024-04-13T22:30:33.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yeRoAgSjB5vI1uGNhesszxezcII=', 'Content-Length': '0'} 2024-04-13T22:30:33.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020c754fafa1b64d3-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.800 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.800 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.800 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.801 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.801 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.801 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.801 DEBUG:boto:Data: 2024-04-13T22:30:33.801 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.801 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.801 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.801 DEBUG:boto:Params: {} 2024-04-13T22:30:33.801 DEBUG:boto:Token: None 2024-04-13T22:30:33.801 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.801 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yeRoAgSjB5vI1uGNhesszxezcII= 2024-04-13T22:30:33.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yeRoAgSjB5vI1uGNhesszxezcII=', 'Content-Length': '0'} 2024-04-13T22:30:33.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49a8d508b007555-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:30:33.803 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.804 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.804 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.804 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.804 DEBUG:boto:Method: GET 2024-04-13T22:30:33.804 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.804 DEBUG:boto:Data: 2024-04-13T22:30:33.804 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.804 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.804 DEBUG:boto:Params: {} 2024-04-13T22:30:33.805 DEBUG:boto:Token: None 2024-04-13T22:30:33.805 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.805 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ= 2024-04-13T22:30:33.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ=', 'Content-Length': '0'} 2024-04-13T22:30:33.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9c7393126106ef-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.807 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.807 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.807 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.807 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.807 DEBUG:boto:Method: GET 2024-04-13T22:30:33.807 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.807 DEBUG:boto:Data: 2024-04-13T22:30:33.807 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.808 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.808 DEBUG:boto:Params: {} 2024-04-13T22:30:33.808 DEBUG:boto:Token: None 2024-04-13T22:30:33.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:33.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ= 2024-04-13T22:30:33.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IXcPPcwJDncpUUVhKJr/BiQ0PGQ=', 'Content-Length': '0'} 2024-04-13T22:30:33.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f0492636e75600-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.810 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.810 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.810 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.811 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.811 DEBUG:boto:Method: GET 2024-04-13T22:30:33.811 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.811 DEBUG:boto:Data: 2024-04-13T22:30:33.811 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.811 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.811 DEBUG:boto:Params: {} 2024-04-13T22:30:33.811 DEBUG:boto:Token: None 2024-04-13T22:30:33.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091e7584d8f11901a-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.814 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.814 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.814 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.815 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.815 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.815 DEBUG:boto:Method: GET 2024-04-13T22:30:33.815 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.815 DEBUG:boto:Data: 2024-04-13T22:30:33.815 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.815 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.815 DEBUG:boto:Params: {} 2024-04-13T22:30:33.816 DEBUG:boto:Token: None 2024-04-13T22:30:33.816 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005471066b3ef63ac9-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.819 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.820 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.820 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.820 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.820 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:33.820 DEBUG:boto:Data: 2024-04-13T22:30:33.820 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.820 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.821 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.821 DEBUG:boto:Params: {} 2024-04-13T22:30:33.821 DEBUG:boto:Token: None 2024-04-13T22:30:33.821 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38 2024-04-13T22:30:33.821 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ancZKPnNvhkbQRzs8cEa3cdz0Gg= 2024-04-13T22:30:33.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ancZKPnNvhkbQRzs8cEa3cdz0Gg=', 'Content-Length': '0'} 2024-04-13T22:30:33.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100964437105d81c-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.823 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.823 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.823 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.823 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:33.823 DEBUG:boto:Data: 2024-04-13T22:30:33.823 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.823 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.824 DEBUG:boto:Params: {} 2024-04-13T22:30:33.824 DEBUG:boto:Token: None 2024-04-13T22:30:33.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38 2024-04-13T22:30:33.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ancZKPnNvhkbQRzs8cEa3cdz0Gg= 2024-04-13T22:30:33.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ancZKPnNvhkbQRzs8cEa3cdz0Gg=', 'Content-Length': '0'} 2024-04-13T22:30:33.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a382c486ae605749-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:30:33.826 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.826 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.827 DEBUG:boto:Method: GET 2024-04-13T22:30:33.827 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:33.827 DEBUG:boto:Data: 2024-04-13T22:30:33.827 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.827 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.827 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.827 DEBUG:boto:Params: {} 2024-04-13T22:30:33.827 DEBUG:boto:Token: None 2024-04-13T22:30:33.827 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38 2024-04-13T22:30:33.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LFCyRR3b+oPXy4b1XUHrDKTKR7E= 2024-04-13T22:30:33.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LFCyRR3b+oPXy4b1XUHrDKTKR7E=', 'Content-Length': '0'} 2024-04-13T22:30:33.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6e9a1b3ad50eb3-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.870 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:33.871 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:33.871 DEBUG:boto:Method: GET 2024-04-13T22:30:33.871 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:33.871 DEBUG:boto:Data: 2024-04-13T22:30:33.871 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.871 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.871 DEBUG:boto:Params: {} 2024-04-13T22:30:33.871 DEBUG:boto:Token: None 2024-04-13T22:30:33.871 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj38 2024-04-13T22:30:33.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LFCyRR3b+oPXy4b1XUHrDKTKR7E= 2024-04-13T22:30:33.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LFCyRR3b+oPXy4b1XUHrDKTKR7E=', 'Content-Length': '0'} 2024-04-13T22:30:33.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000075380d1d911ce0-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.874 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.874 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.874 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.874 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.874 DEBUG:boto:Method: GET 2024-04-13T22:30:33.874 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.874 DEBUG:boto:Data: 2024-04-13T22:30:33.874 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.875 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.875 DEBUG:boto:Params: {} 2024-04-13T22:30:33.875 DEBUG:boto:Token: None 2024-04-13T22:30:33.875 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.875 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f307b4dd31df6c8-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.877 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.878 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.878 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.878 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.878 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.878 DEBUG:boto:Method: GET 2024-04-13T22:30:33.878 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:33.879 DEBUG:boto:Data: 2024-04-13T22:30:33.879 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.879 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.879 DEBUG:boto:Params: {} 2024-04-13T22:30:33.879 DEBUG:boto:Token: None 2024-04-13T22:30:33.879 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.879 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfd53b6a110d7e3f-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.883 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:30:33.884 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.884 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.884 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.884 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.884 DEBUG:boto:Method: GET 2024-04-13T22:30:33.884 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.884 DEBUG:boto:Data: 2024-04-13T22:30:33.884 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.884 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.884 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.884 DEBUG:boto:Params: {} 2024-04-13T22:30:33.885 DEBUG:boto:Token: None 2024-04-13T22:30:33.885 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.885 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI= 2024-04-13T22:30:33.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI=', 'Content-Length': '0'} 2024-04-13T22:30:33.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a27e8bf1ba3b2bf-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.887 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.887 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.887 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.887 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.887 DEBUG:boto:Method: GET 2024-04-13T22:30:33.887 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.887 DEBUG:boto:Data: 2024-04-13T22:30:33.887 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.887 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.887 DEBUG:boto:Params: {} 2024-04-13T22:30:33.888 DEBUG:boto:Token: None 2024-04-13T22:30:33.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI= 2024-04-13T22:30:33.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI=', 'Content-Length': '0'} 2024-04-13T22:30:33.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98cc7228c9bf92b-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.930 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.931 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.931 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.931 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.931 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.931 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.931 DEBUG:boto:Data: 2024-04-13T22:30:33.931 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.931 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.931 DEBUG:boto:Params: {} 2024-04-13T22:30:33.931 DEBUG:boto:Token: None 2024-04-13T22:30:33.932 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sFdTTntTFEfchbNjPf7QgcOCsxs= 2024-04-13T22:30:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sFdTTntTFEfchbNjPf7QgcOCsxs=', 'Content-Length': '0'} 2024-04-13T22:30:33.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041040fbddf3bce8f-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.934 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.934 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.934 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.934 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.934 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.934 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.934 DEBUG:boto:Data: 2024-04-13T22:30:33.934 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.935 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.935 DEBUG:boto:Params: {} 2024-04-13T22:30:33.935 DEBUG:boto:Token: None 2024-04-13T22:30:33.935 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sFdTTntTFEfchbNjPf7QgcOCsxs= 2024-04-13T22:30:33.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sFdTTntTFEfchbNjPf7QgcOCsxs=', 'Content-Length': '0'} 2024-04-13T22:30:33.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000040517d744ac2eeb-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:30:33.937 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.937 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.938 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.938 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.938 DEBUG:boto:Method: GET 2024-04-13T22:30:33.938 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.938 DEBUG:boto:Data: 2024-04-13T22:30:33.938 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.938 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.938 DEBUG:boto:Params: {} 2024-04-13T22:30:33.938 DEBUG:boto:Token: None 2024-04-13T22:30:33.939 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI= 2024-04-13T22:30:33.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI=', 'Content-Length': '0'} 2024-04-13T22:30:33.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de0fb3943987203-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.940 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.940 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.941 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.941 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.941 DEBUG:boto:Method: GET 2024-04-13T22:30:33.941 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.941 DEBUG:boto:Data: 2024-04-13T22:30:33.941 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.941 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.941 DEBUG:boto:Params: {} 2024-04-13T22:30:33.941 DEBUG:boto:Token: None 2024-04-13T22:30:33.942 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:33.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI= 2024-04-13T22:30:33.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TmzDvYP9owRg18HL1upZCXQxEoI=', 'Content-Length': '0'} 2024-04-13T22:30:33.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000735b02dceac7a973-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.944 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.944 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.944 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:33.944 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:33.944 DEBUG:boto:Method: GET 2024-04-13T22:30:33.944 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:33.944 DEBUG:boto:Data: 2024-04-13T22:30:33.944 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.945 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.945 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.945 DEBUG:boto:Params: {} 2024-04-13T22:30:33.945 DEBUG:boto:Token: None 2024-04-13T22:30:33.945 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.945 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d646b80aa0a1fed0-00661b0789-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.947 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.948 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:33.948 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:33.948 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:33.948 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:33.948 DEBUG:boto:Method: GET 2024-04-13T22:30:33.948 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:33.948 DEBUG:boto:Data: 2024-04-13T22:30:33.948 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.949 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.949 DEBUG:boto:Params: {} 2024-04-13T22:30:33.949 DEBUG:boto:Token: None 2024-04-13T22:30:33.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/ 2024-04-13T22:30:33.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4= 2024-04-13T22:30:33.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjmMWDuuyBDzuOG7HnnlJxVXKQ4=', 'Content-Length': '0'} 2024-04-13T22:30:33.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33cce22ed124428-00661b0789-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.953 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:33.953 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.953 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.953 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.953 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:33.953 DEBUG:boto:Data: 2024-04-13T22:30:33.953 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.953 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.953 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.953 DEBUG:boto:Params: {} 2024-04-13T22:30:33.954 DEBUG:boto:Token: None 2024-04-13T22:30:33.954 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44 2024-04-13T22:30:33.954 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qdHRCtJBrXY1tgtxP/Yu0oxxoh0= 2024-04-13T22:30:33.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qdHRCtJBrXY1tgtxP/Yu0oxxoh0=', 'Content-Length': '0'} 2024-04-13T22:30:33.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe2340778055035-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.956 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.956 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.956 DEBUG:boto:Method: HEAD 2024-04-13T22:30:33.956 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:33.956 DEBUG:boto:Data: 2024-04-13T22:30:33.956 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.956 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.956 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.957 DEBUG:boto:Params: {} 2024-04-13T22:30:33.957 DEBUG:boto:Token: None 2024-04-13T22:30:33.957 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44 2024-04-13T22:30:33.957 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qdHRCtJBrXY1tgtxP/Yu0oxxoh0= 2024-04-13T22:30:33.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qdHRCtJBrXY1tgtxP/Yu0oxxoh0=', 'Content-Length': '0'} 2024-04-13T22:30:33.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a93ab5c075fcc5eb-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:30:33.959 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.959 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.959 DEBUG:boto:Method: GET 2024-04-13T22:30:33.960 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:33.960 DEBUG:boto:Data: 2024-04-13T22:30:33.960 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.960 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:33.960 DEBUG:boto:Port: 8001 2024-04-13T22:30:33.960 DEBUG:boto:Params: {} 2024-04-13T22:30:33.960 DEBUG:boto:Token: None 2024-04-13T22:30:33.960 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44 2024-04-13T22:30:33.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2cqYkP+IBw/dCoBtBLqTyJ4/FSc= 2024-04-13T22:30:33.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2cqYkP+IBw/dCoBtBLqTyJ4/FSc=', 'Content-Length': '0'} 2024-04-13T22:30:33.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f29f37899b8c499-00661b0789-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:33.963 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:33.963 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:33.963 DEBUG:boto:Method: GET 2024-04-13T22:30:33.963 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:33.963 DEBUG:boto:Data: 2024-04-13T22:30:33.963 DEBUG:boto:Headers: {} 2024-04-13T22:30:33.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:33.963 DEBUG:boto:Port: 8000 2024-04-13T22:30:33.963 DEBUG:boto:Params: {} 2024-04-13T22:30:33.963 DEBUG:boto:Token: None 2024-04-13T22:30:33.964 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:33 GMT /vwnfns-61/obj44 2024-04-13T22:30:33.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2cqYkP+IBw/dCoBtBLqTyJ4/FSc= 2024-04-13T22:30:33.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:33 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2cqYkP+IBw/dCoBtBLqTyJ4/FSc=', 'Content-Length': '0'} 2024-04-13T22:30:33.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ec339abaaf2c575-00661b0789-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.006 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.006 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.007 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:34.007 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:34.007 DEBUG:boto:Method: GET 2024-04-13T22:30:34.007 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:34.007 DEBUG:boto:Data: 2024-04-13T22:30:34.007 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.007 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.007 DEBUG:boto:Params: {} 2024-04-13T22:30:34.007 DEBUG:boto:Token: None 2024-04-13T22:30:34.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.008 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e4369bb9ed6e87-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.010 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.010 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.010 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.010 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:34.010 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:34.011 DEBUG:boto:Method: GET 2024-04-13T22:30:34.011 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:34.011 DEBUG:boto:Data: 2024-04-13T22:30:34.011 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.011 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.011 DEBUG:boto:Params: {} 2024-04-13T22:30:34.011 DEBUG:boto:Token: None 2024-04-13T22:30:34.011 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062d5b0efeb516159-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.015 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:30:34.016 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.016 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.016 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.016 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.016 DEBUG:boto:Method: GET 2024-04-13T22:30:34.016 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.016 DEBUG:boto:Data: 2024-04-13T22:30:34.016 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.016 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.016 DEBUG:boto:Params: {} 2024-04-13T22:30:34.017 DEBUG:boto:Token: None 2024-04-13T22:30:34.017 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg= 2024-04-13T22:30:34.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg=', 'Content-Length': '0'} 2024-04-13T22:30:34.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325086d74047f7b3-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.019 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.019 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.019 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.019 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.019 DEBUG:boto:Method: GET 2024-04-13T22:30:34.020 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.020 DEBUG:boto:Data: 2024-04-13T22:30:34.020 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.020 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.020 DEBUG:boto:Params: {} 2024-04-13T22:30:34.020 DEBUG:boto:Token: None 2024-04-13T22:30:34.020 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg= 2024-04-13T22:30:34.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg=', 'Content-Length': '0'} 2024-04-13T22:30:34.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c6b4514db69091-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.023 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.023 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.023 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.023 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.023 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.023 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.023 DEBUG:boto:Data: 2024-04-13T22:30:34.023 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.024 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.024 DEBUG:boto:Params: {} 2024-04-13T22:30:34.024 DEBUG:boto:Token: None 2024-04-13T22:30:34.024 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GHogRBr7gxYp4SFJEWG1dW00mXU= 2024-04-13T22:30:34.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GHogRBr7gxYp4SFJEWG1dW00mXU=', 'Content-Length': '0'} 2024-04-13T22:30:34.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e15e84d88368810-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.026 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.026 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.026 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.026 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.026 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.026 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.026 DEBUG:boto:Data: 2024-04-13T22:30:34.026 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.027 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.027 DEBUG:boto:Params: {} 2024-04-13T22:30:34.027 DEBUG:boto:Token: None 2024-04-13T22:30:34.027 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GHogRBr7gxYp4SFJEWG1dW00mXU= 2024-04-13T22:30:34.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GHogRBr7gxYp4SFJEWG1dW00mXU=', 'Content-Length': '0'} 2024-04-13T22:30:34.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fc4f541387134bc-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:30:34.029 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.029 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.029 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.029 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.029 DEBUG:boto:Method: GET 2024-04-13T22:30:34.029 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.029 DEBUG:boto:Data: 2024-04-13T22:30:34.029 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.030 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.030 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.030 DEBUG:boto:Params: {} 2024-04-13T22:30:34.030 DEBUG:boto:Token: None 2024-04-13T22:30:34.030 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg= 2024-04-13T22:30:34.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg=', 'Content-Length': '0'} 2024-04-13T22:30:34.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cd2ed76e4825ba-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.074 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.075 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.075 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.075 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.075 DEBUG:boto:Method: GET 2024-04-13T22:30:34.075 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.075 DEBUG:boto:Data: 2024-04-13T22:30:34.075 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.075 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.075 DEBUG:boto:Params: {} 2024-04-13T22:30:34.075 DEBUG:boto:Token: None 2024-04-13T22:30:34.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:34.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg= 2024-04-13T22:30:34.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nr1JA0pjqpnwudyycMkilDCcUvg=', 'Content-Length': '0'} 2024-04-13T22:30:34.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a10a5310d593624-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.078 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.078 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.078 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.078 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.078 DEBUG:boto:Method: GET 2024-04-13T22:30:34.078 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.078 DEBUG:boto:Data: 2024-04-13T22:30:34.078 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.078 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.079 DEBUG:boto:Params: {} 2024-04-13T22:30:34.079 DEBUG:boto:Token: None 2024-04-13T22:30:34.079 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b286487c5206de2-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.081 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.082 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.082 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.082 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.082 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.082 DEBUG:boto:Method: GET 2024-04-13T22:30:34.082 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.082 DEBUG:boto:Data: 2024-04-13T22:30:34.082 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.082 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.082 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.083 DEBUG:boto:Params: {} 2024-04-13T22:30:34.083 DEBUG:boto:Token: None 2024-04-13T22:30:34.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000282a4375bcbd4cf5-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.086 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.087 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.087 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.087 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.087 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:34.087 DEBUG:boto:Data: 2024-04-13T22:30:34.087 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.087 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.088 DEBUG:boto:Params: {} 2024-04-13T22:30:34.088 DEBUG:boto:Token: None 2024-04-13T22:30:34.088 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46 2024-04-13T22:30:34.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JrzCmN/QZCGgEESrb0BuB+L5bq0= 2024-04-13T22:30:34.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JrzCmN/QZCGgEESrb0BuB+L5bq0=', 'Content-Length': '0'} 2024-04-13T22:30:34.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004477759d6bb8a334-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.090 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.090 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.090 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.090 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:34.090 DEBUG:boto:Data: 2024-04-13T22:30:34.090 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.090 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.090 DEBUG:boto:Params: {} 2024-04-13T22:30:34.091 DEBUG:boto:Token: None 2024-04-13T22:30:34.091 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46 2024-04-13T22:30:34.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JrzCmN/QZCGgEESrb0BuB+L5bq0= 2024-04-13T22:30:34.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JrzCmN/QZCGgEESrb0BuB+L5bq0=', 'Content-Length': '0'} 2024-04-13T22:30:34.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d2472c1734bbae-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:30:34.093 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.093 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.093 DEBUG:boto:Method: GET 2024-04-13T22:30:34.093 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:34.093 DEBUG:boto:Data: 2024-04-13T22:30:34.093 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.093 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.093 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.093 DEBUG:boto:Params: {} 2024-04-13T22:30:34.094 DEBUG:boto:Token: None 2024-04-13T22:30:34.094 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46 2024-04-13T22:30:34.094 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KfkHDc2zqYGBjc1Xu2uijXHeEM4= 2024-04-13T22:30:34.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KfkHDc2zqYGBjc1Xu2uijXHeEM4=', 'Content-Length': '0'} 2024-04-13T22:30:34.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03e60f0fda66cb2-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.096 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:34.096 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:34.096 DEBUG:boto:Method: GET 2024-04-13T22:30:34.096 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:34.096 DEBUG:boto:Data: 2024-04-13T22:30:34.096 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.096 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.096 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.096 DEBUG:boto:Params: {} 2024-04-13T22:30:34.097 DEBUG:boto:Token: None 2024-04-13T22:30:34.097 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj46 2024-04-13T22:30:34.097 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KfkHDc2zqYGBjc1Xu2uijXHeEM4= 2024-04-13T22:30:34.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KfkHDc2zqYGBjc1Xu2uijXHeEM4=', 'Content-Length': '0'} 2024-04-13T22:30:34.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083603062b292b645-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.099 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.099 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.099 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.099 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.099 DEBUG:boto:Method: GET 2024-04-13T22:30:34.099 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.099 DEBUG:boto:Data: 2024-04-13T22:30:34.099 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.099 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.100 DEBUG:boto:Params: {} 2024-04-13T22:30:34.100 DEBUG:boto:Token: None 2024-04-13T22:30:34.100 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002265067e238700ff-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.102 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.103 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.103 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.103 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.103 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.103 DEBUG:boto:Method: GET 2024-04-13T22:30:34.103 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:34.103 DEBUG:boto:Data: 2024-04-13T22:30:34.103 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.103 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.104 DEBUG:boto:Params: {} 2024-04-13T22:30:34.104 DEBUG:boto:Token: None 2024-04-13T22:30:34.104 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.104 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17d5bc31434a653-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.108 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:30:34.108 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.108 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.109 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.109 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.109 DEBUG:boto:Method: GET 2024-04-13T22:30:34.109 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.109 DEBUG:boto:Data: 2024-04-13T22:30:34.109 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.109 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.109 DEBUG:boto:Params: {} 2024-04-13T22:30:34.109 DEBUG:boto:Token: None 2024-04-13T22:30:34.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY= 2024-04-13T22:30:34.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY=', 'Content-Length': '0'} 2024-04-13T22:30:34.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c30ba4e556dcdd2-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.111 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.112 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.112 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.112 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.112 DEBUG:boto:Method: GET 2024-04-13T22:30:34.112 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.112 DEBUG:boto:Data: 2024-04-13T22:30:34.112 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.112 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.112 DEBUG:boto:Params: {} 2024-04-13T22:30:34.112 DEBUG:boto:Token: None 2024-04-13T22:30:34.113 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.113 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY= 2024-04-13T22:30:34.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY=', 'Content-Length': '0'} 2024-04-13T22:30:34.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f607873cf867ef4-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.158 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.159 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.159 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.159 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.159 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.159 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.159 DEBUG:boto:Data: 2024-04-13T22:30:34.159 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.159 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.159 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.160 DEBUG:boto:Params: {} 2024-04-13T22:30:34.160 DEBUG:boto:Token: None 2024-04-13T22:30:34.160 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.160 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jkGwAH2s2uE5gSBrYsqwhxDO3AQ= 2024-04-13T22:30:34.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jkGwAH2s2uE5gSBrYsqwhxDO3AQ=', 'Content-Length': '0'} 2024-04-13T22:30:34.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b87fa954a8903c02-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.162 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.163 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.163 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.163 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.163 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.163 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.163 DEBUG:boto:Data: 2024-04-13T22:30:34.163 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.163 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.163 DEBUG:boto:Params: {} 2024-04-13T22:30:34.164 DEBUG:boto:Token: None 2024-04-13T22:30:34.164 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jkGwAH2s2uE5gSBrYsqwhxDO3AQ= 2024-04-13T22:30:34.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jkGwAH2s2uE5gSBrYsqwhxDO3AQ=', 'Content-Length': '0'} 2024-04-13T22:30:34.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db69e0411a5fc8fc-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:30:34.166 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.166 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.166 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.166 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.167 DEBUG:boto:Method: GET 2024-04-13T22:30:34.167 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.167 DEBUG:boto:Data: 2024-04-13T22:30:34.167 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.167 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.167 DEBUG:boto:Params: {} 2024-04-13T22:30:34.167 DEBUG:boto:Token: None 2024-04-13T22:30:34.167 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.167 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY= 2024-04-13T22:30:34.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY=', 'Content-Length': '0'} 2024-04-13T22:30:34.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007af523807c08cb02-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.169 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.169 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.169 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.169 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.170 DEBUG:boto:Method: GET 2024-04-13T22:30:34.170 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.170 DEBUG:boto:Data: 2024-04-13T22:30:34.170 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.170 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.170 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.170 DEBUG:boto:Params: {} 2024-04-13T22:30:34.170 DEBUG:boto:Token: None 2024-04-13T22:30:34.170 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:34.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY= 2024-04-13T22:30:34.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8L2GrRwN7OSeii7p1/e/2a/avJY=', 'Content-Length': '0'} 2024-04-13T22:30:34.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d0f9cb2c3c5ec2-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.172 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.173 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.173 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.173 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.173 DEBUG:boto:Method: GET 2024-04-13T22:30:34.173 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.173 DEBUG:boto:Data: 2024-04-13T22:30:34.173 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.173 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.173 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.173 DEBUG:boto:Params: {} 2024-04-13T22:30:34.174 DEBUG:boto:Token: None 2024-04-13T22:30:34.174 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.174 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dcdeb9b5d16fd42-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.176 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.177 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.177 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.177 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.177 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.177 DEBUG:boto:Method: GET 2024-04-13T22:30:34.177 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.177 DEBUG:boto:Data: 2024-04-13T22:30:34.177 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.177 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.178 DEBUG:boto:Params: {} 2024-04-13T22:30:34.178 DEBUG:boto:Token: None 2024-04-13T22:30:34.178 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023fc91562561ad11-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.182 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.182 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.182 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.182 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.182 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:34.182 DEBUG:boto:Data: 2024-04-13T22:30:34.183 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.183 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.183 DEBUG:boto:Params: {} 2024-04-13T22:30:34.183 DEBUG:boto:Token: None 2024-04-13T22:30:34.183 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51 2024-04-13T22:30:34.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iCvbRlaRDkxcuYP2sTpUk5YEgZw= 2024-04-13T22:30:34.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iCvbRlaRDkxcuYP2sTpUk5YEgZw=', 'Content-Length': '0'} 2024-04-13T22:30:34.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f305e134426fd9-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.185 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.185 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.185 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.185 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:34.186 DEBUG:boto:Data: 2024-04-13T22:30:34.186 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.186 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.186 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.186 DEBUG:boto:Params: {} 2024-04-13T22:30:34.186 DEBUG:boto:Token: None 2024-04-13T22:30:34.187 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51 2024-04-13T22:30:34.187 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iCvbRlaRDkxcuYP2sTpUk5YEgZw= 2024-04-13T22:30:34.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iCvbRlaRDkxcuYP2sTpUk5YEgZw=', 'Content-Length': '0'} 2024-04-13T22:30:34.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81ff2e7928c7e10-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:30:34.189 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.189 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.189 DEBUG:boto:Method: GET 2024-04-13T22:30:34.189 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:34.189 DEBUG:boto:Data: 2024-04-13T22:30:34.189 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.189 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.189 DEBUG:boto:Params: {} 2024-04-13T22:30:34.190 DEBUG:boto:Token: None 2024-04-13T22:30:34.190 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51 2024-04-13T22:30:34.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4u6BchC3BnLDNQu9kMcHuETgWFc= 2024-04-13T22:30:34.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4u6BchC3BnLDNQu9kMcHuETgWFc=', 'Content-Length': '0'} 2024-04-13T22:30:34.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b145a447e2596f-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.192 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:34.192 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:34.192 DEBUG:boto:Method: GET 2024-04-13T22:30:34.192 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:34.192 DEBUG:boto:Data: 2024-04-13T22:30:34.192 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.192 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.192 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.193 DEBUG:boto:Params: {} 2024-04-13T22:30:34.193 DEBUG:boto:Token: None 2024-04-13T22:30:34.193 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj51 2024-04-13T22:30:34.193 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4u6BchC3BnLDNQu9kMcHuETgWFc= 2024-04-13T22:30:34.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4u6BchC3BnLDNQu9kMcHuETgWFc=', 'Content-Length': '0'} 2024-04-13T22:30:34.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097da203203894bc1-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.195 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.195 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.195 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.195 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.195 DEBUG:boto:Method: GET 2024-04-13T22:30:34.195 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.195 DEBUG:boto:Data: 2024-04-13T22:30:34.195 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.195 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.195 DEBUG:boto:Params: {} 2024-04-13T22:30:34.198 DEBUG:boto:Token: None 2024-04-13T22:30:34.198 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c883e9a476b41537-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.201 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.201 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.201 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.201 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.201 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.201 DEBUG:boto:Method: GET 2024-04-13T22:30:34.201 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:34.201 DEBUG:boto:Data: 2024-04-13T22:30:34.202 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.202 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.202 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.202 DEBUG:boto:Params: {} 2024-04-13T22:30:34.202 DEBUG:boto:Token: None 2024-04-13T22:30:34.202 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.202 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd338eccb30cf42-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.206 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:30:34.206 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.207 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.207 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.207 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.207 DEBUG:boto:Method: GET 2024-04-13T22:30:34.207 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.207 DEBUG:boto:Data: 2024-04-13T22:30:34.207 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.207 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.207 DEBUG:boto:Params: {} 2024-04-13T22:30:34.207 DEBUG:boto:Token: None 2024-04-13T22:30:34.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0= 2024-04-13T22:30:34.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0=', 'Content-Length': '0'} 2024-04-13T22:30:34.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3490844326f7f3-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.210 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.210 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.210 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.210 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.210 DEBUG:boto:Method: GET 2024-04-13T22:30:34.210 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.210 DEBUG:boto:Data: 2024-04-13T22:30:34.210 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.211 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.211 DEBUG:boto:Params: {} 2024-04-13T22:30:34.211 DEBUG:boto:Token: None 2024-04-13T22:30:34.211 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.211 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0= 2024-04-13T22:30:34.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0=', 'Content-Length': '0'} 2024-04-13T22:30:34.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011861990afd8a8e4-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.213 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.213 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.213 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.213 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.214 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.214 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.214 DEBUG:boto:Data: 2024-04-13T22:30:34.214 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.214 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.214 DEBUG:boto:Params: {} 2024-04-13T22:30:34.214 DEBUG:boto:Token: None 2024-04-13T22:30:34.214 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.214 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qZnisz/p+4OhRhLerrcmWkm7BqU= 2024-04-13T22:30:34.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qZnisz/p+4OhRhLerrcmWkm7BqU=', 'Content-Length': '0'} 2024-04-13T22:30:34.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086711b3e3da4ef5d-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.216 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.216 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.216 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.216 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.216 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.217 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.217 DEBUG:boto:Data: 2024-04-13T22:30:34.217 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.217 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.217 DEBUG:boto:Params: {} 2024-04-13T22:30:34.217 DEBUG:boto:Token: None 2024-04-13T22:30:34.217 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qZnisz/p+4OhRhLerrcmWkm7BqU= 2024-04-13T22:30:34.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qZnisz/p+4OhRhLerrcmWkm7BqU=', 'Content-Length': '0'} 2024-04-13T22:30:34.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129406841e1fc2dc-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:30:34.219 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.219 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.219 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.219 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.219 DEBUG:boto:Method: GET 2024-04-13T22:30:34.219 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.220 DEBUG:boto:Data: 2024-04-13T22:30:34.220 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.220 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.220 DEBUG:boto:Params: {} 2024-04-13T22:30:34.220 DEBUG:boto:Token: None 2024-04-13T22:30:34.221 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0= 2024-04-13T22:30:34.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0=', 'Content-Length': '0'} 2024-04-13T22:30:34.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a849b3bd5c4a6570-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.223 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.223 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.223 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.223 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.223 DEBUG:boto:Method: GET 2024-04-13T22:30:34.223 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.223 DEBUG:boto:Data: 2024-04-13T22:30:34.223 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.224 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.224 DEBUG:boto:Params: {} 2024-04-13T22:30:34.224 DEBUG:boto:Token: None 2024-04-13T22:30:34.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:34.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0= 2024-04-13T22:30:34.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:76MXTOu8y1jHUGP07pAottpOoo0=', 'Content-Length': '0'} 2024-04-13T22:30:34.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad0dc8e413d1a42-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.266 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.267 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.267 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.267 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.267 DEBUG:boto:Method: GET 2024-04-13T22:30:34.267 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.267 DEBUG:boto:Data: 2024-04-13T22:30:34.267 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.267 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.267 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.267 DEBUG:boto:Params: {} 2024-04-13T22:30:34.267 DEBUG:boto:Token: None 2024-04-13T22:30:34.268 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.268 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b4877bbd86b674-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.270 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.271 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.271 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.271 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.271 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.271 DEBUG:boto:Method: GET 2024-04-13T22:30:34.271 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.271 DEBUG:boto:Data: 2024-04-13T22:30:34.271 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.271 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.271 DEBUG:boto:Params: {} 2024-04-13T22:30:34.272 DEBUG:boto:Token: None 2024-04-13T22:30:34.272 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008acd9845545bacdc-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.275 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.276 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.276 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.276 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.276 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:34.276 DEBUG:boto:Data: 2024-04-13T22:30:34.276 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.276 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.276 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.276 DEBUG:boto:Params: {} 2024-04-13T22:30:34.277 DEBUG:boto:Token: None 2024-04-13T22:30:34.277 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55 2024-04-13T22:30:34.277 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S227hvo6TEmkQcik+n05hscIYsU= 2024-04-13T22:30:34.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S227hvo6TEmkQcik+n05hscIYsU=', 'Content-Length': '0'} 2024-04-13T22:30:34.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5f90e01ac52e56-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.278 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.279 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.279 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.279 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:34.279 DEBUG:boto:Data: 2024-04-13T22:30:34.279 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.279 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.279 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.279 DEBUG:boto:Params: {} 2024-04-13T22:30:34.279 DEBUG:boto:Token: None 2024-04-13T22:30:34.279 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55 2024-04-13T22:30:34.280 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S227hvo6TEmkQcik+n05hscIYsU= 2024-04-13T22:30:34.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S227hvo6TEmkQcik+n05hscIYsU=', 'Content-Length': '0'} 2024-04-13T22:30:34.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef6973f42ad5283d-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:30:34.282 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.283 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.283 DEBUG:boto:Method: GET 2024-04-13T22:30:34.283 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:34.283 DEBUG:boto:Data: 2024-04-13T22:30:34.283 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.283 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.283 DEBUG:boto:Params: {} 2024-04-13T22:30:34.283 DEBUG:boto:Token: None 2024-04-13T22:30:34.283 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55 2024-04-13T22:30:34.284 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tFAIw51YL1L8cy1n4WkXE9s/wT0= 2024-04-13T22:30:34.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tFAIw51YL1L8cy1n4WkXE9s/wT0=', 'Content-Length': '0'} 2024-04-13T22:30:34.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009babd094eb68f54a-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.286 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:34.286 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:34.286 DEBUG:boto:Method: GET 2024-04-13T22:30:34.286 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:34.286 DEBUG:boto:Data: 2024-04-13T22:30:34.286 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.286 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.286 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.286 DEBUG:boto:Params: {} 2024-04-13T22:30:34.287 DEBUG:boto:Token: None 2024-04-13T22:30:34.287 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj55 2024-04-13T22:30:34.287 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tFAIw51YL1L8cy1n4WkXE9s/wT0= 2024-04-13T22:30:34.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tFAIw51YL1L8cy1n4WkXE9s/wT0=', 'Content-Length': '0'} 2024-04-13T22:30:34.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce20cc2c5e03b1d-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.289 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.289 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.289 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.289 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.289 DEBUG:boto:Method: GET 2024-04-13T22:30:34.289 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.289 DEBUG:boto:Data: 2024-04-13T22:30:34.289 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.289 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.289 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.290 DEBUG:boto:Params: {} 2024-04-13T22:30:34.290 DEBUG:boto:Token: None 2024-04-13T22:30:34.290 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.290 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008968d591dd99cab1-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.293 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.293 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.293 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.293 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.293 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.294 DEBUG:boto:Method: GET 2024-04-13T22:30:34.294 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:34.294 DEBUG:boto:Data: 2024-04-13T22:30:34.294 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.294 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.294 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.294 DEBUG:boto:Params: {} 2024-04-13T22:30:34.294 DEBUG:boto:Token: None 2024-04-13T22:30:34.294 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e37ef6b5bb1a568-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.298 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:30:34.299 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.299 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.299 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.299 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.299 DEBUG:boto:Method: GET 2024-04-13T22:30:34.299 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.300 DEBUG:boto:Data: 2024-04-13T22:30:34.300 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.300 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.300 DEBUG:boto:Params: {} 2024-04-13T22:30:34.300 DEBUG:boto:Token: None 2024-04-13T22:30:34.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8= 2024-04-13T22:30:34.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8=', 'Content-Length': '0'} 2024-04-13T22:30:34.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ccadd51c1e55f4c-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.303 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.303 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.303 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.303 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.303 DEBUG:boto:Method: GET 2024-04-13T22:30:34.304 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.304 DEBUG:boto:Data: 2024-04-13T22:30:34.304 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.304 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.304 DEBUG:boto:Params: {} 2024-04-13T22:30:34.304 DEBUG:boto:Token: None 2024-04-13T22:30:34.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8= 2024-04-13T22:30:34.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8=', 'Content-Length': '0'} 2024-04-13T22:30:34.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9cc2ff89195c948-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.346 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.347 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.347 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.347 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.347 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.347 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.347 DEBUG:boto:Data: 2024-04-13T22:30:34.347 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.347 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.347 DEBUG:boto:Params: {} 2024-04-13T22:30:34.347 DEBUG:boto:Token: None 2024-04-13T22:30:34.348 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aJVCxuLMTpS0RCzleAR45teBUgU= 2024-04-13T22:30:34.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aJVCxuLMTpS0RCzleAR45teBUgU=', 'Content-Length': '0'} 2024-04-13T22:30:34.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c133d6acdfd9aa06-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.350 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.350 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.350 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.350 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.350 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.350 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.350 DEBUG:boto:Data: 2024-04-13T22:30:34.350 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.351 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.351 DEBUG:boto:Params: {} 2024-04-13T22:30:34.351 DEBUG:boto:Token: None 2024-04-13T22:30:34.351 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aJVCxuLMTpS0RCzleAR45teBUgU= 2024-04-13T22:30:34.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aJVCxuLMTpS0RCzleAR45teBUgU=', 'Content-Length': '0'} 2024-04-13T22:30:34.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000716f42ca395868fc-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:30:34.353 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.353 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.353 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.353 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.353 DEBUG:boto:Method: GET 2024-04-13T22:30:34.354 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.354 DEBUG:boto:Data: 2024-04-13T22:30:34.354 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.354 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.354 DEBUG:boto:Params: {} 2024-04-13T22:30:34.354 DEBUG:boto:Token: None 2024-04-13T22:30:34.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8= 2024-04-13T22:30:34.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8=', 'Content-Length': '0'} 2024-04-13T22:30:34.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69e2a9f198b6da6-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.356 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.356 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.356 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.357 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.357 DEBUG:boto:Method: GET 2024-04-13T22:30:34.357 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.357 DEBUG:boto:Data: 2024-04-13T22:30:34.357 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.357 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.357 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.357 DEBUG:boto:Params: {} 2024-04-13T22:30:34.357 DEBUG:boto:Token: None 2024-04-13T22:30:34.357 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:34.358 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8= 2024-04-13T22:30:34.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aIHqEg+wWU2WkClFXSxhrOu08Y8=', 'Content-Length': '0'} 2024-04-13T22:30:34.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028da5a678b61c59e-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.360 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.360 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.360 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.360 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.360 DEBUG:boto:Method: GET 2024-04-13T22:30:34.361 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.361 DEBUG:boto:Data: 2024-04-13T22:30:34.361 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.361 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.361 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.361 DEBUG:boto:Params: {} 2024-04-13T22:30:34.361 DEBUG:boto:Token: None 2024-04-13T22:30:34.361 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.361 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b29579909c6cd9a3-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.364 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.364 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.364 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.364 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.364 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.364 DEBUG:boto:Method: GET 2024-04-13T22:30:34.365 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.365 DEBUG:boto:Data: 2024-04-13T22:30:34.365 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.365 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.365 DEBUG:boto:Params: {} 2024-04-13T22:30:34.365 DEBUG:boto:Token: None 2024-04-13T22:30:34.365 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.365 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a078b9d429ffe51-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.369 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.369 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.369 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.370 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.370 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:34.370 DEBUG:boto:Data: 2024-04-13T22:30:34.370 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.370 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.370 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.370 DEBUG:boto:Params: {} 2024-04-13T22:30:34.370 DEBUG:boto:Token: None 2024-04-13T22:30:34.370 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57 2024-04-13T22:30:34.371 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXlKxbtGEUiU7T7BXT196ngZO0A= 2024-04-13T22:30:34.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXlKxbtGEUiU7T7BXT196ngZO0A=', 'Content-Length': '0'} 2024-04-13T22:30:34.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001726c54408272b0d-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.372 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.372 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.372 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.373 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:34.373 DEBUG:boto:Data: 2024-04-13T22:30:34.373 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.373 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.373 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.373 DEBUG:boto:Params: {} 2024-04-13T22:30:34.373 DEBUG:boto:Token: None 2024-04-13T22:30:34.373 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57 2024-04-13T22:30:34.373 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MXlKxbtGEUiU7T7BXT196ngZO0A= 2024-04-13T22:30:34.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MXlKxbtGEUiU7T7BXT196ngZO0A=', 'Content-Length': '0'} 2024-04-13T22:30:34.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331fa3f7f80e056c-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:30:34.375 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.375 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.375 DEBUG:boto:Method: GET 2024-04-13T22:30:34.375 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:34.375 DEBUG:boto:Data: 2024-04-13T22:30:34.376 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.376 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.376 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.376 DEBUG:boto:Params: {} 2024-04-13T22:30:34.376 DEBUG:boto:Token: None 2024-04-13T22:30:34.376 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57 2024-04-13T22:30:34.376 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K5eaJ9TjIC7pNvSiPqM/9SI+unY= 2024-04-13T22:30:34.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K5eaJ9TjIC7pNvSiPqM/9SI+unY=', 'Content-Length': '0'} 2024-04-13T22:30:34.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187a34a81c458b51-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.418 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:34.418 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:34.419 DEBUG:boto:Method: GET 2024-04-13T22:30:34.419 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:34.419 DEBUG:boto:Data: 2024-04-13T22:30:34.419 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.419 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.419 DEBUG:boto:Params: {} 2024-04-13T22:30:34.419 DEBUG:boto:Token: None 2024-04-13T22:30:34.419 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj57 2024-04-13T22:30:34.420 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K5eaJ9TjIC7pNvSiPqM/9SI+unY= 2024-04-13T22:30:34.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K5eaJ9TjIC7pNvSiPqM/9SI+unY=', 'Content-Length': '0'} 2024-04-13T22:30:34.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf865b685dda0e5-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.422 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.422 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.422 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.422 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.422 DEBUG:boto:Method: GET 2024-04-13T22:30:34.422 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.422 DEBUG:boto:Data: 2024-04-13T22:30:34.422 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.423 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.423 DEBUG:boto:Params: {} 2024-04-13T22:30:34.423 DEBUG:boto:Token: None 2024-04-13T22:30:34.423 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.423 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068853a8fe37c5114-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.426 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.426 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.426 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.426 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.426 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.426 DEBUG:boto:Method: GET 2024-04-13T22:30:34.427 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:34.427 DEBUG:boto:Data: 2024-04-13T22:30:34.427 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.427 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.427 DEBUG:boto:Params: {} 2024-04-13T22:30:34.427 DEBUG:boto:Token: None 2024-04-13T22:30:34.427 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092e985caf6f9efea-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.431 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:30:34.431 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.431 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.431 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.431 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.432 DEBUG:boto:Method: GET 2024-04-13T22:30:34.432 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.432 DEBUG:boto:Data: 2024-04-13T22:30:34.432 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.432 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.432 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.432 DEBUG:boto:Params: {} 2024-04-13T22:30:34.432 DEBUG:boto:Token: None 2024-04-13T22:30:34.432 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.432 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc= 2024-04-13T22:30:34.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc=', 'Content-Length': '0'} 2024-04-13T22:30:34.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693ac1767bfd43ef-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.434 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.434 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.434 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.435 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.435 DEBUG:boto:Method: GET 2024-04-13T22:30:34.435 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.435 DEBUG:boto:Data: 2024-04-13T22:30:34.435 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.435 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.435 DEBUG:boto:Params: {} 2024-04-13T22:30:34.435 DEBUG:boto:Token: None 2024-04-13T22:30:34.435 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc= 2024-04-13T22:30:34.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc=', 'Content-Length': '0'} 2024-04-13T22:30:34.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595a15a0ed1d05c9-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.478 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.478 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.479 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.479 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.479 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.479 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.479 DEBUG:boto:Data: 2024-04-13T22:30:34.479 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.479 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.479 DEBUG:boto:Params: {} 2024-04-13T22:30:34.479 DEBUG:boto:Token: None 2024-04-13T22:30:34.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QGIMYpBcQkZE6GA7GRdOTnG750A= 2024-04-13T22:30:34.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QGIMYpBcQkZE6GA7GRdOTnG750A=', 'Content-Length': '0'} 2024-04-13T22:30:34.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8566ca0e29589fe-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.481 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.481 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.481 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.482 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.482 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.482 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.482 DEBUG:boto:Data: 2024-04-13T22:30:34.482 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.482 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.482 DEBUG:boto:Params: {} 2024-04-13T22:30:34.482 DEBUG:boto:Token: None 2024-04-13T22:30:34.482 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QGIMYpBcQkZE6GA7GRdOTnG750A= 2024-04-13T22:30:34.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QGIMYpBcQkZE6GA7GRdOTnG750A=', 'Content-Length': '0'} 2024-04-13T22:30:34.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cc1e6f2094e746-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:30:34.484 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.485 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.485 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.485 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.485 DEBUG:boto:Method: GET 2024-04-13T22:30:34.485 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.485 DEBUG:boto:Data: 2024-04-13T22:30:34.485 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.485 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.485 DEBUG:boto:Params: {} 2024-04-13T22:30:34.485 DEBUG:boto:Token: None 2024-04-13T22:30:34.486 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc= 2024-04-13T22:30:34.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc=', 'Content-Length': '0'} 2024-04-13T22:30:34.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6c2bcda7b2330c-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.487 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.488 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.488 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.488 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.488 DEBUG:boto:Method: GET 2024-04-13T22:30:34.488 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.488 DEBUG:boto:Data: 2024-04-13T22:30:34.488 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.488 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.488 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.488 DEBUG:boto:Params: {} 2024-04-13T22:30:34.489 DEBUG:boto:Token: None 2024-04-13T22:30:34.489 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:34.489 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc= 2024-04-13T22:30:34.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Nw+Qa8M2IDAG7Y1OwI0rIspLJPc=', 'Content-Length': '0'} 2024-04-13T22:30:34.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f638ab6fac1fad12-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.491 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.491 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.491 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.491 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.491 DEBUG:boto:Method: GET 2024-04-13T22:30:34.491 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.491 DEBUG:boto:Data: 2024-04-13T22:30:34.491 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.491 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.491 DEBUG:boto:Params: {} 2024-04-13T22:30:34.492 DEBUG:boto:Token: None 2024-04-13T22:30:34.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579fdf6ba38c39b2-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.494 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.494 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.495 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.495 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.495 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.495 DEBUG:boto:Method: GET 2024-04-13T22:30:34.495 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.495 DEBUG:boto:Data: 2024-04-13T22:30:34.495 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.495 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.495 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.495 DEBUG:boto:Params: {} 2024-04-13T22:30:34.495 DEBUG:boto:Token: None 2024-04-13T22:30:34.496 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.496 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a013cc086725d0e-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.500 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.500 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.500 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.500 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.500 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:34.500 DEBUG:boto:Data: 2024-04-13T22:30:34.500 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.500 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.501 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.501 DEBUG:boto:Params: {} 2024-04-13T22:30:34.501 DEBUG:boto:Token: None 2024-04-13T22:30:34.501 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58 2024-04-13T22:30:34.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OROP5EOK/gu7Fe6zJOjij9NSlho= 2024-04-13T22:30:34.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OROP5EOK/gu7Fe6zJOjij9NSlho=', 'Content-Length': '0'} 2024-04-13T22:30:34.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3d1f493caeb8d8-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.503 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.503 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.503 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.503 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:34.503 DEBUG:boto:Data: 2024-04-13T22:30:34.503 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.503 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.503 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.503 DEBUG:boto:Params: {} 2024-04-13T22:30:34.504 DEBUG:boto:Token: None 2024-04-13T22:30:34.504 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58 2024-04-13T22:30:34.504 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OROP5EOK/gu7Fe6zJOjij9NSlho= 2024-04-13T22:30:34.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OROP5EOK/gu7Fe6zJOjij9NSlho=', 'Content-Length': '0'} 2024-04-13T22:30:34.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f50f1d2d92637f2-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:30:34.506 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.506 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.506 DEBUG:boto:Method: GET 2024-04-13T22:30:34.506 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:34.506 DEBUG:boto:Data: 2024-04-13T22:30:34.506 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.506 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.506 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.506 DEBUG:boto:Params: {} 2024-04-13T22:30:34.507 DEBUG:boto:Token: None 2024-04-13T22:30:34.507 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58 2024-04-13T22:30:34.507 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tLP99FYPGHJgA/x6PoLr+ozIKMI= 2024-04-13T22:30:34.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tLP99FYPGHJgA/x6PoLr+ozIKMI=', 'Content-Length': '0'} 2024-04-13T22:30:34.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1216ab1ad4564de-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.509 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:34.509 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:34.509 DEBUG:boto:Method: GET 2024-04-13T22:30:34.509 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:34.509 DEBUG:boto:Data: 2024-04-13T22:30:34.509 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.509 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.510 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.510 DEBUG:boto:Params: {} 2024-04-13T22:30:34.510 DEBUG:boto:Token: None 2024-04-13T22:30:34.510 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj58 2024-04-13T22:30:34.510 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tLP99FYPGHJgA/x6PoLr+ozIKMI= 2024-04-13T22:30:34.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tLP99FYPGHJgA/x6PoLr+ozIKMI=', 'Content-Length': '0'} 2024-04-13T22:30:34.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e484d62a080122ad-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.554 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.555 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.555 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.555 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.555 DEBUG:boto:Method: GET 2024-04-13T22:30:34.555 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.555 DEBUG:boto:Data: 2024-04-13T22:30:34.555 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.555 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.555 DEBUG:boto:Params: {} 2024-04-13T22:30:34.556 DEBUG:boto:Token: None 2024-04-13T22:30:34.556 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.556 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039d56b5feda3e85e-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.558 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.559 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.559 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.559 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.559 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.559 DEBUG:boto:Method: GET 2024-04-13T22:30:34.559 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:34.559 DEBUG:boto:Data: 2024-04-13T22:30:34.559 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.560 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.560 DEBUG:boto:Params: {} 2024-04-13T22:30:34.560 DEBUG:boto:Token: None 2024-04-13T22:30:34.560 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078774630db6720f9-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.564 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:30:34.564 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.564 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.564 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.564 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.564 DEBUG:boto:Method: GET 2024-04-13T22:30:34.565 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.565 DEBUG:boto:Data: 2024-04-13T22:30:34.565 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.565 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.565 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.565 DEBUG:boto:Params: {} 2024-04-13T22:30:34.565 DEBUG:boto:Token: None 2024-04-13T22:30:34.565 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo= 2024-04-13T22:30:34.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo=', 'Content-Length': '0'} 2024-04-13T22:30:34.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea396c411739729-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.610 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.611 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.611 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.611 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.611 DEBUG:boto:Method: GET 2024-04-13T22:30:34.611 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.611 DEBUG:boto:Data: 2024-04-13T22:30:34.611 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.611 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.611 DEBUG:boto:Params: {} 2024-04-13T22:30:34.612 DEBUG:boto:Token: None 2024-04-13T22:30:34.612 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo= 2024-04-13T22:30:34.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo=', 'Content-Length': '0'} 2024-04-13T22:30:34.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006928cada4599e51f-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.614 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.614 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.614 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.614 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.615 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.615 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.615 DEBUG:boto:Data: 2024-04-13T22:30:34.615 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.615 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.615 DEBUG:boto:Params: {} 2024-04-13T22:30:34.615 DEBUG:boto:Token: None 2024-04-13T22:30:34.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vJDD+p21/yPLImWoos1F6uXLq98= 2024-04-13T22:30:34.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vJDD+p21/yPLImWoos1F6uXLq98=', 'Content-Length': '0'} 2024-04-13T22:30:34.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011288385b103a856-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.618 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.618 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.618 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.618 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.618 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.618 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.618 DEBUG:boto:Data: 2024-04-13T22:30:34.619 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.619 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.619 DEBUG:boto:Params: {} 2024-04-13T22:30:34.619 DEBUG:boto:Token: None 2024-04-13T22:30:34.619 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vJDD+p21/yPLImWoos1F6uXLq98= 2024-04-13T22:30:34.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vJDD+p21/yPLImWoos1F6uXLq98=', 'Content-Length': '0'} 2024-04-13T22:30:34.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d11c9258124a83-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:30:34.621 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.621 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.621 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.621 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.621 DEBUG:boto:Method: GET 2024-04-13T22:30:34.621 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.621 DEBUG:boto:Data: 2024-04-13T22:30:34.621 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.622 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.622 DEBUG:boto:Params: {} 2024-04-13T22:30:34.622 DEBUG:boto:Token: None 2024-04-13T22:30:34.622 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo= 2024-04-13T22:30:34.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo=', 'Content-Length': '0'} 2024-04-13T22:30:34.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d4fb749a3d6eda4-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.624 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.624 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.624 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.624 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.624 DEBUG:boto:Method: GET 2024-04-13T22:30:34.625 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.625 DEBUG:boto:Data: 2024-04-13T22:30:34.625 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.625 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.625 DEBUG:boto:Params: {} 2024-04-13T22:30:34.625 DEBUG:boto:Token: None 2024-04-13T22:30:34.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:34.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo= 2024-04-13T22:30:34.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QSiP5v/lpIBsPMZVyAamM27ycvo=', 'Content-Length': '0'} 2024-04-13T22:30:34.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15e2f4b7771180a-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.627 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.628 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.628 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.628 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.628 DEBUG:boto:Method: GET 2024-04-13T22:30:34.628 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.628 DEBUG:boto:Data: 2024-04-13T22:30:34.628 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.628 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.628 DEBUG:boto:Params: {} 2024-04-13T22:30:34.629 DEBUG:boto:Token: None 2024-04-13T22:30:34.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073730c1a3e74cdd3-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.632 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.632 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.632 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.632 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.632 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.632 DEBUG:boto:Method: GET 2024-04-13T22:30:34.632 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.633 DEBUG:boto:Data: 2024-04-13T22:30:34.633 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.633 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.633 DEBUG:boto:Params: {} 2024-04-13T22:30:34.633 DEBUG:boto:Token: None 2024-04-13T22:30:34.633 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d77315e62dfafc-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.637 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.637 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.637 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.637 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.637 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:34.637 DEBUG:boto:Data: 2024-04-13T22:30:34.637 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.637 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.638 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.638 DEBUG:boto:Params: {} 2024-04-13T22:30:34.638 DEBUG:boto:Token: None 2024-04-13T22:30:34.638 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62 2024-04-13T22:30:34.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JdEXuQ9n4HTt5FoVYv216h6DGzM= 2024-04-13T22:30:34.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JdEXuQ9n4HTt5FoVYv216h6DGzM=', 'Content-Length': '0'} 2024-04-13T22:30:34.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62168585e108d27-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.640 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.640 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.640 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.640 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:34.640 DEBUG:boto:Data: 2024-04-13T22:30:34.640 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.640 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.640 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.640 DEBUG:boto:Params: {} 2024-04-13T22:30:34.640 DEBUG:boto:Token: None 2024-04-13T22:30:34.641 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62 2024-04-13T22:30:34.641 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JdEXuQ9n4HTt5FoVYv216h6DGzM= 2024-04-13T22:30:34.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JdEXuQ9n4HTt5FoVYv216h6DGzM=', 'Content-Length': '0'} 2024-04-13T22:30:34.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683b8938986f46fd-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:30:34.643 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.643 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.643 DEBUG:boto:Method: GET 2024-04-13T22:30:34.643 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:34.643 DEBUG:boto:Data: 2024-04-13T22:30:34.643 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.643 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.643 DEBUG:boto:Params: {} 2024-04-13T22:30:34.644 DEBUG:boto:Token: None 2024-04-13T22:30:34.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62 2024-04-13T22:30:34.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BdYi2ypvGHwhSxgGOt+Ah675PSY= 2024-04-13T22:30:34.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BdYi2ypvGHwhSxgGOt+Ah675PSY=', 'Content-Length': '0'} 2024-04-13T22:30:34.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b0221a645f1ec9-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.646 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:34.646 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:34.646 DEBUG:boto:Method: GET 2024-04-13T22:30:34.646 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:34.646 DEBUG:boto:Data: 2024-04-13T22:30:34.646 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.647 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.647 DEBUG:boto:Params: {} 2024-04-13T22:30:34.647 DEBUG:boto:Token: None 2024-04-13T22:30:34.647 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj62 2024-04-13T22:30:34.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BdYi2ypvGHwhSxgGOt+Ah675PSY= 2024-04-13T22:30:34.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BdYi2ypvGHwhSxgGOt+Ah675PSY=', 'Content-Length': '0'} 2024-04-13T22:30:34.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004780eb3c23a50348-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.690 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.691 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.691 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.691 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.691 DEBUG:boto:Method: GET 2024-04-13T22:30:34.691 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.691 DEBUG:boto:Data: 2024-04-13T22:30:34.691 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.691 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.691 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.691 DEBUG:boto:Params: {} 2024-04-13T22:30:34.691 DEBUG:boto:Token: None 2024-04-13T22:30:34.692 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.692 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035996c3180b75515-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.694 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.695 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.695 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.695 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.695 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.695 DEBUG:boto:Method: GET 2024-04-13T22:30:34.695 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:34.695 DEBUG:boto:Data: 2024-04-13T22:30:34.695 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.696 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.696 DEBUG:boto:Params: {} 2024-04-13T22:30:34.696 DEBUG:boto:Token: None 2024-04-13T22:30:34.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018878d6c748dc1ac-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.700 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:30:34.700 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.700 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.700 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.701 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.701 DEBUG:boto:Method: GET 2024-04-13T22:30:34.701 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.701 DEBUG:boto:Data: 2024-04-13T22:30:34.701 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.701 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.701 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.701 DEBUG:boto:Params: {} 2024-04-13T22:30:34.701 DEBUG:boto:Token: None 2024-04-13T22:30:34.701 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.702 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8= 2024-04-13T22:30:34.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8=', 'Content-Length': '0'} 2024-04-13T22:30:34.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1201efaa35e0fc-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.746 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.746 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.747 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.747 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.747 DEBUG:boto:Method: GET 2024-04-13T22:30:34.747 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.747 DEBUG:boto:Data: 2024-04-13T22:30:34.747 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.747 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.747 DEBUG:boto:Params: {} 2024-04-13T22:30:34.747 DEBUG:boto:Token: None 2024-04-13T22:30:34.748 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8= 2024-04-13T22:30:34.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8=', 'Content-Length': '0'} 2024-04-13T22:30:34.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f3d3d405e489cd-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.750 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.750 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.750 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.750 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.750 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.750 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.750 DEBUG:boto:Data: 2024-04-13T22:30:34.751 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.751 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.751 DEBUG:boto:Params: {} 2024-04-13T22:30:34.751 DEBUG:boto:Token: None 2024-04-13T22:30:34.751 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IJe3trtge3RfgJuMzsAiYMbOznc= 2024-04-13T22:30:34.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IJe3trtge3RfgJuMzsAiYMbOznc=', 'Content-Length': '0'} 2024-04-13T22:30:34.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dadd68e590b133d3-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.753 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.753 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.754 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.754 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.754 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.754 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.754 DEBUG:boto:Data: 2024-04-13T22:30:34.754 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.754 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.754 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.754 DEBUG:boto:Params: {} 2024-04-13T22:30:34.754 DEBUG:boto:Token: None 2024-04-13T22:30:34.755 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.755 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IJe3trtge3RfgJuMzsAiYMbOznc= 2024-04-13T22:30:34.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IJe3trtge3RfgJuMzsAiYMbOznc=', 'Content-Length': '0'} 2024-04-13T22:30:34.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1052b56b0c1fa2-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:30:34.757 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.757 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.757 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.757 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.757 DEBUG:boto:Method: GET 2024-04-13T22:30:34.757 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.757 DEBUG:boto:Data: 2024-04-13T22:30:34.757 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.757 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.758 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.758 DEBUG:boto:Params: {} 2024-04-13T22:30:34.758 DEBUG:boto:Token: None 2024-04-13T22:30:34.758 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8= 2024-04-13T22:30:34.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8=', 'Content-Length': '0'} 2024-04-13T22:30:34.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099cc132262ca98c5-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.760 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.760 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.760 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.760 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.761 DEBUG:boto:Method: GET 2024-04-13T22:30:34.761 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.761 DEBUG:boto:Data: 2024-04-13T22:30:34.761 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.761 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.761 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.761 DEBUG:boto:Params: {} 2024-04-13T22:30:34.761 DEBUG:boto:Token: None 2024-04-13T22:30:34.761 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:34.761 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8= 2024-04-13T22:30:34.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bLqyhJTYdZ/YTzWE2okXJ+8ufA8=', 'Content-Length': '0'} 2024-04-13T22:30:34.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fa201110b61ac40-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.764 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.764 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.764 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.764 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.764 DEBUG:boto:Method: GET 2024-04-13T22:30:34.764 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.764 DEBUG:boto:Data: 2024-04-13T22:30:34.764 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.764 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.764 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.764 DEBUG:boto:Params: {} 2024-04-13T22:30:34.765 DEBUG:boto:Token: None 2024-04-13T22:30:34.765 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc3d3b283efca813-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.767 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.767 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.767 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.768 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.768 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.768 DEBUG:boto:Method: GET 2024-04-13T22:30:34.768 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.768 DEBUG:boto:Data: 2024-04-13T22:30:34.768 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.768 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.768 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.768 DEBUG:boto:Params: {} 2024-04-13T22:30:34.768 DEBUG:boto:Token: None 2024-04-13T22:30:34.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.769 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007544873d770550ce-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.773 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.773 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.773 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.773 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.773 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:34.773 DEBUG:boto:Data: 2024-04-13T22:30:34.773 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.774 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.774 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.774 DEBUG:boto:Params: {} 2024-04-13T22:30:34.774 DEBUG:boto:Token: None 2024-04-13T22:30:34.774 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68 2024-04-13T22:30:34.774 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TnDcpgA1W9R9p2Me0AY+KfWrN+I= 2024-04-13T22:30:34.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TnDcpgA1W9R9p2Me0AY+KfWrN+I=', 'Content-Length': '0'} 2024-04-13T22:30:34.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b9464bb487f373-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.776 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.776 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.776 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.776 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:34.776 DEBUG:boto:Data: 2024-04-13T22:30:34.776 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.776 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.777 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.777 DEBUG:boto:Params: {} 2024-04-13T22:30:34.777 DEBUG:boto:Token: None 2024-04-13T22:30:34.777 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68 2024-04-13T22:30:34.777 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TnDcpgA1W9R9p2Me0AY+KfWrN+I= 2024-04-13T22:30:34.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TnDcpgA1W9R9p2Me0AY+KfWrN+I=', 'Content-Length': '0'} 2024-04-13T22:30:34.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429e271dafbcb1f4-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:30:34.779 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.779 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.779 DEBUG:boto:Method: GET 2024-04-13T22:30:34.779 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:34.779 DEBUG:boto:Data: 2024-04-13T22:30:34.779 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.780 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.780 DEBUG:boto:Params: {} 2024-04-13T22:30:34.780 DEBUG:boto:Token: None 2024-04-13T22:30:34.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68 2024-04-13T22:30:34.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Mu14AP4Gf3eJoDmxst6f3Dme0Nw= 2024-04-13T22:30:34.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Mu14AP4Gf3eJoDmxst6f3Dme0Nw=', 'Content-Length': '0'} 2024-04-13T22:30:34.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3071a49c2c1b76-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.782 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:34.782 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:34.782 DEBUG:boto:Method: GET 2024-04-13T22:30:34.782 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:34.782 DEBUG:boto:Data: 2024-04-13T22:30:34.782 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.782 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.783 DEBUG:boto:Params: {} 2024-04-13T22:30:34.783 DEBUG:boto:Token: None 2024-04-13T22:30:34.783 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj68 2024-04-13T22:30:34.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Mu14AP4Gf3eJoDmxst6f3Dme0Nw= 2024-04-13T22:30:34.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Mu14AP4Gf3eJoDmxst6f3Dme0Nw=', 'Content-Length': '0'} 2024-04-13T22:30:34.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a134af0a8ecfab14-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.831 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.831 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.831 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.831 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.831 DEBUG:boto:Method: GET 2024-04-13T22:30:34.831 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.831 DEBUG:boto:Data: 2024-04-13T22:30:34.831 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.832 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.832 DEBUG:boto:Params: {} 2024-04-13T22:30:34.832 DEBUG:boto:Token: None 2024-04-13T22:30:34.832 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.832 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b3341cfd77f3625-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.834 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.835 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.835 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.835 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.835 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.835 DEBUG:boto:Method: GET 2024-04-13T22:30:34.835 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:34.835 DEBUG:boto:Data: 2024-04-13T22:30:34.835 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.836 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.836 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.836 DEBUG:boto:Params: {} 2024-04-13T22:30:34.836 DEBUG:boto:Token: None 2024-04-13T22:30:34.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093444f6186603948-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.840 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:30:34.840 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.840 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.840 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.840 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.840 DEBUG:boto:Method: GET 2024-04-13T22:30:34.841 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.841 DEBUG:boto:Data: 2024-04-13T22:30:34.841 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.841 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.841 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.841 DEBUG:boto:Params: {} 2024-04-13T22:30:34.841 DEBUG:boto:Token: None 2024-04-13T22:30:34.841 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.841 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM= 2024-04-13T22:30:34.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM=', 'Content-Length': '0'} 2024-04-13T22:30:34.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040218dc835419c30-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.886 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.887 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.887 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.887 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.887 DEBUG:boto:Method: GET 2024-04-13T22:30:34.887 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.887 DEBUG:boto:Data: 2024-04-13T22:30:34.887 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.887 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.887 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.887 DEBUG:boto:Params: {} 2024-04-13T22:30:34.888 DEBUG:boto:Token: None 2024-04-13T22:30:34.888 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.888 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM= 2024-04-13T22:30:34.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM=', 'Content-Length': '0'} 2024-04-13T22:30:34.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22a497503d2ce59-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.890 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.890 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.890 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.890 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.890 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.890 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.890 DEBUG:boto:Data: 2024-04-13T22:30:34.890 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.891 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.891 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.891 DEBUG:boto:Params: {} 2024-04-13T22:30:34.891 DEBUG:boto:Token: None 2024-04-13T22:30:34.891 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.891 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DO0Qak8KoVB0HRuincK5KOGSKog= 2024-04-13T22:30:34.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DO0Qak8KoVB0HRuincK5KOGSKog=', 'Content-Length': '0'} 2024-04-13T22:30:34.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd72045a1b32612c-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.893 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.893 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.893 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.893 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.893 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.893 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.893 DEBUG:boto:Data: 2024-04-13T22:30:34.893 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.894 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.894 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.894 DEBUG:boto:Params: {} 2024-04-13T22:30:34.894 DEBUG:boto:Token: None 2024-04-13T22:30:34.894 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.894 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DO0Qak8KoVB0HRuincK5KOGSKog= 2024-04-13T22:30:34.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DO0Qak8KoVB0HRuincK5KOGSKog=', 'Content-Length': '0'} 2024-04-13T22:30:34.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005435549721b5159e-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:30:34.896 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.896 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.896 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.896 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.896 DEBUG:boto:Method: GET 2024-04-13T22:30:34.896 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.896 DEBUG:boto:Data: 2024-04-13T22:30:34.897 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.897 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.897 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.897 DEBUG:boto:Params: {} 2024-04-13T22:30:34.897 DEBUG:boto:Token: None 2024-04-13T22:30:34.897 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.897 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM= 2024-04-13T22:30:34.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM=', 'Content-Length': '0'} 2024-04-13T22:30:34.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd286f5cdec3d95-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.899 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.900 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.900 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.900 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.900 DEBUG:boto:Method: GET 2024-04-13T22:30:34.900 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.900 DEBUG:boto:Data: 2024-04-13T22:30:34.900 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.900 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.900 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.900 DEBUG:boto:Params: {} 2024-04-13T22:30:34.901 DEBUG:boto:Token: None 2024-04-13T22:30:34.901 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:34.901 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM= 2024-04-13T22:30:34.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CA3Z3+/3vvwWWmOkBrjPvJXssEM=', 'Content-Length': '0'} 2024-04-13T22:30:34.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004729cff37781067a-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.903 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.903 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.903 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.903 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.903 DEBUG:boto:Method: GET 2024-04-13T22:30:34.903 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.903 DEBUG:boto:Data: 2024-04-13T22:30:34.904 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.904 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.904 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.904 DEBUG:boto:Params: {} 2024-04-13T22:30:34.904 DEBUG:boto:Token: None 2024-04-13T22:30:34.904 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099321b966c9c65af-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.907 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.907 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.907 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.907 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.907 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.908 DEBUG:boto:Method: GET 2024-04-13T22:30:34.908 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.908 DEBUG:boto:Data: 2024-04-13T22:30:34.908 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.908 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.908 DEBUG:boto:Params: {} 2024-04-13T22:30:34.908 DEBUG:boto:Token: None 2024-04-13T22:30:34.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006073dc8e2b5c1639-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.912 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.913 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.913 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.913 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.913 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:34.913 DEBUG:boto:Data: 2024-04-13T22:30:34.913 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.913 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.913 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.913 DEBUG:boto:Params: {} 2024-04-13T22:30:34.914 DEBUG:boto:Token: None 2024-04-13T22:30:34.914 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69 2024-04-13T22:30:34.914 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MnZ80TEJaVqRkcgAyRGLjh7PzNA= 2024-04-13T22:30:34.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MnZ80TEJaVqRkcgAyRGLjh7PzNA=', 'Content-Length': '0'} 2024-04-13T22:30:34.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3bd0d05d252ed5a-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.916 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.916 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.916 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.916 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:34.916 DEBUG:boto:Data: 2024-04-13T22:30:34.916 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.916 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.916 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.916 DEBUG:boto:Params: {} 2024-04-13T22:30:34.917 DEBUG:boto:Token: None 2024-04-13T22:30:34.917 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69 2024-04-13T22:30:34.917 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MnZ80TEJaVqRkcgAyRGLjh7PzNA= 2024-04-13T22:30:34.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MnZ80TEJaVqRkcgAyRGLjh7PzNA=', 'Content-Length': '0'} 2024-04-13T22:30:34.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007046c31aaa4cb091-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:30:34.919 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.919 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.919 DEBUG:boto:Method: GET 2024-04-13T22:30:34.919 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:34.919 DEBUG:boto:Data: 2024-04-13T22:30:34.919 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.920 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.920 DEBUG:boto:Params: {} 2024-04-13T22:30:34.920 DEBUG:boto:Token: None 2024-04-13T22:30:34.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69 2024-04-13T22:30:34.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qxLOXOKnRSHogNHxqnoqMNMKrfM= 2024-04-13T22:30:34.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qxLOXOKnRSHogNHxqnoqMNMKrfM=', 'Content-Length': '0'} 2024-04-13T22:30:34.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9b9d901aa00282-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.962 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:34.963 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:34.963 DEBUG:boto:Method: GET 2024-04-13T22:30:34.963 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:34.963 DEBUG:boto:Data: 2024-04-13T22:30:34.963 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.963 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.963 DEBUG:boto:Params: {} 2024-04-13T22:30:34.963 DEBUG:boto:Token: None 2024-04-13T22:30:34.963 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj69 2024-04-13T22:30:34.964 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qxLOXOKnRSHogNHxqnoqMNMKrfM= 2024-04-13T22:30:34.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qxLOXOKnRSHogNHxqnoqMNMKrfM=', 'Content-Length': '0'} 2024-04-13T22:30:34.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3e49d215c87b54-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.966 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.966 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.966 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.966 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.966 DEBUG:boto:Method: GET 2024-04-13T22:30:34.966 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.966 DEBUG:boto:Data: 2024-04-13T22:30:34.967 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.967 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.967 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.967 DEBUG:boto:Params: {} 2024-04-13T22:30:34.967 DEBUG:boto:Token: None 2024-04-13T22:30:34.967 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.967 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd1fdbb4f8a5f080-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.970 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.970 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.970 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.970 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.970 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.971 DEBUG:boto:Method: GET 2024-04-13T22:30:34.971 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:34.971 DEBUG:boto:Data: 2024-04-13T22:30:34.971 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.971 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.971 DEBUG:boto:Params: {} 2024-04-13T22:30:34.971 DEBUG:boto:Token: None 2024-04-13T22:30:34.971 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6d0e9469fc640c5-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.975 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-13T22:30:34.975 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:34.976 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:34.976 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.976 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.976 DEBUG:boto:Method: GET 2024-04-13T22:30:34.976 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.976 DEBUG:boto:Data: 2024-04-13T22:30:34.976 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.976 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.976 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.976 DEBUG:boto:Params: {} 2024-04-13T22:30:34.977 DEBUG:boto:Token: None 2024-04-13T22:30:34.977 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.977 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w= 2024-04-13T22:30:34.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w=', 'Content-Length': '0'} 2024-04-13T22:30:34.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028179810a9a312b0-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.979 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:34.979 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:34.979 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.979 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.979 DEBUG:boto:Method: GET 2024-04-13T22:30:34.979 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.979 DEBUG:boto:Data: 2024-04-13T22:30:34.980 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.980 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.980 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.980 DEBUG:boto:Params: {} 2024-04-13T22:30:34.980 DEBUG:boto:Token: None 2024-04-13T22:30:34.980 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w= 2024-04-13T22:30:34.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w=', 'Content-Length': '0'} 2024-04-13T22:30:34.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667e7e9ddb5ce366-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.982 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:34.982 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:34.982 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.982 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.983 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.983 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.983 DEBUG:boto:Data: 2024-04-13T22:30:34.983 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.983 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.983 DEBUG:boto:Params: {} 2024-04-13T22:30:34.983 DEBUG:boto:Token: None 2024-04-13T22:30:34.983 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rgud4DEQcrWH29kTFyrPhYMA6kM= 2024-04-13T22:30:34.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rgud4DEQcrWH29kTFyrPhYMA6kM=', 'Content-Length': '0'} 2024-04-13T22:30:34.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1806a92a5ab1cd1-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.985 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:34.986 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:34.986 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.986 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.986 DEBUG:boto:Method: HEAD 2024-04-13T22:30:34.986 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.986 DEBUG:boto:Data: 2024-04-13T22:30:34.986 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.986 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.986 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.986 DEBUG:boto:Params: {} 2024-04-13T22:30:34.987 DEBUG:boto:Token: None 2024-04-13T22:30:34.987 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.987 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rgud4DEQcrWH29kTFyrPhYMA6kM= 2024-04-13T22:30:34.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rgud4DEQcrWH29kTFyrPhYMA6kM=', 'Content-Length': '0'} 2024-04-13T22:30:34.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15f73f27def4e50-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-13T22:30:34.989 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:34.989 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:34.989 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.989 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.989 DEBUG:boto:Method: GET 2024-04-13T22:30:34.989 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.989 DEBUG:boto:Data: 2024-04-13T22:30:34.990 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.990 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.990 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.990 DEBUG:boto:Params: {} 2024-04-13T22:30:34.990 DEBUG:boto:Token: None 2024-04-13T22:30:34.990 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.990 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w= 2024-04-13T22:30:34.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w=', 'Content-Length': '0'} 2024-04-13T22:30:34.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001315eee13d94c6e9-00661b078a-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.992 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:34.992 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:34.992 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.992 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.992 DEBUG:boto:Method: GET 2024-04-13T22:30:34.993 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.993 DEBUG:boto:Data: 2024-04-13T22:30:34.993 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.993 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:34.993 DEBUG:boto:Port: 8000 2024-04-13T22:30:34.993 DEBUG:boto:Params: {} 2024-04-13T22:30:34.993 DEBUG:boto:Token: None 2024-04-13T22:30:34.993 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:34.993 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w= 2024-04-13T22:30:34.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MD7S+foU+NL7MdSWnHSg46OQh2w=', 'Content-Length': '0'} 2024-04-13T22:30:34.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab8b6ac1ee12b8d-00661b078a-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.995 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.995 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.995 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:34.996 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:34.996 DEBUG:boto:Method: GET 2024-04-13T22:30:34.996 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:34.996 DEBUG:boto:Data: 2024-04-13T22:30:34.996 DEBUG:boto:Headers: {} 2024-04-13T22:30:34.996 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:34.996 DEBUG:boto:Port: 8001 2024-04-13T22:30:34.996 DEBUG:boto:Params: {} 2024-04-13T22:30:34.996 DEBUG:boto:Token: None 2024-04-13T22:30:34.996 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:34 GMT /vwnfns-61/ 2024-04-13T22:30:34.996 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA= 2024-04-13T22:30:34.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:34 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5dKHr+6aocqEglQto4nuYrqvYRA=', 'Content-Length': '0'} 2024-04-13T22:30:34.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585c10ffa7514b3b-00661b078a-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:34.999 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:34.999 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:34.999 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:34.999 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:34.999 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.000 DEBUG:boto:Method: GET 2024-04-13T22:30:35.000 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.000 DEBUG:boto:Data: 2024-04-13T22:30:35.000 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.000 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.000 DEBUG:boto:Params: {} 2024-04-13T22:30:35.000 DEBUG:boto:Token: None 2024-04-13T22:30:35.000 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e6dfb9a33bf4a1-00661b078a-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.005 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.005 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:35.005 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:35.005 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.005 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:35.005 DEBUG:boto:Data: 2024-04-13T22:30:35.005 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.006 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.006 DEBUG:boto:Params: {} 2024-04-13T22:30:35.006 DEBUG:boto:Token: None 2024-04-13T22:30:35.006 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj74 2024-04-13T22:30:35.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVOnphK4OTxkOy4mdxDgvttGiJs= 2024-04-13T22:30:35.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVOnphK4OTxkOy4mdxDgvttGiJs=', 'Content-Length': '0'} 2024-04-13T22:30:35.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d99afc99364502-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.008 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:35.008 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:35.008 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.008 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:35.008 DEBUG:boto:Data: 2024-04-13T22:30:35.008 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.008 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.009 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.009 DEBUG:boto:Params: {} 2024-04-13T22:30:35.009 DEBUG:boto:Token: None 2024-04-13T22:30:35.009 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj74 2024-04-13T22:30:35.009 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KVOnphK4OTxkOy4mdxDgvttGiJs= 2024-04-13T22:30:35.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KVOnphK4OTxkOy4mdxDgvttGiJs=', 'Content-Length': '0'} 2024-04-13T22:30:35.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687fc53796dc1bf9-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-13T22:30:35.014 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:35.014 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:35.014 DEBUG:boto:Method: GET 2024-04-13T22:30:35.014 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:35.014 DEBUG:boto:Data: 2024-04-13T22:30:35.014 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.014 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.014 DEBUG:boto:Params: {} 2024-04-13T22:30:35.014 DEBUG:boto:Token: None 2024-04-13T22:30:35.015 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj74 2024-04-13T22:30:35.015 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jEHIAilYQHmPFKz0LJhnySn72WU= 2024-04-13T22:30:35.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jEHIAilYQHmPFKz0LJhnySn72WU=', 'Content-Length': '0'} 2024-04-13T22:30:35.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b6a0872ef1d2c5-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.058 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:35.058 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:35.059 DEBUG:boto:Method: GET 2024-04-13T22:30:35.059 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:35.059 DEBUG:boto:Data: 2024-04-13T22:30:35.059 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.059 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.059 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.059 DEBUG:boto:Params: {} 2024-04-13T22:30:35.059 DEBUG:boto:Token: None 2024-04-13T22:30:35.059 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj74 2024-04-13T22:30:35.060 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jEHIAilYQHmPFKz0LJhnySn72WU= 2024-04-13T22:30:35.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jEHIAilYQHmPFKz0LJhnySn72WU=', 'Content-Length': '0'} 2024-04-13T22:30:35.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b2f358e0412e53-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.062 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.062 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.062 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.062 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.062 DEBUG:boto:Method: GET 2024-04-13T22:30:35.062 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.063 DEBUG:boto:Data: 2024-04-13T22:30:35.063 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.063 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.063 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.063 DEBUG:boto:Params: {} 2024-04-13T22:30:35.063 DEBUG:boto:Token: None 2024-04-13T22:30:35.063 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.063 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ce63f898d120920-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.065 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.066 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.066 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.066 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.066 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.066 DEBUG:boto:Method: GET 2024-04-13T22:30:35.066 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:35.067 DEBUG:boto:Data: 2024-04-13T22:30:35.067 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.067 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.067 DEBUG:boto:Params: {} 2024-04-13T22:30:35.067 DEBUG:boto:Token: None 2024-04-13T22:30:35.067 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.067 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f0c9776cb25fe7c-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.071 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:30:35.071 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.071 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.071 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.071 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.072 DEBUG:boto:Method: GET 2024-04-13T22:30:35.072 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.072 DEBUG:boto:Data: 2024-04-13T22:30:35.072 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.072 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.072 DEBUG:boto:Params: {} 2024-04-13T22:30:35.072 DEBUG:boto:Token: None 2024-04-13T22:30:35.072 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.073 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I= 2024-04-13T22:30:35.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I=', 'Content-Length': '0'} 2024-04-13T22:30:35.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abdfbb7f2bb4edc-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.074 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.074 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.075 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.075 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.075 DEBUG:boto:Method: GET 2024-04-13T22:30:35.075 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.075 DEBUG:boto:Data: 2024-04-13T22:30:35.075 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.075 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.075 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.075 DEBUG:boto:Params: {} 2024-04-13T22:30:35.075 DEBUG:boto:Token: None 2024-04-13T22:30:35.075 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.076 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I= 2024-04-13T22:30:35.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I=', 'Content-Length': '0'} 2024-04-13T22:30:35.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94685f3a4f2665c-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.077 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.077 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.078 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.078 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.078 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.078 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.078 DEBUG:boto:Data: 2024-04-13T22:30:35.078 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.078 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.078 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.078 DEBUG:boto:Params: {} 2024-04-13T22:30:35.078 DEBUG:boto:Token: None 2024-04-13T22:30:35.078 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hdeycRziSWReCqwkipj08SozPMM= 2024-04-13T22:30:35.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hdeycRziSWReCqwkipj08SozPMM=', 'Content-Length': '0'} 2024-04-13T22:30:35.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575071596742f087-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.081 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.081 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.081 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.081 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.081 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.081 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.081 DEBUG:boto:Data: 2024-04-13T22:30:35.081 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.081 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.082 DEBUG:boto:Params: {} 2024-04-13T22:30:35.082 DEBUG:boto:Token: None 2024-04-13T22:30:35.082 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hdeycRziSWReCqwkipj08SozPMM= 2024-04-13T22:30:35.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hdeycRziSWReCqwkipj08SozPMM=', 'Content-Length': '0'} 2024-04-13T22:30:35.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0ed3d4b0b1b9ad-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:30:35.084 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.084 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.084 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.084 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.084 DEBUG:boto:Method: GET 2024-04-13T22:30:35.084 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.084 DEBUG:boto:Data: 2024-04-13T22:30:35.084 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.085 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.085 DEBUG:boto:Params: {} 2024-04-13T22:30:35.085 DEBUG:boto:Token: None 2024-04-13T22:30:35.085 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.085 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I= 2024-04-13T22:30:35.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I=', 'Content-Length': '0'} 2024-04-13T22:30:35.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dc8ffd93707040b-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.087 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.087 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.087 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.087 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.087 DEBUG:boto:Method: GET 2024-04-13T22:30:35.087 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.087 DEBUG:boto:Data: 2024-04-13T22:30:35.087 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.088 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.088 DEBUG:boto:Params: {} 2024-04-13T22:30:35.088 DEBUG:boto:Token: None 2024-04-13T22:30:35.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:35.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I= 2024-04-13T22:30:35.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ImDYAAg8YcID3bnWq5gZjVjtN+I=', 'Content-Length': '0'} 2024-04-13T22:30:35.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070a286c439f2fc6e-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.090 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.090 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.090 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.090 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.090 DEBUG:boto:Method: GET 2024-04-13T22:30:35.091 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.091 DEBUG:boto:Data: 2024-04-13T22:30:35.091 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.091 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.091 DEBUG:boto:Params: {} 2024-04-13T22:30:35.091 DEBUG:boto:Token: None 2024-04-13T22:30:35.091 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a8816b10ef4e817-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.093 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.094 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.094 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.094 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.094 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.094 DEBUG:boto:Method: GET 2024-04-13T22:30:35.094 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.094 DEBUG:boto:Data: 2024-04-13T22:30:35.094 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.095 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.095 DEBUG:boto:Params: {} 2024-04-13T22:30:35.095 DEBUG:boto:Token: None 2024-04-13T22:30:35.095 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004821d7be2b2fe78-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.099 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.099 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.099 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.099 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.100 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:35.100 DEBUG:boto:Data: 2024-04-13T22:30:35.100 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.100 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.100 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.100 DEBUG:boto:Params: {} 2024-04-13T22:30:35.100 DEBUG:boto:Token: None 2024-04-13T22:30:35.100 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76 2024-04-13T22:30:35.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fBYR9qAOnF/0WugHN6uGXMpImg8= 2024-04-13T22:30:35.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fBYR9qAOnF/0WugHN6uGXMpImg8=', 'Content-Length': '0'} 2024-04-13T22:30:35.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432758db8bd94d6d-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.102 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.102 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.102 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.102 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:35.103 DEBUG:boto:Data: 2024-04-13T22:30:35.103 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.103 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.103 DEBUG:boto:Params: {} 2024-04-13T22:30:35.103 DEBUG:boto:Token: None 2024-04-13T22:30:35.103 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76 2024-04-13T22:30:35.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fBYR9qAOnF/0WugHN6uGXMpImg8= 2024-04-13T22:30:35.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fBYR9qAOnF/0WugHN6uGXMpImg8=', 'Content-Length': '0'} 2024-04-13T22:30:35.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253a79222ee4134f-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:30:35.105 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.105 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.105 DEBUG:boto:Method: GET 2024-04-13T22:30:35.105 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:35.105 DEBUG:boto:Data: 2024-04-13T22:30:35.105 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.106 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.106 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.106 DEBUG:boto:Params: {} 2024-04-13T22:30:35.106 DEBUG:boto:Token: None 2024-04-13T22:30:35.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76 2024-04-13T22:30:35.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bf9cUhVHXLqUPbVznbOBSaCkFAo= 2024-04-13T22:30:35.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bf9cUhVHXLqUPbVznbOBSaCkFAo=', 'Content-Length': '0'} 2024-04-13T22:30:35.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e0e9647e723eb1-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.108 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:35.108 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:35.108 DEBUG:boto:Method: GET 2024-04-13T22:30:35.109 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:35.109 DEBUG:boto:Data: 2024-04-13T22:30:35.109 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.109 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.109 DEBUG:boto:Params: {} 2024-04-13T22:30:35.109 DEBUG:boto:Token: None 2024-04-13T22:30:35.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj76 2024-04-13T22:30:35.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Bf9cUhVHXLqUPbVznbOBSaCkFAo= 2024-04-13T22:30:35.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Bf9cUhVHXLqUPbVznbOBSaCkFAo=', 'Content-Length': '0'} 2024-04-13T22:30:35.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165a6387370f743f-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.111 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.111 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.111 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.112 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.112 DEBUG:boto:Method: GET 2024-04-13T22:30:35.112 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.112 DEBUG:boto:Data: 2024-04-13T22:30:35.112 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.112 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.112 DEBUG:boto:Params: {} 2024-04-13T22:30:35.112 DEBUG:boto:Token: None 2024-04-13T22:30:35.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c53fdc7eb267b95-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.115 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.115 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.115 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.115 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.115 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.115 DEBUG:boto:Method: GET 2024-04-13T22:30:35.115 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:35.116 DEBUG:boto:Data: 2024-04-13T22:30:35.116 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.116 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.116 DEBUG:boto:Params: {} 2024-04-13T22:30:35.116 DEBUG:boto:Token: None 2024-04-13T22:30:35.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000076e60d5a175877c-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.162 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:30:35.163 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.163 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.163 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.163 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.163 DEBUG:boto:Method: GET 2024-04-13T22:30:35.163 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.164 DEBUG:boto:Data: 2024-04-13T22:30:35.164 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.164 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.164 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.164 DEBUG:boto:Params: {} 2024-04-13T22:30:35.164 DEBUG:boto:Token: None 2024-04-13T22:30:35.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg= 2024-04-13T22:30:35.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg=', 'Content-Length': '0'} 2024-04-13T22:30:35.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e4bab93fabf2c3-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.167 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.167 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.167 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.167 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.167 DEBUG:boto:Method: GET 2024-04-13T22:30:35.167 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.167 DEBUG:boto:Data: 2024-04-13T22:30:35.167 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.167 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.168 DEBUG:boto:Params: {} 2024-04-13T22:30:35.168 DEBUG:boto:Token: None 2024-04-13T22:30:35.168 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg= 2024-04-13T22:30:35.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg=', 'Content-Length': '0'} 2024-04-13T22:30:35.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c3d67a175ec0fd-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.170 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.170 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.170 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.170 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.170 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.170 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.170 DEBUG:boto:Data: 2024-04-13T22:30:35.171 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.171 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.171 DEBUG:boto:Params: {} 2024-04-13T22:30:35.171 DEBUG:boto:Token: None 2024-04-13T22:30:35.171 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H6Y/laGPUv0sb+Q10AyhxJm7lAg= 2024-04-13T22:30:35.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H6Y/laGPUv0sb+Q10AyhxJm7lAg=', 'Content-Length': '0'} 2024-04-13T22:30:35.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec0017b644363dc-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.173 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.173 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.173 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.173 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.173 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.174 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.174 DEBUG:boto:Data: 2024-04-13T22:30:35.174 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.174 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.174 DEBUG:boto:Params: {} 2024-04-13T22:30:35.174 DEBUG:boto:Token: None 2024-04-13T22:30:35.174 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H6Y/laGPUv0sb+Q10AyhxJm7lAg= 2024-04-13T22:30:35.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H6Y/laGPUv0sb+Q10AyhxJm7lAg=', 'Content-Length': '0'} 2024-04-13T22:30:35.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224b774307d868d7-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:30:35.176 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.177 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.177 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.177 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.177 DEBUG:boto:Method: GET 2024-04-13T22:30:35.177 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.177 DEBUG:boto:Data: 2024-04-13T22:30:35.177 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.177 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.177 DEBUG:boto:Params: {} 2024-04-13T22:30:35.177 DEBUG:boto:Token: None 2024-04-13T22:30:35.178 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.178 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg= 2024-04-13T22:30:35.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg=', 'Content-Length': '0'} 2024-04-13T22:30:35.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce0294b4def5aa8-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.222 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.223 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.223 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.223 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.223 DEBUG:boto:Method: GET 2024-04-13T22:30:35.223 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.223 DEBUG:boto:Data: 2024-04-13T22:30:35.223 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.223 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.224 DEBUG:boto:Params: {} 2024-04-13T22:30:35.224 DEBUG:boto:Token: None 2024-04-13T22:30:35.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:35.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg= 2024-04-13T22:30:35.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CrhX96fOMfCdEG5/cWKNT1KUqWg=', 'Content-Length': '0'} 2024-04-13T22:30:35.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb314eaf7787ee24-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.226 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.226 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.226 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.226 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.227 DEBUG:boto:Method: GET 2024-04-13T22:30:35.227 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.227 DEBUG:boto:Data: 2024-04-13T22:30:35.227 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.227 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.227 DEBUG:boto:Params: {} 2024-04-13T22:30:35.227 DEBUG:boto:Token: None 2024-04-13T22:30:35.227 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.227 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006439c6de897906dd-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.230 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.230 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.231 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.231 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.231 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.231 DEBUG:boto:Method: GET 2024-04-13T22:30:35.231 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.231 DEBUG:boto:Data: 2024-04-13T22:30:35.231 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.231 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.231 DEBUG:boto:Params: {} 2024-04-13T22:30:35.232 DEBUG:boto:Token: None 2024-04-13T22:30:35.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e2cbf621ef9845-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.235 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.236 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.236 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.236 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.236 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:35.236 DEBUG:boto:Data: 2024-04-13T22:30:35.236 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.236 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.236 DEBUG:boto:Params: {} 2024-04-13T22:30:35.236 DEBUG:boto:Token: None 2024-04-13T22:30:35.237 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85 2024-04-13T22:30:35.237 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u0VjdtFi+bH+v2PrJwFG3i44aow= 2024-04-13T22:30:35.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u0VjdtFi+bH+v2PrJwFG3i44aow=', 'Content-Length': '0'} 2024-04-13T22:30:35.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94bbe853265dc49-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.239 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.239 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.239 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.239 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:35.239 DEBUG:boto:Data: 2024-04-13T22:30:35.239 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.239 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.239 DEBUG:boto:Params: {} 2024-04-13T22:30:35.239 DEBUG:boto:Token: None 2024-04-13T22:30:35.240 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85 2024-04-13T22:30:35.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:u0VjdtFi+bH+v2PrJwFG3i44aow= 2024-04-13T22:30:35.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:u0VjdtFi+bH+v2PrJwFG3i44aow=', 'Content-Length': '0'} 2024-04-13T22:30:35.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043dd5cc264756248-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:30:35.242 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.242 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.242 DEBUG:boto:Method: GET 2024-04-13T22:30:35.242 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:35.242 DEBUG:boto:Data: 2024-04-13T22:30:35.242 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.242 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.243 DEBUG:boto:Params: {} 2024-04-13T22:30:35.243 DEBUG:boto:Token: None 2024-04-13T22:30:35.243 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85 2024-04-13T22:30:35.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I0mCktkfRxLsoNaG++jT8rLoPDU= 2024-04-13T22:30:35.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I0mCktkfRxLsoNaG++jT8rLoPDU=', 'Content-Length': '0'} 2024-04-13T22:30:35.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1779ce8e1e10784-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.245 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:35.245 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:35.245 DEBUG:boto:Method: GET 2024-04-13T22:30:35.245 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:35.245 DEBUG:boto:Data: 2024-04-13T22:30:35.245 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.245 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.245 DEBUG:boto:Params: {} 2024-04-13T22:30:35.245 DEBUG:boto:Token: None 2024-04-13T22:30:35.246 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj85 2024-04-13T22:30:35.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I0mCktkfRxLsoNaG++jT8rLoPDU= 2024-04-13T22:30:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I0mCktkfRxLsoNaG++jT8rLoPDU=', 'Content-Length': '0'} 2024-04-13T22:30:35.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f241151b54abbc4-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.248 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.248 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.248 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.248 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.248 DEBUG:boto:Method: GET 2024-04-13T22:30:35.248 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.248 DEBUG:boto:Data: 2024-04-13T22:30:35.248 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.249 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.249 DEBUG:boto:Params: {} 2024-04-13T22:30:35.249 DEBUG:boto:Token: None 2024-04-13T22:30:35.249 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.249 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097cf5c8dfe4fd24e-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.251 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.251 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.252 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.252 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.252 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.252 DEBUG:boto:Method: GET 2024-04-13T22:30:35.252 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:35.252 DEBUG:boto:Data: 2024-04-13T22:30:35.252 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.252 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.252 DEBUG:boto:Params: {} 2024-04-13T22:30:35.252 DEBUG:boto:Token: None 2024-04-13T22:30:35.253 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.253 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637405fa9e9a982a-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.257 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:30:35.257 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.257 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.257 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.257 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.257 DEBUG:boto:Method: GET 2024-04-13T22:30:35.258 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.258 DEBUG:boto:Data: 2024-04-13T22:30:35.258 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.258 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.258 DEBUG:boto:Params: {} 2024-04-13T22:30:35.258 DEBUG:boto:Token: None 2024-04-13T22:30:35.258 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E= 2024-04-13T22:30:35.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E=', 'Content-Length': '0'} 2024-04-13T22:30:35.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632604a21d5deac5-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.302 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.302 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.303 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.303 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.303 DEBUG:boto:Method: GET 2024-04-13T22:30:35.303 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.303 DEBUG:boto:Data: 2024-04-13T22:30:35.303 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.303 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.303 DEBUG:boto:Params: {} 2024-04-13T22:30:35.303 DEBUG:boto:Token: None 2024-04-13T22:30:35.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E= 2024-04-13T22:30:35.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E=', 'Content-Length': '0'} 2024-04-13T22:30:35.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b10d89131b2d509-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.306 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.306 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.306 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.306 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.306 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.306 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.306 DEBUG:boto:Data: 2024-04-13T22:30:35.306 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.306 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.306 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.307 DEBUG:boto:Params: {} 2024-04-13T22:30:35.307 DEBUG:boto:Token: None 2024-04-13T22:30:35.307 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S8mnUduziqV9jQEuGvskDQceqPw= 2024-04-13T22:30:35.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S8mnUduziqV9jQEuGvskDQceqPw=', 'Content-Length': '0'} 2024-04-13T22:30:35.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c01020c3c576be6-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.309 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.309 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.309 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.309 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.309 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.309 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.309 DEBUG:boto:Data: 2024-04-13T22:30:35.309 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.310 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.310 DEBUG:boto:Params: {} 2024-04-13T22:30:35.310 DEBUG:boto:Token: None 2024-04-13T22:30:35.310 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S8mnUduziqV9jQEuGvskDQceqPw= 2024-04-13T22:30:35.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S8mnUduziqV9jQEuGvskDQceqPw=', 'Content-Length': '0'} 2024-04-13T22:30:35.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7922f269e65c23c-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:30:35.312 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.312 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.312 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.312 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.312 DEBUG:boto:Method: GET 2024-04-13T22:30:35.312 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.312 DEBUG:boto:Data: 2024-04-13T22:30:35.312 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.312 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.313 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.313 DEBUG:boto:Params: {} 2024-04-13T22:30:35.313 DEBUG:boto:Token: None 2024-04-13T22:30:35.313 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.313 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E= 2024-04-13T22:30:35.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E=', 'Content-Length': '0'} 2024-04-13T22:30:35.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812dbbad3e44170d-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.315 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.315 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.315 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.315 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.315 DEBUG:boto:Method: GET 2024-04-13T22:30:35.315 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.316 DEBUG:boto:Data: 2024-04-13T22:30:35.316 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.316 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.316 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.316 DEBUG:boto:Params: {} 2024-04-13T22:30:35.316 DEBUG:boto:Token: None 2024-04-13T22:30:35.316 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:35.316 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E= 2024-04-13T22:30:35.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kmv+KDAxvpGtlGS/ZmXTPLnSZ/E=', 'Content-Length': '0'} 2024-04-13T22:30:35.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d535996a0f190f9-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.318 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.318 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.318 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.319 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.319 DEBUG:boto:Method: GET 2024-04-13T22:30:35.319 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.319 DEBUG:boto:Data: 2024-04-13T22:30:35.319 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.319 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.319 DEBUG:boto:Params: {} 2024-04-13T22:30:35.319 DEBUG:boto:Token: None 2024-04-13T22:30:35.319 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a0687429d498b8-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.322 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.322 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.322 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.322 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.323 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.323 DEBUG:boto:Method: GET 2024-04-13T22:30:35.323 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.323 DEBUG:boto:Data: 2024-04-13T22:30:35.323 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.323 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.323 DEBUG:boto:Params: {} 2024-04-13T22:30:35.323 DEBUG:boto:Token: None 2024-04-13T22:30:35.323 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a333d67cb3c5496-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.327 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.328 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.328 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.328 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.328 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:35.328 DEBUG:boto:Data: 2024-04-13T22:30:35.328 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.328 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.328 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.328 DEBUG:boto:Params: {} 2024-04-13T22:30:35.329 DEBUG:boto:Token: None 2024-04-13T22:30:35.329 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87 2024-04-13T22:30:35.329 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3g1OA64RsM6fGuS7mxde/iujU4s= 2024-04-13T22:30:35.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3g1OA64RsM6fGuS7mxde/iujU4s=', 'Content-Length': '0'} 2024-04-13T22:30:35.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5226781d6ebe681-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.331 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.331 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.331 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.331 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:35.331 DEBUG:boto:Data: 2024-04-13T22:30:35.331 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.331 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.331 DEBUG:boto:Params: {} 2024-04-13T22:30:35.331 DEBUG:boto:Token: None 2024-04-13T22:30:35.332 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87 2024-04-13T22:30:35.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3g1OA64RsM6fGuS7mxde/iujU4s= 2024-04-13T22:30:35.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3g1OA64RsM6fGuS7mxde/iujU4s=', 'Content-Length': '0'} 2024-04-13T22:30:35.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf8b93626afe067-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:30:35.334 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.334 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.334 DEBUG:boto:Method: GET 2024-04-13T22:30:35.334 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:35.334 DEBUG:boto:Data: 2024-04-13T22:30:35.334 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.334 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.335 DEBUG:boto:Params: {} 2024-04-13T22:30:35.335 DEBUG:boto:Token: None 2024-04-13T22:30:35.335 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87 2024-04-13T22:30:35.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ut77srw3X8bAqZzSIZpxKhj9gP8= 2024-04-13T22:30:35.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ut77srw3X8bAqZzSIZpxKhj9gP8=', 'Content-Length': '0'} 2024-04-13T22:30:35.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ab21e263cea759a-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.382 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:35.383 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:35.383 DEBUG:boto:Method: GET 2024-04-13T22:30:35.383 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:35.383 DEBUG:boto:Data: 2024-04-13T22:30:35.383 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.383 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.383 DEBUG:boto:Params: {} 2024-04-13T22:30:35.383 DEBUG:boto:Token: None 2024-04-13T22:30:35.383 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj87 2024-04-13T22:30:35.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ut77srw3X8bAqZzSIZpxKhj9gP8= 2024-04-13T22:30:35.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ut77srw3X8bAqZzSIZpxKhj9gP8=', 'Content-Length': '0'} 2024-04-13T22:30:35.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726c163f62a2aa77-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.386 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.386 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.386 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.386 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.386 DEBUG:boto:Method: GET 2024-04-13T22:30:35.387 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.387 DEBUG:boto:Data: 2024-04-13T22:30:35.387 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.387 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.387 DEBUG:boto:Params: {} 2024-04-13T22:30:35.387 DEBUG:boto:Token: None 2024-04-13T22:30:35.387 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.387 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7257ad91ecd71b-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.390 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.390 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.390 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.391 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.391 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.391 DEBUG:boto:Method: GET 2024-04-13T22:30:35.391 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:35.391 DEBUG:boto:Data: 2024-04-13T22:30:35.391 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.391 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.391 DEBUG:boto:Params: {} 2024-04-13T22:30:35.391 DEBUG:boto:Token: None 2024-04-13T22:30:35.391 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.392 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfbc2ddf37e172f7-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.395 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:30:35.395 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.395 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.395 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.396 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.396 DEBUG:boto:Method: GET 2024-04-13T22:30:35.396 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.396 DEBUG:boto:Data: 2024-04-13T22:30:35.396 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.396 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.396 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.396 DEBUG:boto:Params: {} 2024-04-13T22:30:35.396 DEBUG:boto:Token: None 2024-04-13T22:30:35.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.397 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4= 2024-04-13T22:30:35.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4=', 'Content-Length': '0'} 2024-04-13T22:30:35.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea99739e56894a17-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.399 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.399 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.399 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.399 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.399 DEBUG:boto:Method: GET 2024-04-13T22:30:35.399 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.399 DEBUG:boto:Data: 2024-04-13T22:30:35.399 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.399 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.399 DEBUG:boto:Params: {} 2024-04-13T22:30:35.400 DEBUG:boto:Token: None 2024-04-13T22:30:35.400 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4= 2024-04-13T22:30:35.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4=', 'Content-Length': '0'} 2024-04-13T22:30:35.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9bd9a4256f9a3d-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.402 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.402 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.402 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.402 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.402 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.402 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.402 DEBUG:boto:Data: 2024-04-13T22:30:35.402 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.403 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.403 DEBUG:boto:Params: {} 2024-04-13T22:30:35.403 DEBUG:boto:Token: None 2024-04-13T22:30:35.403 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.403 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Owbo/o5QnqIYSq2K/UH/rk5AL8M= 2024-04-13T22:30:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Owbo/o5QnqIYSq2K/UH/rk5AL8M=', 'Content-Length': '0'} 2024-04-13T22:30:35.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf15580f483ec7e2-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.405 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.405 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.405 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.405 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.405 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.405 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.405 DEBUG:boto:Data: 2024-04-13T22:30:35.406 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.406 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.406 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.406 DEBUG:boto:Params: {} 2024-04-13T22:30:35.406 DEBUG:boto:Token: None 2024-04-13T22:30:35.406 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Owbo/o5QnqIYSq2K/UH/rk5AL8M= 2024-04-13T22:30:35.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Owbo/o5QnqIYSq2K/UH/rk5AL8M=', 'Content-Length': '0'} 2024-04-13T22:30:35.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd3add805266c61-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:30:35.408 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.408 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.408 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.408 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.409 DEBUG:boto:Method: GET 2024-04-13T22:30:35.409 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.409 DEBUG:boto:Data: 2024-04-13T22:30:35.409 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.409 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.409 DEBUG:boto:Params: {} 2024-04-13T22:30:35.409 DEBUG:boto:Token: None 2024-04-13T22:30:35.409 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.409 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4= 2024-04-13T22:30:35.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4=', 'Content-Length': '0'} 2024-04-13T22:30:35.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072846038d782b92f-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.411 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.411 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.411 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.412 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.412 DEBUG:boto:Method: GET 2024-04-13T22:30:35.412 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.412 DEBUG:boto:Data: 2024-04-13T22:30:35.412 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.412 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.412 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.412 DEBUG:boto:Params: {} 2024-04-13T22:30:35.412 DEBUG:boto:Token: None 2024-04-13T22:30:35.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:35.413 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4= 2024-04-13T22:30:35.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qElmJHum761m9g9OMvlJYKmiM/4=', 'Content-Length': '0'} 2024-04-13T22:30:35.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4989320e7a135e-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.415 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.415 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.415 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.415 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.415 DEBUG:boto:Method: GET 2024-04-13T22:30:35.415 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.415 DEBUG:boto:Data: 2024-04-13T22:30:35.415 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.416 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.416 DEBUG:boto:Params: {} 2024-04-13T22:30:35.416 DEBUG:boto:Token: None 2024-04-13T22:30:35.416 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a990a9b00abc554-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.466 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.467 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.467 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.467 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.467 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.467 DEBUG:boto:Method: GET 2024-04-13T22:30:35.468 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.468 DEBUG:boto:Data: 2024-04-13T22:30:35.468 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.468 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.468 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.468 DEBUG:boto:Params: {} 2024-04-13T22:30:35.468 DEBUG:boto:Token: None 2024-04-13T22:30:35.469 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.469 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043cd46df249be59a-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.473 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.473 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.473 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.473 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.473 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:35.474 DEBUG:boto:Data: 2024-04-13T22:30:35.474 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.474 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.474 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.474 DEBUG:boto:Params: {} 2024-04-13T22:30:35.474 DEBUG:boto:Token: None 2024-04-13T22:30:35.474 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92 2024-04-13T22:30:35.475 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vlRVBBtAvG/RK7XhiFzYCwlrNgA= 2024-04-13T22:30:35.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vlRVBBtAvG/RK7XhiFzYCwlrNgA=', 'Content-Length': '0'} 2024-04-13T22:30:35.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084eaabbe81ba7472-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.477 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.477 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.477 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.477 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:35.477 DEBUG:boto:Data: 2024-04-13T22:30:35.477 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.477 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.477 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.477 DEBUG:boto:Params: {} 2024-04-13T22:30:35.477 DEBUG:boto:Token: None 2024-04-13T22:30:35.478 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92 2024-04-13T22:30:35.478 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vlRVBBtAvG/RK7XhiFzYCwlrNgA= 2024-04-13T22:30:35.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vlRVBBtAvG/RK7XhiFzYCwlrNgA=', 'Content-Length': '0'} 2024-04-13T22:30:35.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c6f7fbc46c0c1f-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:30:35.480 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.480 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.480 DEBUG:boto:Method: GET 2024-04-13T22:30:35.480 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:35.480 DEBUG:boto:Data: 2024-04-13T22:30:35.480 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.481 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.481 DEBUG:boto:Params: {} 2024-04-13T22:30:35.481 DEBUG:boto:Token: None 2024-04-13T22:30:35.481 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92 2024-04-13T22:30:35.481 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jHW5BMtKbQXjb5txsC/ryAhOaRw= 2024-04-13T22:30:35.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jHW5BMtKbQXjb5txsC/ryAhOaRw=', 'Content-Length': '0'} 2024-04-13T22:30:35.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c97a8a1df56127-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.483 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:35.483 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:35.483 DEBUG:boto:Method: GET 2024-04-13T22:30:35.484 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:35.484 DEBUG:boto:Data: 2024-04-13T22:30:35.484 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.484 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.484 DEBUG:boto:Params: {} 2024-04-13T22:30:35.484 DEBUG:boto:Token: None 2024-04-13T22:30:35.484 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj92 2024-04-13T22:30:35.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jHW5BMtKbQXjb5txsC/ryAhOaRw= 2024-04-13T22:30:35.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jHW5BMtKbQXjb5txsC/ryAhOaRw=', 'Content-Length': '0'} 2024-04-13T22:30:35.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b01a849ff98af813-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.487 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.487 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.487 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.487 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.487 DEBUG:boto:Method: GET 2024-04-13T22:30:35.487 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.487 DEBUG:boto:Data: 2024-04-13T22:30:35.487 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.487 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.487 DEBUG:boto:Params: {} 2024-04-13T22:30:35.488 DEBUG:boto:Token: None 2024-04-13T22:30:35.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145a458a93ae500f-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.490 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.490 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.491 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.491 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.491 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.491 DEBUG:boto:Method: GET 2024-04-13T22:30:35.491 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:35.491 DEBUG:boto:Data: 2024-04-13T22:30:35.491 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.491 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.491 DEBUG:boto:Params: {} 2024-04-13T22:30:35.492 DEBUG:boto:Token: None 2024-04-13T22:30:35.492 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.492 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e70aa45d3b35ab2-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.542 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:30:35.543 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.543 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.543 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.543 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.544 DEBUG:boto:Method: GET 2024-04-13T22:30:35.544 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.544 DEBUG:boto:Data: 2024-04-13T22:30:35.544 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.544 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.544 DEBUG:boto:Params: {} 2024-04-13T22:30:35.544 DEBUG:boto:Token: None 2024-04-13T22:30:35.545 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk= 2024-04-13T22:30:35.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk=', 'Content-Length': '0'} 2024-04-13T22:30:35.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5bbfe0cfb6effc-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.547 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.547 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.547 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.547 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.548 DEBUG:boto:Method: GET 2024-04-13T22:30:35.548 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.548 DEBUG:boto:Data: 2024-04-13T22:30:35.548 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.548 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.548 DEBUG:boto:Params: {} 2024-04-13T22:30:35.548 DEBUG:boto:Token: None 2024-04-13T22:30:35.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk= 2024-04-13T22:30:35.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk=', 'Content-Length': '0'} 2024-04-13T22:30:35.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419187e8ec9e8c8e-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.551 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.551 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.551 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.551 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.551 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.551 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.551 DEBUG:boto:Data: 2024-04-13T22:30:35.551 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.552 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.552 DEBUG:boto:Params: {} 2024-04-13T22:30:35.552 DEBUG:boto:Token: None 2024-04-13T22:30:35.552 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DAU6hl9RsGi6pggBFcdVmuwa/Uc= 2024-04-13T22:30:35.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DAU6hl9RsGi6pggBFcdVmuwa/Uc=', 'Content-Length': '0'} 2024-04-13T22:30:35.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903d3fbe8955480f-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.554 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.554 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.554 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.554 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.554 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.555 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.555 DEBUG:boto:Data: 2024-04-13T22:30:35.555 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.555 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.555 DEBUG:boto:Params: {} 2024-04-13T22:30:35.555 DEBUG:boto:Token: None 2024-04-13T22:30:35.555 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DAU6hl9RsGi6pggBFcdVmuwa/Uc= 2024-04-13T22:30:35.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DAU6hl9RsGi6pggBFcdVmuwa/Uc=', 'Content-Length': '0'} 2024-04-13T22:30:35.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8bf7a5ba2f9e169-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:30:35.557 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.557 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.558 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.558 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.558 DEBUG:boto:Method: GET 2024-04-13T22:30:35.558 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.558 DEBUG:boto:Data: 2024-04-13T22:30:35.558 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.558 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.558 DEBUG:boto:Params: {} 2024-04-13T22:30:35.558 DEBUG:boto:Token: None 2024-04-13T22:30:35.559 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk= 2024-04-13T22:30:35.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk=', 'Content-Length': '0'} 2024-04-13T22:30:35.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec64796bbf512fe-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.602 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.603 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.603 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.603 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.603 DEBUG:boto:Method: GET 2024-04-13T22:30:35.603 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.603 DEBUG:boto:Data: 2024-04-13T22:30:35.603 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.603 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.604 DEBUG:boto:Params: {} 2024-04-13T22:30:35.604 DEBUG:boto:Token: None 2024-04-13T22:30:35.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:35.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk= 2024-04-13T22:30:35.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hbP1m55UKiUrpMOepgZABtzlvQk=', 'Content-Length': '0'} 2024-04-13T22:30:35.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd56b5f41a2f8594-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.606 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.606 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.607 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.607 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.607 DEBUG:boto:Method: GET 2024-04-13T22:30:35.607 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.607 DEBUG:boto:Data: 2024-04-13T22:30:35.607 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.607 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.607 DEBUG:boto:Params: {} 2024-04-13T22:30:35.607 DEBUG:boto:Token: None 2024-04-13T22:30:35.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed9949f01f62a81-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.610 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.610 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.610 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.611 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.611 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.611 DEBUG:boto:Method: GET 2024-04-13T22:30:35.611 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.611 DEBUG:boto:Data: 2024-04-13T22:30:35.611 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.611 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.611 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.611 DEBUG:boto:Params: {} 2024-04-13T22:30:35.611 DEBUG:boto:Token: None 2024-04-13T22:30:35.612 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.612 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e2868a66573384-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.615 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.616 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.616 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.616 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.616 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:35.616 DEBUG:boto:Data: 2024-04-13T22:30:35.616 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.616 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.616 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.616 DEBUG:boto:Params: {} 2024-04-13T22:30:35.616 DEBUG:boto:Token: None 2024-04-13T22:30:35.617 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93 2024-04-13T22:30:35.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TjLhUGTRELtx8vttJgtTXvcFOm0= 2024-04-13T22:30:35.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TjLhUGTRELtx8vttJgtTXvcFOm0=', 'Content-Length': '0'} 2024-04-13T22:30:35.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008903e63299eeba2e-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.619 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.619 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.619 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.619 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:35.619 DEBUG:boto:Data: 2024-04-13T22:30:35.619 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.619 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.619 DEBUG:boto:Params: {} 2024-04-13T22:30:35.619 DEBUG:boto:Token: None 2024-04-13T22:30:35.620 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93 2024-04-13T22:30:35.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TjLhUGTRELtx8vttJgtTXvcFOm0= 2024-04-13T22:30:35.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TjLhUGTRELtx8vttJgtTXvcFOm0=', 'Content-Length': '0'} 2024-04-13T22:30:35.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000395151032df6f39a-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:30:35.621 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.622 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.622 DEBUG:boto:Method: GET 2024-04-13T22:30:35.622 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:35.622 DEBUG:boto:Data: 2024-04-13T22:30:35.622 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.622 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.622 DEBUG:boto:Params: {} 2024-04-13T22:30:35.622 DEBUG:boto:Token: None 2024-04-13T22:30:35.623 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93 2024-04-13T22:30:35.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ALUyoE9adBzxCHYj2Dk9k5vNC0Y= 2024-04-13T22:30:35.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ALUyoE9adBzxCHYj2Dk9k5vNC0Y=', 'Content-Length': '0'} 2024-04-13T22:30:35.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9a8b13f7d93ae9-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.624 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:35.625 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:35.625 DEBUG:boto:Method: GET 2024-04-13T22:30:35.625 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:35.625 DEBUG:boto:Data: 2024-04-13T22:30:35.625 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.625 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.625 DEBUG:boto:Params: {} 2024-04-13T22:30:35.625 DEBUG:boto:Token: None 2024-04-13T22:30:35.626 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj93 2024-04-13T22:30:35.626 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ALUyoE9adBzxCHYj2Dk9k5vNC0Y= 2024-04-13T22:30:35.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ALUyoE9adBzxCHYj2Dk9k5vNC0Y=', 'Content-Length': '0'} 2024-04-13T22:30:35.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001228d5fc254e231e-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.671 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.671 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.671 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.671 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.671 DEBUG:boto:Method: GET 2024-04-13T22:30:35.671 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.671 DEBUG:boto:Data: 2024-04-13T22:30:35.671 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.671 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.671 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.672 DEBUG:boto:Params: {} 2024-04-13T22:30:35.672 DEBUG:boto:Token: None 2024-04-13T22:30:35.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006405d71a66c42a5a-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.675 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.675 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.675 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.675 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.675 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.675 DEBUG:boto:Method: GET 2024-04-13T22:30:35.676 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:35.676 DEBUG:boto:Data: 2024-04-13T22:30:35.676 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.676 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.676 DEBUG:boto:Params: {} 2024-04-13T22:30:35.676 DEBUG:boto:Token: None 2024-04-13T22:30:35.676 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c23b274e6cc57c-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.680 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:30:35.680 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.681 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.681 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.681 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.681 DEBUG:boto:Method: GET 2024-04-13T22:30:35.681 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.681 DEBUG:boto:Data: 2024-04-13T22:30:35.681 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.681 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.681 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.681 DEBUG:boto:Params: {} 2024-04-13T22:30:35.681 DEBUG:boto:Token: None 2024-04-13T22:30:35.682 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA= 2024-04-13T22:30:35.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA=', 'Content-Length': '0'} 2024-04-13T22:30:35.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000802ca37f64938eb9-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.726 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.727 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.727 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.727 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.727 DEBUG:boto:Method: GET 2024-04-13T22:30:35.727 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.727 DEBUG:boto:Data: 2024-04-13T22:30:35.727 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.727 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.727 DEBUG:boto:Params: {} 2024-04-13T22:30:35.727 DEBUG:boto:Token: None 2024-04-13T22:30:35.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA= 2024-04-13T22:30:35.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA=', 'Content-Length': '0'} 2024-04-13T22:30:35.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042897449123e42f2-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.730 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.730 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.730 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.730 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.730 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.730 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.730 DEBUG:boto:Data: 2024-04-13T22:30:35.730 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.730 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.731 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.731 DEBUG:boto:Params: {} 2024-04-13T22:30:35.731 DEBUG:boto:Token: None 2024-04-13T22:30:35.731 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.731 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S2V1HHTJ02/7qiQ2dj23A0+MHKQ= 2024-04-13T22:30:35.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S2V1HHTJ02/7qiQ2dj23A0+MHKQ=', 'Content-Length': '0'} 2024-04-13T22:30:35.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6424d215ca5dc72-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.733 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.733 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.733 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.733 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.733 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.733 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.733 DEBUG:boto:Data: 2024-04-13T22:30:35.733 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.734 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.734 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.734 DEBUG:boto:Params: {} 2024-04-13T22:30:35.734 DEBUG:boto:Token: None 2024-04-13T22:30:35.734 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S2V1HHTJ02/7qiQ2dj23A0+MHKQ= 2024-04-13T22:30:35.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S2V1HHTJ02/7qiQ2dj23A0+MHKQ=', 'Content-Length': '0'} 2024-04-13T22:30:35.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb1300c16cfbf33-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:30:35.736 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.736 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.736 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.737 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.737 DEBUG:boto:Method: GET 2024-04-13T22:30:35.737 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.737 DEBUG:boto:Data: 2024-04-13T22:30:35.737 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.737 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.737 DEBUG:boto:Params: {} 2024-04-13T22:30:35.737 DEBUG:boto:Token: None 2024-04-13T22:30:35.737 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA= 2024-04-13T22:30:35.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA=', 'Content-Length': '0'} 2024-04-13T22:30:35.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7026b112398905-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.739 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.740 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.740 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.740 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.740 DEBUG:boto:Method: GET 2024-04-13T22:30:35.740 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.740 DEBUG:boto:Data: 2024-04-13T22:30:35.740 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.740 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.740 DEBUG:boto:Params: {} 2024-04-13T22:30:35.740 DEBUG:boto:Token: None 2024-04-13T22:30:35.741 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:35.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA= 2024-04-13T22:30:35.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vp2ZUrq6m9ikyjsqa56vjJoBnqA=', 'Content-Length': '0'} 2024-04-13T22:30:35.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065aa3377fb8af936-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.743 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.743 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.743 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.743 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.743 DEBUG:boto:Method: GET 2024-04-13T22:30:35.743 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.743 DEBUG:boto:Data: 2024-04-13T22:30:35.743 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.743 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.743 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.744 DEBUG:boto:Params: {} 2024-04-13T22:30:35.744 DEBUG:boto:Token: None 2024-04-13T22:30:35.744 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb13cd4695c82d24-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.746 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.747 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.747 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.747 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.747 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.747 DEBUG:boto:Method: GET 2024-04-13T22:30:35.747 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.747 DEBUG:boto:Data: 2024-04-13T22:30:35.747 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.747 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.748 DEBUG:boto:Params: {} 2024-04-13T22:30:35.748 DEBUG:boto:Token: None 2024-04-13T22:30:35.748 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c912f54ed4df9a-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.751 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.752 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.752 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.752 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.752 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:35.752 DEBUG:boto:Data: 2024-04-13T22:30:35.752 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.752 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.752 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.752 DEBUG:boto:Params: {} 2024-04-13T22:30:35.753 DEBUG:boto:Token: None 2024-04-13T22:30:35.753 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95 2024-04-13T22:30:35.753 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:75U8eycvZM0ZBcCzrLcnPMdwqys= 2024-04-13T22:30:35.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:75U8eycvZM0ZBcCzrLcnPMdwqys=', 'Content-Length': '0'} 2024-04-13T22:30:35.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a25f82bfd169e0-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.755 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.755 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.755 DEBUG:boto:Method: HEAD 2024-04-13T22:30:35.755 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:35.755 DEBUG:boto:Data: 2024-04-13T22:30:35.755 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.756 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.756 DEBUG:boto:Params: {} 2024-04-13T22:30:35.756 DEBUG:boto:Token: None 2024-04-13T22:30:35.756 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95 2024-04-13T22:30:35.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:75U8eycvZM0ZBcCzrLcnPMdwqys= 2024-04-13T22:30:35.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:75U8eycvZM0ZBcCzrLcnPMdwqys=', 'Content-Length': '0'} 2024-04-13T22:30:35.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff28844d7d9228d-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:30:35.758 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.758 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.758 DEBUG:boto:Method: GET 2024-04-13T22:30:35.758 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:35.758 DEBUG:boto:Data: 2024-04-13T22:30:35.758 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.758 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.759 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.759 DEBUG:boto:Params: {} 2024-04-13T22:30:35.759 DEBUG:boto:Token: None 2024-04-13T22:30:35.759 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95 2024-04-13T22:30:35.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Xvb2c7MZhUXL/FJso7sNDsedCc= 2024-04-13T22:30:35.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Xvb2c7MZhUXL/FJso7sNDsedCc=', 'Content-Length': '0'} 2024-04-13T22:30:35.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f238b67709e0d12-00661b078b-1f77-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.806 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:35.806 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:35.807 DEBUG:boto:Method: GET 2024-04-13T22:30:35.807 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:35.807 DEBUG:boto:Data: 2024-04-13T22:30:35.807 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.807 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.807 DEBUG:boto:Params: {} 2024-04-13T22:30:35.807 DEBUG:boto:Token: None 2024-04-13T22:30:35.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/obj95 2024-04-13T22:30:35.808 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8Xvb2c7MZhUXL/FJso7sNDsedCc= 2024-04-13T22:30:35.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Xvb2c7MZhUXL/FJso7sNDsedCc=', 'Content-Length': '0'} 2024-04-13T22:30:35.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aadb1ffe04b8f5dd-00661b078b-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.810 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.810 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.810 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.810 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.810 DEBUG:boto:Method: GET 2024-04-13T22:30:35.810 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.810 DEBUG:boto:Data: 2024-04-13T22:30:35.810 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:35.811 DEBUG:boto:Port: 8001 2024-04-13T22:30:35.811 DEBUG:boto:Params: {} 2024-04-13T22:30:35.811 DEBUG:boto:Token: None 2024-04-13T22:30:35.811 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219589a8baaba9f3-00661b078b-1f77-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.813 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.814 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:35.814 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:35.814 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.814 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.814 DEBUG:boto:Method: GET 2024-04-13T22:30:35.814 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:35.815 DEBUG:boto:Data: 2024-04-13T22:30:35.815 DEBUG:boto:Headers: {} 2024-04-13T22:30:35.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:35.815 DEBUG:boto:Port: 8000 2024-04-13T22:30:35.815 DEBUG:boto:Params: {} 2024-04-13T22:30:35.815 DEBUG:boto:Token: None 2024-04-13T22:30:35.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:35 GMT /vwnfns-61/ 2024-04-13T22:30:35.815 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co= 2024-04-13T22:30:35.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:35 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AYVFTQBOlmcEWgSIUbYKz7EQ6Co=', 'Content-Length': '0'} 2024-04-13T22:30:35.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cd13dfcc7346f48-00661b078b-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:35.819 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:35.819 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:30:35.820 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:30:35.820 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:30:35.821 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:30:35.817+0000 7fcb3dd35700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 100313) UID: 0 2024-04-13T22:30:35.821 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:30:35.817+0000 7fcb42f0a7c0 -1 shutting down 2024-04-13T22:30:36.842 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:30:41.920 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:30:41.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:41.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:41.921 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:30:41.979 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:41.979 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj11", 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.080268Z", 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_U6n3uyL8iAGK0itaJfG0vGrZSWrZeKe", 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj111", 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.195853Z", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_ff3hnBaf7mZvXbfMXbQrcFs2AvoMX54", 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj115", 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.695128Z", 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_CwXjQ7xguyvXYwuRAjdOT84McwCUOxw", 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj116", 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.773219Z", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_XgMtT9JI4s8EwEEAPZGR61liOTydY1p", 2024-04-13T22:30:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj124", 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:42.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.237334Z", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.027 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_eUmwMoatDLJvkg2A-B7ngY_CIBVe54e", 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj145", 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.028 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 16 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.786931Z", 2024-04-13T22:30:42.029 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_Q4ZZU1tfiLanz4Pg-p8lkvJWXi-BHTy", 2024-04-13T22:30:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj148", 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.206936Z", 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_E1u_1WIlVrcIpsWpkplXPRddIN4zWP3", 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj155", 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.252143Z", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_baUaM4YLjFaQlNJnJlvwmWDMh_PehlB", 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj17", 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.339596Z", 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_1X-2bJIFh0dnl3MN9E0S4CfwjzZfWHH", 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj185", 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.256473Z", 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_fSXH90e-fLpFgVfzqtgf64-cMnRr-C5", 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj19", 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.160674Z", 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_kNQ7CWjoIY_AOLGOJ-TUvZSxhATeGtS", 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj22", 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.091623Z", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_cj4gczwaJXgvSz1dPqQkQ9awGwUSY38", 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj23", 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.577907Z", 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_NcYOIfEfxU3colAm2VInVHCTEZT9Yhy", 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj29", 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.699020Z", 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_GsHhViSfs88sAF7gcEg53G9Wx5TKIvU", 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj31", 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.122012Z", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_zBiLDWQV7mp_sSEA4aB-2aZCNJJkmw4", 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj33", 2024-04-13T22:30:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.139233Z", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_aMAx70BVqauf4TmQIsCcsPBxr-nfwpR", 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj34", 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.351692Z", 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_gY3Y-kA66aRKg7rcgpiJwD4PX7t9G2n", 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj37", 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.136916Z", 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_IE3r35aWOz6uoeYTgo2LSIWOj_G1q3a", 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj38", 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.211637Z", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_vmlWmV9NmCZXQmDwQ3VMR6OPYR2dOdc", 2024-04-13T22:30:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj44", 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 16 2024-04-13T22:30:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.148902Z", 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_fkxT-0xaXOYuvMYwxvO8yd0oZNtExuf", 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj46", 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.587952Z", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_NeoogSfQn4eBpNIzsKx-YudjFg3xw-Q", 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj51", 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.366001Z", 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_B8Ix4mZ9w0KXDsyY-V7cMT2zSb6O_gW", 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj55", 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.200001Z", 2024-04-13T22:30:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_8ZgvanbzyzPmBmfpnhfOO0Z5nji0r6T", 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj57", 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.220440Z", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.075 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.076 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.076 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.076 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_E3Qyiy-2npqnuLVhiI1D1o3qTe_KHNH", 2024-04-13T22:30:42.076 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.076 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj58", 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.077 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.709709Z", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_QPOopxaH6-lSb0-l-Oqv0aCVVe8j-2w", 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj62", 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.172563Z", 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_FsgUTKRnO65d2Rxf-Tk_pjLcKVqBicv", 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj68", 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.417071Z", 2024-04-13T22:30:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_yyy-yDV9uhAp1Jyn_OKd6byUsz1jTvV", 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj69", 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.084 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.637045Z", 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_oEadf22GP26JkcIGqYPImWthl_Lkrgl", 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj74", 2024-04-13T22:30:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 1 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.147565Z", 2024-04-13T22:30:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_i7_mToNMiq4AIghNSV-chgc0laQTtuv", 2024-04-13T22:30:42.088 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj76", 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.271532Z", 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_-QYwGyLTn1Yw1rJd7j2kHMWYJYo8dMH", 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj85", 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.427907Z", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_FIAD0qYe2w44EkZST_BkAOggI4TvP2f", 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj87", 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.096 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.096 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.096 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.096 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.719965Z", 2024-04-13T22:30:42.096 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_ZxdGqeIce8ododDiFcy0HBQR8r6qmAp", 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.097 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj92", 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.098 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.647609Z", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_outFfz2_zyCjUzRxNjMnJlTv2TUROO5", 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj93", 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.183373Z", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_EODCtUaEFxckYSB5x9ZBtwNClLuNSC7", 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj95", 2024-04-13T22:30:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.282329Z", 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_urdV0eyMgFcS1IixHMTcX0hEJWZvEpt", 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:42.105 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:30:42.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.106 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.157 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.157 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.246 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.295 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.295 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.381 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.431 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.432 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:30:42.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.522 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.572 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.572 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.660 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.709 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.796 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.846 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.847 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:42.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:42.933 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:42.983 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:42.983 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.071 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.121 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.121 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.208 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.258 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.258 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.345 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.395 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.395 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.483 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.621 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.671 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.671 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.760 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.810 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.810 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:43.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:43.899 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:43.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:43.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.038 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.087 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.087 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.175 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.312 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.361 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.361 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.449 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.499 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.499 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.588 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.725 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.775 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.775 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.862 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:44.911 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:44.911 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:44.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:44.999 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.050 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.050 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.142 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.192 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.193 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.280 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.418 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.468 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.468 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.558 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.608 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.608 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.696 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.745 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.746 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.833 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:45.883 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:45.883 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:45.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:45.970 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.019 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.020 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.108 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.157 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.158 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.245 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.294 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.295 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.385 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.436 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.436 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.525 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.575 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.575 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.671 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.720 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.720 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.808 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.858 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.858 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:46.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:46.945 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:46.992 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:46.992 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:47.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:47.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:47.044 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:47.093 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:47.093 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:47.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:47.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:30:47.140 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:30:47.190 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:47.191 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:47.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.259+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.259+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: waiting on bucket lease 2024-04-13T22:30:47.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.265+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: status after acquiring the lock is: init 2024-04-13T22:30:47.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.278+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bucket for full sync 2024-04-13T22:30:47.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj11 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj11 op=0 op_state=1 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj11[0] 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj111 op=0 op_state=1 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj115 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj115 op=0 op_state=1 2024-04-13T22:30:47.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj115[0] 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj116 op=0 op_state=1 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj116[0] 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj124 op=0 op_state=1 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj124[0] 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj145 op=0 op_state=1 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj145[0] 2024-04-13T22:30:47.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj148 op=0 op_state=1 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj148[0] 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj155 op=0 op_state=1 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj17 op=0 op_state=1 2024-04-13T22:30:47.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj17[0] 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj185 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj185 op=0 op_state=1 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj185[0] 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj19 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj19 op=0 op_state=1 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj19[0] 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj22 op=0 op_state=1 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2024-04-13T22:30:47.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj23 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj23 op=0 op_state=1 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj23[0] 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.323+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj29 op=0 op_state=1 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj31 2024-04-13T22:30:47.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj31 op=0 op_state=1 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj31[0] 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj33 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj33 op=0 op_state=1 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj33[0] 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj34 op=0 op_state=1 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj37 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj37 op=0 op_state=1 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj37[0] 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj38 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj38 op=0 op_state=1 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj38[0] 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj44 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj44 op=0 op_state=1 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj44[0] 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj46 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj46 op=0 op_state=1 2024-04-13T22:30:47.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.324+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj46[0] 2024-04-13T22:30:47.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj51 2024-04-13T22:30:47.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj51 op=0 op_state=1 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj51[0] 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj55 op=0 op_state=1 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj55[0] 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj57 op=0 op_state=1 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.327+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj57[0] 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.328+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.328+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj58 2024-04-13T22:30:47.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.329+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj58 op=0 op_state=1 2024-04-13T22:30:47.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.329+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj58[0] 2024-04-13T22:30:47.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.330+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.330+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.330+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj62 op=0 op_state=1 2024-04-13T22:30:47.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.330+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj62[0] 2024-04-13T22:30:47.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.347+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: success 2024-04-13T22:30:47.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.347+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.347+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj68 op=0 op_state=1 2024-04-13T22:30:47.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.347+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj68[0] 2024-04-13T22:30:47.352 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.349+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: success 2024-04-13T22:30:47.352 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.349+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj69 2024-04-13T22:30:47.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.349+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj69 op=0 op_state=1 2024-04-13T22:30:47.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.349+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj69[0] 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: success 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: success 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: success 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj74 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj74 op=0 op_state=1 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj74[0] 2024-04-13T22:30:47.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj76 op=0 op_state=1 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: success 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj76[0] 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj85 op=0 op_state=1 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj85[0] 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj87 op=0 op_state=1 2024-04-13T22:30:47.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.351+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj87[0] 2024-04-13T22:30:47.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.352+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: success 2024-04-13T22:30:47.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.352+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj92 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.352+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj92 op=0 op_state=1 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.352+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj92[0] 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: success 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj93 op=0 op_state=1 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj93[0] 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: success 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj95 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj95 op=0 op_state=1 2024-04-13T22:30:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj95[0] 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: success 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: success 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: success 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: success 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: success 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: success 2024-04-13T22:30:47.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.353+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: success 2024-04-13T22:30:47.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.382+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: success 2024-04-13T22:30:47.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.382+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: success 2024-04-13T22:30:47.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.383+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: success 2024-04-13T22:30:47.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.384+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: success 2024-04-13T22:30:47.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.385+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: success 2024-04-13T22:30:47.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.395+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: success 2024-04-13T22:30:47.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.400+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: success 2024-04-13T22:30:47.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.400+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: success 2024-04-13T22:30:47.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.402+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: success 2024-04-13T22:30:47.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.402+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: success 2024-04-13T22:30:47.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.402+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: success 2024-04-13T22:30:47.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.403+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: success 2024-04-13T22:30:47.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.403+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: success 2024-04-13T22:30:47.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.403+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: success 2024-04-13T22:30:47.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.403+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj11 2024-04-13T22:30:47.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.406+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: backing out with sync_status=-5 2024-04-13T22:30:47.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.406+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: ERROR: full sync failed. error: -5 2024-04-13T22:30:47.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.407+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.407+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: waiting on bucket lease 2024-04-13T22:30:47.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.411+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: status after acquiring the lock is: full 2024-04-13T22:30:47.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.424+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bucket for full sync 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj11 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj11 op=0 op_state=1 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj11[0] 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj111 op=0 op_state=1 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj111[0] 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj115 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj115 op=0 op_state=1 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj115[0] 2024-04-13T22:30:47.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj116 op=0 op_state=1 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj116[0] 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj124 op=0 op_state=1 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj124[0] 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj145 op=0 op_state=1 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj145[0] 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj148 op=0 op_state=1 2024-04-13T22:30:47.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj148[0] 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj155 op=0 op_state=1 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj155[0] 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj17 op=0 op_state=1 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj17[0] 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj185 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj185 op=0 op_state=1 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj185[0] 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj19 2024-04-13T22:30:47.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj19 op=0 op_state=1 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj19[0] 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj22 op=0 op_state=1 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj22[0] 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj23 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj23 op=0 op_state=1 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj23[0] 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj29 op=0 op_state=1 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj29[0] 2024-04-13T22:30:47.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj31 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj31 op=0 op_state=1 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj31[0] 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.469+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj33 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj33 op=0 op_state=1 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj33[0] 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj34 op=0 op_state=1 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj34[0] 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj37 2024-04-13T22:30:47.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj37 op=0 op_state=1 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj37[0] 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj38 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj38 op=0 op_state=1 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj38[0] 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj44 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj44 op=0 op_state=1 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj44[0] 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj46 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj46 op=0 op_state=1 2024-04-13T22:30:47.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.470+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj46[0] 2024-04-13T22:30:47.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.473+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: success 2024-04-13T22:30:47.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.473+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj51 2024-04-13T22:30:47.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.473+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj51 op=0 op_state=1 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.473+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj51[0] 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.473+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.473+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj55 op=0 op_state=1 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj55[0] 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: success 2024-04-13T22:30:47.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj57 op=0 op_state=1 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj57[0] 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj58 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj58 op=0 op_state=1 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj58[0] 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj62 op=0 op_state=1 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.474+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj62[0] 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj68 op=0 op_state=1 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj69 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj69 op=0 op_state=1 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj69[0] 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj74 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj74 op=0 op_state=1 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: success 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj74[0] 2024-04-13T22:30:47.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj76 op=0 op_state=1 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: success 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj85 op=0 op_state=1 2024-04-13T22:30:47.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.475+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj85[0] 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: success 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: success 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: success 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj87 op=0 op_state=1 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj92 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj92 op=0 op_state=1 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj92[0] 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj93 op=0 op_state=1 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj93[0] 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj95 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj95 op=0 op_state=1 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.479+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj95[0] 2024-04-13T22:30:47.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: success 2024-04-13T22:30:47.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.481+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: success 2024-04-13T22:30:47.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.482+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: success 2024-04-13T22:30:47.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.482+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.482+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.482+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: success 2024-04-13T22:30:47.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.483+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.483+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.484+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.484+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.484+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.485+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.485+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: success 2024-04-13T22:30:47.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.486+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: success 2024-04-13T22:30:47.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.486+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj23 2024-04-13T22:30:47.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.489+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: success 2024-04-13T22:30:47.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.489+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj62 2024-04-13T22:30:47.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.491+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: backing out with sync_status=-5 2024-04-13T22:30:47.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.491+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: ERROR: full sync failed. error: -5 2024-04-13T22:30:47.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.492+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.492+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: waiting on bucket lease 2024-04-13T22:30:47.540 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.537+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: status after acquiring the lock is: full 2024-04-13T22:30:47.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.548+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bucket for full sync 2024-04-13T22:30:47.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj11 2024-04-13T22:30:47.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj11 op=0 op_state=1 2024-04-13T22:30:47.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj11[0] 2024-04-13T22:30:47.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj111 op=0 op_state=1 2024-04-13T22:30:47.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj111[0] 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj115 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj115 op=0 op_state=1 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj115[0] 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj116 op=0 op_state=1 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj124 op=0 op_state=1 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj124[0] 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj145 op=0 op_state=1 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj145[0] 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj148 op=0 op_state=1 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.594+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj148[0] 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj155 op=0 op_state=1 2024-04-13T22:30:47.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj17 op=0 op_state=1 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj17[0] 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj185 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj185 op=0 op_state=1 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj185[0] 2024-04-13T22:30:47.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj19 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj19 op=0 op_state=1 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj19[0] 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj22 op=0 op_state=1 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj22[0] 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj23 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj23 op=0 op_state=1 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj23[0] 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj29 op=0 op_state=1 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj29[0] 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj31 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj31 op=0 op_state=1 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj31[0] 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj33 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj33 op=0 op_state=1 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj33[0] 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj34 op=0 op_state=1 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj34[0] 2024-04-13T22:30:47.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj37 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj37 op=0 op_state=1 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj37[0] 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj38 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj38 op=0 op_state=1 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj38[0] 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj44 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj44 op=0 op_state=1 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj44[0] 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj46 2024-04-13T22:30:47.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj46 op=0 op_state=1 2024-04-13T22:30:47.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.595+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj46[0] 2024-04-13T22:30:47.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.597+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: success 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.597+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj51 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.597+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj51 op=0 op_state=1 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.597+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj51[0] 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: success 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj55 op=0 op_state=1 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj55[0] 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: success 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj57 op=0 op_state=1 2024-04-13T22:30:47.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj57[0] 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: success 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj58 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj58 op=0 op_state=1 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj58[0] 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: success 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj62 op=0 op_state=1 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.599+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: success 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: success 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: success 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: success 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj68 op=0 op_state=1 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.600+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj68[0] 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj69 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj69 op=0 op_state=1 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj69[0] 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj74 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj74 op=0 op_state=1 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj74[0] 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj76 op=0 op_state=1 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj76[0] 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj85 op=0 op_state=1 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj87 op=0 op_state=1 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.601+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: success 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: success 2024-04-13T22:30:47.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj115 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: success 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj92 2024-04-13T22:30:47.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj92 op=0 op_state=1 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj92[0] 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj93 op=0 op_state=1 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj95 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj95 op=0 op_state=1 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.602+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj95[0] 2024-04-13T22:30:47.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.603+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: success 2024-04-13T22:30:47.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.604+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: success 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: success 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: success 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: success 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: success 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: success 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.605+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: success 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: success 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: success 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: backing out with sync_status=-5 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.606+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: ERROR: full sync failed. error: -5 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.608+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.608+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: waiting on bucket lease 2024-04-13T22:30:47.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.654+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: status after acquiring the lock is: full 2024-04-13T22:30:47.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.665+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bucket for full sync 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj11 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj11 op=0 op_state=1 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj11[0] 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj111 op=0 op_state=1 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj115 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj115 op=0 op_state=1 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj115[0] 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj116 op=0 op_state=1 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj116[0] 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj124 op=0 op_state=1 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj145 op=0 op_state=1 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj145[0] 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj148 op=0 op_state=1 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj155 op=0 op_state=1 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj155[0] 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj17 op=0 op_state=1 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj185 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj185 op=0 op_state=1 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj185[0] 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj19 2024-04-13T22:30:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj19 op=0 op_state=1 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj19[0] 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj22 op=0 op_state=1 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj22[0] 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj23 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj23 op=0 op_state=1 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.711+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj23[0] 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj29 op=0 op_state=1 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj29[0] 2024-04-13T22:30:47.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj31 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj31 op=0 op_state=1 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj31[0] 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj33 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj33 op=0 op_state=1 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj33[0] 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj34 op=0 op_state=1 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj34[0] 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj37 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj37 op=0 op_state=1 2024-04-13T22:30:47.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj37[0] 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj38 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj38 op=0 op_state=1 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj38[0] 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj44 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj44 op=0 op_state=1 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj44[0] 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj46 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj46 op=0 op_state=1 2024-04-13T22:30:47.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.712+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj46[0] 2024-04-13T22:30:47.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.715+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.715+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj51 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.715+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj51 op=0 op_state=1 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.715+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj51[0] 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.715+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.715+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.716+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj55 op=0 op_state=1 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.716+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.716+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.716+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: success 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: success 2024-04-13T22:30:47.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: success 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: success 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj57 op=0 op_state=1 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj58 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj58 op=0 op_state=1 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj58[0] 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj62 op=0 op_state=1 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj62[0] 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj68 op=0 op_state=1 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.717+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj68[0] 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: success 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: success 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj69 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj69 op=0 op_state=1 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj69[0] 2024-04-13T22:30:47.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj74 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj74 op=0 op_state=1 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj74[0] 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: success 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj76 op=0 op_state=1 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj76[0] 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj85 op=0 op_state=1 2024-04-13T22:30:47.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj85[0] 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj87 op=0 op_state=1 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.718+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj87[0] 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: success 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: success 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: success 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: success 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: success 2024-04-13T22:30:47.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: success 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: success 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: success 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: success 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: success 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: success 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj92 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj92 op=0 op_state=1 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj92[0] 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj93 op=0 op_state=1 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: success 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: success 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj93[0] 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj95 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj95 op=0 op_state=1 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj95[0] 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.720+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: success 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.721+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: success 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: success 2024-04-13T22:30:47.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: success 2024-04-13T22:30:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: success 2024-04-13T22:30:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: success 2024-04-13T22:30:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: success 2024-04-13T22:30:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: success 2024-04-13T22:30:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: success 2024-04-13T22:30:47.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.722+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj11 2024-04-13T22:30:47.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.724+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: backing out with sync_status=-5 2024-04-13T22:30:47.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.724+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: ERROR: full sync failed. error: -5 2024-04-13T22:30:47.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.725+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.725+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: waiting on bucket lease 2024-04-13T22:30:47.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.770+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: status after acquiring the lock is: full 2024-04-13T22:30:47.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.782+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bucket for full sync 2024-04-13T22:30:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj11 2024-04-13T22:30:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj11 op=0 op_state=1 2024-04-13T22:30:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj11[0] 2024-04-13T22:30:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj111 op=0 op_state=1 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj111[0] 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj115 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj115 op=0 op_state=1 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj115[0] 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj116 op=0 op_state=1 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj116[0] 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj124 op=0 op_state=1 2024-04-13T22:30:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj124[0] 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj145 op=0 op_state=1 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj148 op=0 op_state=1 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj148[0] 2024-04-13T22:30:47.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj155 op=0 op_state=1 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj155[0] 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj17 op=0 op_state=1 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj17[0] 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj185 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.827+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj185 op=0 op_state=1 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj185[0] 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj19 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj19 op=0 op_state=1 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj19[0] 2024-04-13T22:30:47.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj22 op=0 op_state=1 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj22[0] 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj23 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj23 op=0 op_state=1 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj23[0] 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj29 op=0 op_state=1 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj31 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj31 op=0 op_state=1 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj31[0] 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj33 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj33 op=0 op_state=1 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj33[0] 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj34 op=0 op_state=1 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj34[0] 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj37 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj37 op=0 op_state=1 2024-04-13T22:30:47.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj37[0] 2024-04-13T22:30:47.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj38 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj38 op=0 op_state=1 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj38[0] 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj44 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj44 op=0 op_state=1 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj44[0] 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj46 2024-04-13T22:30:47.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj46 op=0 op_state=1 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.828+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj46[0] 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.831+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: success 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.831+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj51 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.831+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj51 op=0 op_state=1 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.831+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj51[0] 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj55 op=0 op_state=1 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: success 2024-04-13T22:30:47.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: success 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: success 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: success 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: success 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: success 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj55[0] 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj57 op=0 op_state=1 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj57[0] 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj58 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj58 op=0 op_state=1 2024-04-13T22:30:47.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj58[0] 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj62 op=0 op_state=1 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: success 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj62[0] 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj68 op=0 op_state=1 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: success 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj68[0] 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj69 2024-04-13T22:30:47.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj69 op=0 op_state=1 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj69[0] 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj74 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj74 op=0 op_state=1 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj74[0] 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj76 op=0 op_state=1 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj76[0] 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj85 op=0 op_state=1 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-13T22:30:47.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: success 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 0 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: ERROR: failed to sync object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: success 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj87 op=0 op_state=1 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj87[0] 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj92 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj92 op=0 op_state=1 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj92[0] 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj93 op=0 op_state=1 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.833+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj93[0] 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: success 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: success 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: success 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: success 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj95 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj95 op=0 op_state=1 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj95[0] 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.834+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: success 2024-04-13T22:30:47.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.835+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: success 2024-04-13T22:30:47.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.836+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: success 2024-04-13T22:30:47.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.836+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: success 2024-04-13T22:30:47.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.836+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: success 2024-04-13T22:30:47.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.837+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: a sync operation returned error 2024-04-13T22:30:47.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.837+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: success 2024-04-13T22:30:47.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.837+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.838+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.839+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.839+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.839+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: success 2024-04-13T22:30:47.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.839+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj124 2024-04-13T22:30:47.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.840+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: backing out with sync_status=-5 2024-04-13T22:30:47.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.840+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: ERROR: full sync failed. error: -5 2024-04-13T22:30:47.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.841+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.841+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: waiting on bucket lease 2024-04-13T22:30:47.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.886+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: status after acquiring the lock is: full 2024-04-13T22:30:47.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.898+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bucket for full sync 2024-04-13T22:30:47.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj11 2024-04-13T22:30:47.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj11 op=0 op_state=1 2024-04-13T22:30:47.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj11[0] 2024-04-13T22:30:47.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj111 2024-04-13T22:30:47.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj111 op=0 op_state=1 2024-04-13T22:30:47.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj111[0] 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj115 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj115 op=0 op_state=1 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj115[0] 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj116 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj116 op=0 op_state=1 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj116[0] 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj124 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj124 op=0 op_state=1 2024-04-13T22:30:47.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj124[0] 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj145 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj145 op=0 op_state=1 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj145[0] 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj148 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj148 op=0 op_state=1 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj148[0] 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj155 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj155 op=0 op_state=1 2024-04-13T22:30:47.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj155[0] 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj17 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj17 op=0 op_state=1 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj17[0] 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj185 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj185 op=0 op_state=1 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj185[0] 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj19 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj19 op=0 op_state=1 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj19[0] 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj22 2024-04-13T22:30:47.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj22 op=0 op_state=1 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj22[0] 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj23 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj23 op=0 op_state=1 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj23[0] 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj29 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj29 op=0 op_state=1 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj29[0] 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj31 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj31 op=0 op_state=1 2024-04-13T22:30:47.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj31[0] 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj33 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj33 op=0 op_state=1 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.943+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj33[0] 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj34 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj34 op=0 op_state=1 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj34[0] 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj37 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj37 op=0 op_state=1 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj37[0] 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj38 2024-04-13T22:30:47.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj38 op=0 op_state=1 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj38[0] 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj44 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj44 op=0 op_state=1 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj44[0] 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj46 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj46 op=0 op_state=1 2024-04-13T22:30:47.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.944+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj46[0] 2024-04-13T22:30:47.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj115]: success 2024-04-13T22:30:47.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj116]: success 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj51 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj51 op=0 op_state=1 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj51[0] 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj55 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj55 op=0 op_state=1 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj55[0] 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj145]: success 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj57 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj57 op=0 op_state=1 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj57[0] 2024-04-13T22:30:47.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj155]: success 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.947+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj58 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj58 op=0 op_state=1 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj58[0] 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj111]: success 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj11]: success 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj62 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj62 op=0 op_state=1 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj62[0] 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj68 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj68 op=0 op_state=1 2024-04-13T22:30:47.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj68[0] 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj124]: success 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj69 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj69 op=0 op_state=1 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj69[0] 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj31]: success 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj74 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj74 op=0 op_state=1 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj74[0] 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj185]: success 2024-04-13T22:30:47.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj76 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj76 op=0 op_state=1 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.948+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj76[0] 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.949+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj46]: success 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.949+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj85 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.949+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj85 op=0 op_state=1 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.949+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj85[0] 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj37]: success 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj44]: success 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj23]: success 2024-04-13T22:30:47.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj87 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj87 op=0 op_state=1 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj87[0] 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj92 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj92 op=0 op_state=1 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj92[0] 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj93 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj93 op=0 op_state=1 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj93[0] 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj17]: success 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: [full sync] syncing object: vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0/obj95 2024-04-13T22:30:47.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b=bucket sync single entry (source_zone=f6f208d6-f791-479b-bc3d-112512eec2d4) b= log_entry=obj95 op=0 op_state=1 2024-04-13T22:30:47.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.950+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: bucket sync: sync obj: f6f208d6-f791-479b-bc3d-112512eec2d4/:vwnfns-61[f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24])/obj95[0] 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.952+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj33]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.952+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj19]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.952+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj29]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.952+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj22]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj51]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj62]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj76]: success 2024-04-13T22:30:47.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj148]: success 2024-04-13T22:30:47.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj33 2024-04-13T22:30:47.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj38]: success 2024-04-13T22:30:47.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj34]: success 2024-04-13T22:30:47.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj95]: success 2024-04-13T22:30:47.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj68]: success 2024-04-13T22:30:47.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.953+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj69]: success 2024-04-13T22:30:47.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj57]: success 2024-04-13T22:30:47.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj74]: success 2024-04-13T22:30:47.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj55]: success 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj87]: success 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj85]: success 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj92]: success 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj58]: success 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj92 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:entry[obj93]: success 2024-04-13T22:30:47.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.954+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: updating marker oid=bucket.full-sync-status.f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24 marker=obj95 2024-04-13T22:30:47.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.956+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: set bucket state=incremental 2024-04-13T22:30:47.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.957+0000 7fee2b625780 5 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:full_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: bucket status objv={r=_2YGwJxuzPQb_tslc_aBYnk8:149,w=:0} 2024-04-13T22:30:47.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.958+0000 7fee2b625780 10 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2024-04-13T22:30:47.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.960+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:2]: sync status for source bucket shard: 2 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:2]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:2]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:4]: sync status for source bucket shard: 2 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:4]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:4]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:5]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:5]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:5]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.961+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:0]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:3]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:6]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:9]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:1]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:8]: sync status for source bucket shard: 2 2024-04-13T22:30:47.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:7]: sync status for source bucket shard: 2 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:3]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:3]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:6]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:6]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:9]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:9]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:1]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:1]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:8]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:8]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.962+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:7]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:7]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:10]: sync status for source bucket shard: 2 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:10]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:10]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.963+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:12]: sync status for source bucket shard: 2 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:12]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:12]: listing bilog for incremental sync; position=00000000002.87.6 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:14]: sync status for source bucket shard: 2 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:15]: sync status for source bucket shard: 2 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:11]: sync status for source bucket shard: 2 2024-04-13T22:30:47.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:16]: sync status for source bucket shard: 2 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:13]: sync status for source bucket shard: 2 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:17]: sync status for source bucket shard: 2 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:14]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:14]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:15]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:15]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:11]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:11]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:16]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:16]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:13]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:13]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.964+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:17]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:17]: listing bilog for incremental sync; position=00000000002.100.6 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.965+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.965+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.965+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:18]: sync status for source bucket shard: 2 2024-04-13T22:30:47.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:18]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:18]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:19]: sync status for source bucket shard: 2 2024-04-13T22:30:47.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:21]: sync status for source bucket shard: 2 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:20]: sync status for source bucket shard: 2 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:19]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:19]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:21]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:21]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.966+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:20]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:20]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.967+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:24]: sync status for source bucket shard: 2 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.968+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:24]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:24]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:25]: sync status for source bucket shard: 2 2024-04-13T22:30:47.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:22]: sync status for source bucket shard: 2 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:23]: sync status for source bucket shard: 2 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:26]: sync status for source bucket shard: 2 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:30]: sync status for source bucket shard: 2 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:29]: sync status for source bucket shard: 2 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:25]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:25]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:22]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:22]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:23]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:23]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:26]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:26]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:30]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:30]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.969+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:29]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:29]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:32]: sync status for source bucket shard: 2 2024-04-13T22:30:47.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:27]: sync status for source bucket shard: 2 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:28]: sync status for source bucket shard: 2 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:33]: sync status for source bucket shard: 2 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:32]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:32]: listing bilog for incremental sync; position=00000000001.82.6 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:27]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:27]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:28]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:28]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.970+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:33]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:33]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.971+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:31]: sync status for source bucket shard: 2 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.971+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:31]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:31]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.971+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:35]: sync status for source bucket shard: 2 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.971+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:35]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:35]: listing bilog for incremental sync; position= 2024-04-13T22:30:47.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.972+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:36]: sync status for source bucket shard: 2 2024-04-13T22:30:47.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.972+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:36]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:36]: listing bilog for incremental sync; position=00000000001.61.6 2024-04-13T22:30:47.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.972+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:34]: sync status for source bucket shard: 2 2024-04-13T22:30:47.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:47.972+0000 7fee2b625780 20 RGW-SYNC:bucket[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24<-vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:34]:inc_sync[vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24:34]: listing bilog for incremental sync; position= 2024-04-13T22:30:48.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:48.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:48.039 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:30:48.088 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:48.088 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj11", 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.121 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.080268Z", 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.122 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.2736335920438274646", 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.123 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj111", 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.124 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.195853Z", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.125 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4895232112641871450", 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj115", 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.695128Z", 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.127 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16242744964584326870", 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.128 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj116", 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.129 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.773219Z", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.130 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.3357980972682958046", 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj124", 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.131 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.237334Z", 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.132 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.6697530960449421260", 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.133 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj145", 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.134 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.786931Z", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.135 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.1154812115150167318", 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj148", 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.136 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.206936Z", 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.137 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12660076680301392011", 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj155", 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.252143Z", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16478293886796138892", 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj17", 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.141 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.339596Z", 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.142 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.3504007711377721195", 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj185", 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.144 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.256473Z", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13773979257566978168", 2024-04-13T22:30:48.145 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj19", 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:30:48.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.160674Z", 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.147 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.836944743856336907", 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj22", 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 2 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.149 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.091623Z", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.6113495079345428376", 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj23", 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.151 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.577907Z", 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.152 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12562293300913698241", 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj29", 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.699020Z", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.155 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14598905399172336650", 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj31", 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 19 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.156 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.122012Z", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.157 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4022728150037384596", 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.158 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj33", 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.139233Z", 2024-04-13T22:30:48.159 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16440851735355679998", 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.160 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj34", 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.161 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.351692Z", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.162 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8465864414952644947", 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj37", 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.163 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.136916Z", 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16382156171365670300", 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj38", 2024-04-13T22:30:48.165 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.166 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.211637Z", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10660284179544355544", 2024-04-13T22:30:48.167 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj44", 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 17 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.168 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.148902Z", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.169 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.10776813436135018861", 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.170 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj46", 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.587952Z", 2024-04-13T22:30:48.171 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.7254623085383189564", 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.172 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj51", 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.173 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.366001Z", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.174 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8928560277670892708", 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj55", 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.175 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.200001Z", 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.176 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.942818908511228665", 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.177 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj57", 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.220440Z", 2024-04-13T22:30:48.178 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.12372418105539241781", 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.179 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj58", 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.180 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.709709Z", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.3195354117763026480", 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj62", 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.182 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.172563Z", 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.183 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13137608927860194330", 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.184 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj68", 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.417071Z", 2024-04-13T22:30:48.185 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.15076826513965684562", 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.186 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj69", 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 18 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.187 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.637045Z", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.188 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.5239096948326279285", 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj74", 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 1 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.189 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.147565Z", 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.190 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.17984558321806693817", 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj76", 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.191 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 18 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.271532Z", 2024-04-13T22:30:48.192 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.2716893980355941956", 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.194 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj85", 2024-04-13T22:30:48.194 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.194 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.194 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.195 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:30:48.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.195 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.195 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.195 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.427907Z", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.13236562012089696270", 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj87", 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:48.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.719965Z", 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.198 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.8090548589016940940", 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj92", 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.647609Z", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.2861710359877804386", 2024-04-13T22:30:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj93", 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.183373Z", 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.203 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.14774669802864364510", 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj95", 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.204 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 9, 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.282329Z", 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.205 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16660816676279203678", 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.206 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.207 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:48.207 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:30:48.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:48.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:48.208 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:30:48.259 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:48.259 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:48.290 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:30:48.290 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.290 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj11", 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.291 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.080268Z", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.292 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_7rVptkSQcwsohwErNAkSzKm8yxFfYxt", 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj111", 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.195853Z", 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.294 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_4-e55zBQNiufB2U8yr3eboIvwbTNcGn", 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.295 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj115", 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.296 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.695128Z", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_knsK3e1S8jcsd4Irw99r5hxW19Po0NR", 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.297 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj116", 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.298 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.773219Z", 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.299 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_widvmXGifDYIutl2o8An2PgxBSSexT-", 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj124", 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.300 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.237334Z", 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.301 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_Pmbu_eKeLCG7rpS_vxvOxuHiV50V1Fe", 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj145", 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 21 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.303 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.786931Z", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.304 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_ByPEf0EcqBiD948UPTrnc4MS7MsQMpU", 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj148", 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.305 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.206936Z", 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.306 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_yIaFvcxaPYN4PZiuzswyYJ8n0pkFvBM", 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj155", 2024-04-13T22:30:48.307 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.252143Z", 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.308 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_3htDHiDdRAcvn7D8yssV24WF_0aN375", 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.309 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj17", 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 20 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.310 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.311 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.339596Z", 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.313 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_JPwfkyaD9swxk4s5Sgq5fWyByB3H6zC", 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj185", 2024-04-13T22:30:48.314 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.315 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.256473Z", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_p8uTUOsCYdXDdgttExlxeK_SREzkyF8", 2024-04-13T22:30:48.316 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj19", 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.317 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.160674Z", 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.318 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_peaAyErVhdNrGOkjr2NfN_M9F6UL2c-", 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj22", 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 4 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.091623Z", 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.320 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_QY7Qsr0voz7hYYLeeOJXSIv8qZQTiTI", 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj23", 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 22 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.322 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.577907Z", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_Y_lGKTpq3ui-rWSIIHj67jOsLVu8Bhl", 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj29", 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:48.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.699020Z", 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.325 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_TQfB7aQiMbzX_0SfGMV7UPfhmtwA2ux", 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.326 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj31", 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.327 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.122012Z", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_MaraN4N2wflJ7roVdSK5j0B8tk6QM3T", 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.328 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj33", 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 11 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.329 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.139233Z", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.330 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_Fy1s9hQAhiSlWc5PVTy8ObpRhEJUotW", 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj34", 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.331 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.351692Z", 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.332 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_mwhHfG1YozxNuR8llmUn7gBwRIMKBi-", 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj37", 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.334 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.136916Z", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_4PbDzGOpcfhjrPWmGcbgr5eHusHgvco", 2024-04-13T22:30:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj38", 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.336 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.211637Z", 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.337 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_7ePwVzFwkJCYNxnm_hmc9zRC21w_aZH", 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.338 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj44", 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 18 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.148902Z", 2024-04-13T22:30:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_zMu_8DZJWMxsti69zdECRDtLbBlzZYV", 2024-04-13T22:30:48.340 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj46", 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 7 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.341 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.587952Z", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.342 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "__P8vuVcWhcq5KTHSUtmWugBw8DxM_DR", 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj51", 2024-04-13T22:30:48.343 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.366001Z", 2024-04-13T22:30:48.344 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_bNDpmfeB5ZD9FOKbgInpim-Nl_3TrWd", 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.345 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj55", 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.346 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:16:35.200001Z", 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_S-dLMKBvrJAXDu8MJ0TaJ1hfXLn2I8f", 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj57", 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.349 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.220440Z", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_-UdyPn9N8SOMSr73OVxfOBpoTiiJ7FE", 2024-04-13T22:30:48.350 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj58", 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.351 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.709709Z", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.352 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_Rspj-JiysBOCyuOp_oq6NymS5QoAalm", 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj62", 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.353 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.172563Z", 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.354 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_3_O-h1_GSyAGJ6iAB3cEUuBPovppFz4", 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.355 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj68", 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 5 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.356 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.417071Z", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.357 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_V_MhCfU94rlKTVR6RsZIusBPd2JDwkp", 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj69", 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.358 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 9 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 3, 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.637045Z", 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.359 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_-MjCOp7Vdr05UmhHrZaKPVjZWxTXBji", 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj74", 2024-04-13T22:30:48.360 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 3 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:28:56.147565Z", 2024-04-13T22:30:48.361 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_dK1HcLjIDTcDclpuUD6wL-vKL2B3sMl", 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.362 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj76", 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.363 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.271532Z", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_ODSA1TKmqVH_BZi19ude4ALM_T66oY7", 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj85", 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.365 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 12 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:18:11.427907Z", 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.366 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_1WyG-p_jjCPhmyMWUnwob66FxWqD8TD", 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj87", 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 6 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.368 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.369 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.369 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:25:33.719965Z", 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.370 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_vt1t9npvqlfbVn9bmnwVE6t6INY7b3Y", 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj92", 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.371 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 16, 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:22:50.647609Z", 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.372 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_B8w0X9Uz-FiISPSPnm_VwEGrg86CR-V", 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.373 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj93", 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 8 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.374 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 5, 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:27:52.183373Z", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.375 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_PZsfbYi38-BlXzB8orwRlhEpcwaNGMQ", 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "obj95", 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "instance": "", 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "ver": { 2024-04-13T22:30:48.376 INFO:teuthology.orchestra.run.smithi132.stdout: "pool": 13, 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 10 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "locator": "", 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "exists": true, 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "meta": { 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "category": 1, 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 6, 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "mtime": "2024-04-13T22:21:13.282329Z", 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_class": "", 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "owner": "rgw-multisite-test-user", 2024-04-13T22:30:48.377 INFO:teuthology.orchestra.run.smithi132.stdout: "owner_display_name": "Multisite Test User", 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "content_type": "application/octet-stream", 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "accounted_size": 3, 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "user_data": "", 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "appendable": false 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "tag": "_EJy8IeLKqRmVRkKquFS9qDzzUkhErB1", 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "flags": 0, 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "pending_map": [], 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: "versioned_epoch": 0 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:48.378 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:30:48.379 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:30:48.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:30:48.382 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:30:48.382 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:30:48.401 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:30:48.402 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:30:48.402 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:30:49.402 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:49.423 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:30:49.423 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:30:49.423 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:30:49.424 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:30:49.425 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-61 2024-04-13T22:30:49.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:30:49.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:30:49.425 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:30:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:49.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:49.559+0000 7f895050d780 1 bucket sync caught up with source: 2024-04-13T22:30:49.563 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:49.563 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.69.6, , , , , , 00000000002.87.6, , , , , 00000000002.100.6, , , , , , , , , , , , , , , 00000000001.82.6, , , , 00000000001.61.6] 2024-04-13T22:30:49.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:49.559+0000 7f895050d780 0 bucket checkpoint complete 2024-04-13T22:30:49.576 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-61 2024-04-13T22:30:49.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:30:49.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:30:49.577 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:30:49.631 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:49.631 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:49.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:49.665+0000 7f9f88f84780 1 waiting for incremental sync to catch up: 2024-04-13T22:30:49.668 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000009.2095.6, 00000000006.2310.6, 00000000002.3339.6, 00000000015.1243.6, 00000000012.1177.6, 00000000003.1790.6, 00000000001.2066.6, 00000000012.657.6, 00000000021.1944.6, 00000000018.1800.6] 2024-04-13T22:30:49.668 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000009.2095.6, 00000000006.2310.6, 00000000006.3351.6, 00000000015.1243.6, 00000000012.1177.6, 00000000003.1790.6, 00000000003.2096.6, 00000000012.657.6, 00000000021.1944.6, 00000000018.1800.6] 2024-04-13T22:30:54.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:54.667+0000 7f9f88f84780 1 bucket sync caught up with source: 2024-04-13T22:30:54.669 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, 00000000009.2095.6, 00000000006.2310.6, 00000000006.3351.6, 00000000015.1243.6, 00000000012.1177.6, 00000000003.1790.6, 00000000003.2096.6, 00000000012.657.6, 00000000021.1944.6, 00000000018.1800.6] 2024-04-13T22:30:54.670 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, 00000000009.2095.6, 00000000006.2310.6, 00000000006.3351.6, 00000000015.1243.6, 00000000012.1177.6, 00000000003.1790.6, 00000000003.2096.6, 00000000012.657.6, 00000000021.1944.6, 00000000018.1800.6] 2024-04-13T22:30:54.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:54.667+0000 7f9f88f84780 0 bucket checkpoint complete 2024-04-13T22:30:54.688 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:30:54.688 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:54.689 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:54.689 DEBUG:boto:Method: HEAD 2024-04-13T22:30:54.689 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:30:54.689 DEBUG:boto:Data: 2024-04-13T22:30:54.689 DEBUG:boto:Headers: {} 2024-04-13T22:30:54.689 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:54.689 DEBUG:boto:Port: 8001 2024-04-13T22:30:54.689 DEBUG:boto:Params: {} 2024-04-13T22:30:54.689 DEBUG:boto:Token: None 2024-04-13T22:30:54.690 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:54 GMT /vwnfns-61/ 2024-04-13T22:30:54.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xdmCUc3+PdsmSAySTrtjZnpVIWY= 2024-04-13T22:30:54.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xdmCUc3+PdsmSAySTrtjZnpVIWY=', 'Content-Length': '0'} 2024-04-13T22:30:54.691 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-13T22:30:54.691 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:30:55.004 DEBUG:boto:Token: None 2024-04-13T22:30:55.004 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:54 GMT /vwnfns-61/ 2024-04-13T22:30:55.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xdmCUc3+PdsmSAySTrtjZnpVIWY= 2024-04-13T22:30:55.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xdmCUc3+PdsmSAySTrtjZnpVIWY=', 'Content-Length': '0'} 2024-04-13T22:30:55.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072ddbd0cabd7c92d-00661b079f-21de-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.011 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.011 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.011 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.012 DEBUG:boto:Path: /vwnfns-61/ 2024-04-13T22:30:55.012 DEBUG:boto:Data: 2024-04-13T22:30:55.012 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.012 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.012 DEBUG:boto:Params: {} 2024-04-13T22:30:55.012 DEBUG:boto:Token: None 2024-04-13T22:30:55.012 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BWZlnAD4fj8gJczjM5zsq6p4k+U= 2024-04-13T22:30:55.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BWZlnAD4fj8gJczjM5zsq6p4k+U=', 'Content-Length': '0'} 2024-04-13T22:30:55.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b86a3f3f2d169135-00661b079f-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:30:55.017 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.017 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.017 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.017 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.018 DEBUG:boto:Method: GET 2024-04-13T22:30:55.018 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.018 DEBUG:boto:Data: 2024-04-13T22:30:55.018 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.018 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.018 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.018 DEBUG:boto:Params: {} 2024-04-13T22:30:55.018 DEBUG:boto:Token: None 2024-04-13T22:30:55.018 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/?versions 2024-04-13T22:30:55.019 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ= 2024-04-13T22:30:55.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ=', 'Content-Length': '0'} 2024-04-13T22:30:55.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000655545dc6d67b76c-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.022 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-13T22:30:55.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:30:55.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:30:55.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:30:55.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-13T22:30:55.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:30:55.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:30:55.027 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.027 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.027 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.027 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.027 DEBUG:boto:Method: GET 2024-04-13T22:30:55.027 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.027 DEBUG:boto:Data: 2024-04-13T22:30:55.027 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.028 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.028 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.028 DEBUG:boto:Params: {} 2024-04-13T22:30:55.028 DEBUG:boto:Token: None 2024-04-13T22:30:55.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/?versions 2024-04-13T22:30:55.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ= 2024-04-13T22:30:55.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ=', 'Content-Length': '0'} 2024-04-13T22:30:55.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000160dc82b6d6a7758-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.033 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-13T22:30:55.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-13T22:30:55.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-13T22:30:55.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-13T22:30:55.038 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.038 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.038 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.038 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.038 DEBUG:boto:Method: GET 2024-04-13T22:30:55.038 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.038 DEBUG:boto:Data: 2024-04-13T22:30:55.038 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.038 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.039 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.039 DEBUG:boto:Params: {} 2024-04-13T22:30:55.039 DEBUG:boto:Token: None 2024-04-13T22:30:55.039 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/?versions 2024-04-13T22:30:55.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ= 2024-04-13T22:30:55.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ=', 'Content-Length': '0'} 2024-04-13T22:30:55.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005747adbbe8a0cc77-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.042 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.044 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.044 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.044 DEBUG:boto:path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.045 DEBUG:boto:auth_path=/vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.045 DEBUG:boto:Method: GET 2024-04-13T22:30:55.045 DEBUG:boto:Path: /vwnfns-61/?versions&max-keys=999 2024-04-13T22:30:55.045 DEBUG:boto:Data: 2024-04-13T22:30:55.045 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.045 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.045 DEBUG:boto:Params: {} 2024-04-13T22:30:55.045 DEBUG:boto:Token: None 2024-04-13T22:30:55.045 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/?versions 2024-04-13T22:30:55.046 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ= 2024-04-13T22:30:55.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:78JyaoNOGjfn3MlaEnfcV72sRpQ=', 'Content-Length': '0'} 2024-04-13T22:30:55.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea7d2dbfd6d5ef0-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.050 DEBUG:boto:b'vwnfns-61999falseobj11nulltrue2024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:30:55.052 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.052 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.052 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.052 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.053 DEBUG:boto:Method: GET 2024-04-13T22:30:55.053 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.053 DEBUG:boto:Data: 2024-04-13T22:30:55.053 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.053 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.053 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.053 DEBUG:boto:Params: {} 2024-04-13T22:30:55.053 DEBUG:boto:Token: None 2024-04-13T22:30:55.053 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.054 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM= 2024-04-13T22:30:55.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM=', 'Content-Length': '0'} 2024-04-13T22:30:55.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099f3473efdca3eaa-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.056 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.056 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.056 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.057 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.057 DEBUG:boto:Method: GET 2024-04-13T22:30:55.057 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.057 DEBUG:boto:Data: 2024-04-13T22:30:55.057 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.057 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.057 DEBUG:boto:Params: {} 2024-04-13T22:30:55.057 DEBUG:boto:Token: None 2024-04-13T22:30:55.057 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.058 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM= 2024-04-13T22:30:55.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM=', 'Content-Length': '0'} 2024-04-13T22:30:55.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afb614ccf769fb6-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.102 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.103 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.103 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.103 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.103 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.103 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.103 DEBUG:boto:Data: 2024-04-13T22:30:55.103 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.103 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.103 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.103 DEBUG:boto:Params: {} 2024-04-13T22:30:55.104 DEBUG:boto:Token: None 2024-04-13T22:30:55.104 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.104 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F7bxsVp8gvoGm8j/pSGuehysRm4= 2024-04-13T22:30:55.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F7bxsVp8gvoGm8j/pSGuehysRm4=', 'Content-Length': '0'} 2024-04-13T22:30:55.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4fff5758c9b7f6f-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.106 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.106 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.106 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.107 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.107 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.107 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.107 DEBUG:boto:Data: 2024-04-13T22:30:55.107 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.107 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.107 DEBUG:boto:Params: {} 2024-04-13T22:30:55.107 DEBUG:boto:Token: None 2024-04-13T22:30:55.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F7bxsVp8gvoGm8j/pSGuehysRm4= 2024-04-13T22:30:55.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F7bxsVp8gvoGm8j/pSGuehysRm4=', 'Content-Length': '0'} 2024-04-13T22:30:55.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743acc282f797cac-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:30:55.110 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.110 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.110 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.110 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.110 DEBUG:boto:Method: GET 2024-04-13T22:30:55.111 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.111 DEBUG:boto:Data: 2024-04-13T22:30:55.111 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.111 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.111 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.111 DEBUG:boto:Params: {} 2024-04-13T22:30:55.111 DEBUG:boto:Token: None 2024-04-13T22:30:55.111 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.111 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM= 2024-04-13T22:30:55.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM=', 'Content-Length': '0'} 2024-04-13T22:30:55.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081fc5d15b6973996-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.113 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.114 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.114 DEBUG:boto:path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.114 DEBUG:boto:auth_path=/vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.114 DEBUG:boto:Method: GET 2024-04-13T22:30:55.114 DEBUG:boto:Path: /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.114 DEBUG:boto:Data: 2024-04-13T22:30:55.114 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.114 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.114 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.114 DEBUG:boto:Params: {} 2024-04-13T22:30:55.115 DEBUG:boto:Token: None 2024-04-13T22:30:55.115 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11?versionId=null 2024-04-13T22:30:55.115 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM= 2024-04-13T22:30:55.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/yPen+KCszhOIe0tQNV6uTZHDTM=', 'Content-Length': '0'} 2024-04-13T22:30:55.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab9047466c44ad4-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.117 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.117 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.117 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.117 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.117 DEBUG:boto:Method: GET 2024-04-13T22:30:55.118 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.118 DEBUG:boto:Data: 2024-04-13T22:30:55.118 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.118 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.118 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.118 DEBUG:boto:Params: {} 2024-04-13T22:30:55.118 DEBUG:boto:Token: None 2024-04-13T22:30:55.118 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.118 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b454b7aafa7c5e97-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.121 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.122 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.122 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.122 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.122 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.122 DEBUG:boto:Method: GET 2024-04-13T22:30:55.122 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.123 DEBUG:boto:Data: 2024-04-13T22:30:55.123 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.123 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.123 DEBUG:boto:Params: {} 2024-04-13T22:30:55.123 DEBUG:boto:Token: None 2024-04-13T22:30:55.123 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bba37a7ce50733f-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.127 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.128 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.128 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.128 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.128 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:55.128 DEBUG:boto:Data: 2024-04-13T22:30:55.128 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.128 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.128 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.128 DEBUG:boto:Params: {} 2024-04-13T22:30:55.129 DEBUG:boto:Token: None 2024-04-13T22:30:55.129 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11 2024-04-13T22:30:55.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGbx5Gz7NnxRnwtwpXIkl/qq/Tk= 2024-04-13T22:30:55.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGbx5Gz7NnxRnwtwpXIkl/qq/Tk=', 'Content-Length': '0'} 2024-04-13T22:30:55.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000922a3ccc5d549a15-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.131 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.131 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.131 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.131 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:55.131 DEBUG:boto:Data: 2024-04-13T22:30:55.131 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.132 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.132 DEBUG:boto:Params: {} 2024-04-13T22:30:55.132 DEBUG:boto:Token: None 2024-04-13T22:30:55.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11 2024-04-13T22:30:55.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CGbx5Gz7NnxRnwtwpXIkl/qq/Tk= 2024-04-13T22:30:55.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CGbx5Gz7NnxRnwtwpXIkl/qq/Tk=', 'Content-Length': '0'} 2024-04-13T22:30:55.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d3fce267b2d257-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:30:55.134 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.134 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.134 DEBUG:boto:Method: GET 2024-04-13T22:30:55.134 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:55.135 DEBUG:boto:Data: 2024-04-13T22:30:55.135 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.135 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.135 DEBUG:boto:Params: {} 2024-04-13T22:30:55.135 DEBUG:boto:Token: None 2024-04-13T22:30:55.135 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11 2024-04-13T22:30:55.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KSWUpCLfR3OCnoBmoeMMidJms6Y= 2024-04-13T22:30:55.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KSWUpCLfR3OCnoBmoeMMidJms6Y=', 'Content-Length': '0'} 2024-04-13T22:30:55.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000076434b569f06632-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.178 DEBUG:boto:path=/vwnfns-61/obj11 2024-04-13T22:30:55.178 DEBUG:boto:auth_path=/vwnfns-61/obj11 2024-04-13T22:30:55.179 DEBUG:boto:Method: GET 2024-04-13T22:30:55.179 DEBUG:boto:Path: /vwnfns-61/obj11 2024-04-13T22:30:55.179 DEBUG:boto:Data: 2024-04-13T22:30:55.179 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.179 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.179 DEBUG:boto:Params: {} 2024-04-13T22:30:55.179 DEBUG:boto:Token: None 2024-04-13T22:30:55.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj11 2024-04-13T22:30:55.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KSWUpCLfR3OCnoBmoeMMidJms6Y= 2024-04-13T22:30:55.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KSWUpCLfR3OCnoBmoeMMidJms6Y=', 'Content-Length': '0'} 2024-04-13T22:30:55.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1601c55144b0669-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.182 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.182 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.182 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.182 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.182 DEBUG:boto:Method: GET 2024-04-13T22:30:55.182 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.182 DEBUG:boto:Data: 2024-04-13T22:30:55.182 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.183 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.183 DEBUG:boto:Params: {} 2024-04-13T22:30:55.183 DEBUG:boto:Token: None 2024-04-13T22:30:55.183 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdffa36d5c89a9f1-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.186 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.187 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.187 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.187 DEBUG:boto:path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.187 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.187 DEBUG:boto:Method: GET 2024-04-13T22:30:55.187 DEBUG:boto:Path: /vwnfns-61/?prefix=obj11 2024-04-13T22:30:55.187 DEBUG:boto:Data: 2024-04-13T22:30:55.187 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.187 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.187 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.187 DEBUG:boto:Params: {} 2024-04-13T22:30:55.187 DEBUG:boto:Token: None 2024-04-13T22:30:55.188 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4d5ef65bf6ea97-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.192 DEBUG:boto:b'vwnfns-61obj111000falseobj112024-04-13T22:16:35.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-13T22:30:55.192 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.193 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.193 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.193 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.193 DEBUG:boto:Method: GET 2024-04-13T22:30:55.193 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.193 DEBUG:boto:Data: 2024-04-13T22:30:55.193 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.193 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.193 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.193 DEBUG:boto:Params: {} 2024-04-13T22:30:55.193 DEBUG:boto:Token: None 2024-04-13T22:30:55.194 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.194 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A= 2024-04-13T22:30:55.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A=', 'Content-Length': '0'} 2024-04-13T22:30:55.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c34cf9ced832207-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.196 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.196 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.196 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.197 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.197 DEBUG:boto:Method: GET 2024-04-13T22:30:55.197 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.197 DEBUG:boto:Data: 2024-04-13T22:30:55.197 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.197 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.197 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.197 DEBUG:boto:Params: {} 2024-04-13T22:30:55.197 DEBUG:boto:Token: None 2024-04-13T22:30:55.197 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A= 2024-04-13T22:30:55.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A=', 'Content-Length': '0'} 2024-04-13T22:30:55.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd1c54bec9ab43e-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.246 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.246 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.247 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.247 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.247 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.247 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.247 DEBUG:boto:Data: 2024-04-13T22:30:55.247 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.247 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.247 DEBUG:boto:Params: {} 2024-04-13T22:30:55.247 DEBUG:boto:Token: None 2024-04-13T22:30:55.248 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GCTdAbI5PirhzrUl4IyGn2ls1yE= 2024-04-13T22:30:55.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GCTdAbI5PirhzrUl4IyGn2ls1yE=', 'Content-Length': '0'} 2024-04-13T22:30:55.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a2eb5f3f89dd7a1-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.250 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.250 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.250 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.250 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.250 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.250 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.250 DEBUG:boto:Data: 2024-04-13T22:30:55.251 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.251 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.251 DEBUG:boto:Params: {} 2024-04-13T22:30:55.251 DEBUG:boto:Token: None 2024-04-13T22:30:55.251 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GCTdAbI5PirhzrUl4IyGn2ls1yE= 2024-04-13T22:30:55.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GCTdAbI5PirhzrUl4IyGn2ls1yE=', 'Content-Length': '0'} 2024-04-13T22:30:55.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000937c52ec406160b0-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-13T22:30:55.253 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.253 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.254 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.254 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.254 DEBUG:boto:Method: GET 2024-04-13T22:30:55.254 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.254 DEBUG:boto:Data: 2024-04-13T22:30:55.254 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.254 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.254 DEBUG:boto:Params: {} 2024-04-13T22:30:55.254 DEBUG:boto:Token: None 2024-04-13T22:30:55.255 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A= 2024-04-13T22:30:55.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A=', 'Content-Length': '0'} 2024-04-13T22:30:55.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b5c340757737ab-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.257 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.257 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.257 DEBUG:boto:path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.257 DEBUG:boto:auth_path=/vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.257 DEBUG:boto:Method: GET 2024-04-13T22:30:55.257 DEBUG:boto:Path: /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.257 DEBUG:boto:Data: 2024-04-13T22:30:55.257 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.257 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.257 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.257 DEBUG:boto:Params: {} 2024-04-13T22:30:55.258 DEBUG:boto:Token: None 2024-04-13T22:30:55.258 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111?versionId=null 2024-04-13T22:30:55.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A= 2024-04-13T22:30:55.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5GJRc3Xy28kW5TH/TnA11Jtdt2A=', 'Content-Length': '0'} 2024-04-13T22:30:55.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe54e968f8372cc-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.260 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.260 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.260 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.260 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.260 DEBUG:boto:Method: GET 2024-04-13T22:30:55.261 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.261 DEBUG:boto:Data: 2024-04-13T22:30:55.261 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.261 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.261 DEBUG:boto:Params: {} 2024-04-13T22:30:55.261 DEBUG:boto:Token: None 2024-04-13T22:30:55.261 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.261 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca948e0893baa18-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.264 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.264 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.264 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.264 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.264 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.265 DEBUG:boto:Method: GET 2024-04-13T22:30:55.265 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.265 DEBUG:boto:Data: 2024-04-13T22:30:55.265 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.265 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.265 DEBUG:boto:Params: {} 2024-04-13T22:30:55.265 DEBUG:boto:Token: None 2024-04-13T22:30:55.265 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.265 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059499220f50c82b8-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.270 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.270 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.270 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.270 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.271 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:55.271 DEBUG:boto:Data: 2024-04-13T22:30:55.271 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.271 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.271 DEBUG:boto:Params: {} 2024-04-13T22:30:55.271 DEBUG:boto:Token: None 2024-04-13T22:30:55.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111 2024-04-13T22:30:55.271 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M5EiqDQ0STlVovU6kogZ066S1W8= 2024-04-13T22:30:55.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M5EiqDQ0STlVovU6kogZ066S1W8=', 'Content-Length': '0'} 2024-04-13T22:30:55.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5f38e02da52766-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.273 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.273 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.274 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.274 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:55.274 DEBUG:boto:Data: 2024-04-13T22:30:55.274 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.274 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.274 DEBUG:boto:Params: {} 2024-04-13T22:30:55.274 DEBUG:boto:Token: None 2024-04-13T22:30:55.274 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111 2024-04-13T22:30:55.274 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M5EiqDQ0STlVovU6kogZ066S1W8= 2024-04-13T22:30:55.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M5EiqDQ0STlVovU6kogZ066S1W8=', 'Content-Length': '0'} 2024-04-13T22:30:55.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674a8b1280b85d0a-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-13T22:30:55.277 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.277 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.277 DEBUG:boto:Method: GET 2024-04-13T22:30:55.277 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:55.277 DEBUG:boto:Data: 2024-04-13T22:30:55.277 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.277 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.277 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.277 DEBUG:boto:Params: {} 2024-04-13T22:30:55.277 DEBUG:boto:Token: None 2024-04-13T22:30:55.277 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111 2024-04-13T22:30:55.278 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HjktO3ngiLB7LxVU27vKhoMI4lo= 2024-04-13T22:30:55.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HjktO3ngiLB7LxVU27vKhoMI4lo=', 'Content-Length': '0'} 2024-04-13T22:30:55.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc057088624ef4f-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.282 DEBUG:boto:path=/vwnfns-61/obj111 2024-04-13T22:30:55.282 DEBUG:boto:auth_path=/vwnfns-61/obj111 2024-04-13T22:30:55.282 DEBUG:boto:Method: GET 2024-04-13T22:30:55.282 DEBUG:boto:Path: /vwnfns-61/obj111 2024-04-13T22:30:55.282 DEBUG:boto:Data: 2024-04-13T22:30:55.283 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.283 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.283 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.283 DEBUG:boto:Params: {} 2024-04-13T22:30:55.283 DEBUG:boto:Token: None 2024-04-13T22:30:55.283 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj111 2024-04-13T22:30:55.283 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HjktO3ngiLB7LxVU27vKhoMI4lo= 2024-04-13T22:30:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HjktO3ngiLB7LxVU27vKhoMI4lo=', 'Content-Length': '0'} 2024-04-13T22:30:55.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf62302591082e7c-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.285 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.285 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.285 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.285 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.286 DEBUG:boto:Method: GET 2024-04-13T22:30:55.286 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.286 DEBUG:boto:Data: 2024-04-13T22:30:55.286 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.286 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.286 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.286 DEBUG:boto:Params: {} 2024-04-13T22:30:55.286 DEBUG:boto:Token: None 2024-04-13T22:30:55.286 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.286 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd191d60034e737f-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.330 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.331 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.331 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.331 DEBUG:boto:path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.331 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.331 DEBUG:boto:Method: GET 2024-04-13T22:30:55.331 DEBUG:boto:Path: /vwnfns-61/?prefix=obj111 2024-04-13T22:30:55.331 DEBUG:boto:Data: 2024-04-13T22:30:55.331 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.331 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.331 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.332 DEBUG:boto:Params: {} 2024-04-13T22:30:55.332 DEBUG:boto:Token: None 2024-04-13T22:30:55.332 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.332 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a52feb85456c9f4-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.336 DEBUG:boto:b'vwnfns-61obj1111000falseobj1112024-04-13T22:28:56.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:30:55.336 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.336 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.336 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.336 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.336 DEBUG:boto:Method: GET 2024-04-13T22:30:55.337 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.337 DEBUG:boto:Data: 2024-04-13T22:30:55.337 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.337 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.337 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.337 DEBUG:boto:Params: {} 2024-04-13T22:30:55.337 DEBUG:boto:Token: None 2024-04-13T22:30:55.337 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.337 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg= 2024-04-13T22:30:55.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg=', 'Content-Length': '0'} 2024-04-13T22:30:55.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396036f051edb12b-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.340 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.340 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.341 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.341 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.341 DEBUG:boto:Method: GET 2024-04-13T22:30:55.341 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.341 DEBUG:boto:Data: 2024-04-13T22:30:55.341 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.341 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.341 DEBUG:boto:Params: {} 2024-04-13T22:30:55.341 DEBUG:boto:Token: None 2024-04-13T22:30:55.342 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg= 2024-04-13T22:30:55.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg=', 'Content-Length': '0'} 2024-04-13T22:30:55.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf0c0be26393ed1-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.386 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.387 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.387 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.387 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.387 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.387 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.387 DEBUG:boto:Data: 2024-04-13T22:30:55.387 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.387 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.387 DEBUG:boto:Params: {} 2024-04-13T22:30:55.388 DEBUG:boto:Token: None 2024-04-13T22:30:55.388 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/X9Ds1ttvvhCNtjGAa7sdNxeaYg= 2024-04-13T22:30:55.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/X9Ds1ttvvhCNtjGAa7sdNxeaYg=', 'Content-Length': '0'} 2024-04-13T22:30:55.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8de52af149a06b-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.390 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.390 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.390 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.390 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.390 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.390 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.390 DEBUG:boto:Data: 2024-04-13T22:30:55.391 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.391 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.391 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.391 DEBUG:boto:Params: {} 2024-04-13T22:30:55.391 DEBUG:boto:Token: None 2024-04-13T22:30:55.391 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.391 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/X9Ds1ttvvhCNtjGAa7sdNxeaYg= 2024-04-13T22:30:55.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/X9Ds1ttvvhCNtjGAa7sdNxeaYg=', 'Content-Length': '0'} 2024-04-13T22:30:55.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f811fac8e8acc1a-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:30:55.393 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.393 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.394 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.394 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.394 DEBUG:boto:Method: GET 2024-04-13T22:30:55.394 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.394 DEBUG:boto:Data: 2024-04-13T22:30:55.394 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.394 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.394 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.394 DEBUG:boto:Params: {} 2024-04-13T22:30:55.394 DEBUG:boto:Token: None 2024-04-13T22:30:55.395 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.395 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg= 2024-04-13T22:30:55.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg=', 'Content-Length': '0'} 2024-04-13T22:30:55.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f6c10e7c1e94f2-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.397 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.397 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.397 DEBUG:boto:path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.397 DEBUG:boto:auth_path=/vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.397 DEBUG:boto:Method: GET 2024-04-13T22:30:55.397 DEBUG:boto:Path: /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.397 DEBUG:boto:Data: 2024-04-13T22:30:55.398 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.398 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.398 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.398 DEBUG:boto:Params: {} 2024-04-13T22:30:55.398 DEBUG:boto:Token: None 2024-04-13T22:30:55.398 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115?versionId=null 2024-04-13T22:30:55.398 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg= 2024-04-13T22:30:55.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IGuKllLnia/Ddo/6a/cGdWjSbfg=', 'Content-Length': '0'} 2024-04-13T22:30:55.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87f6e9eca74c585-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.400 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.400 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.400 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.400 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.401 DEBUG:boto:Method: GET 2024-04-13T22:30:55.401 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.401 DEBUG:boto:Data: 2024-04-13T22:30:55.401 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.401 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.401 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.401 DEBUG:boto:Params: {} 2024-04-13T22:30:55.401 DEBUG:boto:Token: None 2024-04-13T22:30:55.401 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.401 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008619c7f5f4c90434-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.404 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.404 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.404 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.405 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.405 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.405 DEBUG:boto:Method: GET 2024-04-13T22:30:55.405 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.405 DEBUG:boto:Data: 2024-04-13T22:30:55.405 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.405 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.405 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.405 DEBUG:boto:Params: {} 2024-04-13T22:30:55.405 DEBUG:boto:Token: None 2024-04-13T22:30:55.406 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.406 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ebb4b6ab9cea77-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.409 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.410 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.410 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.410 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.410 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:55.410 DEBUG:boto:Data: 2024-04-13T22:30:55.410 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.410 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.410 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.410 DEBUG:boto:Params: {} 2024-04-13T22:30:55.411 DEBUG:boto:Token: None 2024-04-13T22:30:55.411 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115 2024-04-13T22:30:55.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1GAUva4tfB8h+fGlSyRXQCzG0Cs= 2024-04-13T22:30:55.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1GAUva4tfB8h+fGlSyRXQCzG0Cs=', 'Content-Length': '0'} 2024-04-13T22:30:55.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc4096e817f9ac7-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.413 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.413 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.413 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.413 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:55.413 DEBUG:boto:Data: 2024-04-13T22:30:55.413 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.414 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.414 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.414 DEBUG:boto:Params: {} 2024-04-13T22:30:55.414 DEBUG:boto:Token: None 2024-04-13T22:30:55.414 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115 2024-04-13T22:30:55.414 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1GAUva4tfB8h+fGlSyRXQCzG0Cs= 2024-04-13T22:30:55.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1GAUva4tfB8h+fGlSyRXQCzG0Cs=', 'Content-Length': '0'} 2024-04-13T22:30:55.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4752e2020b5ecc-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-13T22:30:55.416 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.416 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.417 DEBUG:boto:Method: GET 2024-04-13T22:30:55.417 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:55.417 DEBUG:boto:Data: 2024-04-13T22:30:55.417 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.417 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.417 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.417 DEBUG:boto:Params: {} 2024-04-13T22:30:55.417 DEBUG:boto:Token: None 2024-04-13T22:30:55.417 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115 2024-04-13T22:30:55.417 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iulFdLnyjANwDFvuHujFvj06+7g= 2024-04-13T22:30:55.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iulFdLnyjANwDFvuHujFvj06+7g=', 'Content-Length': '0'} 2024-04-13T22:30:55.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000384d7c84ed1eca7e-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.419 DEBUG:boto:path=/vwnfns-61/obj115 2024-04-13T22:30:55.419 DEBUG:boto:auth_path=/vwnfns-61/obj115 2024-04-13T22:30:55.420 DEBUG:boto:Method: GET 2024-04-13T22:30:55.420 DEBUG:boto:Path: /vwnfns-61/obj115 2024-04-13T22:30:55.420 DEBUG:boto:Data: 2024-04-13T22:30:55.420 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.420 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.420 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.420 DEBUG:boto:Params: {} 2024-04-13T22:30:55.420 DEBUG:boto:Token: None 2024-04-13T22:30:55.420 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj115 2024-04-13T22:30:55.420 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iulFdLnyjANwDFvuHujFvj06+7g= 2024-04-13T22:30:55.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iulFdLnyjANwDFvuHujFvj06+7g=', 'Content-Length': '0'} 2024-04-13T22:30:55.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7cd57b5d54a98a-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.422 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.423 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.423 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.423 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.423 DEBUG:boto:Method: GET 2024-04-13T22:30:55.423 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.423 DEBUG:boto:Data: 2024-04-13T22:30:55.423 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.423 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.423 DEBUG:boto:Params: {} 2024-04-13T22:30:55.423 DEBUG:boto:Token: None 2024-04-13T22:30:55.424 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.424 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2497eba451b278a-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.470 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.471 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.471 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.471 DEBUG:boto:path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.471 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.471 DEBUG:boto:Method: GET 2024-04-13T22:30:55.471 DEBUG:boto:Path: /vwnfns-61/?prefix=obj115 2024-04-13T22:30:55.471 DEBUG:boto:Data: 2024-04-13T22:30:55.471 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.471 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.472 DEBUG:boto:Params: {} 2024-04-13T22:30:55.472 DEBUG:boto:Token: None 2024-04-13T22:30:55.472 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.472 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f710bfa2fceb9dad-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.477 DEBUG:boto:b'vwnfns-61obj1151000falseobj1152024-04-13T22:22:50.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:30:55.477 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.477 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.477 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.477 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.477 DEBUG:boto:Method: GET 2024-04-13T22:30:55.477 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.478 DEBUG:boto:Data: 2024-04-13T22:30:55.478 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.478 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.478 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.478 DEBUG:boto:Params: {} 2024-04-13T22:30:55.478 DEBUG:boto:Token: None 2024-04-13T22:30:55.478 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.478 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o= 2024-04-13T22:30:55.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o=', 'Content-Length': '0'} 2024-04-13T22:30:55.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e34e4c98c81e8a-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.481 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.481 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.481 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.482 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.482 DEBUG:boto:Method: GET 2024-04-13T22:30:55.482 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.482 DEBUG:boto:Data: 2024-04-13T22:30:55.482 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.482 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.482 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.482 DEBUG:boto:Params: {} 2024-04-13T22:30:55.482 DEBUG:boto:Token: None 2024-04-13T22:30:55.483 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.483 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o= 2024-04-13T22:30:55.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o=', 'Content-Length': '0'} 2024-04-13T22:30:55.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9b625fdc450493-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.485 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.485 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.485 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.485 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.485 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.485 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.485 DEBUG:boto:Data: 2024-04-13T22:30:55.485 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.485 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.486 DEBUG:boto:Params: {} 2024-04-13T22:30:55.486 DEBUG:boto:Token: None 2024-04-13T22:30:55.486 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z4n4or1oMldGOHKP9YUlAf2M/T0= 2024-04-13T22:30:55.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z4n4or1oMldGOHKP9YUlAf2M/T0=', 'Content-Length': '0'} 2024-04-13T22:30:55.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4036707e6b7f836-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.490 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.490 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.490 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.490 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.490 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.490 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.490 DEBUG:boto:Data: 2024-04-13T22:30:55.490 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.491 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.491 DEBUG:boto:Params: {} 2024-04-13T22:30:55.491 DEBUG:boto:Token: None 2024-04-13T22:30:55.491 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z4n4or1oMldGOHKP9YUlAf2M/T0= 2024-04-13T22:30:55.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z4n4or1oMldGOHKP9YUlAf2M/T0=', 'Content-Length': '0'} 2024-04-13T22:30:55.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51d121c7aaf3cd3-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:30:55.493 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.493 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.494 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.494 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.494 DEBUG:boto:Method: GET 2024-04-13T22:30:55.494 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.494 DEBUG:boto:Data: 2024-04-13T22:30:55.494 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.494 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.494 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.494 DEBUG:boto:Params: {} 2024-04-13T22:30:55.494 DEBUG:boto:Token: None 2024-04-13T22:30:55.495 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.495 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o= 2024-04-13T22:30:55.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o=', 'Content-Length': '0'} 2024-04-13T22:30:55.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e2f39f6246673f-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.497 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.497 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.497 DEBUG:boto:path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.497 DEBUG:boto:auth_path=/vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.497 DEBUG:boto:Method: GET 2024-04-13T22:30:55.497 DEBUG:boto:Path: /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.497 DEBUG:boto:Data: 2024-04-13T22:30:55.497 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.497 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.498 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.498 DEBUG:boto:Params: {} 2024-04-13T22:30:55.498 DEBUG:boto:Token: None 2024-04-13T22:30:55.498 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116?versionId=null 2024-04-13T22:30:55.498 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o= 2024-04-13T22:30:55.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6ISoee71+XEHnf8YqofJ7Ut/u7o=', 'Content-Length': '0'} 2024-04-13T22:30:55.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ddae7fda61ae0a-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.500 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.500 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.500 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.500 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.500 DEBUG:boto:Method: GET 2024-04-13T22:30:55.500 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.501 DEBUG:boto:Data: 2024-04-13T22:30:55.501 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.501 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.501 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.501 DEBUG:boto:Params: {} 2024-04-13T22:30:55.501 DEBUG:boto:Token: None 2024-04-13T22:30:55.501 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.501 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb7ec16b8fa0b915-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.504 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.505 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.505 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.505 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.505 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.505 DEBUG:boto:Method: GET 2024-04-13T22:30:55.505 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.505 DEBUG:boto:Data: 2024-04-13T22:30:55.505 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.505 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.506 DEBUG:boto:Params: {} 2024-04-13T22:30:55.506 DEBUG:boto:Token: None 2024-04-13T22:30:55.506 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.506 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbdf3e53cb70724d-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.511 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.511 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.511 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.511 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.511 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:55.511 DEBUG:boto:Data: 2024-04-13T22:30:55.511 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.512 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.512 DEBUG:boto:Params: {} 2024-04-13T22:30:55.512 DEBUG:boto:Token: None 2024-04-13T22:30:55.512 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116 2024-04-13T22:30:55.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gd2uOrHp5A/kgC4IyqQgXELAc18= 2024-04-13T22:30:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gd2uOrHp5A/kgC4IyqQgXELAc18=', 'Content-Length': '0'} 2024-04-13T22:30:55.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b750fc536ea76e-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.514 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.514 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.514 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.514 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:55.514 DEBUG:boto:Data: 2024-04-13T22:30:55.514 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.515 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.515 DEBUG:boto:Params: {} 2024-04-13T22:30:55.515 DEBUG:boto:Token: None 2024-04-13T22:30:55.515 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116 2024-04-13T22:30:55.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gd2uOrHp5A/kgC4IyqQgXELAc18= 2024-04-13T22:30:55.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gd2uOrHp5A/kgC4IyqQgXELAc18=', 'Content-Length': '0'} 2024-04-13T22:30:55.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000497b3289c87a4a36-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-13T22:30:55.517 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.518 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.518 DEBUG:boto:Method: GET 2024-04-13T22:30:55.518 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:55.518 DEBUG:boto:Data: 2024-04-13T22:30:55.518 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.518 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.518 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.518 DEBUG:boto:Params: {} 2024-04-13T22:30:55.518 DEBUG:boto:Token: None 2024-04-13T22:30:55.518 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116 2024-04-13T22:30:55.519 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h04s2Y6cmkM7UIlhosOC1GNLMlE= 2024-04-13T22:30:55.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h04s2Y6cmkM7UIlhosOC1GNLMlE=', 'Content-Length': '0'} 2024-04-13T22:30:55.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000760399abf4af3103-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.520 DEBUG:boto:path=/vwnfns-61/obj116 2024-04-13T22:30:55.520 DEBUG:boto:auth_path=/vwnfns-61/obj116 2024-04-13T22:30:55.521 DEBUG:boto:Method: GET 2024-04-13T22:30:55.521 DEBUG:boto:Path: /vwnfns-61/obj116 2024-04-13T22:30:55.521 DEBUG:boto:Data: 2024-04-13T22:30:55.521 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.521 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.521 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.521 DEBUG:boto:Params: {} 2024-04-13T22:30:55.521 DEBUG:boto:Token: None 2024-04-13T22:30:55.521 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj116 2024-04-13T22:30:55.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h04s2Y6cmkM7UIlhosOC1GNLMlE= 2024-04-13T22:30:55.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h04s2Y6cmkM7UIlhosOC1GNLMlE=', 'Content-Length': '0'} 2024-04-13T22:30:55.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7dbd4a1235ce4f7-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.566 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.566 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.566 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.567 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.567 DEBUG:boto:Method: GET 2024-04-13T22:30:55.567 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.567 DEBUG:boto:Data: 2024-04-13T22:30:55.567 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.567 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.567 DEBUG:boto:Params: {} 2024-04-13T22:30:55.567 DEBUG:boto:Token: None 2024-04-13T22:30:55.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eff707bec67c25a-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.570 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.570 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.571 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.571 DEBUG:boto:path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.571 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.571 DEBUG:boto:Method: GET 2024-04-13T22:30:55.571 DEBUG:boto:Path: /vwnfns-61/?prefix=obj116 2024-04-13T22:30:55.571 DEBUG:boto:Data: 2024-04-13T22:30:55.571 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.571 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.571 DEBUG:boto:Params: {} 2024-04-13T22:30:55.571 DEBUG:boto:Token: None 2024-04-13T22:30:55.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b5394520b3e93e-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.576 DEBUG:boto:b'vwnfns-61obj1161000falseobj1162024-04-13T22:25:33.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:30:55.576 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.576 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.577 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.577 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.577 DEBUG:boto:Method: GET 2024-04-13T22:30:55.577 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.577 DEBUG:boto:Data: 2024-04-13T22:30:55.577 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.577 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.577 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.577 DEBUG:boto:Params: {} 2024-04-13T22:30:55.577 DEBUG:boto:Token: None 2024-04-13T22:30:55.577 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.578 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I= 2024-04-13T22:30:55.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I=', 'Content-Length': '0'} 2024-04-13T22:30:55.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006665cd8ab19cfacf-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.580 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.580 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.580 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.580 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.580 DEBUG:boto:Method: GET 2024-04-13T22:30:55.580 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.580 DEBUG:boto:Data: 2024-04-13T22:30:55.580 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.581 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.581 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.581 DEBUG:boto:Params: {} 2024-04-13T22:30:55.581 DEBUG:boto:Token: None 2024-04-13T22:30:55.581 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.581 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I= 2024-04-13T22:30:55.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I=', 'Content-Length': '0'} 2024-04-13T22:30:55.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741bb588bde953f1-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.584 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.584 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.584 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.584 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.584 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.584 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.585 DEBUG:boto:Data: 2024-04-13T22:30:55.585 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.585 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.585 DEBUG:boto:Params: {} 2024-04-13T22:30:55.585 DEBUG:boto:Token: None 2024-04-13T22:30:55.585 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.585 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UXg8K24p8vT3dVIcE/cHcrSI9oo= 2024-04-13T22:30:55.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UXg8K24p8vT3dVIcE/cHcrSI9oo=', 'Content-Length': '0'} 2024-04-13T22:30:55.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096fb70fecf838d1f-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.587 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.588 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.588 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.588 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.588 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.588 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.588 DEBUG:boto:Data: 2024-04-13T22:30:55.588 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.588 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.588 DEBUG:boto:Params: {} 2024-04-13T22:30:55.588 DEBUG:boto:Token: None 2024-04-13T22:30:55.588 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UXg8K24p8vT3dVIcE/cHcrSI9oo= 2024-04-13T22:30:55.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UXg8K24p8vT3dVIcE/cHcrSI9oo=', 'Content-Length': '0'} 2024-04-13T22:30:55.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03a8d69074cf104-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:30:55.591 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.591 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.591 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.591 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.591 DEBUG:boto:Method: GET 2024-04-13T22:30:55.591 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.591 DEBUG:boto:Data: 2024-04-13T22:30:55.591 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.591 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.592 DEBUG:boto:Params: {} 2024-04-13T22:30:55.592 DEBUG:boto:Token: None 2024-04-13T22:30:55.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I= 2024-04-13T22:30:55.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I=', 'Content-Length': '0'} 2024-04-13T22:30:55.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07f1a91ef58d45f-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.634 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.634 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.635 DEBUG:boto:path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.635 DEBUG:boto:auth_path=/vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.635 DEBUG:boto:Method: GET 2024-04-13T22:30:55.635 DEBUG:boto:Path: /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.635 DEBUG:boto:Data: 2024-04-13T22:30:55.635 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.635 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.635 DEBUG:boto:Params: {} 2024-04-13T22:30:55.635 DEBUG:boto:Token: None 2024-04-13T22:30:55.635 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124?versionId=null 2024-04-13T22:30:55.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I= 2024-04-13T22:30:55.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xsXhvqbP1VqA2m0jAbED3Jkk+4I=', 'Content-Length': '0'} 2024-04-13T22:30:55.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b4e17249a8d7b1-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.638 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.638 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.638 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.638 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.638 DEBUG:boto:Method: GET 2024-04-13T22:30:55.638 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.638 DEBUG:boto:Data: 2024-04-13T22:30:55.638 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.639 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.639 DEBUG:boto:Params: {} 2024-04-13T22:30:55.639 DEBUG:boto:Token: None 2024-04-13T22:30:55.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015630231d8be900a-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.642 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.642 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.642 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.642 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.643 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.643 DEBUG:boto:Method: GET 2024-04-13T22:30:55.643 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.643 DEBUG:boto:Data: 2024-04-13T22:30:55.643 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.643 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.643 DEBUG:boto:Params: {} 2024-04-13T22:30:55.643 DEBUG:boto:Token: None 2024-04-13T22:30:55.643 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c6930438a1adccd-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.648 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.648 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.648 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.648 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.649 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:55.649 DEBUG:boto:Data: 2024-04-13T22:30:55.649 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.649 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.649 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.649 DEBUG:boto:Params: {} 2024-04-13T22:30:55.649 DEBUG:boto:Token: None 2024-04-13T22:30:55.649 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124 2024-04-13T22:30:55.649 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iPzEmRbOIOCrzt4FfsfSwhfm9/Q= 2024-04-13T22:30:55.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iPzEmRbOIOCrzt4FfsfSwhfm9/Q=', 'Content-Length': '0'} 2024-04-13T22:30:55.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb1770cef38d45c-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.651 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.651 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.652 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.652 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:55.652 DEBUG:boto:Data: 2024-04-13T22:30:55.652 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.652 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.652 DEBUG:boto:Params: {} 2024-04-13T22:30:55.652 DEBUG:boto:Token: None 2024-04-13T22:30:55.652 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124 2024-04-13T22:30:55.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iPzEmRbOIOCrzt4FfsfSwhfm9/Q= 2024-04-13T22:30:55.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iPzEmRbOIOCrzt4FfsfSwhfm9/Q=', 'Content-Length': '0'} 2024-04-13T22:30:55.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056da2fd9c18fc2b3-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-13T22:30:55.655 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.655 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.655 DEBUG:boto:Method: GET 2024-04-13T22:30:55.655 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:55.655 DEBUG:boto:Data: 2024-04-13T22:30:55.655 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.655 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.655 DEBUG:boto:Params: {} 2024-04-13T22:30:55.655 DEBUG:boto:Token: None 2024-04-13T22:30:55.656 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124 2024-04-13T22:30:55.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b02DmMmboE64CmDJEq9Ee3Lj3hU= 2024-04-13T22:30:55.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b02DmMmboE64CmDJEq9Ee3Lj3hU=', 'Content-Length': '0'} 2024-04-13T22:30:55.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb3f173b6939c9a-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.658 DEBUG:boto:path=/vwnfns-61/obj124 2024-04-13T22:30:55.658 DEBUG:boto:auth_path=/vwnfns-61/obj124 2024-04-13T22:30:55.658 DEBUG:boto:Method: GET 2024-04-13T22:30:55.658 DEBUG:boto:Path: /vwnfns-61/obj124 2024-04-13T22:30:55.658 DEBUG:boto:Data: 2024-04-13T22:30:55.658 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.659 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.659 DEBUG:boto:Params: {} 2024-04-13T22:30:55.659 DEBUG:boto:Token: None 2024-04-13T22:30:55.659 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj124 2024-04-13T22:30:55.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:b02DmMmboE64CmDJEq9Ee3Lj3hU= 2024-04-13T22:30:55.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:b02DmMmboE64CmDJEq9Ee3Lj3hU=', 'Content-Length': '0'} 2024-04-13T22:30:55.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cf6e2d95577698-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.661 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.661 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.661 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.662 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.662 DEBUG:boto:Method: GET 2024-04-13T22:30:55.662 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.662 DEBUG:boto:Data: 2024-04-13T22:30:55.662 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.662 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.662 DEBUG:boto:Params: {} 2024-04-13T22:30:55.662 DEBUG:boto:Token: None 2024-04-13T22:30:55.662 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005541545c930f1648-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.665 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.665 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.665 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.665 DEBUG:boto:path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.666 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.666 DEBUG:boto:Method: GET 2024-04-13T22:30:55.666 DEBUG:boto:Path: /vwnfns-61/?prefix=obj124 2024-04-13T22:30:55.666 DEBUG:boto:Data: 2024-04-13T22:30:55.666 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.666 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.666 DEBUG:boto:Params: {} 2024-04-13T22:30:55.666 DEBUG:boto:Token: None 2024-04-13T22:30:55.666 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3015175cb146772-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.670 DEBUG:boto:b'vwnfns-61obj1241000falseobj1242024-04-13T22:27:52.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:30:55.671 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.671 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.671 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.671 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.671 DEBUG:boto:Method: GET 2024-04-13T22:30:55.671 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.671 DEBUG:boto:Data: 2024-04-13T22:30:55.672 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.672 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.672 DEBUG:boto:Params: {} 2024-04-13T22:30:55.672 DEBUG:boto:Token: None 2024-04-13T22:30:55.672 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.672 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc= 2024-04-13T22:30:55.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc=', 'Content-Length': '0'} 2024-04-13T22:30:55.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d329c7916bd650-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.674 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.674 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.674 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.675 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.675 DEBUG:boto:Method: GET 2024-04-13T22:30:55.675 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.675 DEBUG:boto:Data: 2024-04-13T22:30:55.675 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.675 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.675 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.675 DEBUG:boto:Params: {} 2024-04-13T22:30:55.675 DEBUG:boto:Token: None 2024-04-13T22:30:55.675 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc= 2024-04-13T22:30:55.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc=', 'Content-Length': '0'} 2024-04-13T22:30:55.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083888333e09b99d8-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.718 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.718 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.719 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.719 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.719 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.719 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.719 DEBUG:boto:Data: 2024-04-13T22:30:55.719 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.719 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.719 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.719 DEBUG:boto:Params: {} 2024-04-13T22:30:55.719 DEBUG:boto:Token: None 2024-04-13T22:30:55.719 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.720 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d3vS/zegGrbYO8mjLxK9bwN6M5U= 2024-04-13T22:30:55.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d3vS/zegGrbYO8mjLxK9bwN6M5U=', 'Content-Length': '0'} 2024-04-13T22:30:55.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dbde7b9c67b971c-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.721 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.722 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.722 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.722 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.722 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.722 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.722 DEBUG:boto:Data: 2024-04-13T22:30:55.722 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.722 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.722 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.722 DEBUG:boto:Params: {} 2024-04-13T22:30:55.723 DEBUG:boto:Token: None 2024-04-13T22:30:55.723 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d3vS/zegGrbYO8mjLxK9bwN6M5U= 2024-04-13T22:30:55.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d3vS/zegGrbYO8mjLxK9bwN6M5U=', 'Content-Length': '0'} 2024-04-13T22:30:55.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e4591025969b47-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:30:55.725 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.725 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.725 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.725 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.725 DEBUG:boto:Method: GET 2024-04-13T22:30:55.725 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.725 DEBUG:boto:Data: 2024-04-13T22:30:55.725 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.725 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.726 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.726 DEBUG:boto:Params: {} 2024-04-13T22:30:55.726 DEBUG:boto:Token: None 2024-04-13T22:30:55.726 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.726 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc= 2024-04-13T22:30:55.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc=', 'Content-Length': '0'} 2024-04-13T22:30:55.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e91dee5fb46d09-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.728 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.728 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.728 DEBUG:boto:path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.728 DEBUG:boto:auth_path=/vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.728 DEBUG:boto:Method: GET 2024-04-13T22:30:55.728 DEBUG:boto:Path: /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.729 DEBUG:boto:Data: 2024-04-13T22:30:55.729 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.729 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.729 DEBUG:boto:Params: {} 2024-04-13T22:30:55.729 DEBUG:boto:Token: None 2024-04-13T22:30:55.729 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145?versionId=null 2024-04-13T22:30:55.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc= 2024-04-13T22:30:55.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1pTUcwNdZeE+iktpvxRqS8GoEsc=', 'Content-Length': '0'} 2024-04-13T22:30:55.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e26ad4f21df3f10-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.731 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.731 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.731 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.732 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.732 DEBUG:boto:Method: GET 2024-04-13T22:30:55.732 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.732 DEBUG:boto:Data: 2024-04-13T22:30:55.732 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.732 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.732 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.732 DEBUG:boto:Params: {} 2024-04-13T22:30:55.732 DEBUG:boto:Token: None 2024-04-13T22:30:55.732 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.733 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000209d36ba6073a483-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.735 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.735 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.735 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.735 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.735 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.736 DEBUG:boto:Method: GET 2024-04-13T22:30:55.736 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.736 DEBUG:boto:Data: 2024-04-13T22:30:55.736 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.736 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.736 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.736 DEBUG:boto:Params: {} 2024-04-13T22:30:55.736 DEBUG:boto:Token: None 2024-04-13T22:30:55.736 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de499d0b6ca7bc52-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.740 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.740 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.740 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.741 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.741 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:55.741 DEBUG:boto:Data: 2024-04-13T22:30:55.741 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.741 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.741 DEBUG:boto:Params: {} 2024-04-13T22:30:55.741 DEBUG:boto:Token: None 2024-04-13T22:30:55.741 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145 2024-04-13T22:30:55.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ADcuT0k2GNmTQz5XD/UIOSFKjr0= 2024-04-13T22:30:55.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ADcuT0k2GNmTQz5XD/UIOSFKjr0=', 'Content-Length': '0'} 2024-04-13T22:30:55.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35c61b61eb82c5e-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.743 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.743 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.743 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.743 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:55.744 DEBUG:boto:Data: 2024-04-13T22:30:55.744 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.744 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.744 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.744 DEBUG:boto:Params: {} 2024-04-13T22:30:55.744 DEBUG:boto:Token: None 2024-04-13T22:30:55.744 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145 2024-04-13T22:30:55.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ADcuT0k2GNmTQz5XD/UIOSFKjr0= 2024-04-13T22:30:55.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ADcuT0k2GNmTQz5XD/UIOSFKjr0=', 'Content-Length': '0'} 2024-04-13T22:30:55.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a0ddd0700437de-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-13T22:30:55.746 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.746 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.747 DEBUG:boto:Method: GET 2024-04-13T22:30:55.747 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:55.747 DEBUG:boto:Data: 2024-04-13T22:30:55.747 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.747 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.747 DEBUG:boto:Params: {} 2024-04-13T22:30:55.747 DEBUG:boto:Token: None 2024-04-13T22:30:55.747 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145 2024-04-13T22:30:55.747 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PWNvAon9EggK+WNBPnQ9bZFppvc= 2024-04-13T22:30:55.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PWNvAon9EggK+WNBPnQ9bZFppvc=', 'Content-Length': '0'} 2024-04-13T22:30:55.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc3f1b155558737-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.794 DEBUG:boto:path=/vwnfns-61/obj145 2024-04-13T22:30:55.795 DEBUG:boto:auth_path=/vwnfns-61/obj145 2024-04-13T22:30:55.795 DEBUG:boto:Method: GET 2024-04-13T22:30:55.795 DEBUG:boto:Path: /vwnfns-61/obj145 2024-04-13T22:30:55.795 DEBUG:boto:Data: 2024-04-13T22:30:55.795 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.795 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.795 DEBUG:boto:Params: {} 2024-04-13T22:30:55.795 DEBUG:boto:Token: None 2024-04-13T22:30:55.795 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj145 2024-04-13T22:30:55.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PWNvAon9EggK+WNBPnQ9bZFppvc= 2024-04-13T22:30:55.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PWNvAon9EggK+WNBPnQ9bZFppvc=', 'Content-Length': '0'} 2024-04-13T22:30:55.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8fe93273ee79175-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.798 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.798 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.798 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.798 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.798 DEBUG:boto:Method: GET 2024-04-13T22:30:55.798 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.799 DEBUG:boto:Data: 2024-04-13T22:30:55.799 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.799 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.799 DEBUG:boto:Params: {} 2024-04-13T22:30:55.799 DEBUG:boto:Token: None 2024-04-13T22:30:55.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.799 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64eabb0549c6f3e-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.802 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.802 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.802 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.803 DEBUG:boto:path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.803 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.803 DEBUG:boto:Method: GET 2024-04-13T22:30:55.803 DEBUG:boto:Path: /vwnfns-61/?prefix=obj145 2024-04-13T22:30:55.803 DEBUG:boto:Data: 2024-04-13T22:30:55.803 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.803 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.803 DEBUG:boto:Params: {} 2024-04-13T22:30:55.803 DEBUG:boto:Token: None 2024-04-13T22:30:55.804 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.804 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000264a57a71c5875d8-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.807 DEBUG:boto:b'vwnfns-61obj1451000falseobj1452024-04-13T22:25:33.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-13T22:30:55.808 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.808 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.808 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.808 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.808 DEBUG:boto:Method: GET 2024-04-13T22:30:55.808 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.808 DEBUG:boto:Data: 2024-04-13T22:30:55.808 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.808 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.809 DEBUG:boto:Params: {} 2024-04-13T22:30:55.809 DEBUG:boto:Token: None 2024-04-13T22:30:55.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU= 2024-04-13T22:30:55.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU=', 'Content-Length': '0'} 2024-04-13T22:30:55.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec3b6ae130e1cc7c-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.811 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.811 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.811 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.811 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.811 DEBUG:boto:Method: GET 2024-04-13T22:30:55.811 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.812 DEBUG:boto:Data: 2024-04-13T22:30:55.812 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.812 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.812 DEBUG:boto:Params: {} 2024-04-13T22:30:55.812 DEBUG:boto:Token: None 2024-04-13T22:30:55.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU= 2024-04-13T22:30:55.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU=', 'Content-Length': '0'} 2024-04-13T22:30:55.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044eac5c98f149608-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.814 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.815 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.815 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.815 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.815 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.815 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.815 DEBUG:boto:Data: 2024-04-13T22:30:55.815 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.815 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.815 DEBUG:boto:Params: {} 2024-04-13T22:30:55.815 DEBUG:boto:Token: None 2024-04-13T22:30:55.816 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ayqYULphddsg4M79ppS8r9q7fFk= 2024-04-13T22:30:55.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ayqYULphddsg4M79ppS8r9q7fFk=', 'Content-Length': '0'} 2024-04-13T22:30:55.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57f779c311176b1-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.818 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.818 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.818 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.818 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.818 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.818 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.818 DEBUG:boto:Data: 2024-04-13T22:30:55.818 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.818 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.818 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.819 DEBUG:boto:Params: {} 2024-04-13T22:30:55.819 DEBUG:boto:Token: None 2024-04-13T22:30:55.819 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ayqYULphddsg4M79ppS8r9q7fFk= 2024-04-13T22:30:55.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ayqYULphddsg4M79ppS8r9q7fFk=', 'Content-Length': '0'} 2024-04-13T22:30:55.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637c47385a5fec79-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-13T22:30:55.821 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.821 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.821 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.821 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.821 DEBUG:boto:Method: GET 2024-04-13T22:30:55.821 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.821 DEBUG:boto:Data: 2024-04-13T22:30:55.821 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.822 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.822 DEBUG:boto:Params: {} 2024-04-13T22:30:55.822 DEBUG:boto:Token: None 2024-04-13T22:30:55.822 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.822 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU= 2024-04-13T22:30:55.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU=', 'Content-Length': '0'} 2024-04-13T22:30:55.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8811b8346a600d-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.824 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.824 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.824 DEBUG:boto:path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.824 DEBUG:boto:auth_path=/vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.824 DEBUG:boto:Method: GET 2024-04-13T22:30:55.824 DEBUG:boto:Path: /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.824 DEBUG:boto:Data: 2024-04-13T22:30:55.825 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.825 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.825 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.825 DEBUG:boto:Params: {} 2024-04-13T22:30:55.825 DEBUG:boto:Token: None 2024-04-13T22:30:55.825 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148?versionId=null 2024-04-13T22:30:55.825 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU= 2024-04-13T22:30:55.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RMIPZJrhk8CF/gM1zclSyo1UYkU=', 'Content-Length': '0'} 2024-04-13T22:30:55.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005c05d831977a74-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.827 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.827 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.827 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.827 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.828 DEBUG:boto:Method: GET 2024-04-13T22:30:55.828 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.828 DEBUG:boto:Data: 2024-04-13T22:30:55.828 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.828 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.828 DEBUG:boto:Params: {} 2024-04-13T22:30:55.828 DEBUG:boto:Token: None 2024-04-13T22:30:55.828 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9148da4846eea15-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.831 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.831 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.831 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.832 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.832 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.832 DEBUG:boto:Method: GET 2024-04-13T22:30:55.832 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.832 DEBUG:boto:Data: 2024-04-13T22:30:55.832 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.832 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.832 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.832 DEBUG:boto:Params: {} 2024-04-13T22:30:55.832 DEBUG:boto:Token: None 2024-04-13T22:30:55.832 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.833 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b134f94b575d9b-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.878 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.879 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.879 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.879 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.879 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:55.879 DEBUG:boto:Data: 2024-04-13T22:30:55.879 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.879 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.879 DEBUG:boto:Params: {} 2024-04-13T22:30:55.879 DEBUG:boto:Token: None 2024-04-13T22:30:55.880 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148 2024-04-13T22:30:55.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ljjf57CgfYMqdGSwMMNmzxrRbj0= 2024-04-13T22:30:55.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ljjf57CgfYMqdGSwMMNmzxrRbj0=', 'Content-Length': '0'} 2024-04-13T22:30:55.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f58423e79901b7-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.882 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.882 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.882 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.882 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:55.882 DEBUG:boto:Data: 2024-04-13T22:30:55.882 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.882 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.882 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.882 DEBUG:boto:Params: {} 2024-04-13T22:30:55.883 DEBUG:boto:Token: None 2024-04-13T22:30:55.883 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148 2024-04-13T22:30:55.883 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ljjf57CgfYMqdGSwMMNmzxrRbj0= 2024-04-13T22:30:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ljjf57CgfYMqdGSwMMNmzxrRbj0=', 'Content-Length': '0'} 2024-04-13T22:30:55.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0cf557e74d4fa9-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-13T22:30:55.885 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.885 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.885 DEBUG:boto:Method: GET 2024-04-13T22:30:55.885 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:55.885 DEBUG:boto:Data: 2024-04-13T22:30:55.885 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.885 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.886 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.886 DEBUG:boto:Params: {} 2024-04-13T22:30:55.886 DEBUG:boto:Token: None 2024-04-13T22:30:55.886 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148 2024-04-13T22:30:55.886 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pn4665/HPGc3Fzg7ZjzNXeosXQ4= 2024-04-13T22:30:55.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pn4665/HPGc3Fzg7ZjzNXeosXQ4=', 'Content-Length': '0'} 2024-04-13T22:30:55.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965d4dfab5e26155-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.888 DEBUG:boto:path=/vwnfns-61/obj148 2024-04-13T22:30:55.888 DEBUG:boto:auth_path=/vwnfns-61/obj148 2024-04-13T22:30:55.888 DEBUG:boto:Method: GET 2024-04-13T22:30:55.888 DEBUG:boto:Path: /vwnfns-61/obj148 2024-04-13T22:30:55.888 DEBUG:boto:Data: 2024-04-13T22:30:55.888 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.888 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.889 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.889 DEBUG:boto:Params: {} 2024-04-13T22:30:55.889 DEBUG:boto:Token: None 2024-04-13T22:30:55.889 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj148 2024-04-13T22:30:55.889 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pn4665/HPGc3Fzg7ZjzNXeosXQ4= 2024-04-13T22:30:55.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pn4665/HPGc3Fzg7ZjzNXeosXQ4=', 'Content-Length': '0'} 2024-04-13T22:30:55.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569e286af867cf5f-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.891 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.891 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.891 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.891 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.892 DEBUG:boto:Method: GET 2024-04-13T22:30:55.892 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.892 DEBUG:boto:Data: 2024-04-13T22:30:55.892 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.892 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.892 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.892 DEBUG:boto:Params: {} 2024-04-13T22:30:55.892 DEBUG:boto:Token: None 2024-04-13T22:30:55.892 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.892 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e90e64c5bbd0ecc-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.897 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.898 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.898 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.898 DEBUG:boto:path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.898 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.898 DEBUG:boto:Method: GET 2024-04-13T22:30:55.898 DEBUG:boto:Path: /vwnfns-61/?prefix=obj148 2024-04-13T22:30:55.898 DEBUG:boto:Data: 2024-04-13T22:30:55.898 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.898 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.899 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.899 DEBUG:boto:Params: {} 2024-04-13T22:30:55.899 DEBUG:boto:Token: None 2024-04-13T22:30:55.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.900 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5efe8879b59f999-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.903 DEBUG:boto:b'vwnfns-61obj1481000falseobj1482024-04-13T22:28:56.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:30:55.904 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:55.904 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:55.904 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.904 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.904 DEBUG:boto:Method: GET 2024-04-13T22:30:55.904 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.904 DEBUG:boto:Data: 2024-04-13T22:30:55.904 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.905 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.905 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.905 DEBUG:boto:Params: {} 2024-04-13T22:30:55.905 DEBUG:boto:Token: None 2024-04-13T22:30:55.905 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.905 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A= 2024-04-13T22:30:55.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A=', 'Content-Length': '0'} 2024-04-13T22:30:55.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028412432785afaaf-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.951 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:55.951 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:55.951 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.951 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.951 DEBUG:boto:Method: GET 2024-04-13T22:30:55.951 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.951 DEBUG:boto:Data: 2024-04-13T22:30:55.951 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.952 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.952 DEBUG:boto:Params: {} 2024-04-13T22:30:55.952 DEBUG:boto:Token: None 2024-04-13T22:30:55.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A= 2024-04-13T22:30:55.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A=', 'Content-Length': '0'} 2024-04-13T22:30:55.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a597c8b489a7409-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.954 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:55.954 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:55.955 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.955 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.955 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.955 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.955 DEBUG:boto:Data: 2024-04-13T22:30:55.955 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.955 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.955 DEBUG:boto:Params: {} 2024-04-13T22:30:55.955 DEBUG:boto:Token: None 2024-04-13T22:30:55.956 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pLZTnB6QS4Pq5BxltUT1wXHG0xw= 2024-04-13T22:30:55.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pLZTnB6QS4Pq5BxltUT1wXHG0xw=', 'Content-Length': '0'} 2024-04-13T22:30:55.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2482054f1926ee-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.957 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:55.958 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:55.958 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.958 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.958 DEBUG:boto:Method: HEAD 2024-04-13T22:30:55.958 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.958 DEBUG:boto:Data: 2024-04-13T22:30:55.958 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.958 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.958 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.958 DEBUG:boto:Params: {} 2024-04-13T22:30:55.959 DEBUG:boto:Token: None 2024-04-13T22:30:55.959 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.959 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pLZTnB6QS4Pq5BxltUT1wXHG0xw= 2024-04-13T22:30:55.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pLZTnB6QS4Pq5BxltUT1wXHG0xw=', 'Content-Length': '0'} 2024-04-13T22:30:55.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e476fb7aef6f28-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:30:55.961 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:55.961 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:55.961 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.961 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.961 DEBUG:boto:Method: GET 2024-04-13T22:30:55.961 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.961 DEBUG:boto:Data: 2024-04-13T22:30:55.961 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.961 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.961 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.962 DEBUG:boto:Params: {} 2024-04-13T22:30:55.962 DEBUG:boto:Token: None 2024-04-13T22:30:55.962 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.962 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A= 2024-04-13T22:30:55.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A=', 'Content-Length': '0'} 2024-04-13T22:30:55.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000683aa474232178-00661b079f-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.964 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:55.964 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:55.964 DEBUG:boto:path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.964 DEBUG:boto:auth_path=/vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.964 DEBUG:boto:Method: GET 2024-04-13T22:30:55.964 DEBUG:boto:Path: /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.964 DEBUG:boto:Data: 2024-04-13T22:30:55.965 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.965 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.965 DEBUG:boto:Params: {} 2024-04-13T22:30:55.965 DEBUG:boto:Token: None 2024-04-13T22:30:55.965 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/obj155?versionId=null 2024-04-13T22:30:55.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A= 2024-04-13T22:30:55.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CXLJ3g0WHz/wCKimK5/gOM2FM9A=', 'Content-Length': '0'} 2024-04-13T22:30:55.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740947f74a84334f-00661b079f-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.967 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.967 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.967 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:55.968 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:55.968 DEBUG:boto:Method: GET 2024-04-13T22:30:55.968 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:55.968 DEBUG:boto:Data: 2024-04-13T22:30:55.968 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:55.968 DEBUG:boto:Port: 8001 2024-04-13T22:30:55.968 DEBUG:boto:Params: {} 2024-04-13T22:30:55.968 DEBUG:boto:Token: None 2024-04-13T22:30:55.968 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.969 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023bb5abeb3985e83-00661b079f-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:55.971 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:55.971 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:55.971 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:55.971 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:55.971 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:55.972 DEBUG:boto:Method: GET 2024-04-13T22:30:55.972 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:55.972 DEBUG:boto:Data: 2024-04-13T22:30:55.972 DEBUG:boto:Headers: {} 2024-04-13T22:30:55.972 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:55.972 DEBUG:boto:Port: 8000 2024-04-13T22:30:55.972 DEBUG:boto:Params: {} 2024-04-13T22:30:55.972 DEBUG:boto:Token: None 2024-04-13T22:30:55.972 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:55 GMT /vwnfns-61/ 2024-04-13T22:30:55.973 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk= 2024-04-13T22:30:55.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nadlvD031BsMbUhSsRO0IMQGldk=', 'Content-Length': '0'} 2024-04-13T22:30:55.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7af7701e2dec1c-00661b079f-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.018 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.019 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:56.019 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:56.019 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.019 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:56.019 DEBUG:boto:Data: 2024-04-13T22:30:56.019 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.019 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.020 DEBUG:boto:Params: {} 2024-04-13T22:30:56.020 DEBUG:boto:Token: None 2024-04-13T22:30:56.020 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj155 2024-04-13T22:30:56.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E3OPOoor/OYUyNZpVwPrWp1sRnE= 2024-04-13T22:30:56.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E3OPOoor/OYUyNZpVwPrWp1sRnE=', 'Content-Length': '0'} 2024-04-13T22:30:56.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfa1a9791e7d649-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.022 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:56.022 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:56.022 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.022 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:56.022 DEBUG:boto:Data: 2024-04-13T22:30:56.023 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.023 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.023 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.023 DEBUG:boto:Params: {} 2024-04-13T22:30:56.023 DEBUG:boto:Token: None 2024-04-13T22:30:56.023 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj155 2024-04-13T22:30:56.023 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:E3OPOoor/OYUyNZpVwPrWp1sRnE= 2024-04-13T22:30:56.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E3OPOoor/OYUyNZpVwPrWp1sRnE=', 'Content-Length': '0'} 2024-04-13T22:30:56.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e0f362ed92213d8-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-13T22:30:56.025 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:56.025 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:56.025 DEBUG:boto:Method: GET 2024-04-13T22:30:56.025 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:56.025 DEBUG:boto:Data: 2024-04-13T22:30:56.026 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.026 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.026 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.026 DEBUG:boto:Params: {} 2024-04-13T22:30:56.026 DEBUG:boto:Token: None 2024-04-13T22:30:56.026 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj155 2024-04-13T22:30:56.026 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xVF4lLfUil+V0zDVBtOmnqIF1dI= 2024-04-13T22:30:56.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xVF4lLfUil+V0zDVBtOmnqIF1dI=', 'Content-Length': '0'} 2024-04-13T22:30:56.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41c5d3133e65ccf-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.028 DEBUG:boto:path=/vwnfns-61/obj155 2024-04-13T22:30:56.028 DEBUG:boto:auth_path=/vwnfns-61/obj155 2024-04-13T22:30:56.029 DEBUG:boto:Method: GET 2024-04-13T22:30:56.029 DEBUG:boto:Path: /vwnfns-61/obj155 2024-04-13T22:30:56.029 DEBUG:boto:Data: 2024-04-13T22:30:56.029 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.029 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.029 DEBUG:boto:Params: {} 2024-04-13T22:30:56.029 DEBUG:boto:Token: None 2024-04-13T22:30:56.029 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj155 2024-04-13T22:30:56.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xVF4lLfUil+V0zDVBtOmnqIF1dI= 2024-04-13T22:30:56.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xVF4lLfUil+V0zDVBtOmnqIF1dI=', 'Content-Length': '0'} 2024-04-13T22:30:56.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008827249163095dc2-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.032 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.032 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.032 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:56.032 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:56.032 DEBUG:boto:Method: GET 2024-04-13T22:30:56.032 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:56.032 DEBUG:boto:Data: 2024-04-13T22:30:56.032 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.032 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.032 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.033 DEBUG:boto:Params: {} 2024-04-13T22:30:56.033 DEBUG:boto:Token: None 2024-04-13T22:30:56.033 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.033 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1d7c8979408ea06-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.035 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.036 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.036 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.036 DEBUG:boto:path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:56.036 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj155 2024-04-13T22:30:56.036 DEBUG:boto:Method: GET 2024-04-13T22:30:56.036 DEBUG:boto:Path: /vwnfns-61/?prefix=obj155 2024-04-13T22:30:56.036 DEBUG:boto:Data: 2024-04-13T22:30:56.036 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.037 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.037 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.037 DEBUG:boto:Params: {} 2024-04-13T22:30:56.037 DEBUG:boto:Token: None 2024-04-13T22:30:56.037 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.037 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f8fe72ae2e58e1b-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.041 DEBUG:boto:b'vwnfns-61obj1551000falseobj1552024-04-13T22:27:52.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:30:56.042 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.042 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.042 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.042 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.042 DEBUG:boto:Method: GET 2024-04-13T22:30:56.042 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.042 DEBUG:boto:Data: 2024-04-13T22:30:56.042 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.043 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.043 DEBUG:boto:Params: {} 2024-04-13T22:30:56.043 DEBUG:boto:Token: None 2024-04-13T22:30:56.043 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.043 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA= 2024-04-13T22:30:56.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA=', 'Content-Length': '0'} 2024-04-13T22:30:56.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6a4c3771bb1e5f-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.045 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.045 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.046 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.046 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.046 DEBUG:boto:Method: GET 2024-04-13T22:30:56.046 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.046 DEBUG:boto:Data: 2024-04-13T22:30:56.046 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.046 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.046 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.046 DEBUG:boto:Params: {} 2024-04-13T22:30:56.046 DEBUG:boto:Token: None 2024-04-13T22:30:56.047 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.047 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA= 2024-04-13T22:30:56.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA=', 'Content-Length': '0'} 2024-04-13T22:30:56.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8844a46487cad5-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.090 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.090 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.091 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.091 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.091 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.091 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.091 DEBUG:boto:Data: 2024-04-13T22:30:56.091 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.091 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.091 DEBUG:boto:Params: {} 2024-04-13T22:30:56.091 DEBUG:boto:Token: None 2024-04-13T22:30:56.092 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5PwVwkE2LP53fMvTu4mcT8rEuNs= 2024-04-13T22:30:56.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5PwVwkE2LP53fMvTu4mcT8rEuNs=', 'Content-Length': '0'} 2024-04-13T22:30:56.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004112e8f57cae40cf-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.093 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.094 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.094 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.094 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.094 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.094 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.094 DEBUG:boto:Data: 2024-04-13T22:30:56.094 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.094 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.094 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.095 DEBUG:boto:Params: {} 2024-04-13T22:30:56.095 DEBUG:boto:Token: None 2024-04-13T22:30:56.095 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.095 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5PwVwkE2LP53fMvTu4mcT8rEuNs= 2024-04-13T22:30:56.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5PwVwkE2LP53fMvTu4mcT8rEuNs=', 'Content-Length': '0'} 2024-04-13T22:30:56.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc7773b6bac13878-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:30:56.097 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.097 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.097 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.097 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.097 DEBUG:boto:Method: GET 2024-04-13T22:30:56.097 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.097 DEBUG:boto:Data: 2024-04-13T22:30:56.098 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.098 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.098 DEBUG:boto:Params: {} 2024-04-13T22:30:56.098 DEBUG:boto:Token: None 2024-04-13T22:30:56.098 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA= 2024-04-13T22:30:56.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA=', 'Content-Length': '0'} 2024-04-13T22:30:56.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b483342d53e1d2-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.100 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.100 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.100 DEBUG:boto:path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.101 DEBUG:boto:auth_path=/vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.101 DEBUG:boto:Method: GET 2024-04-13T22:30:56.101 DEBUG:boto:Path: /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.101 DEBUG:boto:Data: 2024-04-13T22:30:56.101 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.101 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.101 DEBUG:boto:Params: {} 2024-04-13T22:30:56.101 DEBUG:boto:Token: None 2024-04-13T22:30:56.102 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17?versionId=null 2024-04-13T22:30:56.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA= 2024-04-13T22:30:56.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rQXzjUgSOJ0CB+1vCmugUKqnAiA=', 'Content-Length': '0'} 2024-04-13T22:30:56.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007177d5031ea5d87f-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.104 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.104 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.104 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.104 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.104 DEBUG:boto:Method: GET 2024-04-13T22:30:56.104 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.104 DEBUG:boto:Data: 2024-04-13T22:30:56.104 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.104 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.104 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.105 DEBUG:boto:Params: {} 2024-04-13T22:30:56.105 DEBUG:boto:Token: None 2024-04-13T22:30:56.105 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.105 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc89a96a3190962-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.107 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.108 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.108 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.108 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.108 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.108 DEBUG:boto:Method: GET 2024-04-13T22:30:56.108 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.108 DEBUG:boto:Data: 2024-04-13T22:30:56.109 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.109 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.109 DEBUG:boto:Params: {} 2024-04-13T22:30:56.109 DEBUG:boto:Token: None 2024-04-13T22:30:56.109 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054ccfd5568e558cb-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.113 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.113 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.113 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.113 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.114 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:56.114 DEBUG:boto:Data: 2024-04-13T22:30:56.114 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.114 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.114 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.114 DEBUG:boto:Params: {} 2024-04-13T22:30:56.114 DEBUG:boto:Token: None 2024-04-13T22:30:56.114 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17 2024-04-13T22:30:56.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rivhmU7ZoMQCa6sfvLGn/Ac6DeM= 2024-04-13T22:30:56.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rivhmU7ZoMQCa6sfvLGn/Ac6DeM=', 'Content-Length': '0'} 2024-04-13T22:30:56.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6ae36c6949b571-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.116 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.116 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.116 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.117 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:56.117 DEBUG:boto:Data: 2024-04-13T22:30:56.117 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.117 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.117 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.117 DEBUG:boto:Params: {} 2024-04-13T22:30:56.117 DEBUG:boto:Token: None 2024-04-13T22:30:56.117 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17 2024-04-13T22:30:56.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rivhmU7ZoMQCa6sfvLGn/Ac6DeM= 2024-04-13T22:30:56.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rivhmU7ZoMQCa6sfvLGn/Ac6DeM=', 'Content-Length': '0'} 2024-04-13T22:30:56.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033df7d975bdf8ccc-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-13T22:30:56.119 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.119 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.119 DEBUG:boto:Method: GET 2024-04-13T22:30:56.120 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:56.120 DEBUG:boto:Data: 2024-04-13T22:30:56.120 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.120 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.120 DEBUG:boto:Params: {} 2024-04-13T22:30:56.120 DEBUG:boto:Token: None 2024-04-13T22:30:56.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17 2024-04-13T22:30:56.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjrGhWbrR0UUxwOMiRXeaN6EDoI= 2024-04-13T22:30:56.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjrGhWbrR0UUxwOMiRXeaN6EDoI=', 'Content-Length': '0'} 2024-04-13T22:30:56.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d4595bff1c0e5f3-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.162 DEBUG:boto:path=/vwnfns-61/obj17 2024-04-13T22:30:56.163 DEBUG:boto:auth_path=/vwnfns-61/obj17 2024-04-13T22:30:56.163 DEBUG:boto:Method: GET 2024-04-13T22:30:56.163 DEBUG:boto:Path: /vwnfns-61/obj17 2024-04-13T22:30:56.163 DEBUG:boto:Data: 2024-04-13T22:30:56.163 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.163 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.163 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.163 DEBUG:boto:Params: {} 2024-04-13T22:30:56.163 DEBUG:boto:Token: None 2024-04-13T22:30:56.164 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj17 2024-04-13T22:30:56.164 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DjrGhWbrR0UUxwOMiRXeaN6EDoI= 2024-04-13T22:30:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DjrGhWbrR0UUxwOMiRXeaN6EDoI=', 'Content-Length': '0'} 2024-04-13T22:30:56.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40e89d1dcbcc086-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.166 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.166 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.166 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.166 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.166 DEBUG:boto:Method: GET 2024-04-13T22:30:56.166 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.166 DEBUG:boto:Data: 2024-04-13T22:30:56.167 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.167 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.167 DEBUG:boto:Params: {} 2024-04-13T22:30:56.167 DEBUG:boto:Token: None 2024-04-13T22:30:56.167 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.167 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000430c6f263e79de5a-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.170 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.170 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.170 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.170 DEBUG:boto:path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.170 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.170 DEBUG:boto:Method: GET 2024-04-13T22:30:56.171 DEBUG:boto:Path: /vwnfns-61/?prefix=obj17 2024-04-13T22:30:56.171 DEBUG:boto:Data: 2024-04-13T22:30:56.171 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.171 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.171 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.171 DEBUG:boto:Params: {} 2024-04-13T22:30:56.171 DEBUG:boto:Token: None 2024-04-13T22:30:56.171 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.171 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a1e981394c87ea-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.175 DEBUG:boto:b'vwnfns-61obj171000falseobj172024-04-13T22:18:11.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-13T22:30:56.176 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.176 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.176 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.176 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.176 DEBUG:boto:Method: GET 2024-04-13T22:30:56.176 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.176 DEBUG:boto:Data: 2024-04-13T22:30:56.176 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.177 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.177 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.177 DEBUG:boto:Params: {} 2024-04-13T22:30:56.177 DEBUG:boto:Token: None 2024-04-13T22:30:56.177 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.177 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8= 2024-04-13T22:30:56.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8=', 'Content-Length': '0'} 2024-04-13T22:30:56.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef110f9bd5ef1e73-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.179 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.179 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.179 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.180 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.180 DEBUG:boto:Method: GET 2024-04-13T22:30:56.180 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.180 DEBUG:boto:Data: 2024-04-13T22:30:56.180 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.180 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.180 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.180 DEBUG:boto:Params: {} 2024-04-13T22:30:56.180 DEBUG:boto:Token: None 2024-04-13T22:30:56.180 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.181 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8= 2024-04-13T22:30:56.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8=', 'Content-Length': '0'} 2024-04-13T22:30:56.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5bd130649669afc-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.226 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.227 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.227 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.227 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.227 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.227 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.227 DEBUG:boto:Data: 2024-04-13T22:30:56.227 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.227 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.227 DEBUG:boto:Params: {} 2024-04-13T22:30:56.228 DEBUG:boto:Token: None 2024-04-13T22:30:56.228 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5LsdkT99sEcyqMZAVzhfscmaoYc= 2024-04-13T22:30:56.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5LsdkT99sEcyqMZAVzhfscmaoYc=', 'Content-Length': '0'} 2024-04-13T22:30:56.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19206492d708324-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.230 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.230 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.230 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.230 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.230 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.231 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.231 DEBUG:boto:Data: 2024-04-13T22:30:56.231 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.231 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.231 DEBUG:boto:Params: {} 2024-04-13T22:30:56.231 DEBUG:boto:Token: None 2024-04-13T22:30:56.231 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.231 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5LsdkT99sEcyqMZAVzhfscmaoYc= 2024-04-13T22:30:56.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5LsdkT99sEcyqMZAVzhfscmaoYc=', 'Content-Length': '0'} 2024-04-13T22:30:56.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51ad46534746edc-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-13T22:30:56.234 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.234 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.234 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.234 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.234 DEBUG:boto:Method: GET 2024-04-13T22:30:56.235 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.235 DEBUG:boto:Data: 2024-04-13T22:30:56.235 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.235 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.235 DEBUG:boto:Params: {} 2024-04-13T22:30:56.235 DEBUG:boto:Token: None 2024-04-13T22:30:56.235 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8= 2024-04-13T22:30:56.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8=', 'Content-Length': '0'} 2024-04-13T22:30:56.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebdae944b2a93f5c-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.237 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.237 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.237 DEBUG:boto:path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.237 DEBUG:boto:auth_path=/vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.238 DEBUG:boto:Method: GET 2024-04-13T22:30:56.238 DEBUG:boto:Path: /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.238 DEBUG:boto:Data: 2024-04-13T22:30:56.238 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.238 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.238 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.238 DEBUG:boto:Params: {} 2024-04-13T22:30:56.238 DEBUG:boto:Token: None 2024-04-13T22:30:56.238 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185?versionId=null 2024-04-13T22:30:56.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8= 2024-04-13T22:30:56.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3H96m9P3g+ZHKgyuEwUz3qitiN8=', 'Content-Length': '0'} 2024-04-13T22:30:56.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e5015c6fe5104b-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.241 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.241 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.241 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.242 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.242 DEBUG:boto:Method: GET 2024-04-13T22:30:56.242 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.242 DEBUG:boto:Data: 2024-04-13T22:30:56.242 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.242 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.242 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.242 DEBUG:boto:Params: {} 2024-04-13T22:30:56.242 DEBUG:boto:Token: None 2024-04-13T22:30:56.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.243 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e02fe527501eea5d-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.245 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.245 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.246 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.246 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.246 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.246 DEBUG:boto:Method: GET 2024-04-13T22:30:56.246 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.246 DEBUG:boto:Data: 2024-04-13T22:30:56.246 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.246 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.246 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.246 DEBUG:boto:Params: {} 2024-04-13T22:30:56.247 DEBUG:boto:Token: None 2024-04-13T22:30:56.247 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c0eb0276064e969-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.251 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.251 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.251 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.251 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.251 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:56.251 DEBUG:boto:Data: 2024-04-13T22:30:56.251 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.252 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.252 DEBUG:boto:Params: {} 2024-04-13T22:30:56.252 DEBUG:boto:Token: None 2024-04-13T22:30:56.252 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185 2024-04-13T22:30:56.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C6hzuypf7CRVZ3jtowCuD8L3i7o= 2024-04-13T22:30:56.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C6hzuypf7CRVZ3jtowCuD8L3i7o=', 'Content-Length': '0'} 2024-04-13T22:30:56.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b082fe4a8e006347-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.254 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.254 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.254 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.254 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:56.255 DEBUG:boto:Data: 2024-04-13T22:30:56.255 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.255 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.255 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.255 DEBUG:boto:Params: {} 2024-04-13T22:30:56.255 DEBUG:boto:Token: None 2024-04-13T22:30:56.255 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185 2024-04-13T22:30:56.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C6hzuypf7CRVZ3jtowCuD8L3i7o= 2024-04-13T22:30:56.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C6hzuypf7CRVZ3jtowCuD8L3i7o=', 'Content-Length': '0'} 2024-04-13T22:30:56.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324558050416b74e-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-13T22:30:56.257 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.257 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.257 DEBUG:boto:Method: GET 2024-04-13T22:30:56.258 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:56.258 DEBUG:boto:Data: 2024-04-13T22:30:56.258 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.258 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.258 DEBUG:boto:Params: {} 2024-04-13T22:30:56.258 DEBUG:boto:Token: None 2024-04-13T22:30:56.258 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185 2024-04-13T22:30:56.258 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vc10yL8Ta7EVY6gfzGUS6UTldvM= 2024-04-13T22:30:56.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vc10yL8Ta7EVY6gfzGUS6UTldvM=', 'Content-Length': '0'} 2024-04-13T22:30:56.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9095f3ae87416c1-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.261 DEBUG:boto:path=/vwnfns-61/obj185 2024-04-13T22:30:56.261 DEBUG:boto:auth_path=/vwnfns-61/obj185 2024-04-13T22:30:56.261 DEBUG:boto:Method: GET 2024-04-13T22:30:56.261 DEBUG:boto:Path: /vwnfns-61/obj185 2024-04-13T22:30:56.261 DEBUG:boto:Data: 2024-04-13T22:30:56.261 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.261 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.261 DEBUG:boto:Params: {} 2024-04-13T22:30:56.261 DEBUG:boto:Token: None 2024-04-13T22:30:56.262 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj185 2024-04-13T22:30:56.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Vc10yL8Ta7EVY6gfzGUS6UTldvM= 2024-04-13T22:30:56.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Vc10yL8Ta7EVY6gfzGUS6UTldvM=', 'Content-Length': '0'} 2024-04-13T22:30:56.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a7b2e675066162-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.264 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.264 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.264 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.265 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.265 DEBUG:boto:Method: GET 2024-04-13T22:30:56.265 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.265 DEBUG:boto:Data: 2024-04-13T22:30:56.265 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.265 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.265 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.266 DEBUG:boto:Params: {} 2024-04-13T22:30:56.266 DEBUG:boto:Token: None 2024-04-13T22:30:56.266 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.266 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b8328e8f1db68ff-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.268 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.268 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.269 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.269 DEBUG:boto:path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.269 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.269 DEBUG:boto:Method: GET 2024-04-13T22:30:56.269 DEBUG:boto:Path: /vwnfns-61/?prefix=obj185 2024-04-13T22:30:56.269 DEBUG:boto:Data: 2024-04-13T22:30:56.269 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.269 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.269 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.269 DEBUG:boto:Params: {} 2024-04-13T22:30:56.269 DEBUG:boto:Token: None 2024-04-13T22:30:56.270 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.270 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef621d226774f6a-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.274 DEBUG:boto:b'vwnfns-61obj1851000falseobj1852024-04-13T22:28:56.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:30:56.274 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.274 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.274 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.274 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.274 DEBUG:boto:Method: GET 2024-04-13T22:30:56.275 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.275 DEBUG:boto:Data: 2024-04-13T22:30:56.275 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.275 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.275 DEBUG:boto:Params: {} 2024-04-13T22:30:56.275 DEBUG:boto:Token: None 2024-04-13T22:30:56.275 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM= 2024-04-13T22:30:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM=', 'Content-Length': '0'} 2024-04-13T22:30:56.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d028d6bd134334-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.277 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.277 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.277 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.278 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.278 DEBUG:boto:Method: GET 2024-04-13T22:30:56.278 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.278 DEBUG:boto:Data: 2024-04-13T22:30:56.278 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.278 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.278 DEBUG:boto:Params: {} 2024-04-13T22:30:56.278 DEBUG:boto:Token: None 2024-04-13T22:30:56.278 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM= 2024-04-13T22:30:56.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM=', 'Content-Length': '0'} 2024-04-13T22:30:56.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0eda2991ee7fd7c-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.280 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.281 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.281 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.281 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.281 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.281 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.281 DEBUG:boto:Data: 2024-04-13T22:30:56.281 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.281 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.281 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.281 DEBUG:boto:Params: {} 2024-04-13T22:30:56.281 DEBUG:boto:Token: None 2024-04-13T22:30:56.282 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.282 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hoIuIZ5ebsDWZJAaibTf4GSa4r0= 2024-04-13T22:30:56.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hoIuIZ5ebsDWZJAaibTf4GSa4r0=', 'Content-Length': '0'} 2024-04-13T22:30:56.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa867e851da15d8c-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.283 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.284 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.284 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.284 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.284 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.284 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.284 DEBUG:boto:Data: 2024-04-13T22:30:56.284 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.284 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.284 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.284 DEBUG:boto:Params: {} 2024-04-13T22:30:56.284 DEBUG:boto:Token: None 2024-04-13T22:30:56.285 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.285 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hoIuIZ5ebsDWZJAaibTf4GSa4r0= 2024-04-13T22:30:56.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hoIuIZ5ebsDWZJAaibTf4GSa4r0=', 'Content-Length': '0'} 2024-04-13T22:30:56.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000588bac2d917c6f12-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:30:56.287 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.287 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.287 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.287 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.287 DEBUG:boto:Method: GET 2024-04-13T22:30:56.287 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.287 DEBUG:boto:Data: 2024-04-13T22:30:56.288 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.288 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.288 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.288 DEBUG:boto:Params: {} 2024-04-13T22:30:56.288 DEBUG:boto:Token: None 2024-04-13T22:30:56.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM= 2024-04-13T22:30:56.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM=', 'Content-Length': '0'} 2024-04-13T22:30:56.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33d2b637ce6307c-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.290 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.290 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.290 DEBUG:boto:path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.291 DEBUG:boto:auth_path=/vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.291 DEBUG:boto:Method: GET 2024-04-13T22:30:56.291 DEBUG:boto:Path: /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.291 DEBUG:boto:Data: 2024-04-13T22:30:56.291 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.291 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.291 DEBUG:boto:Params: {} 2024-04-13T22:30:56.291 DEBUG:boto:Token: None 2024-04-13T22:30:56.291 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19?versionId=null 2024-04-13T22:30:56.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM= 2024-04-13T22:30:56.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EEaRv7acMTy3L+FuACpxJtFaPKM=', 'Content-Length': '0'} 2024-04-13T22:30:56.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae2b9e4f2aa0efad-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.334 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.335 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.335 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.335 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.335 DEBUG:boto:Method: GET 2024-04-13T22:30:56.335 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.335 DEBUG:boto:Data: 2024-04-13T22:30:56.335 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.335 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.336 DEBUG:boto:Params: {} 2024-04-13T22:30:56.336 DEBUG:boto:Token: None 2024-04-13T22:30:56.336 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.336 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000027b37d3452f602f-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.339 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.339 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.339 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.339 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.339 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.339 DEBUG:boto:Method: GET 2024-04-13T22:30:56.340 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.340 DEBUG:boto:Data: 2024-04-13T22:30:56.340 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.340 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.340 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.340 DEBUG:boto:Params: {} 2024-04-13T22:30:56.340 DEBUG:boto:Token: None 2024-04-13T22:30:56.340 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.340 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e99f9f6c3a9202-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.344 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.344 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.344 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.344 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.345 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:56.345 DEBUG:boto:Data: 2024-04-13T22:30:56.345 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.345 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.345 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.345 DEBUG:boto:Params: {} 2024-04-13T22:30:56.345 DEBUG:boto:Token: None 2024-04-13T22:30:56.345 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19 2024-04-13T22:30:56.345 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eLrx/FmOIGgznFaivUWwoosIPl8= 2024-04-13T22:30:56.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eLrx/FmOIGgznFaivUWwoosIPl8=', 'Content-Length': '0'} 2024-04-13T22:30:56.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012663d5b22f3a1d3-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.347 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.348 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.348 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.348 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:56.348 DEBUG:boto:Data: 2024-04-13T22:30:56.348 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.348 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.348 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.348 DEBUG:boto:Params: {} 2024-04-13T22:30:56.348 DEBUG:boto:Token: None 2024-04-13T22:30:56.349 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19 2024-04-13T22:30:56.349 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eLrx/FmOIGgznFaivUWwoosIPl8= 2024-04-13T22:30:56.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eLrx/FmOIGgznFaivUWwoosIPl8=', 'Content-Length': '0'} 2024-04-13T22:30:56.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e430e1e2f9c216b-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-13T22:30:56.351 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.351 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.351 DEBUG:boto:Method: GET 2024-04-13T22:30:56.351 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:56.351 DEBUG:boto:Data: 2024-04-13T22:30:56.351 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.351 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.351 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.351 DEBUG:boto:Params: {} 2024-04-13T22:30:56.351 DEBUG:boto:Token: None 2024-04-13T22:30:56.352 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19 2024-04-13T22:30:56.352 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OUB5StuHLCh0XbR1OV7cjYvO4JY= 2024-04-13T22:30:56.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OUB5StuHLCh0XbR1OV7cjYvO4JY=', 'Content-Length': '0'} 2024-04-13T22:30:56.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38ad8e2532212dc-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.394 DEBUG:boto:path=/vwnfns-61/obj19 2024-04-13T22:30:56.395 DEBUG:boto:auth_path=/vwnfns-61/obj19 2024-04-13T22:30:56.395 DEBUG:boto:Method: GET 2024-04-13T22:30:56.395 DEBUG:boto:Path: /vwnfns-61/obj19 2024-04-13T22:30:56.395 DEBUG:boto:Data: 2024-04-13T22:30:56.395 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.395 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.395 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.395 DEBUG:boto:Params: {} 2024-04-13T22:30:56.396 DEBUG:boto:Token: None 2024-04-13T22:30:56.396 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj19 2024-04-13T22:30:56.396 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OUB5StuHLCh0XbR1OV7cjYvO4JY= 2024-04-13T22:30:56.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OUB5StuHLCh0XbR1OV7cjYvO4JY=', 'Content-Length': '0'} 2024-04-13T22:30:56.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000627b882b2e650ab0-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.398 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.398 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.398 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.398 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.398 DEBUG:boto:Method: GET 2024-04-13T22:30:56.398 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.399 DEBUG:boto:Data: 2024-04-13T22:30:56.399 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.399 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.399 DEBUG:boto:Params: {} 2024-04-13T22:30:56.399 DEBUG:boto:Token: None 2024-04-13T22:30:56.399 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.399 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001531a394870e9dd0-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.402 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.402 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.402 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.402 DEBUG:boto:path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.402 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.402 DEBUG:boto:Method: GET 2024-04-13T22:30:56.403 DEBUG:boto:Path: /vwnfns-61/?prefix=obj19 2024-04-13T22:30:56.403 DEBUG:boto:Data: 2024-04-13T22:30:56.403 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.403 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.403 DEBUG:boto:Params: {} 2024-04-13T22:30:56.403 DEBUG:boto:Token: None 2024-04-13T22:30:56.403 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.403 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbe7727860b7d4d-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.407 DEBUG:boto:b'vwnfns-61obj191000falseobj192024-04-13T22:21:13.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:30:56.408 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.408 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.408 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.408 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.408 DEBUG:boto:Method: GET 2024-04-13T22:30:56.408 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.408 DEBUG:boto:Data: 2024-04-13T22:30:56.408 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.408 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.408 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.408 DEBUG:boto:Params: {} 2024-04-13T22:30:56.409 DEBUG:boto:Token: None 2024-04-13T22:30:56.409 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.409 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA= 2024-04-13T22:30:56.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA=', 'Content-Length': '0'} 2024-04-13T22:30:56.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6841eb25de9c47-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.411 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.411 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.411 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.411 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.411 DEBUG:boto:Method: GET 2024-04-13T22:30:56.411 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.411 DEBUG:boto:Data: 2024-04-13T22:30:56.412 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.412 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.412 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.412 DEBUG:boto:Params: {} 2024-04-13T22:30:56.412 DEBUG:boto:Token: None 2024-04-13T22:30:56.412 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.412 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA= 2024-04-13T22:30:56.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA=', 'Content-Length': '0'} 2024-04-13T22:30:56.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ae1dd2ab9a18c3-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.458 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.459 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.459 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.459 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.459 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.459 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.459 DEBUG:boto:Data: 2024-04-13T22:30:56.459 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.459 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.459 DEBUG:boto:Params: {} 2024-04-13T22:30:56.460 DEBUG:boto:Token: None 2024-04-13T22:30:56.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OHl7P5NgQMWHzmBi1rNn/nmjjYE= 2024-04-13T22:30:56.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OHl7P5NgQMWHzmBi1rNn/nmjjYE=', 'Content-Length': '0'} 2024-04-13T22:30:56.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddcc29eb167b1492-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.462 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.462 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.462 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.462 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.462 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.462 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.462 DEBUG:boto:Data: 2024-04-13T22:30:56.462 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.463 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.463 DEBUG:boto:Params: {} 2024-04-13T22:30:56.463 DEBUG:boto:Token: None 2024-04-13T22:30:56.463 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.463 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OHl7P5NgQMWHzmBi1rNn/nmjjYE= 2024-04-13T22:30:56.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OHl7P5NgQMWHzmBi1rNn/nmjjYE=', 'Content-Length': '0'} 2024-04-13T22:30:56.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032f6f1931902d24-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:30:56.465 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.465 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.465 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.465 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.466 DEBUG:boto:Method: GET 2024-04-13T22:30:56.466 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.466 DEBUG:boto:Data: 2024-04-13T22:30:56.466 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.466 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.466 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.466 DEBUG:boto:Params: {} 2024-04-13T22:30:56.466 DEBUG:boto:Token: None 2024-04-13T22:30:56.466 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA= 2024-04-13T22:30:56.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA=', 'Content-Length': '0'} 2024-04-13T22:30:56.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7e76cb875c2fae-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.468 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.468 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.469 DEBUG:boto:path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.469 DEBUG:boto:auth_path=/vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.469 DEBUG:boto:Method: GET 2024-04-13T22:30:56.469 DEBUG:boto:Path: /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.469 DEBUG:boto:Data: 2024-04-13T22:30:56.469 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.469 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.469 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.469 DEBUG:boto:Params: {} 2024-04-13T22:30:56.469 DEBUG:boto:Token: None 2024-04-13T22:30:56.470 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22?versionId=null 2024-04-13T22:30:56.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA= 2024-04-13T22:30:56.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:C6Lka0ZBpQDLT4Es2JZyZ5/RiwA=', 'Content-Length': '0'} 2024-04-13T22:30:56.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c4b9cb65f0e1585-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.472 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.472 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.472 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.472 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.472 DEBUG:boto:Method: GET 2024-04-13T22:30:56.472 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.472 DEBUG:boto:Data: 2024-04-13T22:30:56.472 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.472 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.472 DEBUG:boto:Params: {} 2024-04-13T22:30:56.473 DEBUG:boto:Token: None 2024-04-13T22:30:56.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e67fe5fa93ab687a-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.475 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.476 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.476 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.476 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.476 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.476 DEBUG:boto:Method: GET 2024-04-13T22:30:56.476 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.476 DEBUG:boto:Data: 2024-04-13T22:30:56.476 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.476 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.476 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.476 DEBUG:boto:Params: {} 2024-04-13T22:30:56.477 DEBUG:boto:Token: None 2024-04-13T22:30:56.477 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.477 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a7e7b3ec1a1ee92-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.480 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.481 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.481 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.481 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.481 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:56.481 DEBUG:boto:Data: 2024-04-13T22:30:56.481 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.481 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.481 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.481 DEBUG:boto:Params: {} 2024-04-13T22:30:56.482 DEBUG:boto:Token: None 2024-04-13T22:30:56.482 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22 2024-04-13T22:30:56.482 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O093qD/ZQb2qf+XrdVyww2r0WNs= 2024-04-13T22:30:56.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O093qD/ZQb2qf+XrdVyww2r0WNs=', 'Content-Length': '0'} 2024-04-13T22:30:56.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23da1034f38bda7-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.484 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.484 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.484 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.484 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:56.484 DEBUG:boto:Data: 2024-04-13T22:30:56.484 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.484 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.484 DEBUG:boto:Params: {} 2024-04-13T22:30:56.484 DEBUG:boto:Token: None 2024-04-13T22:30:56.485 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22 2024-04-13T22:30:56.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O093qD/ZQb2qf+XrdVyww2r0WNs= 2024-04-13T22:30:56.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O093qD/ZQb2qf+XrdVyww2r0WNs=', 'Content-Length': '0'} 2024-04-13T22:30:56.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b85ebfb847819f23-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:30:56.487 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.487 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.487 DEBUG:boto:Method: GET 2024-04-13T22:30:56.487 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:56.487 DEBUG:boto:Data: 2024-04-13T22:30:56.487 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.487 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.487 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.487 DEBUG:boto:Params: {} 2024-04-13T22:30:56.488 DEBUG:boto:Token: None 2024-04-13T22:30:56.488 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22 2024-04-13T22:30:56.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:igZOkXnn6apxGEOS60nKzfbg9TY= 2024-04-13T22:30:56.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:igZOkXnn6apxGEOS60nKzfbg9TY=', 'Content-Length': '0'} 2024-04-13T22:30:56.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410cc77357c2c98f-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.490 DEBUG:boto:path=/vwnfns-61/obj22 2024-04-13T22:30:56.490 DEBUG:boto:auth_path=/vwnfns-61/obj22 2024-04-13T22:30:56.490 DEBUG:boto:Method: GET 2024-04-13T22:30:56.490 DEBUG:boto:Path: /vwnfns-61/obj22 2024-04-13T22:30:56.490 DEBUG:boto:Data: 2024-04-13T22:30:56.490 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.490 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.490 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.490 DEBUG:boto:Params: {} 2024-04-13T22:30:56.491 DEBUG:boto:Token: None 2024-04-13T22:30:56.491 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj22 2024-04-13T22:30:56.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:igZOkXnn6apxGEOS60nKzfbg9TY= 2024-04-13T22:30:56.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:igZOkXnn6apxGEOS60nKzfbg9TY=', 'Content-Length': '0'} 2024-04-13T22:30:56.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000086b1b77be046153-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.534 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.535 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.535 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.535 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.535 DEBUG:boto:Method: GET 2024-04-13T22:30:56.535 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.535 DEBUG:boto:Data: 2024-04-13T22:30:56.535 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.535 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.535 DEBUG:boto:Params: {} 2024-04-13T22:30:56.535 DEBUG:boto:Token: None 2024-04-13T22:30:56.536 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e052d8c53d59eb-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.538 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.539 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.539 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.539 DEBUG:boto:path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.539 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.539 DEBUG:boto:Method: GET 2024-04-13T22:30:56.539 DEBUG:boto:Path: /vwnfns-61/?prefix=obj22 2024-04-13T22:30:56.539 DEBUG:boto:Data: 2024-04-13T22:30:56.539 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.539 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.540 DEBUG:boto:Params: {} 2024-04-13T22:30:56.540 DEBUG:boto:Token: None 2024-04-13T22:30:56.540 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116bae24a33d5791-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.544 DEBUG:boto:b'vwnfns-61obj221000falseobj222024-04-13T22:16:35.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:30:56.544 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.544 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.544 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.544 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.544 DEBUG:boto:Method: GET 2024-04-13T22:30:56.545 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.545 DEBUG:boto:Data: 2024-04-13T22:30:56.545 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.545 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.545 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.545 DEBUG:boto:Params: {} 2024-04-13T22:30:56.545 DEBUG:boto:Token: None 2024-04-13T22:30:56.545 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0= 2024-04-13T22:30:56.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0=', 'Content-Length': '0'} 2024-04-13T22:30:56.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fbe913f916ad33f-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.547 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.547 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.547 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.548 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.548 DEBUG:boto:Method: GET 2024-04-13T22:30:56.548 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.548 DEBUG:boto:Data: 2024-04-13T22:30:56.548 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.548 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.548 DEBUG:boto:Params: {} 2024-04-13T22:30:56.548 DEBUG:boto:Token: None 2024-04-13T22:30:56.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0= 2024-04-13T22:30:56.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0=', 'Content-Length': '0'} 2024-04-13T22:30:56.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088ecf447d5818b28-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.550 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.550 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.551 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.551 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.551 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.551 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.551 DEBUG:boto:Data: 2024-04-13T22:30:56.551 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.551 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.551 DEBUG:boto:Params: {} 2024-04-13T22:30:56.551 DEBUG:boto:Token: None 2024-04-13T22:30:56.552 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FVieE7sTUUzJkBvWOeJcDMHxqCE= 2024-04-13T22:30:56.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FVieE7sTUUzJkBvWOeJcDMHxqCE=', 'Content-Length': '0'} 2024-04-13T22:30:56.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002764d8dcfe207296-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.553 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.553 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.554 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.554 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.554 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.554 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.554 DEBUG:boto:Data: 2024-04-13T22:30:56.554 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.554 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.554 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.554 DEBUG:boto:Params: {} 2024-04-13T22:30:56.554 DEBUG:boto:Token: None 2024-04-13T22:30:56.555 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FVieE7sTUUzJkBvWOeJcDMHxqCE= 2024-04-13T22:30:56.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FVieE7sTUUzJkBvWOeJcDMHxqCE=', 'Content-Length': '0'} 2024-04-13T22:30:56.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c840e72999106c64-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:30:56.556 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.556 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.557 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.557 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.557 DEBUG:boto:Method: GET 2024-04-13T22:30:56.557 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.557 DEBUG:boto:Data: 2024-04-13T22:30:56.557 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.557 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.557 DEBUG:boto:Params: {} 2024-04-13T22:30:56.557 DEBUG:boto:Token: None 2024-04-13T22:30:56.558 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.558 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0= 2024-04-13T22:30:56.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0=', 'Content-Length': '0'} 2024-04-13T22:30:56.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3bd92e7d714c41-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.602 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.603 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.603 DEBUG:boto:path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.603 DEBUG:boto:auth_path=/vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.603 DEBUG:boto:Method: GET 2024-04-13T22:30:56.603 DEBUG:boto:Path: /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.603 DEBUG:boto:Data: 2024-04-13T22:30:56.603 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.603 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.603 DEBUG:boto:Params: {} 2024-04-13T22:30:56.603 DEBUG:boto:Token: None 2024-04-13T22:30:56.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23?versionId=null 2024-04-13T22:30:56.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0= 2024-04-13T22:30:56.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vNnUykCtA91D5a4sko+jdhPgqt0=', 'Content-Length': '0'} 2024-04-13T22:30:56.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f788308e3eeb32-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.606 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.606 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.606 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.606 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.606 DEBUG:boto:Method: GET 2024-04-13T22:30:56.606 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.606 DEBUG:boto:Data: 2024-04-13T22:30:56.606 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.606 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.606 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.607 DEBUG:boto:Params: {} 2024-04-13T22:30:56.607 DEBUG:boto:Token: None 2024-04-13T22:30:56.607 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dfa9fedea496532-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.609 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.610 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.610 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.610 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.610 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.610 DEBUG:boto:Method: GET 2024-04-13T22:30:56.610 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.610 DEBUG:boto:Data: 2024-04-13T22:30:56.610 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.610 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.611 DEBUG:boto:Params: {} 2024-04-13T22:30:56.611 DEBUG:boto:Token: None 2024-04-13T22:30:56.611 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000463e11cc910e1942-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.615 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.615 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.615 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.615 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.615 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:56.616 DEBUG:boto:Data: 2024-04-13T22:30:56.616 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.616 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.616 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.616 DEBUG:boto:Params: {} 2024-04-13T22:30:56.616 DEBUG:boto:Token: None 2024-04-13T22:30:56.616 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23 2024-04-13T22:30:56.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pkXMXkZ2Y7S3VGgfHCZ+RM6eLd4= 2024-04-13T22:30:56.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pkXMXkZ2Y7S3VGgfHCZ+RM6eLd4=', 'Content-Length': '0'} 2024-04-13T22:30:56.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d17b180e8d3d06-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.618 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.618 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.618 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.618 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:56.619 DEBUG:boto:Data: 2024-04-13T22:30:56.619 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.619 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.619 DEBUG:boto:Params: {} 2024-04-13T22:30:56.619 DEBUG:boto:Token: None 2024-04-13T22:30:56.619 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23 2024-04-13T22:30:56.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pkXMXkZ2Y7S3VGgfHCZ+RM6eLd4= 2024-04-13T22:30:56.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pkXMXkZ2Y7S3VGgfHCZ+RM6eLd4=', 'Content-Length': '0'} 2024-04-13T22:30:56.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7c91d9b8a5cd7f-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-13T22:30:56.621 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.621 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.621 DEBUG:boto:Method: GET 2024-04-13T22:30:56.622 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:56.622 DEBUG:boto:Data: 2024-04-13T22:30:56.622 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.622 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.622 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.622 DEBUG:boto:Params: {} 2024-04-13T22:30:56.622 DEBUG:boto:Token: None 2024-04-13T22:30:56.622 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23 2024-04-13T22:30:56.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BIbZxRhuIa2w48/nAeh8zThQZ2Q= 2024-04-13T22:30:56.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BIbZxRhuIa2w48/nAeh8zThQZ2Q=', 'Content-Length': '0'} 2024-04-13T22:30:56.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0224dec58039ac-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.624 DEBUG:boto:path=/vwnfns-61/obj23 2024-04-13T22:30:56.624 DEBUG:boto:auth_path=/vwnfns-61/obj23 2024-04-13T22:30:56.625 DEBUG:boto:Method: GET 2024-04-13T22:30:56.625 DEBUG:boto:Path: /vwnfns-61/obj23 2024-04-13T22:30:56.625 DEBUG:boto:Data: 2024-04-13T22:30:56.625 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.625 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.625 DEBUG:boto:Params: {} 2024-04-13T22:30:56.625 DEBUG:boto:Token: None 2024-04-13T22:30:56.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj23 2024-04-13T22:30:56.626 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BIbZxRhuIa2w48/nAeh8zThQZ2Q= 2024-04-13T22:30:56.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BIbZxRhuIa2w48/nAeh8zThQZ2Q=', 'Content-Length': '0'} 2024-04-13T22:30:56.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dca935e85f9be16-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.627 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.627 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.628 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.628 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.628 DEBUG:boto:Method: GET 2024-04-13T22:30:56.628 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.628 DEBUG:boto:Data: 2024-04-13T22:30:56.628 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.628 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.628 DEBUG:boto:Params: {} 2024-04-13T22:30:56.628 DEBUG:boto:Token: None 2024-04-13T22:30:56.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a646549a1045f3-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.632 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.632 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.633 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.633 DEBUG:boto:path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.633 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.633 DEBUG:boto:Method: GET 2024-04-13T22:30:56.633 DEBUG:boto:Path: /vwnfns-61/?prefix=obj23 2024-04-13T22:30:56.633 DEBUG:boto:Data: 2024-04-13T22:30:56.633 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.633 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.633 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.633 DEBUG:boto:Params: {} 2024-04-13T22:30:56.633 DEBUG:boto:Token: None 2024-04-13T22:30:56.634 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.634 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ceaad7252ab452-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.637 DEBUG:boto:b'vwnfns-61obj231000falseobj232024-04-13T22:22:50.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:30:56.638 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.638 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.638 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.638 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.638 DEBUG:boto:Method: GET 2024-04-13T22:30:56.638 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.638 DEBUG:boto:Data: 2024-04-13T22:30:56.638 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.639 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.639 DEBUG:boto:Params: {} 2024-04-13T22:30:56.639 DEBUG:boto:Token: None 2024-04-13T22:30:56.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U= 2024-04-13T22:30:56.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U=', 'Content-Length': '0'} 2024-04-13T22:30:56.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005942eeb10d7ef451-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.641 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.641 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.641 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.641 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.641 DEBUG:boto:Method: GET 2024-04-13T22:30:56.641 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.642 DEBUG:boto:Data: 2024-04-13T22:30:56.642 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.642 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.642 DEBUG:boto:Params: {} 2024-04-13T22:30:56.642 DEBUG:boto:Token: None 2024-04-13T22:30:56.642 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U= 2024-04-13T22:30:56.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U=', 'Content-Length': '0'} 2024-04-13T22:30:56.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006935337ea2bc5239-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.644 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.644 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.644 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.644 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.644 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.645 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.645 DEBUG:boto:Data: 2024-04-13T22:30:56.645 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.645 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.645 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.645 DEBUG:boto:Params: {} 2024-04-13T22:30:56.645 DEBUG:boto:Token: None 2024-04-13T22:30:56.645 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.645 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2nlIbxuFv4/KLSR73Ki6EsUkFQI= 2024-04-13T22:30:56.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2nlIbxuFv4/KLSR73Ki6EsUkFQI=', 'Content-Length': '0'} 2024-04-13T22:30:56.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4f2621cd6843d8-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.647 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.647 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.647 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.647 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.647 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.647 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.647 DEBUG:boto:Data: 2024-04-13T22:30:56.648 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.648 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.648 DEBUG:boto:Params: {} 2024-04-13T22:30:56.648 DEBUG:boto:Token: None 2024-04-13T22:30:56.648 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2nlIbxuFv4/KLSR73Ki6EsUkFQI= 2024-04-13T22:30:56.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2nlIbxuFv4/KLSR73Ki6EsUkFQI=', 'Content-Length': '0'} 2024-04-13T22:30:56.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb5e3ee5e371b475-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:30:56.650 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.650 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.650 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.650 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.651 DEBUG:boto:Method: GET 2024-04-13T22:30:56.651 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.651 DEBUG:boto:Data: 2024-04-13T22:30:56.651 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.651 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.651 DEBUG:boto:Params: {} 2024-04-13T22:30:56.651 DEBUG:boto:Token: None 2024-04-13T22:30:56.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U= 2024-04-13T22:30:56.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U=', 'Content-Length': '0'} 2024-04-13T22:30:56.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a626f747f7f3f13-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.694 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.694 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.695 DEBUG:boto:path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.695 DEBUG:boto:auth_path=/vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.695 DEBUG:boto:Method: GET 2024-04-13T22:30:56.695 DEBUG:boto:Path: /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.695 DEBUG:boto:Data: 2024-04-13T22:30:56.695 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.695 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.695 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.695 DEBUG:boto:Params: {} 2024-04-13T22:30:56.695 DEBUG:boto:Token: None 2024-04-13T22:30:56.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29?versionId=null 2024-04-13T22:30:56.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U= 2024-04-13T22:30:56.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cA5LPar3V8jTuIKKCWIfpxepy5U=', 'Content-Length': '0'} 2024-04-13T22:30:56.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1837f6efa4f568-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.698 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.698 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.698 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.698 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.698 DEBUG:boto:Method: GET 2024-04-13T22:30:56.698 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.698 DEBUG:boto:Data: 2024-04-13T22:30:56.698 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.699 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.699 DEBUG:boto:Params: {} 2024-04-13T22:30:56.699 DEBUG:boto:Token: None 2024-04-13T22:30:56.699 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.699 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3bdb41968018f5e-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.701 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.702 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.702 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.702 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.702 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.702 DEBUG:boto:Method: GET 2024-04-13T22:30:56.702 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.702 DEBUG:boto:Data: 2024-04-13T22:30:56.703 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.703 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.703 DEBUG:boto:Params: {} 2024-04-13T22:30:56.703 DEBUG:boto:Token: None 2024-04-13T22:30:56.703 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.703 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d07b33e54f5382e-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.707 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.707 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.707 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.707 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.708 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:56.708 DEBUG:boto:Data: 2024-04-13T22:30:56.708 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.708 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.708 DEBUG:boto:Params: {} 2024-04-13T22:30:56.708 DEBUG:boto:Token: None 2024-04-13T22:30:56.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29 2024-04-13T22:30:56.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3/yeK/lZ9y9R7Q4J2oxJ78H5lc4= 2024-04-13T22:30:56.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3/yeK/lZ9y9R7Q4J2oxJ78H5lc4=', 'Content-Length': '0'} 2024-04-13T22:30:56.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d9495bbd5933ece-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.710 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.710 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.710 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.710 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:56.711 DEBUG:boto:Data: 2024-04-13T22:30:56.711 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.711 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.711 DEBUG:boto:Params: {} 2024-04-13T22:30:56.711 DEBUG:boto:Token: None 2024-04-13T22:30:56.711 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29 2024-04-13T22:30:56.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3/yeK/lZ9y9R7Q4J2oxJ78H5lc4= 2024-04-13T22:30:56.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3/yeK/lZ9y9R7Q4J2oxJ78H5lc4=', 'Content-Length': '0'} 2024-04-13T22:30:56.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9972513ddcdc39-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-13T22:30:56.713 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.713 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.713 DEBUG:boto:Method: GET 2024-04-13T22:30:56.713 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:56.714 DEBUG:boto:Data: 2024-04-13T22:30:56.714 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.714 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.714 DEBUG:boto:Params: {} 2024-04-13T22:30:56.714 DEBUG:boto:Token: None 2024-04-13T22:30:56.714 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29 2024-04-13T22:30:56.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5RLKm54BW9x1M1/YRrX6U0nN3I= 2024-04-13T22:30:56.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5RLKm54BW9x1M1/YRrX6U0nN3I=', 'Content-Length': '0'} 2024-04-13T22:30:56.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae7175f87c2fd59-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.716 DEBUG:boto:path=/vwnfns-61/obj29 2024-04-13T22:30:56.716 DEBUG:boto:auth_path=/vwnfns-61/obj29 2024-04-13T22:30:56.716 DEBUG:boto:Method: GET 2024-04-13T22:30:56.716 DEBUG:boto:Path: /vwnfns-61/obj29 2024-04-13T22:30:56.717 DEBUG:boto:Data: 2024-04-13T22:30:56.717 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.717 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.717 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.717 DEBUG:boto:Params: {} 2024-04-13T22:30:56.717 DEBUG:boto:Token: None 2024-04-13T22:30:56.717 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj29 2024-04-13T22:30:56.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S5RLKm54BW9x1M1/YRrX6U0nN3I= 2024-04-13T22:30:56.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5RLKm54BW9x1M1/YRrX6U0nN3I=', 'Content-Length': '0'} 2024-04-13T22:30:56.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9011f906d84bee-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.766 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.767 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.767 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.767 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.767 DEBUG:boto:Method: GET 2024-04-13T22:30:56.767 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.767 DEBUG:boto:Data: 2024-04-13T22:30:56.767 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.767 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.767 DEBUG:boto:Params: {} 2024-04-13T22:30:56.767 DEBUG:boto:Token: None 2024-04-13T22:30:56.768 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0ab908adcb301e-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.770 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.771 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.771 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.771 DEBUG:boto:path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.771 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.771 DEBUG:boto:Method: GET 2024-04-13T22:30:56.771 DEBUG:boto:Path: /vwnfns-61/?prefix=obj29 2024-04-13T22:30:56.771 DEBUG:boto:Data: 2024-04-13T22:30:56.771 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.772 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.772 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.772 DEBUG:boto:Params: {} 2024-04-13T22:30:56.772 DEBUG:boto:Token: None 2024-04-13T22:30:56.772 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83c2d808cb0b2cf-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.776 DEBUG:boto:b'vwnfns-61obj291000falseobj292024-04-13T22:25:33.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:30:56.776 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.776 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.776 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.776 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.777 DEBUG:boto:Method: GET 2024-04-13T22:30:56.777 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.777 DEBUG:boto:Data: 2024-04-13T22:30:56.777 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.777 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.777 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.777 DEBUG:boto:Params: {} 2024-04-13T22:30:56.777 DEBUG:boto:Token: None 2024-04-13T22:30:56.777 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.778 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE= 2024-04-13T22:30:56.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE=', 'Content-Length': '0'} 2024-04-13T22:30:56.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae484413dbbca9bd-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.780 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.780 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.780 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.780 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.780 DEBUG:boto:Method: GET 2024-04-13T22:30:56.780 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.780 DEBUG:boto:Data: 2024-04-13T22:30:56.780 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.781 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.781 DEBUG:boto:Params: {} 2024-04-13T22:30:56.781 DEBUG:boto:Token: None 2024-04-13T22:30:56.781 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.781 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE= 2024-04-13T22:30:56.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE=', 'Content-Length': '0'} 2024-04-13T22:30:56.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d9cdb0c7391847-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.783 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.783 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.783 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.783 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.784 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.784 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.784 DEBUG:boto:Data: 2024-04-13T22:30:56.784 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.784 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.784 DEBUG:boto:Params: {} 2024-04-13T22:30:56.784 DEBUG:boto:Token: None 2024-04-13T22:30:56.784 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.784 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Rr0yXw5/4ZY5dsMk91IeUvM7w0= 2024-04-13T22:30:56.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Rr0yXw5/4ZY5dsMk91IeUvM7w0=', 'Content-Length': '0'} 2024-04-13T22:30:56.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e457476aaac9e0a3-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.786 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.786 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.786 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.786 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.787 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.787 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.787 DEBUG:boto:Data: 2024-04-13T22:30:56.787 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.787 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.787 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.787 DEBUG:boto:Params: {} 2024-04-13T22:30:56.787 DEBUG:boto:Token: None 2024-04-13T22:30:56.787 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.787 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Rr0yXw5/4ZY5dsMk91IeUvM7w0= 2024-04-13T22:30:56.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Rr0yXw5/4ZY5dsMk91IeUvM7w0=', 'Content-Length': '0'} 2024-04-13T22:30:56.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009d1d3fe36228d24-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:30:56.789 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.789 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.789 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.790 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.790 DEBUG:boto:Method: GET 2024-04-13T22:30:56.790 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.790 DEBUG:boto:Data: 2024-04-13T22:30:56.790 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.790 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.790 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.790 DEBUG:boto:Params: {} 2024-04-13T22:30:56.790 DEBUG:boto:Token: None 2024-04-13T22:30:56.790 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.791 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE= 2024-04-13T22:30:56.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE=', 'Content-Length': '0'} 2024-04-13T22:30:56.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da563deb3eb621d-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.834 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.834 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.835 DEBUG:boto:path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.835 DEBUG:boto:auth_path=/vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.835 DEBUG:boto:Method: GET 2024-04-13T22:30:56.835 DEBUG:boto:Path: /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.835 DEBUG:boto:Data: 2024-04-13T22:30:56.835 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.835 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.835 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.835 DEBUG:boto:Params: {} 2024-04-13T22:30:56.835 DEBUG:boto:Token: None 2024-04-13T22:30:56.836 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31?versionId=null 2024-04-13T22:30:56.836 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE= 2024-04-13T22:30:56.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jkIyKsvfkQgUr86f8qUsUNaVHuE=', 'Content-Length': '0'} 2024-04-13T22:30:56.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b3efdf87c4608a-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.838 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.838 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.838 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.838 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.838 DEBUG:boto:Method: GET 2024-04-13T22:30:56.838 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.838 DEBUG:boto:Data: 2024-04-13T22:30:56.838 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.838 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.839 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.839 DEBUG:boto:Params: {} 2024-04-13T22:30:56.839 DEBUG:boto:Token: None 2024-04-13T22:30:56.839 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.839 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025e22cfcb066e0a4-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.841 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.842 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.842 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.842 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.842 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.842 DEBUG:boto:Method: GET 2024-04-13T22:30:56.842 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.842 DEBUG:boto:Data: 2024-04-13T22:30:56.842 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.842 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.843 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.843 DEBUG:boto:Params: {} 2024-04-13T22:30:56.843 DEBUG:boto:Token: None 2024-04-13T22:30:56.843 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.843 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006212e684056e9ff-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.847 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.847 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.847 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.847 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.847 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:56.847 DEBUG:boto:Data: 2024-04-13T22:30:56.847 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.848 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.848 DEBUG:boto:Params: {} 2024-04-13T22:30:56.848 DEBUG:boto:Token: None 2024-04-13T22:30:56.848 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31 2024-04-13T22:30:56.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WAC0h0g1qf3JvFybdH/9NHd3ryM= 2024-04-13T22:30:56.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WAC0h0g1qf3JvFybdH/9NHd3ryM=', 'Content-Length': '0'} 2024-04-13T22:30:56.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ac15b83edc06a9-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.850 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.850 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.850 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.850 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:56.850 DEBUG:boto:Data: 2024-04-13T22:30:56.850 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.850 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.850 DEBUG:boto:Params: {} 2024-04-13T22:30:56.851 DEBUG:boto:Token: None 2024-04-13T22:30:56.851 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31 2024-04-13T22:30:56.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WAC0h0g1qf3JvFybdH/9NHd3ryM= 2024-04-13T22:30:56.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WAC0h0g1qf3JvFybdH/9NHd3ryM=', 'Content-Length': '0'} 2024-04-13T22:30:56.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008183cd08e206ee58-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-13T22:30:56.853 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.853 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.853 DEBUG:boto:Method: GET 2024-04-13T22:30:56.853 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:56.853 DEBUG:boto:Data: 2024-04-13T22:30:56.853 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.853 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.853 DEBUG:boto:Params: {} 2024-04-13T22:30:56.854 DEBUG:boto:Token: None 2024-04-13T22:30:56.854 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31 2024-04-13T22:30:56.854 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lUNcYwMJAOVqsX/fV4qvlZeWxPo= 2024-04-13T22:30:56.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lUNcYwMJAOVqsX/fV4qvlZeWxPo=', 'Content-Length': '0'} 2024-04-13T22:30:56.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e90f7f3ecc9b84c2-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.856 DEBUG:boto:path=/vwnfns-61/obj31 2024-04-13T22:30:56.856 DEBUG:boto:auth_path=/vwnfns-61/obj31 2024-04-13T22:30:56.856 DEBUG:boto:Method: GET 2024-04-13T22:30:56.856 DEBUG:boto:Path: /vwnfns-61/obj31 2024-04-13T22:30:56.856 DEBUG:boto:Data: 2024-04-13T22:30:56.856 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.856 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.856 DEBUG:boto:Params: {} 2024-04-13T22:30:56.856 DEBUG:boto:Token: None 2024-04-13T22:30:56.857 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj31 2024-04-13T22:30:56.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lUNcYwMJAOVqsX/fV4qvlZeWxPo= 2024-04-13T22:30:56.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lUNcYwMJAOVqsX/fV4qvlZeWxPo=', 'Content-Length': '0'} 2024-04-13T22:30:56.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b90a7c35ddda38-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.859 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.859 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.859 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.859 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.859 DEBUG:boto:Method: GET 2024-04-13T22:30:56.859 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.859 DEBUG:boto:Data: 2024-04-13T22:30:56.859 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.860 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.860 DEBUG:boto:Params: {} 2024-04-13T22:30:56.860 DEBUG:boto:Token: None 2024-04-13T22:30:56.860 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8eb361ad0b03fac-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.862 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.862 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.863 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.863 DEBUG:boto:path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.863 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.863 DEBUG:boto:Method: GET 2024-04-13T22:30:56.863 DEBUG:boto:Path: /vwnfns-61/?prefix=obj31 2024-04-13T22:30:56.863 DEBUG:boto:Data: 2024-04-13T22:30:56.863 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.863 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.863 DEBUG:boto:Params: {} 2024-04-13T22:30:56.864 DEBUG:boto:Token: None 2024-04-13T22:30:56.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000028631fbe65e696f-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.867 DEBUG:boto:b'vwnfns-61obj311000falseobj312024-04-13T22:27:52.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:30:56.868 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.868 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.868 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.868 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.868 DEBUG:boto:Method: GET 2024-04-13T22:30:56.868 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.868 DEBUG:boto:Data: 2024-04-13T22:30:56.868 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.868 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.869 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.869 DEBUG:boto:Params: {} 2024-04-13T22:30:56.869 DEBUG:boto:Token: None 2024-04-13T22:30:56.869 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY= 2024-04-13T22:30:56.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY=', 'Content-Length': '0'} 2024-04-13T22:30:56.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091465ca8ffc3a468-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.871 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.871 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.871 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.871 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.871 DEBUG:boto:Method: GET 2024-04-13T22:30:56.871 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.871 DEBUG:boto:Data: 2024-04-13T22:30:56.872 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.872 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.872 DEBUG:boto:Params: {} 2024-04-13T22:30:56.872 DEBUG:boto:Token: None 2024-04-13T22:30:56.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY= 2024-04-13T22:30:56.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY=', 'Content-Length': '0'} 2024-04-13T22:30:56.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb27478b52fa300c-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.918 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.919 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.919 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.919 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.919 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.919 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.919 DEBUG:boto:Data: 2024-04-13T22:30:56.919 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.919 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.919 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.919 DEBUG:boto:Params: {} 2024-04-13T22:30:56.920 DEBUG:boto:Token: None 2024-04-13T22:30:56.920 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kQwd9gXH/v8/5rOJKH2umwmQw4o= 2024-04-13T22:30:56.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kQwd9gXH/v8/5rOJKH2umwmQw4o=', 'Content-Length': '0'} 2024-04-13T22:30:56.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd6a2c46b565c00-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.922 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.922 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.922 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.922 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.922 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.922 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.922 DEBUG:boto:Data: 2024-04-13T22:30:56.923 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.923 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.923 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.923 DEBUG:boto:Params: {} 2024-04-13T22:30:56.923 DEBUG:boto:Token: None 2024-04-13T22:30:56.923 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kQwd9gXH/v8/5rOJKH2umwmQw4o= 2024-04-13T22:30:56.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kQwd9gXH/v8/5rOJKH2umwmQw4o=', 'Content-Length': '0'} 2024-04-13T22:30:56.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f279077205cf47cd-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:30:56.926 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.926 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.926 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.926 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.926 DEBUG:boto:Method: GET 2024-04-13T22:30:56.926 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.926 DEBUG:boto:Data: 2024-04-13T22:30:56.926 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.927 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.927 DEBUG:boto:Params: {} 2024-04-13T22:30:56.927 DEBUG:boto:Token: None 2024-04-13T22:30:56.927 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.927 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY= 2024-04-13T22:30:56.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY=', 'Content-Length': '0'} 2024-04-13T22:30:56.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007860a065be045334-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.929 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.929 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.929 DEBUG:boto:path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.929 DEBUG:boto:auth_path=/vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.929 DEBUG:boto:Method: GET 2024-04-13T22:30:56.930 DEBUG:boto:Path: /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.930 DEBUG:boto:Data: 2024-04-13T22:30:56.930 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.930 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.930 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.930 DEBUG:boto:Params: {} 2024-04-13T22:30:56.930 DEBUG:boto:Token: None 2024-04-13T22:30:56.930 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33?versionId=null 2024-04-13T22:30:56.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY= 2024-04-13T22:30:56.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QVMMaqfahYda2GzyUZHYY/fz5RY=', 'Content-Length': '0'} 2024-04-13T22:30:56.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025758586f74d57be-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.933 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.933 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.933 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.933 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.933 DEBUG:boto:Method: GET 2024-04-13T22:30:56.933 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.933 DEBUG:boto:Data: 2024-04-13T22:30:56.933 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.933 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.933 DEBUG:boto:Params: {} 2024-04-13T22:30:56.934 DEBUG:boto:Token: None 2024-04-13T22:30:56.934 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3dfbc81dc267dce-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.936 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.937 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.937 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.937 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.937 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.937 DEBUG:boto:Method: GET 2024-04-13T22:30:56.937 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.937 DEBUG:boto:Data: 2024-04-13T22:30:56.937 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.938 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.938 DEBUG:boto:Params: {} 2024-04-13T22:30:56.938 DEBUG:boto:Token: None 2024-04-13T22:30:56.938 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa5792972d803da-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.942 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.942 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.942 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.943 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.943 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:56.943 DEBUG:boto:Data: 2024-04-13T22:30:56.943 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.943 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.943 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.943 DEBUG:boto:Params: {} 2024-04-13T22:30:56.943 DEBUG:boto:Token: None 2024-04-13T22:30:56.943 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33 2024-04-13T22:30:56.943 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhBsFeKnlJy4RlRS57iMo+OPPrI= 2024-04-13T22:30:56.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhBsFeKnlJy4RlRS57iMo+OPPrI=', 'Content-Length': '0'} 2024-04-13T22:30:56.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ca75f33ae58331-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.945 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.945 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.945 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.945 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:56.946 DEBUG:boto:Data: 2024-04-13T22:30:56.946 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.946 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.946 DEBUG:boto:Params: {} 2024-04-13T22:30:56.946 DEBUG:boto:Token: None 2024-04-13T22:30:56.946 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33 2024-04-13T22:30:56.946 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uhBsFeKnlJy4RlRS57iMo+OPPrI= 2024-04-13T22:30:56.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhBsFeKnlJy4RlRS57iMo+OPPrI=', 'Content-Length': '0'} 2024-04-13T22:30:56.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096993be0f1f1d80e-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:30:56.948 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.948 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.948 DEBUG:boto:Method: GET 2024-04-13T22:30:56.949 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:56.949 DEBUG:boto:Data: 2024-04-13T22:30:56.949 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.949 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.949 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.949 DEBUG:boto:Params: {} 2024-04-13T22:30:56.949 DEBUG:boto:Token: None 2024-04-13T22:30:56.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33 2024-04-13T22:30:56.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqO/MkYfAiGyKjl2qr60v8PQGHA= 2024-04-13T22:30:56.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqO/MkYfAiGyKjl2qr60v8PQGHA=', 'Content-Length': '0'} 2024-04-13T22:30:56.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f0455ae2f46103-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.951 DEBUG:boto:path=/vwnfns-61/obj33 2024-04-13T22:30:56.951 DEBUG:boto:auth_path=/vwnfns-61/obj33 2024-04-13T22:30:56.951 DEBUG:boto:Method: GET 2024-04-13T22:30:56.951 DEBUG:boto:Path: /vwnfns-61/obj33 2024-04-13T22:30:56.952 DEBUG:boto:Data: 2024-04-13T22:30:56.952 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.952 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.952 DEBUG:boto:Params: {} 2024-04-13T22:30:56.952 DEBUG:boto:Token: None 2024-04-13T22:30:56.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj33 2024-04-13T22:30:56.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SqO/MkYfAiGyKjl2qr60v8PQGHA= 2024-04-13T22:30:56.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqO/MkYfAiGyKjl2qr60v8PQGHA=', 'Content-Length': '0'} 2024-04-13T22:30:56.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7bc232062678c3-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.954 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.954 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.954 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.954 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.955 DEBUG:boto:Method: GET 2024-04-13T22:30:56.955 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.955 DEBUG:boto:Data: 2024-04-13T22:30:56.955 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.955 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.955 DEBUG:boto:Params: {} 2024-04-13T22:30:56.955 DEBUG:boto:Token: None 2024-04-13T22:30:56.955 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.955 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000415a6531a5eade70-00661b07a0-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.958 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.958 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:56.958 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:56.959 DEBUG:boto:path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.959 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.959 DEBUG:boto:Method: GET 2024-04-13T22:30:56.959 DEBUG:boto:Path: /vwnfns-61/?prefix=obj33 2024-04-13T22:30:56.959 DEBUG:boto:Data: 2024-04-13T22:30:56.959 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.959 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.959 DEBUG:boto:Params: {} 2024-04-13T22:30:56.959 DEBUG:boto:Token: None 2024-04-13T22:30:56.959 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/ 2024-04-13T22:30:56.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0= 2024-04-13T22:30:56.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GNBeEeflL3tcjDosLwd1EPR/f/0=', 'Content-Length': '0'} 2024-04-13T22:30:56.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002177ef9c0c48bcb8-00661b07a0-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.963 DEBUG:boto:b'vwnfns-61obj331000falseobj332024-04-13T22:16:35.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:56.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:30:56.964 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:56.964 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:56.964 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.964 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.964 DEBUG:boto:Method: GET 2024-04-13T22:30:56.964 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.964 DEBUG:boto:Data: 2024-04-13T22:30:56.964 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.964 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.964 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.964 DEBUG:boto:Params: {} 2024-04-13T22:30:56.965 DEBUG:boto:Token: None 2024-04-13T22:30:56.965 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iuqmkuiefq0p5GFhL3KT2ILboQI= 2024-04-13T22:30:56.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iuqmkuiefq0p5GFhL3KT2ILboQI=', 'Content-Length': '0'} 2024-04-13T22:30:56.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683617bdcdca8869-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.967 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:56.967 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:56.967 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.967 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.967 DEBUG:boto:Method: GET 2024-04-13T22:30:56.967 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.967 DEBUG:boto:Data: 2024-04-13T22:30:56.967 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.968 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.968 DEBUG:boto:Params: {} 2024-04-13T22:30:56.968 DEBUG:boto:Token: None 2024-04-13T22:30:56.968 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iuqmkuiefq0p5GFhL3KT2ILboQI= 2024-04-13T22:30:56.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iuqmkuiefq0p5GFhL3KT2ILboQI=', 'Content-Length': '0'} 2024-04-13T22:30:56.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5c69dd3ce6c785-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.970 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:56.970 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:56.970 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.970 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.971 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.971 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.971 DEBUG:boto:Data: 2024-04-13T22:30:56.971 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.971 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.971 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.971 DEBUG:boto:Params: {} 2024-04-13T22:30:56.971 DEBUG:boto:Token: None 2024-04-13T22:30:56.971 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fPV2VV9OGUbYPAaivUE/hGwHnNk= 2024-04-13T22:30:56.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fPV2VV9OGUbYPAaivUE/hGwHnNk=', 'Content-Length': '0'} 2024-04-13T22:30:56.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ecce53b0091731-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.973 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:56.973 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:56.973 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.974 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.974 DEBUG:boto:Method: HEAD 2024-04-13T22:30:56.974 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.974 DEBUG:boto:Data: 2024-04-13T22:30:56.974 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.974 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:56.974 DEBUG:boto:Port: 8000 2024-04-13T22:30:56.974 DEBUG:boto:Params: {} 2024-04-13T22:30:56.974 DEBUG:boto:Token: None 2024-04-13T22:30:56.974 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.975 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fPV2VV9OGUbYPAaivUE/hGwHnNk= 2024-04-13T22:30:56.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fPV2VV9OGUbYPAaivUE/hGwHnNk=', 'Content-Length': '0'} 2024-04-13T22:30:56.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b92b06a48ae3eb0-00661b07a0-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:56.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:30:56.976 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:56.976 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:56.976 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.977 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.977 DEBUG:boto:Method: GET 2024-04-13T22:30:56.977 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.977 DEBUG:boto:Data: 2024-04-13T22:30:56.977 DEBUG:boto:Headers: {} 2024-04-13T22:30:56.977 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:56.977 DEBUG:boto:Port: 8001 2024-04-13T22:30:56.977 DEBUG:boto:Params: {} 2024-04-13T22:30:56.977 DEBUG:boto:Token: None 2024-04-13T22:30:56.977 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:56 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:56.978 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iuqmkuiefq0p5GFhL3KT2ILboQI= 2024-04-13T22:30:56.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iuqmkuiefq0p5GFhL3KT2ILboQI=', 'Content-Length': '0'} 2024-04-13T22:30:56.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ee3bcfe178c5568-00661b07a0-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.026 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:57.026 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:57.026 DEBUG:boto:path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:57.027 DEBUG:boto:auth_path=/vwnfns-61/obj34?versionId=null 2024-04-13T22:30:57.027 DEBUG:boto:Method: GET 2024-04-13T22:30:57.027 DEBUG:boto:Path: /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:57.027 DEBUG:boto:Data: 2024-04-13T22:30:57.027 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.027 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.027 DEBUG:boto:Params: {} 2024-04-13T22:30:57.027 DEBUG:boto:Token: None 2024-04-13T22:30:57.027 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj34?versionId=null 2024-04-13T22:30:57.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+Suf0cFluZBw2kX7qihrKEKZkCs= 2024-04-13T22:30:57.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+Suf0cFluZBw2kX7qihrKEKZkCs=', 'Content-Length': '0'} 2024-04-13T22:30:57.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233668c90559cc84-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.029 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.029 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.030 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.030 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.030 DEBUG:boto:Method: GET 2024-04-13T22:30:57.030 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.030 DEBUG:boto:Data: 2024-04-13T22:30:57.030 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.030 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.030 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.030 DEBUG:boto:Params: {} 2024-04-13T22:30:57.030 DEBUG:boto:Token: None 2024-04-13T22:30:57.031 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.031 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5f53854f4b98595-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.033 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.033 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.033 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.033 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.033 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.033 DEBUG:boto:Method: GET 2024-04-13T22:30:57.034 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.034 DEBUG:boto:Data: 2024-04-13T22:30:57.034 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.034 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.034 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.034 DEBUG:boto:Params: {} 2024-04-13T22:30:57.034 DEBUG:boto:Token: None 2024-04-13T22:30:57.034 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.034 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006be449e781d08a54-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.038 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.038 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:57.039 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:57.039 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.039 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:57.039 DEBUG:boto:Data: 2024-04-13T22:30:57.039 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.039 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.039 DEBUG:boto:Params: {} 2024-04-13T22:30:57.039 DEBUG:boto:Token: None 2024-04-13T22:30:57.039 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj34 2024-04-13T22:30:57.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cnc/Yve81GdVZ4BpaEItBmCtY3E= 2024-04-13T22:30:57.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cnc/Yve81GdVZ4BpaEItBmCtY3E=', 'Content-Length': '0'} 2024-04-13T22:30:57.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8b3635bafd6abe-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.041 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:57.041 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:57.041 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.041 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:57.042 DEBUG:boto:Data: 2024-04-13T22:30:57.042 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.042 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.042 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.042 DEBUG:boto:Params: {} 2024-04-13T22:30:57.042 DEBUG:boto:Token: None 2024-04-13T22:30:57.042 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj34 2024-04-13T22:30:57.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Cnc/Yve81GdVZ4BpaEItBmCtY3E= 2024-04-13T22:30:57.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Cnc/Yve81GdVZ4BpaEItBmCtY3E=', 'Content-Length': '0'} 2024-04-13T22:30:57.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99569451a12d0bc-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-13T22:30:57.044 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:57.044 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:57.044 DEBUG:boto:Method: GET 2024-04-13T22:30:57.044 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:57.045 DEBUG:boto:Data: 2024-04-13T22:30:57.045 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.045 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.045 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.045 DEBUG:boto:Params: {} 2024-04-13T22:30:57.045 DEBUG:boto:Token: None 2024-04-13T22:30:57.045 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj34 2024-04-13T22:30:57.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dEJYkJoneneGHMwatg9wfghlTq8= 2024-04-13T22:30:57.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dEJYkJoneneGHMwatg9wfghlTq8=', 'Content-Length': '0'} 2024-04-13T22:30:57.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcddf8fb6ed2619-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.047 DEBUG:boto:path=/vwnfns-61/obj34 2024-04-13T22:30:57.047 DEBUG:boto:auth_path=/vwnfns-61/obj34 2024-04-13T22:30:57.047 DEBUG:boto:Method: GET 2024-04-13T22:30:57.047 DEBUG:boto:Path: /vwnfns-61/obj34 2024-04-13T22:30:57.047 DEBUG:boto:Data: 2024-04-13T22:30:57.047 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.048 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.048 DEBUG:boto:Params: {} 2024-04-13T22:30:57.048 DEBUG:boto:Token: None 2024-04-13T22:30:57.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj34 2024-04-13T22:30:57.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dEJYkJoneneGHMwatg9wfghlTq8= 2024-04-13T22:30:57.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dEJYkJoneneGHMwatg9wfghlTq8=', 'Content-Length': '0'} 2024-04-13T22:30:57.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed2560aaad6ff0b-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.050 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.050 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.050 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.050 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.051 DEBUG:boto:Method: GET 2024-04-13T22:30:57.051 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.051 DEBUG:boto:Data: 2024-04-13T22:30:57.051 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.051 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.051 DEBUG:boto:Params: {} 2024-04-13T22:30:57.051 DEBUG:boto:Token: None 2024-04-13T22:30:57.051 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.051 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc74cdfe3f3267b-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.054 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.054 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.054 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.054 DEBUG:boto:path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.054 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.055 DEBUG:boto:Method: GET 2024-04-13T22:30:57.055 DEBUG:boto:Path: /vwnfns-61/?prefix=obj34 2024-04-13T22:30:57.055 DEBUG:boto:Data: 2024-04-13T22:30:57.055 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.055 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.055 DEBUG:boto:Params: {} 2024-04-13T22:30:57.055 DEBUG:boto:Token: None 2024-04-13T22:30:57.055 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e885bbd9a72ec0-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.059 DEBUG:boto:b'vwnfns-61obj341000falseobj342024-04-13T22:18:11.351Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-13T22:30:57.060 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.060 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.060 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.060 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.060 DEBUG:boto:Method: GET 2024-04-13T22:30:57.060 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.060 DEBUG:boto:Data: 2024-04-13T22:30:57.061 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.061 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.061 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.061 DEBUG:boto:Params: {} 2024-04-13T22:30:57.061 DEBUG:boto:Token: None 2024-04-13T22:30:57.061 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY= 2024-04-13T22:30:57.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY=', 'Content-Length': '0'} 2024-04-13T22:30:57.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbd02631cdcade6-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.063 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.063 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.063 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.063 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.063 DEBUG:boto:Method: GET 2024-04-13T22:30:57.064 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.064 DEBUG:boto:Data: 2024-04-13T22:30:57.064 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.064 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.064 DEBUG:boto:Params: {} 2024-04-13T22:30:57.064 DEBUG:boto:Token: None 2024-04-13T22:30:57.064 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.064 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY= 2024-04-13T22:30:57.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY=', 'Content-Length': '0'} 2024-04-13T22:30:57.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15e6ee6580f6960-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.106 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.106 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.106 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.107 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.107 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.107 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.107 DEBUG:boto:Data: 2024-04-13T22:30:57.107 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.107 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.107 DEBUG:boto:Params: {} 2024-04-13T22:30:57.107 DEBUG:boto:Token: None 2024-04-13T22:30:57.107 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.107 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hjsKvNHCIUDfpx6VXJkNTzB8888= 2024-04-13T22:30:57.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hjsKvNHCIUDfpx6VXJkNTzB8888=', 'Content-Length': '0'} 2024-04-13T22:30:57.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3807acc81fde01b-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.109 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.109 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.109 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.109 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.110 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.110 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.110 DEBUG:boto:Data: 2024-04-13T22:30:57.110 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.110 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.110 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.110 DEBUG:boto:Params: {} 2024-04-13T22:30:57.110 DEBUG:boto:Token: None 2024-04-13T22:30:57.110 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.110 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hjsKvNHCIUDfpx6VXJkNTzB8888= 2024-04-13T22:30:57.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hjsKvNHCIUDfpx6VXJkNTzB8888=', 'Content-Length': '0'} 2024-04-13T22:30:57.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003919f8f14be6482d-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-13T22:30:57.113 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.113 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.113 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.113 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.113 DEBUG:boto:Method: GET 2024-04-13T22:30:57.113 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.113 DEBUG:boto:Data: 2024-04-13T22:30:57.113 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.113 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.113 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.113 DEBUG:boto:Params: {} 2024-04-13T22:30:57.114 DEBUG:boto:Token: None 2024-04-13T22:30:57.114 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.114 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY= 2024-04-13T22:30:57.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY=', 'Content-Length': '0'} 2024-04-13T22:30:57.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d88cad4c9731b9-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.116 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.116 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.116 DEBUG:boto:path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.116 DEBUG:boto:auth_path=/vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.116 DEBUG:boto:Method: GET 2024-04-13T22:30:57.116 DEBUG:boto:Path: /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.116 DEBUG:boto:Data: 2024-04-13T22:30:57.116 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.116 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.116 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.117 DEBUG:boto:Params: {} 2024-04-13T22:30:57.117 DEBUG:boto:Token: None 2024-04-13T22:30:57.117 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37?versionId=null 2024-04-13T22:30:57.117 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY= 2024-04-13T22:30:57.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4kJkykYVllf+LNrK39zrZkg8juY=', 'Content-Length': '0'} 2024-04-13T22:30:57.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020510f13d225fa94-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.119 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.119 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.119 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.119 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.119 DEBUG:boto:Method: GET 2024-04-13T22:30:57.119 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.119 DEBUG:boto:Data: 2024-04-13T22:30:57.120 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.120 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.120 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.120 DEBUG:boto:Params: {} 2024-04-13T22:30:57.120 DEBUG:boto:Token: None 2024-04-13T22:30:57.120 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.120 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bbe1d1c0dda7eab-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.122 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.123 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.123 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.123 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.123 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.123 DEBUG:boto:Method: GET 2024-04-13T22:30:57.123 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.123 DEBUG:boto:Data: 2024-04-13T22:30:57.123 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.123 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.124 DEBUG:boto:Params: {} 2024-04-13T22:30:57.124 DEBUG:boto:Token: None 2024-04-13T22:30:57.124 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.124 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000935875ccdbdc56f0-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.128 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.128 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.128 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.128 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.128 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:57.128 DEBUG:boto:Data: 2024-04-13T22:30:57.128 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.129 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.129 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.129 DEBUG:boto:Params: {} 2024-04-13T22:30:57.129 DEBUG:boto:Token: None 2024-04-13T22:30:57.129 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37 2024-04-13T22:30:57.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qu71wYofIXrpWtJs0qb4VS1aTzw= 2024-04-13T22:30:57.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qu71wYofIXrpWtJs0qb4VS1aTzw=', 'Content-Length': '0'} 2024-04-13T22:30:57.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff0266fdf1837d9-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.131 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.131 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.131 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.131 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:57.131 DEBUG:boto:Data: 2024-04-13T22:30:57.131 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.131 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.131 DEBUG:boto:Params: {} 2024-04-13T22:30:57.132 DEBUG:boto:Token: None 2024-04-13T22:30:57.132 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37 2024-04-13T22:30:57.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qu71wYofIXrpWtJs0qb4VS1aTzw= 2024-04-13T22:30:57.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qu71wYofIXrpWtJs0qb4VS1aTzw=', 'Content-Length': '0'} 2024-04-13T22:30:57.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0e37b8fe9ffdeca-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-13T22:30:57.134 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.134 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.134 DEBUG:boto:Method: GET 2024-04-13T22:30:57.134 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:57.134 DEBUG:boto:Data: 2024-04-13T22:30:57.134 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.134 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.134 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.135 DEBUG:boto:Params: {} 2024-04-13T22:30:57.135 DEBUG:boto:Token: None 2024-04-13T22:30:57.135 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37 2024-04-13T22:30:57.135 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k0tKdPDrf+qtiihpIrQSKj4GorA= 2024-04-13T22:30:57.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k0tKdPDrf+qtiihpIrQSKj4GorA=', 'Content-Length': '0'} 2024-04-13T22:30:57.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a193c80ad417e575-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.178 DEBUG:boto:path=/vwnfns-61/obj37 2024-04-13T22:30:57.178 DEBUG:boto:auth_path=/vwnfns-61/obj37 2024-04-13T22:30:57.178 DEBUG:boto:Method: GET 2024-04-13T22:30:57.179 DEBUG:boto:Path: /vwnfns-61/obj37 2024-04-13T22:30:57.179 DEBUG:boto:Data: 2024-04-13T22:30:57.179 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.179 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.179 DEBUG:boto:Params: {} 2024-04-13T22:30:57.179 DEBUG:boto:Token: None 2024-04-13T22:30:57.179 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj37 2024-04-13T22:30:57.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:k0tKdPDrf+qtiihpIrQSKj4GorA= 2024-04-13T22:30:57.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k0tKdPDrf+qtiihpIrQSKj4GorA=', 'Content-Length': '0'} 2024-04-13T22:30:57.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055be144f0d57aa61-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.181 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.181 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.181 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.182 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.182 DEBUG:boto:Method: GET 2024-04-13T22:30:57.182 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.182 DEBUG:boto:Data: 2024-04-13T22:30:57.182 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.182 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.182 DEBUG:boto:Params: {} 2024-04-13T22:30:57.182 DEBUG:boto:Token: None 2024-04-13T22:30:57.183 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.183 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000035d111398a42f3c-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.185 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.185 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.185 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.185 DEBUG:boto:path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.185 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.186 DEBUG:boto:Method: GET 2024-04-13T22:30:57.186 DEBUG:boto:Path: /vwnfns-61/?prefix=obj37 2024-04-13T22:30:57.186 DEBUG:boto:Data: 2024-04-13T22:30:57.186 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.186 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.186 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.186 DEBUG:boto:Params: {} 2024-04-13T22:30:57.186 DEBUG:boto:Token: None 2024-04-13T22:30:57.186 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d071dc8904c740e6-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.190 DEBUG:boto:b'vwnfns-61obj371000falseobj372024-04-13T22:28:56.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:30:57.190 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.190 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.190 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.190 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.191 DEBUG:boto:Method: GET 2024-04-13T22:30:57.191 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.191 DEBUG:boto:Data: 2024-04-13T22:30:57.191 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.191 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.191 DEBUG:boto:Params: {} 2024-04-13T22:30:57.191 DEBUG:boto:Token: None 2024-04-13T22:30:57.191 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.191 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o= 2024-04-13T22:30:57.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o=', 'Content-Length': '0'} 2024-04-13T22:30:57.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e7c1d8cf3016b54-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.193 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.194 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.194 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.194 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.194 DEBUG:boto:Method: GET 2024-04-13T22:30:57.194 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.194 DEBUG:boto:Data: 2024-04-13T22:30:57.194 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.194 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.194 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.194 DEBUG:boto:Params: {} 2024-04-13T22:30:57.195 DEBUG:boto:Token: None 2024-04-13T22:30:57.195 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.195 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o= 2024-04-13T22:30:57.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o=', 'Content-Length': '0'} 2024-04-13T22:30:57.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb470ef35f34b05b-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.197 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.197 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.197 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.197 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.197 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.197 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.197 DEBUG:boto:Data: 2024-04-13T22:30:57.197 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.198 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.198 DEBUG:boto:Params: {} 2024-04-13T22:30:57.198 DEBUG:boto:Token: None 2024-04-13T22:30:57.198 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.198 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xczoDXC0f2y/KM009FvW/2kHZaQ= 2024-04-13T22:30:57.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xczoDXC0f2y/KM009FvW/2kHZaQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087dbf93a56210fa9-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.200 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.200 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.200 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.200 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.200 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.200 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.200 DEBUG:boto:Data: 2024-04-13T22:30:57.201 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.201 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.201 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.201 DEBUG:boto:Params: {} 2024-04-13T22:30:57.201 DEBUG:boto:Token: None 2024-04-13T22:30:57.201 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.201 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xczoDXC0f2y/KM009FvW/2kHZaQ= 2024-04-13T22:30:57.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xczoDXC0f2y/KM009FvW/2kHZaQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdef1c33c29099a3-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:30:57.203 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.203 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.204 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.204 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.204 DEBUG:boto:Method: GET 2024-04-13T22:30:57.204 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.204 DEBUG:boto:Data: 2024-04-13T22:30:57.204 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.204 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.204 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.204 DEBUG:boto:Params: {} 2024-04-13T22:30:57.204 DEBUG:boto:Token: None 2024-04-13T22:30:57.205 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.205 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o= 2024-04-13T22:30:57.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o=', 'Content-Length': '0'} 2024-04-13T22:30:57.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d746afd169e69b-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.206 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.206 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.207 DEBUG:boto:path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.207 DEBUG:boto:auth_path=/vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.207 DEBUG:boto:Method: GET 2024-04-13T22:30:57.207 DEBUG:boto:Path: /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.207 DEBUG:boto:Data: 2024-04-13T22:30:57.207 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.207 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.207 DEBUG:boto:Params: {} 2024-04-13T22:30:57.207 DEBUG:boto:Token: None 2024-04-13T22:30:57.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38?versionId=null 2024-04-13T22:30:57.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o= 2024-04-13T22:30:57.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Xl1Gtf/j5ROMmochwbtK675r6o=', 'Content-Length': '0'} 2024-04-13T22:30:57.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d27d82951231ea5-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.210 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.210 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.210 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.210 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.210 DEBUG:boto:Method: GET 2024-04-13T22:30:57.210 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.210 DEBUG:boto:Data: 2024-04-13T22:30:57.210 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.210 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.210 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.210 DEBUG:boto:Params: {} 2024-04-13T22:30:57.211 DEBUG:boto:Token: None 2024-04-13T22:30:57.211 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.211 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a56f5f19cfe4a9f-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.213 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.213 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.214 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.214 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.214 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.214 DEBUG:boto:Method: GET 2024-04-13T22:30:57.214 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.214 DEBUG:boto:Data: 2024-04-13T22:30:57.214 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.214 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.214 DEBUG:boto:Params: {} 2024-04-13T22:30:57.214 DEBUG:boto:Token: None 2024-04-13T22:30:57.215 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec82ac213bfae03-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.219 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.219 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.219 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.219 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.219 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:57.219 DEBUG:boto:Data: 2024-04-13T22:30:57.219 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.219 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.220 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.220 DEBUG:boto:Params: {} 2024-04-13T22:30:57.220 DEBUG:boto:Token: None 2024-04-13T22:30:57.220 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38 2024-04-13T22:30:57.220 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hREDhBphE56IRZiRrm4kKncGECk= 2024-04-13T22:30:57.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hREDhBphE56IRZiRrm4kKncGECk=', 'Content-Length': '0'} 2024-04-13T22:30:57.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce65608789e06e8-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.222 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.222 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.222 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.222 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:57.222 DEBUG:boto:Data: 2024-04-13T22:30:57.223 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.223 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.223 DEBUG:boto:Params: {} 2024-04-13T22:30:57.223 DEBUG:boto:Token: None 2024-04-13T22:30:57.223 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38 2024-04-13T22:30:57.223 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hREDhBphE56IRZiRrm4kKncGECk= 2024-04-13T22:30:57.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hREDhBphE56IRZiRrm4kKncGECk=', 'Content-Length': '0'} 2024-04-13T22:30:57.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e47ce6abf6435b4-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-13T22:30:57.225 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.225 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.225 DEBUG:boto:Method: GET 2024-04-13T22:30:57.225 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:57.225 DEBUG:boto:Data: 2024-04-13T22:30:57.225 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.225 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.225 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.226 DEBUG:boto:Params: {} 2024-04-13T22:30:57.226 DEBUG:boto:Token: None 2024-04-13T22:30:57.226 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38 2024-04-13T22:30:57.226 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1+Ldml/M7SlAyFpCcOo76KJ2kNc= 2024-04-13T22:30:57.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1+Ldml/M7SlAyFpCcOo76KJ2kNc=', 'Content-Length': '0'} 2024-04-13T22:30:57.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfa17d9107f8fb8-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.228 DEBUG:boto:path=/vwnfns-61/obj38 2024-04-13T22:30:57.228 DEBUG:boto:auth_path=/vwnfns-61/obj38 2024-04-13T22:30:57.228 DEBUG:boto:Method: GET 2024-04-13T22:30:57.228 DEBUG:boto:Path: /vwnfns-61/obj38 2024-04-13T22:30:57.228 DEBUG:boto:Data: 2024-04-13T22:30:57.228 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.228 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.228 DEBUG:boto:Params: {} 2024-04-13T22:30:57.229 DEBUG:boto:Token: None 2024-04-13T22:30:57.229 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj38 2024-04-13T22:30:57.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1+Ldml/M7SlAyFpCcOo76KJ2kNc= 2024-04-13T22:30:57.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1+Ldml/M7SlAyFpCcOo76KJ2kNc=', 'Content-Length': '0'} 2024-04-13T22:30:57.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f0bcf360489388-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.231 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.231 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.231 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.231 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.231 DEBUG:boto:Method: GET 2024-04-13T22:30:57.231 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.231 DEBUG:boto:Data: 2024-04-13T22:30:57.231 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.232 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.232 DEBUG:boto:Params: {} 2024-04-13T22:30:57.232 DEBUG:boto:Token: None 2024-04-13T22:30:57.232 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.232 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2e1edcc3200d53b-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.234 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.234 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.235 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.235 DEBUG:boto:path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.235 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.235 DEBUG:boto:Method: GET 2024-04-13T22:30:57.235 DEBUG:boto:Path: /vwnfns-61/?prefix=obj38 2024-04-13T22:30:57.235 DEBUG:boto:Data: 2024-04-13T22:30:57.235 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.235 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.235 DEBUG:boto:Params: {} 2024-04-13T22:30:57.235 DEBUG:boto:Token: None 2024-04-13T22:30:57.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d3438bb2a21bee6-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.286 DEBUG:boto:b'vwnfns-61obj381000falseobj382024-04-13T22:21:13.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:30:57.287 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.287 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.287 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.287 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.287 DEBUG:boto:Method: GET 2024-04-13T22:30:57.287 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.287 DEBUG:boto:Data: 2024-04-13T22:30:57.287 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.288 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.288 DEBUG:boto:Params: {} 2024-04-13T22:30:57.288 DEBUG:boto:Token: None 2024-04-13T22:30:57.288 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.288 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ= 2024-04-13T22:30:57.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041914d7f4ea9ad8f-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.290 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.291 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.291 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.291 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.291 DEBUG:boto:Method: GET 2024-04-13T22:30:57.291 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.291 DEBUG:boto:Data: 2024-04-13T22:30:57.291 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.291 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.291 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.291 DEBUG:boto:Params: {} 2024-04-13T22:30:57.291 DEBUG:boto:Token: None 2024-04-13T22:30:57.292 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.292 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ= 2024-04-13T22:30:57.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a154fadf1ef35422-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.294 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.294 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.294 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.294 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.294 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.294 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.294 DEBUG:boto:Data: 2024-04-13T22:30:57.294 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.294 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.295 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.295 DEBUG:boto:Params: {} 2024-04-13T22:30:57.295 DEBUG:boto:Token: None 2024-04-13T22:30:57.295 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.295 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Napog24ICb239NNQwCV+cWQ3i/4= 2024-04-13T22:30:57.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Napog24ICb239NNQwCV+cWQ3i/4=', 'Content-Length': '0'} 2024-04-13T22:30:57.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc685cbddda3ddf6-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.297 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.297 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.297 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.297 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.297 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.297 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.297 DEBUG:boto:Data: 2024-04-13T22:30:57.297 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.297 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.297 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.297 DEBUG:boto:Params: {} 2024-04-13T22:30:57.298 DEBUG:boto:Token: None 2024-04-13T22:30:57.298 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.298 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Napog24ICb239NNQwCV+cWQ3i/4= 2024-04-13T22:30:57.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Napog24ICb239NNQwCV+cWQ3i/4=', 'Content-Length': '0'} 2024-04-13T22:30:57.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000767a7995eba0a921-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:30:57.300 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.300 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.300 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.300 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.300 DEBUG:boto:Method: GET 2024-04-13T22:30:57.300 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.301 DEBUG:boto:Data: 2024-04-13T22:30:57.301 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.301 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.301 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.301 DEBUG:boto:Params: {} 2024-04-13T22:30:57.301 DEBUG:boto:Token: None 2024-04-13T22:30:57.301 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.301 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ= 2024-04-13T22:30:57.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f072ce8bc0e5213e-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.346 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.346 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.347 DEBUG:boto:path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.347 DEBUG:boto:auth_path=/vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.347 DEBUG:boto:Method: GET 2024-04-13T22:30:57.347 DEBUG:boto:Path: /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.347 DEBUG:boto:Data: 2024-04-13T22:30:57.347 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.347 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.347 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.347 DEBUG:boto:Params: {} 2024-04-13T22:30:57.347 DEBUG:boto:Token: None 2024-04-13T22:30:57.347 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44?versionId=null 2024-04-13T22:30:57.348 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ= 2024-04-13T22:30:57.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tcA3g2QL2QOsHSom7qYT78JgUFQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3f0af16cd42386-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.349 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.349 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.350 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.350 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.350 DEBUG:boto:Method: GET 2024-04-13T22:30:57.350 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.350 DEBUG:boto:Data: 2024-04-13T22:30:57.350 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.350 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.350 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.350 DEBUG:boto:Params: {} 2024-04-13T22:30:57.350 DEBUG:boto:Token: None 2024-04-13T22:30:57.351 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.351 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000349ea343e367b6ed-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.353 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.353 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.353 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.353 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.353 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.353 DEBUG:boto:Method: GET 2024-04-13T22:30:57.354 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.354 DEBUG:boto:Data: 2024-04-13T22:30:57.354 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.354 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.354 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.354 DEBUG:boto:Params: {} 2024-04-13T22:30:57.354 DEBUG:boto:Token: None 2024-04-13T22:30:57.354 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.354 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f234c741053ba65d-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.358 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.359 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.359 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.359 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.359 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:57.359 DEBUG:boto:Data: 2024-04-13T22:30:57.359 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.359 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.359 DEBUG:boto:Params: {} 2024-04-13T22:30:57.359 DEBUG:boto:Token: None 2024-04-13T22:30:57.360 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44 2024-04-13T22:30:57.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RFHFGblxzPkjxlE34Z/QzNaHhKA= 2024-04-13T22:30:57.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RFHFGblxzPkjxlE34Z/QzNaHhKA=', 'Content-Length': '0'} 2024-04-13T22:30:57.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f4f480a5eec5a4-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.361 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.362 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.362 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.362 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:57.362 DEBUG:boto:Data: 2024-04-13T22:30:57.362 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.362 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.362 DEBUG:boto:Params: {} 2024-04-13T22:30:57.362 DEBUG:boto:Token: None 2024-04-13T22:30:57.363 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44 2024-04-13T22:30:57.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RFHFGblxzPkjxlE34Z/QzNaHhKA= 2024-04-13T22:30:57.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RFHFGblxzPkjxlE34Z/QzNaHhKA=', 'Content-Length': '0'} 2024-04-13T22:30:57.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006159ee12a8d3f4f2-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:30:57.364 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.365 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.365 DEBUG:boto:Method: GET 2024-04-13T22:30:57.365 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:57.365 DEBUG:boto:Data: 2024-04-13T22:30:57.365 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.365 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.365 DEBUG:boto:Params: {} 2024-04-13T22:30:57.365 DEBUG:boto:Token: None 2024-04-13T22:30:57.365 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44 2024-04-13T22:30:57.366 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DF1I1yFvy0n8m260hyrwheybH/0= 2024-04-13T22:30:57.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DF1I1yFvy0n8m260hyrwheybH/0=', 'Content-Length': '0'} 2024-04-13T22:30:57.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c4798160fbde0d-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.367 DEBUG:boto:path=/vwnfns-61/obj44 2024-04-13T22:30:57.367 DEBUG:boto:auth_path=/vwnfns-61/obj44 2024-04-13T22:30:57.368 DEBUG:boto:Method: GET 2024-04-13T22:30:57.368 DEBUG:boto:Path: /vwnfns-61/obj44 2024-04-13T22:30:57.368 DEBUG:boto:Data: 2024-04-13T22:30:57.368 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.368 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.368 DEBUG:boto:Params: {} 2024-04-13T22:30:57.368 DEBUG:boto:Token: None 2024-04-13T22:30:57.368 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj44 2024-04-13T22:30:57.368 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DF1I1yFvy0n8m260hyrwheybH/0= 2024-04-13T22:30:57.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DF1I1yFvy0n8m260hyrwheybH/0=', 'Content-Length': '0'} 2024-04-13T22:30:57.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dffa2921c1666ae-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.370 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.370 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.371 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.371 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.371 DEBUG:boto:Method: GET 2024-04-13T22:30:57.371 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.371 DEBUG:boto:Data: 2024-04-13T22:30:57.371 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.371 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.371 DEBUG:boto:Params: {} 2024-04-13T22:30:57.371 DEBUG:boto:Token: None 2024-04-13T22:30:57.371 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f32e2ac2eb54405-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.374 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.374 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.374 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.374 DEBUG:boto:path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.374 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.374 DEBUG:boto:Method: GET 2024-04-13T22:30:57.375 DEBUG:boto:Path: /vwnfns-61/?prefix=obj44 2024-04-13T22:30:57.375 DEBUG:boto:Data: 2024-04-13T22:30:57.375 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.375 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.375 DEBUG:boto:Params: {} 2024-04-13T22:30:57.375 DEBUG:boto:Token: None 2024-04-13T22:30:57.375 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f42e7c7e5f2884ab-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.379 DEBUG:boto:b'vwnfns-61obj441000falseobj442024-04-13T22:16:35.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:30:57.379 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.379 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.379 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.379 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.379 DEBUG:boto:Method: GET 2024-04-13T22:30:57.380 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.380 DEBUG:boto:Data: 2024-04-13T22:30:57.380 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.380 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.380 DEBUG:boto:Params: {} 2024-04-13T22:30:57.380 DEBUG:boto:Token: None 2024-04-13T22:30:57.380 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.380 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo= 2024-04-13T22:30:57.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo=', 'Content-Length': '0'} 2024-04-13T22:30:57.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be510c34fcaf76b9-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.382 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.382 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.383 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.383 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.383 DEBUG:boto:Method: GET 2024-04-13T22:30:57.383 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.383 DEBUG:boto:Data: 2024-04-13T22:30:57.383 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.383 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.383 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.383 DEBUG:boto:Params: {} 2024-04-13T22:30:57.383 DEBUG:boto:Token: None 2024-04-13T22:30:57.384 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo= 2024-04-13T22:30:57.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo=', 'Content-Length': '0'} 2024-04-13T22:30:57.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2eae198e95945f0-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.426 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.426 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.427 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.427 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.427 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.427 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.427 DEBUG:boto:Data: 2024-04-13T22:30:57.427 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.427 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.427 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.427 DEBUG:boto:Params: {} 2024-04-13T22:30:57.427 DEBUG:boto:Token: None 2024-04-13T22:30:57.428 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.428 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zOb1SSTZ12R96Pxic/JwhL93L3Q= 2024-04-13T22:30:57.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zOb1SSTZ12R96Pxic/JwhL93L3Q=', 'Content-Length': '0'} 2024-04-13T22:30:57.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000671d0491a7bd85c-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.430 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.430 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.430 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.430 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.430 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.430 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.430 DEBUG:boto:Data: 2024-04-13T22:30:57.430 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.431 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.431 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.431 DEBUG:boto:Params: {} 2024-04-13T22:30:57.431 DEBUG:boto:Token: None 2024-04-13T22:30:57.431 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.431 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zOb1SSTZ12R96Pxic/JwhL93L3Q= 2024-04-13T22:30:57.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zOb1SSTZ12R96Pxic/JwhL93L3Q=', 'Content-Length': '0'} 2024-04-13T22:30:57.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea3bb15241282fb-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:30:57.433 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.433 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.433 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.433 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.433 DEBUG:boto:Method: GET 2024-04-13T22:30:57.433 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.434 DEBUG:boto:Data: 2024-04-13T22:30:57.434 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.434 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.434 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.434 DEBUG:boto:Params: {} 2024-04-13T22:30:57.434 DEBUG:boto:Token: None 2024-04-13T22:30:57.434 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.434 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo= 2024-04-13T22:30:57.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo=', 'Content-Length': '0'} 2024-04-13T22:30:57.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b2d95c2cefad14-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.436 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.436 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.436 DEBUG:boto:path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.436 DEBUG:boto:auth_path=/vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.437 DEBUG:boto:Method: GET 2024-04-13T22:30:57.437 DEBUG:boto:Path: /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.437 DEBUG:boto:Data: 2024-04-13T22:30:57.437 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.437 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.437 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.437 DEBUG:boto:Params: {} 2024-04-13T22:30:57.437 DEBUG:boto:Token: None 2024-04-13T22:30:57.437 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46?versionId=null 2024-04-13T22:30:57.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo= 2024-04-13T22:30:57.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wfvT3NkmgWOM/XaUazdpVTz9flo=', 'Content-Length': '0'} 2024-04-13T22:30:57.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005215952f4595c906-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.439 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.440 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.440 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.440 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.440 DEBUG:boto:Method: GET 2024-04-13T22:30:57.440 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.440 DEBUG:boto:Data: 2024-04-13T22:30:57.440 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.440 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.440 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.440 DEBUG:boto:Params: {} 2024-04-13T22:30:57.440 DEBUG:boto:Token: None 2024-04-13T22:30:57.441 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.441 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa522ba468199df3-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.443 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.443 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.444 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.444 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.444 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.444 DEBUG:boto:Method: GET 2024-04-13T22:30:57.444 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.444 DEBUG:boto:Data: 2024-04-13T22:30:57.444 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.444 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.444 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.444 DEBUG:boto:Params: {} 2024-04-13T22:30:57.444 DEBUG:boto:Token: None 2024-04-13T22:30:57.445 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.445 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b9fcff743a384a-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.448 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.449 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.449 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.449 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.449 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:57.449 DEBUG:boto:Data: 2024-04-13T22:30:57.449 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.449 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.449 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.450 DEBUG:boto:Params: {} 2024-04-13T22:30:57.450 DEBUG:boto:Token: None 2024-04-13T22:30:57.450 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46 2024-04-13T22:30:57.450 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZDOlH/ADtKWElDMk547PAR/HVWA= 2024-04-13T22:30:57.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZDOlH/ADtKWElDMk547PAR/HVWA=', 'Content-Length': '0'} 2024-04-13T22:30:57.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d9e1de779395c3-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.452 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.452 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.452 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.452 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:57.452 DEBUG:boto:Data: 2024-04-13T22:30:57.452 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.453 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.453 DEBUG:boto:Params: {} 2024-04-13T22:30:57.453 DEBUG:boto:Token: None 2024-04-13T22:30:57.453 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46 2024-04-13T22:30:57.453 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZDOlH/ADtKWElDMk547PAR/HVWA= 2024-04-13T22:30:57.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZDOlH/ADtKWElDMk547PAR/HVWA=', 'Content-Length': '0'} 2024-04-13T22:30:57.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004feb05ed5df997b9-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-13T22:30:57.455 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.455 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.455 DEBUG:boto:Method: GET 2024-04-13T22:30:57.455 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:57.455 DEBUG:boto:Data: 2024-04-13T22:30:57.455 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.455 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.456 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.456 DEBUG:boto:Params: {} 2024-04-13T22:30:57.456 DEBUG:boto:Token: None 2024-04-13T22:30:57.456 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46 2024-04-13T22:30:57.456 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LGRFSfigAtbxNQPCjdPtzjNzY78= 2024-04-13T22:30:57.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LGRFSfigAtbxNQPCjdPtzjNzY78=', 'Content-Length': '0'} 2024-04-13T22:30:57.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c475615d5ac415e2-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.458 DEBUG:boto:path=/vwnfns-61/obj46 2024-04-13T22:30:57.458 DEBUG:boto:auth_path=/vwnfns-61/obj46 2024-04-13T22:30:57.458 DEBUG:boto:Method: GET 2024-04-13T22:30:57.458 DEBUG:boto:Path: /vwnfns-61/obj46 2024-04-13T22:30:57.458 DEBUG:boto:Data: 2024-04-13T22:30:57.458 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.459 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.459 DEBUG:boto:Params: {} 2024-04-13T22:30:57.459 DEBUG:boto:Token: None 2024-04-13T22:30:57.459 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj46 2024-04-13T22:30:57.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LGRFSfigAtbxNQPCjdPtzjNzY78= 2024-04-13T22:30:57.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LGRFSfigAtbxNQPCjdPtzjNzY78=', 'Content-Length': '0'} 2024-04-13T22:30:57.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc324cfd23f3ce9a-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.506 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.507 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.507 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.507 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.507 DEBUG:boto:Method: GET 2024-04-13T22:30:57.507 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.507 DEBUG:boto:Data: 2024-04-13T22:30:57.507 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.507 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.507 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.507 DEBUG:boto:Params: {} 2024-04-13T22:30:57.508 DEBUG:boto:Token: None 2024-04-13T22:30:57.508 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e51833e79b94c50-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.510 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.511 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.511 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.511 DEBUG:boto:path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.511 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.511 DEBUG:boto:Method: GET 2024-04-13T22:30:57.511 DEBUG:boto:Path: /vwnfns-61/?prefix=obj46 2024-04-13T22:30:57.511 DEBUG:boto:Data: 2024-04-13T22:30:57.511 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.512 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.512 DEBUG:boto:Params: {} 2024-04-13T22:30:57.512 DEBUG:boto:Token: None 2024-04-13T22:30:57.512 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b74cb3533bf259d8-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.516 DEBUG:boto:b'vwnfns-61obj461000falseobj462024-04-13T22:22:50.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:30:57.516 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.516 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.516 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.517 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.517 DEBUG:boto:Method: GET 2024-04-13T22:30:57.517 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.517 DEBUG:boto:Data: 2024-04-13T22:30:57.517 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.517 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.517 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.517 DEBUG:boto:Params: {} 2024-04-13T22:30:57.517 DEBUG:boto:Token: None 2024-04-13T22:30:57.517 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk= 2024-04-13T22:30:57.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk=', 'Content-Length': '0'} 2024-04-13T22:30:57.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ce568bf7f10fb9-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.520 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.520 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.520 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.520 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.520 DEBUG:boto:Method: GET 2024-04-13T22:30:57.520 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.520 DEBUG:boto:Data: 2024-04-13T22:30:57.520 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.520 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.520 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.521 DEBUG:boto:Params: {} 2024-04-13T22:30:57.521 DEBUG:boto:Token: None 2024-04-13T22:30:57.521 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk= 2024-04-13T22:30:57.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk=', 'Content-Length': '0'} 2024-04-13T22:30:57.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c6fdb7615a382fc-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.523 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.523 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.523 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.523 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.523 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.523 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.524 DEBUG:boto:Data: 2024-04-13T22:30:57.524 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.524 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.524 DEBUG:boto:Params: {} 2024-04-13T22:30:57.524 DEBUG:boto:Token: None 2024-04-13T22:30:57.524 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KRtUCs3ue4kw3DZpSNydYUB3ZrA= 2024-04-13T22:30:57.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KRtUCs3ue4kw3DZpSNydYUB3ZrA=', 'Content-Length': '0'} 2024-04-13T22:30:57.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096c584aa27fdfefb-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.526 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.526 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.526 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.526 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.526 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.527 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.527 DEBUG:boto:Data: 2024-04-13T22:30:57.527 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.527 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.527 DEBUG:boto:Params: {} 2024-04-13T22:30:57.527 DEBUG:boto:Token: None 2024-04-13T22:30:57.527 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KRtUCs3ue4kw3DZpSNydYUB3ZrA= 2024-04-13T22:30:57.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KRtUCs3ue4kw3DZpSNydYUB3ZrA=', 'Content-Length': '0'} 2024-04-13T22:30:57.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c58e5778f43f86-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:30:57.529 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.529 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.530 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.530 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.530 DEBUG:boto:Method: GET 2024-04-13T22:30:57.530 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.530 DEBUG:boto:Data: 2024-04-13T22:30:57.530 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.530 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.530 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.530 DEBUG:boto:Params: {} 2024-04-13T22:30:57.530 DEBUG:boto:Token: None 2024-04-13T22:30:57.531 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.531 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk= 2024-04-13T22:30:57.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk=', 'Content-Length': '0'} 2024-04-13T22:30:57.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84923f2128cfa5a-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.532 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.532 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.533 DEBUG:boto:path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.533 DEBUG:boto:auth_path=/vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.533 DEBUG:boto:Method: GET 2024-04-13T22:30:57.533 DEBUG:boto:Path: /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.533 DEBUG:boto:Data: 2024-04-13T22:30:57.533 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.533 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.533 DEBUG:boto:Params: {} 2024-04-13T22:30:57.533 DEBUG:boto:Token: None 2024-04-13T22:30:57.534 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51?versionId=null 2024-04-13T22:30:57.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk= 2024-04-13T22:30:57.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U4nj6DuO95rXUhLAPauk7nCkqXk=', 'Content-Length': '0'} 2024-04-13T22:30:57.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992085f967f7cf3b-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.578 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.579 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.579 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.579 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.579 DEBUG:boto:Method: GET 2024-04-13T22:30:57.579 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.579 DEBUG:boto:Data: 2024-04-13T22:30:57.579 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.579 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.579 DEBUG:boto:Params: {} 2024-04-13T22:30:57.580 DEBUG:boto:Token: None 2024-04-13T22:30:57.580 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b82ec3af2baa2958-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.582 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.583 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.583 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.583 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.583 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.583 DEBUG:boto:Method: GET 2024-04-13T22:30:57.583 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.583 DEBUG:boto:Data: 2024-04-13T22:30:57.584 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.584 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.584 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.584 DEBUG:boto:Params: {} 2024-04-13T22:30:57.584 DEBUG:boto:Token: None 2024-04-13T22:30:57.584 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62ba0e87bf74b59-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.588 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.588 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.588 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.588 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.588 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:57.588 DEBUG:boto:Data: 2024-04-13T22:30:57.589 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.589 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.589 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.589 DEBUG:boto:Params: {} 2024-04-13T22:30:57.589 DEBUG:boto:Token: None 2024-04-13T22:30:57.589 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51 2024-04-13T22:30:57.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lOlZ/8Nq4tBs1lWt0DfCyL2JGSE= 2024-04-13T22:30:57.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lOlZ/8Nq4tBs1lWt0DfCyL2JGSE=', 'Content-Length': '0'} 2024-04-13T22:30:57.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009524216f99901d40-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.591 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.591 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.591 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.591 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:57.591 DEBUG:boto:Data: 2024-04-13T22:30:57.591 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.591 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.592 DEBUG:boto:Params: {} 2024-04-13T22:30:57.592 DEBUG:boto:Token: None 2024-04-13T22:30:57.592 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51 2024-04-13T22:30:57.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lOlZ/8Nq4tBs1lWt0DfCyL2JGSE= 2024-04-13T22:30:57.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lOlZ/8Nq4tBs1lWt0DfCyL2JGSE=', 'Content-Length': '0'} 2024-04-13T22:30:57.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b9066223c0a330-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-13T22:30:57.594 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.594 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.594 DEBUG:boto:Method: GET 2024-04-13T22:30:57.594 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:57.594 DEBUG:boto:Data: 2024-04-13T22:30:57.594 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.594 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.594 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.594 DEBUG:boto:Params: {} 2024-04-13T22:30:57.595 DEBUG:boto:Token: None 2024-04-13T22:30:57.595 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51 2024-04-13T22:30:57.595 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:trMkrdHme8jj654GbiGMY16sYJo= 2024-04-13T22:30:57.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:trMkrdHme8jj654GbiGMY16sYJo=', 'Content-Length': '0'} 2024-04-13T22:30:57.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f14247eca81f86d-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.638 DEBUG:boto:path=/vwnfns-61/obj51 2024-04-13T22:30:57.638 DEBUG:boto:auth_path=/vwnfns-61/obj51 2024-04-13T22:30:57.639 DEBUG:boto:Method: GET 2024-04-13T22:30:57.639 DEBUG:boto:Path: /vwnfns-61/obj51 2024-04-13T22:30:57.639 DEBUG:boto:Data: 2024-04-13T22:30:57.639 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.639 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.639 DEBUG:boto:Params: {} 2024-04-13T22:30:57.639 DEBUG:boto:Token: None 2024-04-13T22:30:57.639 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj51 2024-04-13T22:30:57.639 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:trMkrdHme8jj654GbiGMY16sYJo= 2024-04-13T22:30:57.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:trMkrdHme8jj654GbiGMY16sYJo=', 'Content-Length': '0'} 2024-04-13T22:30:57.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a03a3f161f35b3c-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.641 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.642 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.642 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.642 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.642 DEBUG:boto:Method: GET 2024-04-13T22:30:57.642 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.642 DEBUG:boto:Data: 2024-04-13T22:30:57.642 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.642 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.642 DEBUG:boto:Params: {} 2024-04-13T22:30:57.643 DEBUG:boto:Token: None 2024-04-13T22:30:57.643 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db7faa8bc734ffc-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.645 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.645 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.645 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.646 DEBUG:boto:path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.646 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.646 DEBUG:boto:Method: GET 2024-04-13T22:30:57.646 DEBUG:boto:Path: /vwnfns-61/?prefix=obj51 2024-04-13T22:30:57.646 DEBUG:boto:Data: 2024-04-13T22:30:57.646 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.646 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.646 DEBUG:boto:Params: {} 2024-04-13T22:30:57.646 DEBUG:boto:Token: None 2024-04-13T22:30:57.647 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006710d8eee9e2b40c-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.650 DEBUG:boto:b'vwnfns-61obj511000falseobj512024-04-13T22:18:11.366Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:30:57.651 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.651 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.651 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.651 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.651 DEBUG:boto:Method: GET 2024-04-13T22:30:57.651 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.651 DEBUG:boto:Data: 2024-04-13T22:30:57.652 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.652 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.652 DEBUG:boto:Params: {} 2024-04-13T22:30:57.652 DEBUG:boto:Token: None 2024-04-13T22:30:57.652 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ= 2024-04-13T22:30:57.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e115cf25396bf27-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.654 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.655 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.655 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.655 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.655 DEBUG:boto:Method: GET 2024-04-13T22:30:57.655 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.655 DEBUG:boto:Data: 2024-04-13T22:30:57.655 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.655 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.655 DEBUG:boto:Params: {} 2024-04-13T22:30:57.656 DEBUG:boto:Token: None 2024-04-13T22:30:57.656 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.656 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ= 2024-04-13T22:30:57.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002352e6ca2210bbeb-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.698 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.698 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.699 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.699 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.699 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.699 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.699 DEBUG:boto:Data: 2024-04-13T22:30:57.699 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.699 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.699 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.699 DEBUG:boto:Params: {} 2024-04-13T22:30:57.699 DEBUG:boto:Token: None 2024-04-13T22:30:57.699 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.700 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j2JItvBZVZeHwXUs6Rqh0uYV0AU= 2024-04-13T22:30:57.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j2JItvBZVZeHwXUs6Rqh0uYV0AU=', 'Content-Length': '0'} 2024-04-13T22:30:57.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784e08b2b503e063-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.701 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.701 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.702 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.702 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.702 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.702 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.702 DEBUG:boto:Data: 2024-04-13T22:30:57.702 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.702 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.702 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.702 DEBUG:boto:Params: {} 2024-04-13T22:30:57.703 DEBUG:boto:Token: None 2024-04-13T22:30:57.703 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.703 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j2JItvBZVZeHwXUs6Rqh0uYV0AU= 2024-04-13T22:30:57.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j2JItvBZVZeHwXUs6Rqh0uYV0AU=', 'Content-Length': '0'} 2024-04-13T22:30:57.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017012b66bf8c9a05-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:30:57.705 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.705 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.705 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.705 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.705 DEBUG:boto:Method: GET 2024-04-13T22:30:57.705 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.705 DEBUG:boto:Data: 2024-04-13T22:30:57.706 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.706 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.706 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.706 DEBUG:boto:Params: {} 2024-04-13T22:30:57.706 DEBUG:boto:Token: None 2024-04-13T22:30:57.706 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.706 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ= 2024-04-13T22:30:57.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349442da11119a3e-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.708 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.708 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.708 DEBUG:boto:path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.709 DEBUG:boto:auth_path=/vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.709 DEBUG:boto:Method: GET 2024-04-13T22:30:57.709 DEBUG:boto:Path: /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.709 DEBUG:boto:Data: 2024-04-13T22:30:57.709 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.709 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.709 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.709 DEBUG:boto:Params: {} 2024-04-13T22:30:57.709 DEBUG:boto:Token: None 2024-04-13T22:30:57.709 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55?versionId=null 2024-04-13T22:30:57.710 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ= 2024-04-13T22:30:57.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DmzvmndXGVT4jiWZeOpZxZVAuaQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7f4b20603db7c6-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.711 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.712 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.712 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.712 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.712 DEBUG:boto:Method: GET 2024-04-13T22:30:57.712 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.712 DEBUG:boto:Data: 2024-04-13T22:30:57.712 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.712 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.712 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.712 DEBUG:boto:Params: {} 2024-04-13T22:30:57.713 DEBUG:boto:Token: None 2024-04-13T22:30:57.713 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.713 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d5d4ef79c97a39-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.715 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.715 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.716 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.716 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.716 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.716 DEBUG:boto:Method: GET 2024-04-13T22:30:57.716 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.716 DEBUG:boto:Data: 2024-04-13T22:30:57.716 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.716 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.716 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.716 DEBUG:boto:Params: {} 2024-04-13T22:30:57.717 DEBUG:boto:Token: None 2024-04-13T22:30:57.717 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.717 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aefe770acc92df3-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.721 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.721 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.721 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.721 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.721 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:57.721 DEBUG:boto:Data: 2024-04-13T22:30:57.721 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.721 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.722 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.722 DEBUG:boto:Params: {} 2024-04-13T22:30:57.722 DEBUG:boto:Token: None 2024-04-13T22:30:57.722 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55 2024-04-13T22:30:57.722 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1HPztH8cp2lZ9naw4hyzKDbhhos= 2024-04-13T22:30:57.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1HPztH8cp2lZ9naw4hyzKDbhhos=', 'Content-Length': '0'} 2024-04-13T22:30:57.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fc2013694b3f189-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.724 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.724 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.724 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.724 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:57.724 DEBUG:boto:Data: 2024-04-13T22:30:57.724 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.724 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.724 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.725 DEBUG:boto:Params: {} 2024-04-13T22:30:57.725 DEBUG:boto:Token: None 2024-04-13T22:30:57.725 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55 2024-04-13T22:30:57.725 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1HPztH8cp2lZ9naw4hyzKDbhhos= 2024-04-13T22:30:57.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1HPztH8cp2lZ9naw4hyzKDbhhos=', 'Content-Length': '0'} 2024-04-13T22:30:57.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007abb58f8d8b0e91c-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:30:57.727 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.727 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.727 DEBUG:boto:Method: GET 2024-04-13T22:30:57.727 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:57.727 DEBUG:boto:Data: 2024-04-13T22:30:57.727 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.727 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.727 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.727 DEBUG:boto:Params: {} 2024-04-13T22:30:57.728 DEBUG:boto:Token: None 2024-04-13T22:30:57.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55 2024-04-13T22:30:57.728 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nMEZKL6S92vrXQvl/tQaTdysAXg= 2024-04-13T22:30:57.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nMEZKL6S92vrXQvl/tQaTdysAXg=', 'Content-Length': '0'} 2024-04-13T22:30:57.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000848fd6ea6c903127-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.730 DEBUG:boto:path=/vwnfns-61/obj55 2024-04-13T22:30:57.730 DEBUG:boto:auth_path=/vwnfns-61/obj55 2024-04-13T22:30:57.730 DEBUG:boto:Method: GET 2024-04-13T22:30:57.730 DEBUG:boto:Path: /vwnfns-61/obj55 2024-04-13T22:30:57.730 DEBUG:boto:Data: 2024-04-13T22:30:57.730 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.730 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.730 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.730 DEBUG:boto:Params: {} 2024-04-13T22:30:57.731 DEBUG:boto:Token: None 2024-04-13T22:30:57.731 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj55 2024-04-13T22:30:57.731 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nMEZKL6S92vrXQvl/tQaTdysAXg= 2024-04-13T22:30:57.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nMEZKL6S92vrXQvl/tQaTdysAXg=', 'Content-Length': '0'} 2024-04-13T22:30:57.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:16:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0a2ebc3edc13b9f-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.774 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.775 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.775 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.775 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.775 DEBUG:boto:Method: GET 2024-04-13T22:30:57.775 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.775 DEBUG:boto:Data: 2024-04-13T22:30:57.775 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.775 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.775 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.775 DEBUG:boto:Params: {} 2024-04-13T22:30:57.776 DEBUG:boto:Token: None 2024-04-13T22:30:57.776 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308c02d9adeed476-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.778 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.779 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.779 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.779 DEBUG:boto:path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.779 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.779 DEBUG:boto:Method: GET 2024-04-13T22:30:57.779 DEBUG:boto:Path: /vwnfns-61/?prefix=obj55 2024-04-13T22:30:57.779 DEBUG:boto:Data: 2024-04-13T22:30:57.779 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.779 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.780 DEBUG:boto:Params: {} 2024-04-13T22:30:57.780 DEBUG:boto:Token: None 2024-04-13T22:30:57.780 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.780 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d8f0d1f5f119ba2-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.784 DEBUG:boto:b'vwnfns-61obj551000falseobj552024-04-13T22:16:35.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:30:57.784 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.785 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.785 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.785 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.785 DEBUG:boto:Method: GET 2024-04-13T22:30:57.785 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.785 DEBUG:boto:Data: 2024-04-13T22:30:57.785 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.785 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.785 DEBUG:boto:Params: {} 2024-04-13T22:30:57.785 DEBUG:boto:Token: None 2024-04-13T22:30:57.786 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ= 2024-04-13T22:30:57.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2aea391a43214bd-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.788 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.788 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.788 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.788 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.788 DEBUG:boto:Method: GET 2024-04-13T22:30:57.788 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.788 DEBUG:boto:Data: 2024-04-13T22:30:57.788 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.789 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.789 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.789 DEBUG:boto:Params: {} 2024-04-13T22:30:57.789 DEBUG:boto:Token: None 2024-04-13T22:30:57.789 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ= 2024-04-13T22:30:57.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8403415fd3d662-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.791 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.791 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.791 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.791 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.791 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.792 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.792 DEBUG:boto:Data: 2024-04-13T22:30:57.792 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.792 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.792 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.792 DEBUG:boto:Params: {} 2024-04-13T22:30:57.792 DEBUG:boto:Token: None 2024-04-13T22:30:57.792 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.793 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:otEMTOsl1siqPElR2dxr1nB/bW8= 2024-04-13T22:30:57.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:otEMTOsl1siqPElR2dxr1nB/bW8=', 'Content-Length': '0'} 2024-04-13T22:30:57.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021574f2450bcf1e6-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.794 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.795 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.795 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.795 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.795 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.795 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.795 DEBUG:boto:Data: 2024-04-13T22:30:57.795 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.795 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.795 DEBUG:boto:Params: {} 2024-04-13T22:30:57.795 DEBUG:boto:Token: None 2024-04-13T22:30:57.796 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:otEMTOsl1siqPElR2dxr1nB/bW8= 2024-04-13T22:30:57.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:otEMTOsl1siqPElR2dxr1nB/bW8=', 'Content-Length': '0'} 2024-04-13T22:30:57.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07c6d658ff541a5-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:30:57.798 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.798 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.798 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.798 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.798 DEBUG:boto:Method: GET 2024-04-13T22:30:57.798 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.798 DEBUG:boto:Data: 2024-04-13T22:30:57.798 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.798 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.798 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.798 DEBUG:boto:Params: {} 2024-04-13T22:30:57.799 DEBUG:boto:Token: None 2024-04-13T22:30:57.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.799 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ= 2024-04-13T22:30:57.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d634dbda77cbc8de-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.801 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.801 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.801 DEBUG:boto:path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.801 DEBUG:boto:auth_path=/vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.801 DEBUG:boto:Method: GET 2024-04-13T22:30:57.801 DEBUG:boto:Path: /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.801 DEBUG:boto:Data: 2024-04-13T22:30:57.801 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.802 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.802 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.802 DEBUG:boto:Params: {} 2024-04-13T22:30:57.802 DEBUG:boto:Token: None 2024-04-13T22:30:57.802 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57?versionId=null 2024-04-13T22:30:57.802 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ= 2024-04-13T22:30:57.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFMVPpuWqHpgF8qz1iuFhWV4XCQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002151e6518a88ed88-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.846 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.847 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.847 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.847 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.847 DEBUG:boto:Method: GET 2024-04-13T22:30:57.847 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.847 DEBUG:boto:Data: 2024-04-13T22:30:57.847 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.847 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.847 DEBUG:boto:Params: {} 2024-04-13T22:30:57.848 DEBUG:boto:Token: None 2024-04-13T22:30:57.848 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c3260342dcc7450-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.850 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.851 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.851 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.851 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.851 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.851 DEBUG:boto:Method: GET 2024-04-13T22:30:57.851 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.851 DEBUG:boto:Data: 2024-04-13T22:30:57.851 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.851 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.852 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.852 DEBUG:boto:Params: {} 2024-04-13T22:30:57.852 DEBUG:boto:Token: None 2024-04-13T22:30:57.852 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.852 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d565fda2e04e467-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.856 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.856 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.856 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.856 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.856 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:57.856 DEBUG:boto:Data: 2024-04-13T22:30:57.857 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.857 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.857 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.857 DEBUG:boto:Params: {} 2024-04-13T22:30:57.857 DEBUG:boto:Token: None 2024-04-13T22:30:57.857 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57 2024-04-13T22:30:57.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rUsrTCTmy1SeautR31iKtnbteDQ= 2024-04-13T22:30:57.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rUsrTCTmy1SeautR31iKtnbteDQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004169493548e69202-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.859 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.859 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.859 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.859 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:57.860 DEBUG:boto:Data: 2024-04-13T22:30:57.860 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.860 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.860 DEBUG:boto:Params: {} 2024-04-13T22:30:57.860 DEBUG:boto:Token: None 2024-04-13T22:30:57.860 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57 2024-04-13T22:30:57.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rUsrTCTmy1SeautR31iKtnbteDQ= 2024-04-13T22:30:57.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rUsrTCTmy1SeautR31iKtnbteDQ=', 'Content-Length': '0'} 2024-04-13T22:30:57.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b156f38841c4020d-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-13T22:30:57.862 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.862 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.863 DEBUG:boto:Method: GET 2024-04-13T22:30:57.863 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:57.863 DEBUG:boto:Data: 2024-04-13T22:30:57.863 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.863 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.863 DEBUG:boto:Params: {} 2024-04-13T22:30:57.863 DEBUG:boto:Token: None 2024-04-13T22:30:57.863 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57 2024-04-13T22:30:57.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UiDbLzL/dB/4e9NFsmAPJicvfn8= 2024-04-13T22:30:57.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UiDbLzL/dB/4e9NFsmAPJicvfn8=', 'Content-Length': '0'} 2024-04-13T22:30:57.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341edb20db0df34c-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.906 DEBUG:boto:path=/vwnfns-61/obj57 2024-04-13T22:30:57.907 DEBUG:boto:auth_path=/vwnfns-61/obj57 2024-04-13T22:30:57.907 DEBUG:boto:Method: GET 2024-04-13T22:30:57.907 DEBUG:boto:Path: /vwnfns-61/obj57 2024-04-13T22:30:57.907 DEBUG:boto:Data: 2024-04-13T22:30:57.907 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.907 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.907 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.907 DEBUG:boto:Params: {} 2024-04-13T22:30:57.907 DEBUG:boto:Token: None 2024-04-13T22:30:57.908 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj57 2024-04-13T22:30:57.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UiDbLzL/dB/4e9NFsmAPJicvfn8= 2024-04-13T22:30:57.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UiDbLzL/dB/4e9NFsmAPJicvfn8=', 'Content-Length': '0'} 2024-04-13T22:30:57.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dabd69aea7d88a72-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.910 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.910 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.910 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.910 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.910 DEBUG:boto:Method: GET 2024-04-13T22:30:57.910 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.911 DEBUG:boto:Data: 2024-04-13T22:30:57.911 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.911 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.911 DEBUG:boto:Params: {} 2024-04-13T22:30:57.911 DEBUG:boto:Token: None 2024-04-13T22:30:57.911 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.911 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca0d7c4b4e89d5f-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.914 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.914 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.914 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.914 DEBUG:boto:path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.914 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.914 DEBUG:boto:Method: GET 2024-04-13T22:30:57.914 DEBUG:boto:Path: /vwnfns-61/?prefix=obj57 2024-04-13T22:30:57.915 DEBUG:boto:Data: 2024-04-13T22:30:57.915 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.915 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.915 DEBUG:boto:Params: {} 2024-04-13T22:30:57.915 DEBUG:boto:Token: None 2024-04-13T22:30:57.915 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.915 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad7fd90f930a6c22-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.919 DEBUG:boto:b'vwnfns-61obj571000falseobj572024-04-13T22:21:13.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:30:57.920 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.920 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.920 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.920 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.920 DEBUG:boto:Method: GET 2024-04-13T22:30:57.920 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.920 DEBUG:boto:Data: 2024-04-13T22:30:57.920 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.921 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.921 DEBUG:boto:Params: {} 2024-04-13T22:30:57.921 DEBUG:boto:Token: None 2024-04-13T22:30:57.921 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A= 2024-04-13T22:30:57.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A=', 'Content-Length': '0'} 2024-04-13T22:30:57.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a93cdf7b13bb1d-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.923 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.923 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.923 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.924 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.924 DEBUG:boto:Method: GET 2024-04-13T22:30:57.924 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.924 DEBUG:boto:Data: 2024-04-13T22:30:57.924 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.924 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.924 DEBUG:boto:Params: {} 2024-04-13T22:30:57.924 DEBUG:boto:Token: None 2024-04-13T22:30:57.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A= 2024-04-13T22:30:57.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A=', 'Content-Length': '0'} 2024-04-13T22:30:57.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad19fcd3d5e65007-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.974 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.974 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.975 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.975 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.975 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.975 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.975 DEBUG:boto:Data: 2024-04-13T22:30:57.975 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.975 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.975 DEBUG:boto:Params: {} 2024-04-13T22:30:57.975 DEBUG:boto:Token: None 2024-04-13T22:30:57.975 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GTMZFt4D/tZDkptFWLfWBl4R0LA= 2024-04-13T22:30:57.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GTMZFt4D/tZDkptFWLfWBl4R0LA=', 'Content-Length': '0'} 2024-04-13T22:30:57.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ccf6de9f934e2d7-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.978 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.978 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.978 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.978 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.978 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.978 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.978 DEBUG:boto:Data: 2024-04-13T22:30:57.978 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.978 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.979 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.979 DEBUG:boto:Params: {} 2024-04-13T22:30:57.979 DEBUG:boto:Token: None 2024-04-13T22:30:57.979 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.979 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GTMZFt4D/tZDkptFWLfWBl4R0LA= 2024-04-13T22:30:57.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GTMZFt4D/tZDkptFWLfWBl4R0LA=', 'Content-Length': '0'} 2024-04-13T22:30:57.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3cd7e4f811563a-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:30:57.981 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.981 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.981 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.981 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.981 DEBUG:boto:Method: GET 2024-04-13T22:30:57.981 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.982 DEBUG:boto:Data: 2024-04-13T22:30:57.982 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.982 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.982 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.982 DEBUG:boto:Params: {} 2024-04-13T22:30:57.982 DEBUG:boto:Token: None 2024-04-13T22:30:57.982 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.982 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A= 2024-04-13T22:30:57.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A=', 'Content-Length': '0'} 2024-04-13T22:30:57.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888c4183be13921d-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.984 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.984 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.984 DEBUG:boto:path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.984 DEBUG:boto:auth_path=/vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.984 DEBUG:boto:Method: GET 2024-04-13T22:30:57.985 DEBUG:boto:Path: /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.985 DEBUG:boto:Data: 2024-04-13T22:30:57.985 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.985 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.985 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.985 DEBUG:boto:Params: {} 2024-04-13T22:30:57.985 DEBUG:boto:Token: None 2024-04-13T22:30:57.985 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58?versionId=null 2024-04-13T22:30:57.985 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A= 2024-04-13T22:30:57.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZVW9lLh/daEZVlR6STf3RPt6d4A=', 'Content-Length': '0'} 2024-04-13T22:30:57.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000597483be7b19f2eb-00661b07a1-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.987 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.987 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.987 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:57.988 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:57.988 DEBUG:boto:Method: GET 2024-04-13T22:30:57.988 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:57.988 DEBUG:boto:Data: 2024-04-13T22:30:57.988 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.988 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.988 DEBUG:boto:Params: {} 2024-04-13T22:30:57.988 DEBUG:boto:Token: None 2024-04-13T22:30:57.988 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.989 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2115812f93f809-00661b07a1-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.991 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.991 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:57.991 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:57.992 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:57.992 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:57.992 DEBUG:boto:Method: GET 2024-04-13T22:30:57.992 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:57.992 DEBUG:boto:Data: 2024-04-13T22:30:57.992 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.992 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:57.992 DEBUG:boto:Port: 8000 2024-04-13T22:30:57.992 DEBUG:boto:Params: {} 2024-04-13T22:30:57.992 DEBUG:boto:Token: None 2024-04-13T22:30:57.992 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/ 2024-04-13T22:30:57.993 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68= 2024-04-13T22:30:57.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkE0+1ahd0PjjvngWng+JZyFJ68=', 'Content-Length': '0'} 2024-04-13T22:30:57.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000307a5f6ea235d676-00661b07a1-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:57.996 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:57.996 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:57.997 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:57.997 DEBUG:boto:Method: HEAD 2024-04-13T22:30:57.997 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:57.997 DEBUG:boto:Data: 2024-04-13T22:30:57.997 DEBUG:boto:Headers: {} 2024-04-13T22:30:57.997 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:57.997 DEBUG:boto:Port: 8001 2024-04-13T22:30:57.997 DEBUG:boto:Params: {} 2024-04-13T22:30:57.997 DEBUG:boto:Token: None 2024-04-13T22:30:57.998 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:57 GMT /vwnfns-61/obj58 2024-04-13T22:30:57.998 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xu5wUESv1m9ujrhZmm5ZUT/dXLo= 2024-04-13T22:30:57.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xu5wUESv1m9ujrhZmm5ZUT/dXLo=', 'Content-Length': '0'} 2024-04-13T22:30:57.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a7fc8950cb1912-00661b07a1-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.000 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:58.000 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:58.000 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.000 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:58.000 DEBUG:boto:Data: 2024-04-13T22:30:58.000 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.000 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.000 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.000 DEBUG:boto:Params: {} 2024-04-13T22:30:58.000 DEBUG:boto:Token: None 2024-04-13T22:30:58.001 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj58 2024-04-13T22:30:58.001 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scIQCruGY8R9HqZkhnF5ja/5XrI= 2024-04-13T22:30:58.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scIQCruGY8R9HqZkhnF5ja/5XrI=', 'Content-Length': '0'} 2024-04-13T22:30:58.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007246d82349166239-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-13T22:30:58.003 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:58.003 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:58.003 DEBUG:boto:Method: GET 2024-04-13T22:30:58.003 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:58.003 DEBUG:boto:Data: 2024-04-13T22:30:58.003 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.003 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.003 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.004 DEBUG:boto:Params: {} 2024-04-13T22:30:58.004 DEBUG:boto:Token: None 2024-04-13T22:30:58.004 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj58 2024-04-13T22:30:58.004 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uiY/xyLGJlUFgcEOb8JuRy/e0c0= 2024-04-13T22:30:58.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uiY/xyLGJlUFgcEOb8JuRy/e0c0=', 'Content-Length': '0'} 2024-04-13T22:30:58.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eee0e13c20469681-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.006 DEBUG:boto:path=/vwnfns-61/obj58 2024-04-13T22:30:58.006 DEBUG:boto:auth_path=/vwnfns-61/obj58 2024-04-13T22:30:58.006 DEBUG:boto:Method: GET 2024-04-13T22:30:58.006 DEBUG:boto:Path: /vwnfns-61/obj58 2024-04-13T22:30:58.006 DEBUG:boto:Data: 2024-04-13T22:30:58.006 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.007 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.007 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.007 DEBUG:boto:Params: {} 2024-04-13T22:30:58.007 DEBUG:boto:Token: None 2024-04-13T22:30:58.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj58 2024-04-13T22:30:58.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uiY/xyLGJlUFgcEOb8JuRy/e0c0= 2024-04-13T22:30:58.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uiY/xyLGJlUFgcEOb8JuRy/e0c0=', 'Content-Length': '0'} 2024-04-13T22:30:58.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ccfb6075e313657-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.054 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.055 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.055 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:58.055 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:58.055 DEBUG:boto:Method: GET 2024-04-13T22:30:58.055 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:58.055 DEBUG:boto:Data: 2024-04-13T22:30:58.055 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.055 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.055 DEBUG:boto:Params: {} 2024-04-13T22:30:58.056 DEBUG:boto:Token: None 2024-04-13T22:30:58.056 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b609b1f494e05616-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.058 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.059 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.059 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.059 DEBUG:boto:path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:58.059 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj58 2024-04-13T22:30:58.059 DEBUG:boto:Method: GET 2024-04-13T22:30:58.059 DEBUG:boto:Path: /vwnfns-61/?prefix=obj58 2024-04-13T22:30:58.059 DEBUG:boto:Data: 2024-04-13T22:30:58.059 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.059 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.060 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.060 DEBUG:boto:Params: {} 2024-04-13T22:30:58.060 DEBUG:boto:Token: None 2024-04-13T22:30:58.060 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.060 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611b16f3ee94e8bb-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.063 DEBUG:boto:b'vwnfns-61obj581000falseobj582024-04-13T22:25:33.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:30:58.064 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.064 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.064 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.064 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.064 DEBUG:boto:Method: GET 2024-04-13T22:30:58.064 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.064 DEBUG:boto:Data: 2024-04-13T22:30:58.064 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.065 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.065 DEBUG:boto:Params: {} 2024-04-13T22:30:58.065 DEBUG:boto:Token: None 2024-04-13T22:30:58.065 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.065 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM= 2024-04-13T22:30:58.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM=', 'Content-Length': '0'} 2024-04-13T22:30:58.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bdde66fbb11619d-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.114 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.115 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.115 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.115 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.115 DEBUG:boto:Method: GET 2024-04-13T22:30:58.115 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.115 DEBUG:boto:Data: 2024-04-13T22:30:58.115 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.115 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.115 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.115 DEBUG:boto:Params: {} 2024-04-13T22:30:58.116 DEBUG:boto:Token: None 2024-04-13T22:30:58.116 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.116 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM= 2024-04-13T22:30:58.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM=', 'Content-Length': '0'} 2024-04-13T22:30:58.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003036e78088566165-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.118 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.118 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.118 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.118 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.118 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.118 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.118 DEBUG:boto:Data: 2024-04-13T22:30:58.119 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.119 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.119 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.119 DEBUG:boto:Params: {} 2024-04-13T22:30:58.119 DEBUG:boto:Token: None 2024-04-13T22:30:58.119 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.119 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wGa5PT8J44kzhHDpCwkBkDEEmK0= 2024-04-13T22:30:58.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wGa5PT8J44kzhHDpCwkBkDEEmK0=', 'Content-Length': '0'} 2024-04-13T22:30:58.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb617f590e7e850-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.121 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.121 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.122 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.122 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.122 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.122 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.122 DEBUG:boto:Data: 2024-04-13T22:30:58.122 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.122 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.122 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.122 DEBUG:boto:Params: {} 2024-04-13T22:30:58.122 DEBUG:boto:Token: None 2024-04-13T22:30:58.123 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wGa5PT8J44kzhHDpCwkBkDEEmK0= 2024-04-13T22:30:58.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wGa5PT8J44kzhHDpCwkBkDEEmK0=', 'Content-Length': '0'} 2024-04-13T22:30:58.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002616551ff004a59d-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:30:58.125 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.125 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.125 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.125 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.125 DEBUG:boto:Method: GET 2024-04-13T22:30:58.125 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.125 DEBUG:boto:Data: 2024-04-13T22:30:58.125 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.125 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.125 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.126 DEBUG:boto:Params: {} 2024-04-13T22:30:58.126 DEBUG:boto:Token: None 2024-04-13T22:30:58.126 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.126 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM= 2024-04-13T22:30:58.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM=', 'Content-Length': '0'} 2024-04-13T22:30:58.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fe6700384071bb-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.128 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.128 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.128 DEBUG:boto:path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.128 DEBUG:boto:auth_path=/vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.128 DEBUG:boto:Method: GET 2024-04-13T22:30:58.128 DEBUG:boto:Path: /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.128 DEBUG:boto:Data: 2024-04-13T22:30:58.128 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.128 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.129 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.129 DEBUG:boto:Params: {} 2024-04-13T22:30:58.129 DEBUG:boto:Token: None 2024-04-13T22:30:58.129 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62?versionId=null 2024-04-13T22:30:58.129 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM= 2024-04-13T22:30:58.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Eg97Ax9PyHhEH+Ct8lXmtmlLHZM=', 'Content-Length': '0'} 2024-04-13T22:30:58.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3b664f2c8912d3-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.131 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.131 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.131 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.131 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.131 DEBUG:boto:Method: GET 2024-04-13T22:30:58.131 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.131 DEBUG:boto:Data: 2024-04-13T22:30:58.131 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.132 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.132 DEBUG:boto:Params: {} 2024-04-13T22:30:58.132 DEBUG:boto:Token: None 2024-04-13T22:30:58.132 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3a083ebaffb259-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.134 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.135 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.135 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.135 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.135 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.135 DEBUG:boto:Method: GET 2024-04-13T22:30:58.135 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.135 DEBUG:boto:Data: 2024-04-13T22:30:58.135 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.135 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.135 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.135 DEBUG:boto:Params: {} 2024-04-13T22:30:58.136 DEBUG:boto:Token: None 2024-04-13T22:30:58.136 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.136 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760614553a2b6285-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.139 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.140 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.140 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.140 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.140 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:58.140 DEBUG:boto:Data: 2024-04-13T22:30:58.140 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.140 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.141 DEBUG:boto:Params: {} 2024-04-13T22:30:58.141 DEBUG:boto:Token: None 2024-04-13T22:30:58.141 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62 2024-04-13T22:30:58.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5xjRvMreAwlfEkchRXuTWkuV8/M= 2024-04-13T22:30:58.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5xjRvMreAwlfEkchRXuTWkuV8/M=', 'Content-Length': '0'} 2024-04-13T22:30:58.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4cc35a8427f82d9-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.143 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.143 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.143 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.143 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:58.143 DEBUG:boto:Data: 2024-04-13T22:30:58.143 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.143 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.143 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.144 DEBUG:boto:Params: {} 2024-04-13T22:30:58.144 DEBUG:boto:Token: None 2024-04-13T22:30:58.144 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62 2024-04-13T22:30:58.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5xjRvMreAwlfEkchRXuTWkuV8/M= 2024-04-13T22:30:58.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5xjRvMreAwlfEkchRXuTWkuV8/M=', 'Content-Length': '0'} 2024-04-13T22:30:58.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e59d1861c44c28-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-13T22:30:58.146 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.146 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.146 DEBUG:boto:Method: GET 2024-04-13T22:30:58.146 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:58.146 DEBUG:boto:Data: 2024-04-13T22:30:58.146 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.146 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.146 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.146 DEBUG:boto:Params: {} 2024-04-13T22:30:58.149 DEBUG:boto:Token: None 2024-04-13T22:30:58.149 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62 2024-04-13T22:30:58.149 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IWeWWKF3Qv9r4RB+6FqdJ/gyvZY= 2024-04-13T22:30:58.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IWeWWKF3Qv9r4RB+6FqdJ/gyvZY=', 'Content-Length': '0'} 2024-04-13T22:30:58.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897940ba492e3c51-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.194 DEBUG:boto:path=/vwnfns-61/obj62 2024-04-13T22:30:58.194 DEBUG:boto:auth_path=/vwnfns-61/obj62 2024-04-13T22:30:58.195 DEBUG:boto:Method: GET 2024-04-13T22:30:58.195 DEBUG:boto:Path: /vwnfns-61/obj62 2024-04-13T22:30:58.195 DEBUG:boto:Data: 2024-04-13T22:30:58.195 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.195 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.195 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.195 DEBUG:boto:Params: {} 2024-04-13T22:30:58.195 DEBUG:boto:Token: None 2024-04-13T22:30:58.195 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj62 2024-04-13T22:30:58.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IWeWWKF3Qv9r4RB+6FqdJ/gyvZY= 2024-04-13T22:30:58.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IWeWWKF3Qv9r4RB+6FqdJ/gyvZY=', 'Content-Length': '0'} 2024-04-13T22:30:58.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22b492a46f6fbcb-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.197 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.198 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.198 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.198 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.198 DEBUG:boto:Method: GET 2024-04-13T22:30:58.198 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.198 DEBUG:boto:Data: 2024-04-13T22:30:58.198 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.198 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.198 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.198 DEBUG:boto:Params: {} 2024-04-13T22:30:58.199 DEBUG:boto:Token: None 2024-04-13T22:30:58.199 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.199 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e5c95cb87a5378-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.201 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.201 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.201 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.202 DEBUG:boto:path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.202 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.202 DEBUG:boto:Method: GET 2024-04-13T22:30:58.202 DEBUG:boto:Path: /vwnfns-61/?prefix=obj62 2024-04-13T22:30:58.202 DEBUG:boto:Data: 2024-04-13T22:30:58.202 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.202 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.202 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.202 DEBUG:boto:Params: {} 2024-04-13T22:30:58.202 DEBUG:boto:Token: None 2024-04-13T22:30:58.202 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.203 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a6180de253796b-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.206 DEBUG:boto:b'vwnfns-61obj621000falseobj622024-04-13T22:27:52.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:30:58.207 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.207 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.207 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.207 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.207 DEBUG:boto:Method: GET 2024-04-13T22:30:58.207 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.207 DEBUG:boto:Data: 2024-04-13T22:30:58.207 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.207 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.208 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.208 DEBUG:boto:Params: {} 2024-04-13T22:30:58.208 DEBUG:boto:Token: None 2024-04-13T22:30:58.208 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.208 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0= 2024-04-13T22:30:58.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0=', 'Content-Length': '0'} 2024-04-13T22:30:58.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d4b8db4e15c5e15-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.210 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.210 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.210 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.210 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.210 DEBUG:boto:Method: GET 2024-04-13T22:30:58.210 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.211 DEBUG:boto:Data: 2024-04-13T22:30:58.211 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.211 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.211 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.211 DEBUG:boto:Params: {} 2024-04-13T22:30:58.211 DEBUG:boto:Token: None 2024-04-13T22:30:58.211 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.211 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0= 2024-04-13T22:30:58.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0=', 'Content-Length': '0'} 2024-04-13T22:30:58.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091bf996eace8467b-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.213 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.213 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.213 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.214 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.214 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.214 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.214 DEBUG:boto:Data: 2024-04-13T22:30:58.214 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.214 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.214 DEBUG:boto:Params: {} 2024-04-13T22:30:58.214 DEBUG:boto:Token: None 2024-04-13T22:30:58.214 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9XOWSCeGQDhm2taEYqij34Lq/y0= 2024-04-13T22:30:58.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9XOWSCeGQDhm2taEYqij34Lq/y0=', 'Content-Length': '0'} 2024-04-13T22:30:58.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009687f7ed972e9919-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.216 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.216 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.217 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.217 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.217 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.217 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.217 DEBUG:boto:Data: 2024-04-13T22:30:58.217 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.217 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.217 DEBUG:boto:Params: {} 2024-04-13T22:30:58.217 DEBUG:boto:Token: None 2024-04-13T22:30:58.218 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9XOWSCeGQDhm2taEYqij34Lq/y0= 2024-04-13T22:30:58.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9XOWSCeGQDhm2taEYqij34Lq/y0=', 'Content-Length': '0'} 2024-04-13T22:30:58.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b73ea9640b3a81-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:30:58.220 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.220 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.220 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.220 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.220 DEBUG:boto:Method: GET 2024-04-13T22:30:58.220 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.220 DEBUG:boto:Data: 2024-04-13T22:30:58.220 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.220 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.220 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.220 DEBUG:boto:Params: {} 2024-04-13T22:30:58.220 DEBUG:boto:Token: None 2024-04-13T22:30:58.221 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.221 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0= 2024-04-13T22:30:58.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0=', 'Content-Length': '0'} 2024-04-13T22:30:58.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000010ae086de09fd18-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.222 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.223 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.223 DEBUG:boto:path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.223 DEBUG:boto:auth_path=/vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.223 DEBUG:boto:Method: GET 2024-04-13T22:30:58.223 DEBUG:boto:Path: /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.223 DEBUG:boto:Data: 2024-04-13T22:30:58.223 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.223 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.223 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.223 DEBUG:boto:Params: {} 2024-04-13T22:30:58.223 DEBUG:boto:Token: None 2024-04-13T22:30:58.224 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68?versionId=null 2024-04-13T22:30:58.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0= 2024-04-13T22:30:58.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:StNha0FQ6I77uPT/AKzRnjzbBt0=', 'Content-Length': '0'} 2024-04-13T22:30:58.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b81bc6976ec2983-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.225 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.226 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.226 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.226 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.226 DEBUG:boto:Method: GET 2024-04-13T22:30:58.226 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.226 DEBUG:boto:Data: 2024-04-13T22:30:58.226 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.226 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.226 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.226 DEBUG:boto:Params: {} 2024-04-13T22:30:58.226 DEBUG:boto:Token: None 2024-04-13T22:30:58.227 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.227 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094961833c1c7fefc-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.229 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.229 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.230 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.230 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.230 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.230 DEBUG:boto:Method: GET 2024-04-13T22:30:58.230 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.230 DEBUG:boto:Data: 2024-04-13T22:30:58.230 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.230 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.230 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.230 DEBUG:boto:Params: {} 2024-04-13T22:30:58.231 DEBUG:boto:Token: None 2024-04-13T22:30:58.231 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.231 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005164724fed895535-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.235 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.235 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.235 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.235 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.235 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:58.235 DEBUG:boto:Data: 2024-04-13T22:30:58.235 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.236 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.236 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.236 DEBUG:boto:Params: {} 2024-04-13T22:30:58.236 DEBUG:boto:Token: None 2024-04-13T22:30:58.236 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68 2024-04-13T22:30:58.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JNkXflW4x7PxmrWzaS8B9Na9kNg= 2024-04-13T22:30:58.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JNkXflW4x7PxmrWzaS8B9Na9kNg=', 'Content-Length': '0'} 2024-04-13T22:30:58.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90a326eb4c42c14-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.238 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.238 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.238 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.238 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:58.238 DEBUG:boto:Data: 2024-04-13T22:30:58.238 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.238 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.238 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.238 DEBUG:boto:Params: {} 2024-04-13T22:30:58.239 DEBUG:boto:Token: None 2024-04-13T22:30:58.239 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68 2024-04-13T22:30:58.239 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JNkXflW4x7PxmrWzaS8B9Na9kNg= 2024-04-13T22:30:58.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JNkXflW4x7PxmrWzaS8B9Na9kNg=', 'Content-Length': '0'} 2024-04-13T22:30:58.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fbc206c10fa5e12-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-13T22:30:58.241 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.241 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.241 DEBUG:boto:Method: GET 2024-04-13T22:30:58.241 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:58.241 DEBUG:boto:Data: 2024-04-13T22:30:58.241 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.241 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.241 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.241 DEBUG:boto:Params: {} 2024-04-13T22:30:58.242 DEBUG:boto:Token: None 2024-04-13T22:30:58.242 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68 2024-04-13T22:30:58.242 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NdVu2f1Iw8M7Rk2Oq1a7ixbH480= 2024-04-13T22:30:58.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NdVu2f1Iw8M7Rk2Oq1a7ixbH480=', 'Content-Length': '0'} 2024-04-13T22:30:58.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001782d944c10176b6-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.243 DEBUG:boto:path=/vwnfns-61/obj68 2024-04-13T22:30:58.244 DEBUG:boto:auth_path=/vwnfns-61/obj68 2024-04-13T22:30:58.244 DEBUG:boto:Method: GET 2024-04-13T22:30:58.244 DEBUG:boto:Path: /vwnfns-61/obj68 2024-04-13T22:30:58.244 DEBUG:boto:Data: 2024-04-13T22:30:58.244 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.244 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.244 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.244 DEBUG:boto:Params: {} 2024-04-13T22:30:58.244 DEBUG:boto:Token: None 2024-04-13T22:30:58.245 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj68 2024-04-13T22:30:58.245 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NdVu2f1Iw8M7Rk2Oq1a7ixbH480= 2024-04-13T22:30:58.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NdVu2f1Iw8M7Rk2Oq1a7ixbH480=', 'Content-Length': '0'} 2024-04-13T22:30:58.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a321394727ec980a-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.247 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.247 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.247 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.247 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.247 DEBUG:boto:Method: GET 2024-04-13T22:30:58.247 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.247 DEBUG:boto:Data: 2024-04-13T22:30:58.247 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.248 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.248 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.248 DEBUG:boto:Params: {} 2024-04-13T22:30:58.248 DEBUG:boto:Token: None 2024-04-13T22:30:58.248 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195b99e4b43b06db-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.250 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.251 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.251 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.251 DEBUG:boto:path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.251 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.251 DEBUG:boto:Method: GET 2024-04-13T22:30:58.251 DEBUG:boto:Path: /vwnfns-61/?prefix=obj68 2024-04-13T22:30:58.251 DEBUG:boto:Data: 2024-04-13T22:30:58.251 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.252 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.252 DEBUG:boto:Params: {} 2024-04-13T22:30:58.252 DEBUG:boto:Token: None 2024-04-13T22:30:58.252 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.252 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ae63a8e96b1580-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.298 DEBUG:boto:b'vwnfns-61obj681000falseobj682024-04-13T22:18:11.417Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:30:58.299 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.299 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.299 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.299 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.299 DEBUG:boto:Method: GET 2024-04-13T22:30:58.299 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.300 DEBUG:boto:Data: 2024-04-13T22:30:58.300 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.300 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.300 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.300 DEBUG:boto:Params: {} 2024-04-13T22:30:58.300 DEBUG:boto:Token: None 2024-04-13T22:30:58.300 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg= 2024-04-13T22:30:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg=', 'Content-Length': '0'} 2024-04-13T22:30:58.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a947d0fcaecca8b-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.303 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.303 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.303 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.303 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.303 DEBUG:boto:Method: GET 2024-04-13T22:30:58.303 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.303 DEBUG:boto:Data: 2024-04-13T22:30:58.303 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.303 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.303 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.303 DEBUG:boto:Params: {} 2024-04-13T22:30:58.304 DEBUG:boto:Token: None 2024-04-13T22:30:58.304 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg= 2024-04-13T22:30:58.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg=', 'Content-Length': '0'} 2024-04-13T22:30:58.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a16e1db60325103-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.306 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.306 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.306 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.306 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.307 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.307 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.307 DEBUG:boto:Data: 2024-04-13T22:30:58.307 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.307 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.307 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.307 DEBUG:boto:Params: {} 2024-04-13T22:30:58.307 DEBUG:boto:Token: None 2024-04-13T22:30:58.307 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.307 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoP+VsZdiEQh3FPeUM+ArAWcSWo= 2024-04-13T22:30:58.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoP+VsZdiEQh3FPeUM+ArAWcSWo=', 'Content-Length': '0'} 2024-04-13T22:30:58.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb1d8ecc36cc939-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.309 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.309 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.309 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.309 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.310 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.310 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.310 DEBUG:boto:Data: 2024-04-13T22:30:58.310 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.310 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.310 DEBUG:boto:Params: {} 2024-04-13T22:30:58.310 DEBUG:boto:Token: None 2024-04-13T22:30:58.310 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.311 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qoP+VsZdiEQh3FPeUM+ArAWcSWo= 2024-04-13T22:30:58.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoP+VsZdiEQh3FPeUM+ArAWcSWo=', 'Content-Length': '0'} 2024-04-13T22:30:58.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2a9bd1260d497d5-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:30:58.312 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.313 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.313 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.313 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.313 DEBUG:boto:Method: GET 2024-04-13T22:30:58.313 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.313 DEBUG:boto:Data: 2024-04-13T22:30:58.313 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.313 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.313 DEBUG:boto:Params: {} 2024-04-13T22:30:58.313 DEBUG:boto:Token: None 2024-04-13T22:30:58.314 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg= 2024-04-13T22:30:58.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg=', 'Content-Length': '0'} 2024-04-13T22:30:58.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a8a144bb4656b0-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.358 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.358 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.359 DEBUG:boto:path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.359 DEBUG:boto:auth_path=/vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.359 DEBUG:boto:Method: GET 2024-04-13T22:30:58.359 DEBUG:boto:Path: /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.359 DEBUG:boto:Data: 2024-04-13T22:30:58.359 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.359 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.359 DEBUG:boto:Params: {} 2024-04-13T22:30:58.359 DEBUG:boto:Token: None 2024-04-13T22:30:58.360 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69?versionId=null 2024-04-13T22:30:58.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg= 2024-04-13T22:30:58.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LYeXY8skOuEK+INUdsiN90wD6Mg=', 'Content-Length': '0'} 2024-04-13T22:30:58.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424b8c97d2c974e3-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.362 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.362 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.362 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.362 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.362 DEBUG:boto:Method: GET 2024-04-13T22:30:58.362 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.362 DEBUG:boto:Data: 2024-04-13T22:30:58.362 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.363 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.363 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.363 DEBUG:boto:Params: {} 2024-04-13T22:30:58.363 DEBUG:boto:Token: None 2024-04-13T22:30:58.363 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b868ed98ffecc392-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.365 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.366 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.366 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.366 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.366 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.366 DEBUG:boto:Method: GET 2024-04-13T22:30:58.366 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.366 DEBUG:boto:Data: 2024-04-13T22:30:58.367 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.367 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.367 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.367 DEBUG:boto:Params: {} 2024-04-13T22:30:58.367 DEBUG:boto:Token: None 2024-04-13T22:30:58.367 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000980a3f813179915-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.371 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.371 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.371 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.371 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.372 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:58.372 DEBUG:boto:Data: 2024-04-13T22:30:58.372 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.372 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.372 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.372 DEBUG:boto:Params: {} 2024-04-13T22:30:58.372 DEBUG:boto:Token: None 2024-04-13T22:30:58.372 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69 2024-04-13T22:30:58.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xuTSoiXN3uupbwoOSfJ7bWZXoBU= 2024-04-13T22:30:58.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xuTSoiXN3uupbwoOSfJ7bWZXoBU=', 'Content-Length': '0'} 2024-04-13T22:30:58.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d417412f35179045-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.374 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.374 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.374 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.375 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:58.375 DEBUG:boto:Data: 2024-04-13T22:30:58.375 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.375 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.375 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.375 DEBUG:boto:Params: {} 2024-04-13T22:30:58.375 DEBUG:boto:Token: None 2024-04-13T22:30:58.375 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69 2024-04-13T22:30:58.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xuTSoiXN3uupbwoOSfJ7bWZXoBU= 2024-04-13T22:30:58.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xuTSoiXN3uupbwoOSfJ7bWZXoBU=', 'Content-Length': '0'} 2024-04-13T22:30:58.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad008deccacbca81-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-13T22:30:58.377 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.377 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.377 DEBUG:boto:Method: GET 2024-04-13T22:30:58.377 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:58.377 DEBUG:boto:Data: 2024-04-13T22:30:58.378 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.378 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.378 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.378 DEBUG:boto:Params: {} 2024-04-13T22:30:58.378 DEBUG:boto:Token: None 2024-04-13T22:30:58.378 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69 2024-04-13T22:30:58.378 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gulzopEoqobUz+vp7QE6QoFxHZI= 2024-04-13T22:30:58.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gulzopEoqobUz+vp7QE6QoFxHZI=', 'Content-Length': '0'} 2024-04-13T22:30:58.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b63b41a2ffd65f8-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.380 DEBUG:boto:path=/vwnfns-61/obj69 2024-04-13T22:30:58.380 DEBUG:boto:auth_path=/vwnfns-61/obj69 2024-04-13T22:30:58.380 DEBUG:boto:Method: GET 2024-04-13T22:30:58.380 DEBUG:boto:Path: /vwnfns-61/obj69 2024-04-13T22:30:58.381 DEBUG:boto:Data: 2024-04-13T22:30:58.381 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.381 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.381 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.381 DEBUG:boto:Params: {} 2024-04-13T22:30:58.381 DEBUG:boto:Token: None 2024-04-13T22:30:58.381 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj69 2024-04-13T22:30:58.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gulzopEoqobUz+vp7QE6QoFxHZI= 2024-04-13T22:30:58.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gulzopEoqobUz+vp7QE6QoFxHZI=', 'Content-Length': '0'} 2024-04-13T22:30:58.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1fb4c72cac65a1e-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.383 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.383 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.383 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.383 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.383 DEBUG:boto:Method: GET 2024-04-13T22:30:58.384 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.384 DEBUG:boto:Data: 2024-04-13T22:30:58.384 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.384 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.384 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.384 DEBUG:boto:Params: {} 2024-04-13T22:30:58.384 DEBUG:boto:Token: None 2024-04-13T22:30:58.384 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.384 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0e128d199024db-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.386 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.387 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.387 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.387 DEBUG:boto:path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.387 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.387 DEBUG:boto:Method: GET 2024-04-13T22:30:58.387 DEBUG:boto:Path: /vwnfns-61/?prefix=obj69 2024-04-13T22:30:58.387 DEBUG:boto:Data: 2024-04-13T22:30:58.387 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.388 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.388 DEBUG:boto:Params: {} 2024-04-13T22:30:58.388 DEBUG:boto:Token: None 2024-04-13T22:30:58.388 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000030eb8a4eabdd1d5-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.434 DEBUG:boto:b'vwnfns-61obj691000falseobj692024-04-13T22:22:50.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-13T22:30:58.435 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.435 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.435 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.435 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.435 DEBUG:boto:Method: GET 2024-04-13T22:30:58.436 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.436 DEBUG:boto:Data: 2024-04-13T22:30:58.436 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.436 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.436 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.436 DEBUG:boto:Params: {} 2024-04-13T22:30:58.436 DEBUG:boto:Token: None 2024-04-13T22:30:58.436 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.437 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY= 2024-04-13T22:30:58.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY=', 'Content-Length': '0'} 2024-04-13T22:30:58.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007fd307fdf018d32-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.439 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.439 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.439 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.439 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.439 DEBUG:boto:Method: GET 2024-04-13T22:30:58.439 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.439 DEBUG:boto:Data: 2024-04-13T22:30:58.440 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.440 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.440 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.440 DEBUG:boto:Params: {} 2024-04-13T22:30:58.440 DEBUG:boto:Token: None 2024-04-13T22:30:58.440 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.440 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY= 2024-04-13T22:30:58.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY=', 'Content-Length': '0'} 2024-04-13T22:30:58.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e15e9c628f6e9dc-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.442 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.443 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.443 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.443 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.443 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.443 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.443 DEBUG:boto:Data: 2024-04-13T22:30:58.443 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.443 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.443 DEBUG:boto:Params: {} 2024-04-13T22:30:58.444 DEBUG:boto:Token: None 2024-04-13T22:30:58.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UyaULiGUNLRioM4wG+aK/I2q4nw= 2024-04-13T22:30:58.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UyaULiGUNLRioM4wG+aK/I2q4nw=', 'Content-Length': '0'} 2024-04-13T22:30:58.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001543af80c86015c0-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.445 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.445 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.446 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.446 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.446 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.446 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.446 DEBUG:boto:Data: 2024-04-13T22:30:58.446 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.446 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.446 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.446 DEBUG:boto:Params: {} 2024-04-13T22:30:58.446 DEBUG:boto:Token: None 2024-04-13T22:30:58.447 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.447 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UyaULiGUNLRioM4wG+aK/I2q4nw= 2024-04-13T22:30:58.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UyaULiGUNLRioM4wG+aK/I2q4nw=', 'Content-Length': '0'} 2024-04-13T22:30:58.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b0a29dd045e9b74-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-13T22:30:58.449 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.449 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.449 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.449 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.449 DEBUG:boto:Method: GET 2024-04-13T22:30:58.449 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.449 DEBUG:boto:Data: 2024-04-13T22:30:58.449 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.449 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.449 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.450 DEBUG:boto:Params: {} 2024-04-13T22:30:58.450 DEBUG:boto:Token: None 2024-04-13T22:30:58.450 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.450 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY= 2024-04-13T22:30:58.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY=', 'Content-Length': '0'} 2024-04-13T22:30:58.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987aa292d4a69f70-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.452 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.452 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.452 DEBUG:boto:path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.452 DEBUG:boto:auth_path=/vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.452 DEBUG:boto:Method: GET 2024-04-13T22:30:58.453 DEBUG:boto:Path: /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.453 DEBUG:boto:Data: 2024-04-13T22:30:58.453 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.453 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.453 DEBUG:boto:Params: {} 2024-04-13T22:30:58.453 DEBUG:boto:Token: None 2024-04-13T22:30:58.453 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74?versionId=null 2024-04-13T22:30:58.453 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY= 2024-04-13T22:30:58.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VX5FT/dFs+RIdjf7MZk3OkjkcnY=', 'Content-Length': '0'} 2024-04-13T22:30:58.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26351cb0393dd8f-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.455 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.456 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.456 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.456 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.456 DEBUG:boto:Method: GET 2024-04-13T22:30:58.456 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.456 DEBUG:boto:Data: 2024-04-13T22:30:58.456 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.456 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.456 DEBUG:boto:Params: {} 2024-04-13T22:30:58.456 DEBUG:boto:Token: None 2024-04-13T22:30:58.457 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.457 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c014d67a2feb191-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.459 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.459 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.460 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.460 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.460 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.460 DEBUG:boto:Method: GET 2024-04-13T22:30:58.460 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.460 DEBUG:boto:Data: 2024-04-13T22:30:58.460 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.460 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.460 DEBUG:boto:Params: {} 2024-04-13T22:30:58.461 DEBUG:boto:Token: None 2024-04-13T22:30:58.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000459976c0a6222321-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.510 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.511 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.511 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.511 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.511 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:58.511 DEBUG:boto:Data: 2024-04-13T22:30:58.511 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.512 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.512 DEBUG:boto:Params: {} 2024-04-13T22:30:58.512 DEBUG:boto:Token: None 2024-04-13T22:30:58.512 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74 2024-04-13T22:30:58.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f3hMWSOYVBHHzKk2c/9nYfJLWLM= 2024-04-13T22:30:58.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f3hMWSOYVBHHzKk2c/9nYfJLWLM=', 'Content-Length': '0'} 2024-04-13T22:30:58.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ee698d64ee3e6b-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.514 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.514 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.514 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.514 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:58.515 DEBUG:boto:Data: 2024-04-13T22:30:58.515 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.515 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.515 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.515 DEBUG:boto:Params: {} 2024-04-13T22:30:58.515 DEBUG:boto:Token: None 2024-04-13T22:30:58.515 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74 2024-04-13T22:30:58.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:f3hMWSOYVBHHzKk2c/9nYfJLWLM= 2024-04-13T22:30:58.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f3hMWSOYVBHHzKk2c/9nYfJLWLM=', 'Content-Length': '0'} 2024-04-13T22:30:58.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370917e80fdd5ccf-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-13T22:30:58.517 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.517 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.517 DEBUG:boto:Method: GET 2024-04-13T22:30:58.517 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:58.517 DEBUG:boto:Data: 2024-04-13T22:30:58.518 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.518 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.518 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.518 DEBUG:boto:Params: {} 2024-04-13T22:30:58.518 DEBUG:boto:Token: None 2024-04-13T22:30:58.518 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74 2024-04-13T22:30:58.518 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T+4pyisVnqLye4w24hT64kljhd8= 2024-04-13T22:30:58.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T+4pyisVnqLye4w24hT64kljhd8=', 'Content-Length': '0'} 2024-04-13T22:30:58.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffbb7ba7dc7c8f03-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.520 DEBUG:boto:path=/vwnfns-61/obj74 2024-04-13T22:30:58.520 DEBUG:boto:auth_path=/vwnfns-61/obj74 2024-04-13T22:30:58.520 DEBUG:boto:Method: GET 2024-04-13T22:30:58.521 DEBUG:boto:Path: /vwnfns-61/obj74 2024-04-13T22:30:58.521 DEBUG:boto:Data: 2024-04-13T22:30:58.521 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.521 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.521 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.521 DEBUG:boto:Params: {} 2024-04-13T22:30:58.521 DEBUG:boto:Token: None 2024-04-13T22:30:58.521 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj74 2024-04-13T22:30:58.521 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T+4pyisVnqLye4w24hT64kljhd8= 2024-04-13T22:30:58.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T+4pyisVnqLye4w24hT64kljhd8=', 'Content-Length': '0'} 2024-04-13T22:30:58.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:28:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000097a08dc2fe1a4-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.523 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.523 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.523 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.524 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.524 DEBUG:boto:Method: GET 2024-04-13T22:30:58.524 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.524 DEBUG:boto:Data: 2024-04-13T22:30:58.524 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.524 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.524 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.524 DEBUG:boto:Params: {} 2024-04-13T22:30:58.524 DEBUG:boto:Token: None 2024-04-13T22:30:58.524 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.525 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c847cc0aa4d921-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.527 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.527 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.527 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.527 DEBUG:boto:path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.527 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.528 DEBUG:boto:Method: GET 2024-04-13T22:30:58.528 DEBUG:boto:Path: /vwnfns-61/?prefix=obj74 2024-04-13T22:30:58.528 DEBUG:boto:Data: 2024-04-13T22:30:58.528 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.528 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.528 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.528 DEBUG:boto:Params: {} 2024-04-13T22:30:58.528 DEBUG:boto:Token: None 2024-04-13T22:30:58.528 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a62a68ebbc5221-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.532 DEBUG:boto:b'vwnfns-61obj741000falseobj742024-04-13T22:28:56.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:30:58.533 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.533 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.533 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.533 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.533 DEBUG:boto:Method: GET 2024-04-13T22:30:58.533 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.533 DEBUG:boto:Data: 2024-04-13T22:30:58.533 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.534 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.534 DEBUG:boto:Params: {} 2024-04-13T22:30:58.534 DEBUG:boto:Token: None 2024-04-13T22:30:58.534 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI= 2024-04-13T22:30:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI=', 'Content-Length': '0'} 2024-04-13T22:30:58.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897259973ee8832c-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.582 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.583 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.583 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.583 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.583 DEBUG:boto:Method: GET 2024-04-13T22:30:58.583 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.583 DEBUG:boto:Data: 2024-04-13T22:30:58.583 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.583 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.583 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.583 DEBUG:boto:Params: {} 2024-04-13T22:30:58.584 DEBUG:boto:Token: None 2024-04-13T22:30:58.584 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.584 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI= 2024-04-13T22:30:58.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI=', 'Content-Length': '0'} 2024-04-13T22:30:58.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c793af11c349ecc2-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.586 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.586 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.586 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.586 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.586 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.587 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.587 DEBUG:boto:Data: 2024-04-13T22:30:58.587 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.587 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.587 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.587 DEBUG:boto:Params: {} 2024-04-13T22:30:58.587 DEBUG:boto:Token: None 2024-04-13T22:30:58.587 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.587 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hsYobn9UMZ+MuGTk2fCLcfPQMAw= 2024-04-13T22:30:58.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hsYobn9UMZ+MuGTk2fCLcfPQMAw=', 'Content-Length': '0'} 2024-04-13T22:30:58.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516547c2db00e174-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.589 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.589 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.590 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.590 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.590 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.590 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.590 DEBUG:boto:Data: 2024-04-13T22:30:58.590 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.590 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.590 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.590 DEBUG:boto:Params: {} 2024-04-13T22:30:58.590 DEBUG:boto:Token: None 2024-04-13T22:30:58.591 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.591 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hsYobn9UMZ+MuGTk2fCLcfPQMAw= 2024-04-13T22:30:58.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hsYobn9UMZ+MuGTk2fCLcfPQMAw=', 'Content-Length': '0'} 2024-04-13T22:30:58.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618d2eabf1b2ec60-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:30:58.593 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.593 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.593 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.593 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.593 DEBUG:boto:Method: GET 2024-04-13T22:30:58.593 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.593 DEBUG:boto:Data: 2024-04-13T22:30:58.593 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.593 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.593 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.593 DEBUG:boto:Params: {} 2024-04-13T22:30:58.594 DEBUG:boto:Token: None 2024-04-13T22:30:58.594 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.594 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI= 2024-04-13T22:30:58.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI=', 'Content-Length': '0'} 2024-04-13T22:30:58.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e5ed4e8ae71dc4-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.596 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.596 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.596 DEBUG:boto:path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.596 DEBUG:boto:auth_path=/vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.596 DEBUG:boto:Method: GET 2024-04-13T22:30:58.596 DEBUG:boto:Path: /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.596 DEBUG:boto:Data: 2024-04-13T22:30:58.596 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.596 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.596 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.596 DEBUG:boto:Params: {} 2024-04-13T22:30:58.597 DEBUG:boto:Token: None 2024-04-13T22:30:58.597 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76?versionId=null 2024-04-13T22:30:58.597 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI= 2024-04-13T22:30:58.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7FFa36fwmH+fnj+QZJHbXTjTARI=', 'Content-Length': '0'} 2024-04-13T22:30:58.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f803351155191c17-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.599 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.599 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.599 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.599 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.599 DEBUG:boto:Method: GET 2024-04-13T22:30:58.599 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.599 DEBUG:boto:Data: 2024-04-13T22:30:58.600 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.600 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.600 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.600 DEBUG:boto:Params: {} 2024-04-13T22:30:58.600 DEBUG:boto:Token: None 2024-04-13T22:30:58.600 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.600 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002533d798484fc364-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.602 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.603 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.603 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.603 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.603 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.603 DEBUG:boto:Method: GET 2024-04-13T22:30:58.603 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.603 DEBUG:boto:Data: 2024-04-13T22:30:58.604 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.604 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.604 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.604 DEBUG:boto:Params: {} 2024-04-13T22:30:58.604 DEBUG:boto:Token: None 2024-04-13T22:30:58.604 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b80660147759c9-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.650 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.651 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.651 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.651 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.651 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:58.651 DEBUG:boto:Data: 2024-04-13T22:30:58.651 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.651 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.652 DEBUG:boto:Params: {} 2024-04-13T22:30:58.652 DEBUG:boto:Token: None 2024-04-13T22:30:58.652 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76 2024-04-13T22:30:58.652 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/ObYVYbABzxmqg5GuExMGViUtzo= 2024-04-13T22:30:58.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/ObYVYbABzxmqg5GuExMGViUtzo=', 'Content-Length': '0'} 2024-04-13T22:30:58.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa19fd4b9834933-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.654 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.654 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.654 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.654 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:58.654 DEBUG:boto:Data: 2024-04-13T22:30:58.654 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.654 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.655 DEBUG:boto:Params: {} 2024-04-13T22:30:58.655 DEBUG:boto:Token: None 2024-04-13T22:30:58.655 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76 2024-04-13T22:30:58.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/ObYVYbABzxmqg5GuExMGViUtzo= 2024-04-13T22:30:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/ObYVYbABzxmqg5GuExMGViUtzo=', 'Content-Length': '0'} 2024-04-13T22:30:58.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e551851b03f5cb-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-13T22:30:58.657 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.657 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.657 DEBUG:boto:Method: GET 2024-04-13T22:30:58.657 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:58.657 DEBUG:boto:Data: 2024-04-13T22:30:58.657 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.658 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.658 DEBUG:boto:Params: {} 2024-04-13T22:30:58.658 DEBUG:boto:Token: None 2024-04-13T22:30:58.658 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76 2024-04-13T22:30:58.658 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uz4j9BwgqEAKWcdrQ8jILEm7p4M= 2024-04-13T22:30:58.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uz4j9BwgqEAKWcdrQ8jILEm7p4M=', 'Content-Length': '0'} 2024-04-13T22:30:58.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000527c821e8e949385-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.660 DEBUG:boto:path=/vwnfns-61/obj76 2024-04-13T22:30:58.660 DEBUG:boto:auth_path=/vwnfns-61/obj76 2024-04-13T22:30:58.660 DEBUG:boto:Method: GET 2024-04-13T22:30:58.660 DEBUG:boto:Path: /vwnfns-61/obj76 2024-04-13T22:30:58.660 DEBUG:boto:Data: 2024-04-13T22:30:58.660 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.660 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.660 DEBUG:boto:Params: {} 2024-04-13T22:30:58.661 DEBUG:boto:Token: None 2024-04-13T22:30:58.661 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj76 2024-04-13T22:30:58.661 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uz4j9BwgqEAKWcdrQ8jILEm7p4M= 2024-04-13T22:30:58.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uz4j9BwgqEAKWcdrQ8jILEm7p4M=', 'Content-Length': '0'} 2024-04-13T22:30:58.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c3d18d68d5fa22d-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.663 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.663 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.663 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.663 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.663 DEBUG:boto:Method: GET 2024-04-13T22:30:58.663 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.663 DEBUG:boto:Data: 2024-04-13T22:30:58.664 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.664 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.664 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.664 DEBUG:boto:Params: {} 2024-04-13T22:30:58.664 DEBUG:boto:Token: None 2024-04-13T22:30:58.664 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f8e0cbc2f9b207-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.666 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.667 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.667 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.667 DEBUG:boto:path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.667 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.667 DEBUG:boto:Method: GET 2024-04-13T22:30:58.667 DEBUG:boto:Path: /vwnfns-61/?prefix=obj76 2024-04-13T22:30:58.667 DEBUG:boto:Data: 2024-04-13T22:30:58.667 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.667 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.668 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.668 DEBUG:boto:Params: {} 2024-04-13T22:30:58.668 DEBUG:boto:Token: None 2024-04-13T22:30:58.668 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.668 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dedc2d2716cbac1-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.671 DEBUG:boto:b'vwnfns-61obj761000falseobj762024-04-13T22:21:13.271Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:30:58.672 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.672 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.672 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.672 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.672 DEBUG:boto:Method: GET 2024-04-13T22:30:58.672 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.672 DEBUG:boto:Data: 2024-04-13T22:30:58.673 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.673 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.673 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.673 DEBUG:boto:Params: {} 2024-04-13T22:30:58.673 DEBUG:boto:Token: None 2024-04-13T22:30:58.673 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.673 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw= 2024-04-13T22:30:58.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw=', 'Content-Length': '0'} 2024-04-13T22:30:58.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000433119858c423e4-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.675 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.675 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.675 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.676 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.676 DEBUG:boto:Method: GET 2024-04-13T22:30:58.676 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.676 DEBUG:boto:Data: 2024-04-13T22:30:58.676 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.676 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.676 DEBUG:boto:Params: {} 2024-04-13T22:30:58.676 DEBUG:boto:Token: None 2024-04-13T22:30:58.676 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.677 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw= 2024-04-13T22:30:58.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw=', 'Content-Length': '0'} 2024-04-13T22:30:58.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab28a53a464f2d84-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.722 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.723 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.723 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.723 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.723 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.723 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.723 DEBUG:boto:Data: 2024-04-13T22:30:58.723 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.723 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.723 DEBUG:boto:Params: {} 2024-04-13T22:30:58.724 DEBUG:boto:Token: None 2024-04-13T22:30:58.724 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/N4fggBuAYlrK8RHAhefXzsmI8I= 2024-04-13T22:30:58.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/N4fggBuAYlrK8RHAhefXzsmI8I=', 'Content-Length': '0'} 2024-04-13T22:30:58.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030fb63524c7ea2e3-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.726 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.726 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.726 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.726 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.726 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.726 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.726 DEBUG:boto:Data: 2024-04-13T22:30:58.726 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.726 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.726 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.726 DEBUG:boto:Params: {} 2024-04-13T22:30:58.727 DEBUG:boto:Token: None 2024-04-13T22:30:58.727 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/N4fggBuAYlrK8RHAhefXzsmI8I= 2024-04-13T22:30:58.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/N4fggBuAYlrK8RHAhefXzsmI8I=', 'Content-Length': '0'} 2024-04-13T22:30:58.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000508698ba2710acf5-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:30:58.729 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.729 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.729 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.729 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.729 DEBUG:boto:Method: GET 2024-04-13T22:30:58.730 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.730 DEBUG:boto:Data: 2024-04-13T22:30:58.730 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.730 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.730 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.730 DEBUG:boto:Params: {} 2024-04-13T22:30:58.730 DEBUG:boto:Token: None 2024-04-13T22:30:58.730 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.730 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw= 2024-04-13T22:30:58.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw=', 'Content-Length': '0'} 2024-04-13T22:30:58.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4bccc875b47f44-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.732 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.732 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.733 DEBUG:boto:path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.733 DEBUG:boto:auth_path=/vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.733 DEBUG:boto:Method: GET 2024-04-13T22:30:58.733 DEBUG:boto:Path: /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.733 DEBUG:boto:Data: 2024-04-13T22:30:58.733 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.733 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.733 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.733 DEBUG:boto:Params: {} 2024-04-13T22:30:58.733 DEBUG:boto:Token: None 2024-04-13T22:30:58.733 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85?versionId=null 2024-04-13T22:30:58.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw= 2024-04-13T22:30:58.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ok5HacsZ2/uBO+hUU2TodYrQhJw=', 'Content-Length': '0'} 2024-04-13T22:30:58.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6e29fd8d57a938-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.736 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.736 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.736 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.736 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.736 DEBUG:boto:Method: GET 2024-04-13T22:30:58.736 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.736 DEBUG:boto:Data: 2024-04-13T22:30:58.736 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.736 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.736 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.737 DEBUG:boto:Params: {} 2024-04-13T22:30:58.737 DEBUG:boto:Token: None 2024-04-13T22:30:58.737 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.737 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bfc16653654d89b-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.739 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.739 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.739 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.740 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.740 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.740 DEBUG:boto:Method: GET 2024-04-13T22:30:58.740 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.740 DEBUG:boto:Data: 2024-04-13T22:30:58.740 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.740 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.740 DEBUG:boto:Params: {} 2024-04-13T22:30:58.740 DEBUG:boto:Token: None 2024-04-13T22:30:58.741 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e502f11ab45ed13-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.744 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.745 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.745 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.745 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.745 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:58.745 DEBUG:boto:Data: 2024-04-13T22:30:58.745 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.745 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.745 DEBUG:boto:Params: {} 2024-04-13T22:30:58.745 DEBUG:boto:Token: None 2024-04-13T22:30:58.746 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85 2024-04-13T22:30:58.746 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0lmo2zZGOEkXtB7528uFKzXYioQ= 2024-04-13T22:30:58.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0lmo2zZGOEkXtB7528uFKzXYioQ=', 'Content-Length': '0'} 2024-04-13T22:30:58.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042ef0dc12d25080a-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.748 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.748 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.748 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.748 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:58.748 DEBUG:boto:Data: 2024-04-13T22:30:58.748 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.748 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.748 DEBUG:boto:Params: {} 2024-04-13T22:30:58.748 DEBUG:boto:Token: None 2024-04-13T22:30:58.749 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85 2024-04-13T22:30:58.749 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0lmo2zZGOEkXtB7528uFKzXYioQ= 2024-04-13T22:30:58.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0lmo2zZGOEkXtB7528uFKzXYioQ=', 'Content-Length': '0'} 2024-04-13T22:30:58.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6689dc442fbaef-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-13T22:30:58.750 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.751 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.751 DEBUG:boto:Method: GET 2024-04-13T22:30:58.751 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:58.751 DEBUG:boto:Data: 2024-04-13T22:30:58.751 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.751 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.751 DEBUG:boto:Params: {} 2024-04-13T22:30:58.751 DEBUG:boto:Token: None 2024-04-13T22:30:58.751 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85 2024-04-13T22:30:58.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7PfkwPxCEbv5GMLggDiBONAXwd8= 2024-04-13T22:30:58.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7PfkwPxCEbv5GMLggDiBONAXwd8=', 'Content-Length': '0'} 2024-04-13T22:30:58.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7c25f5b750f090-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.794 DEBUG:boto:path=/vwnfns-61/obj85 2024-04-13T22:30:58.795 DEBUG:boto:auth_path=/vwnfns-61/obj85 2024-04-13T22:30:58.795 DEBUG:boto:Method: GET 2024-04-13T22:30:58.795 DEBUG:boto:Path: /vwnfns-61/obj85 2024-04-13T22:30:58.795 DEBUG:boto:Data: 2024-04-13T22:30:58.795 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.795 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.795 DEBUG:boto:Params: {} 2024-04-13T22:30:58.795 DEBUG:boto:Token: None 2024-04-13T22:30:58.796 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj85 2024-04-13T22:30:58.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7PfkwPxCEbv5GMLggDiBONAXwd8= 2024-04-13T22:30:58.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7PfkwPxCEbv5GMLggDiBONAXwd8=', 'Content-Length': '0'} 2024-04-13T22:30:58.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:18:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de6b6be9434922a1-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.798 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.798 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.798 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.798 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.798 DEBUG:boto:Method: GET 2024-04-13T22:30:58.798 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.798 DEBUG:boto:Data: 2024-04-13T22:30:58.798 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.798 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.799 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.799 DEBUG:boto:Params: {} 2024-04-13T22:30:58.799 DEBUG:boto:Token: None 2024-04-13T22:30:58.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.799 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a5e9efaeab6bf07-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.801 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.802 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.802 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.802 DEBUG:boto:path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.802 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.802 DEBUG:boto:Method: GET 2024-04-13T22:30:58.802 DEBUG:boto:Path: /vwnfns-61/?prefix=obj85 2024-04-13T22:30:58.802 DEBUG:boto:Data: 2024-04-13T22:30:58.803 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.803 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.803 DEBUG:boto:Params: {} 2024-04-13T22:30:58.803 DEBUG:boto:Token: None 2024-04-13T22:30:58.803 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054521876e241547c-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.807 DEBUG:boto:b'vwnfns-61obj851000falseobj852024-04-13T22:18:11.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:30:58.808 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.808 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.808 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.808 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.808 DEBUG:boto:Method: GET 2024-04-13T22:30:58.808 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.808 DEBUG:boto:Data: 2024-04-13T22:30:58.808 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.808 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.808 DEBUG:boto:Params: {} 2024-04-13T22:30:58.809 DEBUG:boto:Token: None 2024-04-13T22:30:58.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00= 2024-04-13T22:30:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00=', 'Content-Length': '0'} 2024-04-13T22:30:58.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102fb682982fdb79-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.811 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.811 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.811 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.811 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.811 DEBUG:boto:Method: GET 2024-04-13T22:30:58.811 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.811 DEBUG:boto:Data: 2024-04-13T22:30:58.811 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.812 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.812 DEBUG:boto:Params: {} 2024-04-13T22:30:58.812 DEBUG:boto:Token: None 2024-04-13T22:30:58.812 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00= 2024-04-13T22:30:58.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00=', 'Content-Length': '0'} 2024-04-13T22:30:58.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74b0e88e4261e65-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.854 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.855 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.855 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.855 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.855 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.855 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.855 DEBUG:boto:Data: 2024-04-13T22:30:58.855 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.855 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.855 DEBUG:boto:Params: {} 2024-04-13T22:30:58.856 DEBUG:boto:Token: None 2024-04-13T22:30:58.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Go9iI2FbpxqDvXW3rtxuRwX0VBo= 2024-04-13T22:30:58.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Go9iI2FbpxqDvXW3rtxuRwX0VBo=', 'Content-Length': '0'} 2024-04-13T22:30:58.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb721ec214bd392f-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.858 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.858 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.858 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.858 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.858 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.858 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.859 DEBUG:boto:Data: 2024-04-13T22:30:58.859 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.859 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.859 DEBUG:boto:Params: {} 2024-04-13T22:30:58.859 DEBUG:boto:Token: None 2024-04-13T22:30:58.859 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Go9iI2FbpxqDvXW3rtxuRwX0VBo= 2024-04-13T22:30:58.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Go9iI2FbpxqDvXW3rtxuRwX0VBo=', 'Content-Length': '0'} 2024-04-13T22:30:58.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002295d5a72ba4de0b-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:30:58.861 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.861 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.862 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.862 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.862 DEBUG:boto:Method: GET 2024-04-13T22:30:58.862 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.862 DEBUG:boto:Data: 2024-04-13T22:30:58.862 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.862 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.862 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.862 DEBUG:boto:Params: {} 2024-04-13T22:30:58.862 DEBUG:boto:Token: None 2024-04-13T22:30:58.863 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00= 2024-04-13T22:30:58.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00=', 'Content-Length': '0'} 2024-04-13T22:30:58.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9afa6018ebc932e-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.865 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.865 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.865 DEBUG:boto:path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.865 DEBUG:boto:auth_path=/vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.865 DEBUG:boto:Method: GET 2024-04-13T22:30:58.865 DEBUG:boto:Path: /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.865 DEBUG:boto:Data: 2024-04-13T22:30:58.865 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.865 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.865 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.865 DEBUG:boto:Params: {} 2024-04-13T22:30:58.866 DEBUG:boto:Token: None 2024-04-13T22:30:58.866 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87?versionId=null 2024-04-13T22:30:58.866 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00= 2024-04-13T22:30:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWbEbIJ6zm9IXQofPVDkYLdKV00=', 'Content-Length': '0'} 2024-04-13T22:30:58.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc0bf3f319795a8-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.868 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.868 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.868 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.868 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.868 DEBUG:boto:Method: GET 2024-04-13T22:30:58.868 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.868 DEBUG:boto:Data: 2024-04-13T22:30:58.868 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.869 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.869 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.869 DEBUG:boto:Params: {} 2024-04-13T22:30:58.869 DEBUG:boto:Token: None 2024-04-13T22:30:58.869 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.869 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001f7d4c96386a0c4-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.871 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.872 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.872 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.872 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.872 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.872 DEBUG:boto:Method: GET 2024-04-13T22:30:58.872 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.872 DEBUG:boto:Data: 2024-04-13T22:30:58.872 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.872 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.873 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.873 DEBUG:boto:Params: {} 2024-04-13T22:30:58.873 DEBUG:boto:Token: None 2024-04-13T22:30:58.873 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.873 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d92b1c2bd5e9bc3-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.876 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.877 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.877 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.877 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.877 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:58.877 DEBUG:boto:Data: 2024-04-13T22:30:58.877 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.877 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.878 DEBUG:boto:Params: {} 2024-04-13T22:30:58.878 DEBUG:boto:Token: None 2024-04-13T22:30:58.878 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87 2024-04-13T22:30:58.878 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c5NVo06uFrTEaFGyVnMjoJNeEvE= 2024-04-13T22:30:58.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c5NVo06uFrTEaFGyVnMjoJNeEvE=', 'Content-Length': '0'} 2024-04-13T22:30:58.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b32c2d3767797d-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.880 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.880 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.880 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.880 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:58.880 DEBUG:boto:Data: 2024-04-13T22:30:58.880 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.880 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.880 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.880 DEBUG:boto:Params: {} 2024-04-13T22:30:58.880 DEBUG:boto:Token: None 2024-04-13T22:30:58.881 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87 2024-04-13T22:30:58.881 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:c5NVo06uFrTEaFGyVnMjoJNeEvE= 2024-04-13T22:30:58.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c5NVo06uFrTEaFGyVnMjoJNeEvE=', 'Content-Length': '0'} 2024-04-13T22:30:58.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b871e8d79c9228-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-13T22:30:58.883 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.883 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.883 DEBUG:boto:Method: GET 2024-04-13T22:30:58.883 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:58.883 DEBUG:boto:Data: 2024-04-13T22:30:58.883 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.883 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.883 DEBUG:boto:Params: {} 2024-04-13T22:30:58.883 DEBUG:boto:Token: None 2024-04-13T22:30:58.883 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87 2024-04-13T22:30:58.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cvXEAaCupdP11o+Tf7Djcgcd0oE= 2024-04-13T22:30:58.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cvXEAaCupdP11o+Tf7Djcgcd0oE=', 'Content-Length': '0'} 2024-04-13T22:30:58.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042332d0652d6603b-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.886 DEBUG:boto:path=/vwnfns-61/obj87 2024-04-13T22:30:58.886 DEBUG:boto:auth_path=/vwnfns-61/obj87 2024-04-13T22:30:58.886 DEBUG:boto:Method: GET 2024-04-13T22:30:58.886 DEBUG:boto:Path: /vwnfns-61/obj87 2024-04-13T22:30:58.886 DEBUG:boto:Data: 2024-04-13T22:30:58.886 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.886 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.886 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.886 DEBUG:boto:Params: {} 2024-04-13T22:30:58.887 DEBUG:boto:Token: None 2024-04-13T22:30:58.887 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj87 2024-04-13T22:30:58.887 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cvXEAaCupdP11o+Tf7Djcgcd0oE= 2024-04-13T22:30:58.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cvXEAaCupdP11o+Tf7Djcgcd0oE=', 'Content-Length': '0'} 2024-04-13T22:30:58.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:25:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee69d17710ca3815-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.931 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.931 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.931 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.931 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.931 DEBUG:boto:Method: GET 2024-04-13T22:30:58.932 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.932 DEBUG:boto:Data: 2024-04-13T22:30:58.932 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.932 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.933 DEBUG:boto:Params: {} 2024-04-13T22:30:58.933 DEBUG:boto:Token: None 2024-04-13T22:30:58.933 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005547bf745db7533f-00661b07a2-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.936 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.937 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:58.937 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:58.937 DEBUG:boto:path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.937 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.938 DEBUG:boto:Method: GET 2024-04-13T22:30:58.938 DEBUG:boto:Path: /vwnfns-61/?prefix=obj87 2024-04-13T22:30:58.938 DEBUG:boto:Data: 2024-04-13T22:30:58.938 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.938 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.939 DEBUG:boto:Params: {} 2024-04-13T22:30:58.939 DEBUG:boto:Token: None 2024-04-13T22:30:58.939 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/ 2024-04-13T22:30:58.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI= 2024-04-13T22:30:58.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qLJ+BT9FuaVF5LRv4RS94lH9MSI=', 'Content-Length': '0'} 2024-04-13T22:30:58.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b4ee9846650041d-00661b07a2-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.944 DEBUG:boto:b'vwnfns-61obj871000falseobj872024-04-13T22:25:33.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:58.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:30:58.945 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:58.945 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:58.945 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.946 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.946 DEBUG:boto:Method: GET 2024-04-13T22:30:58.946 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.946 DEBUG:boto:Data: 2024-04-13T22:30:58.946 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.946 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.946 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.946 DEBUG:boto:Params: {} 2024-04-13T22:30:58.946 DEBUG:boto:Token: None 2024-04-13T22:30:58.947 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.947 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE= 2024-04-13T22:30:58.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE=', 'Content-Length': '0'} 2024-04-13T22:30:58.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046426346c7ff148a-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.949 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:58.949 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:58.949 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.949 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.949 DEBUG:boto:Method: GET 2024-04-13T22:30:58.949 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.949 DEBUG:boto:Data: 2024-04-13T22:30:58.949 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.949 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.949 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.949 DEBUG:boto:Params: {} 2024-04-13T22:30:58.950 DEBUG:boto:Token: None 2024-04-13T22:30:58.950 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.950 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE= 2024-04-13T22:30:58.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE=', 'Content-Length': '0'} 2024-04-13T22:30:58.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7a7fa35eb9e043-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.952 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:58.953 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:58.953 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.953 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.953 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.954 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.954 DEBUG:boto:Data: 2024-04-13T22:30:58.954 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.954 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.955 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.955 DEBUG:boto:Params: {} 2024-04-13T22:30:58.955 DEBUG:boto:Token: None 2024-04-13T22:30:58.955 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0BHy2R+KRKcSLNRL/odxuqkaMws= 2024-04-13T22:30:58.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0BHy2R+KRKcSLNRL/odxuqkaMws=', 'Content-Length': '0'} 2024-04-13T22:30:58.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f6e27e07454d17-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.959 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:58.959 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:58.959 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.960 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.960 DEBUG:boto:Method: HEAD 2024-04-13T22:30:58.960 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.960 DEBUG:boto:Data: 2024-04-13T22:30:58.960 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.961 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.961 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.961 DEBUG:boto:Params: {} 2024-04-13T22:30:58.961 DEBUG:boto:Token: None 2024-04-13T22:30:58.961 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.961 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0BHy2R+KRKcSLNRL/odxuqkaMws= 2024-04-13T22:30:58.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0BHy2R+KRKcSLNRL/odxuqkaMws=', 'Content-Length': '0'} 2024-04-13T22:30:58.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516eb5bbb100d4d2-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:30:58.963 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:58.964 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:58.964 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.964 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.964 DEBUG:boto:Method: GET 2024-04-13T22:30:58.964 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.964 DEBUG:boto:Data: 2024-04-13T22:30:58.965 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:58.965 DEBUG:boto:Port: 8001 2024-04-13T22:30:58.965 DEBUG:boto:Params: {} 2024-04-13T22:30:58.965 DEBUG:boto:Token: None 2024-04-13T22:30:58.965 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.965 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE= 2024-04-13T22:30:58.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE=', 'Content-Length': '0'} 2024-04-13T22:30:58.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa13f32c7f725ad-00661b07a2-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:58.967 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:58.967 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:58.967 DEBUG:boto:path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.967 DEBUG:boto:auth_path=/vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.968 DEBUG:boto:Method: GET 2024-04-13T22:30:58.968 DEBUG:boto:Path: /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.968 DEBUG:boto:Data: 2024-04-13T22:30:58.968 DEBUG:boto:Headers: {} 2024-04-13T22:30:58.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:58.968 DEBUG:boto:Port: 8000 2024-04-13T22:30:58.968 DEBUG:boto:Params: {} 2024-04-13T22:30:58.968 DEBUG:boto:Token: None 2024-04-13T22:30:58.968 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:58 GMT /vwnfns-61/obj92?versionId=null 2024-04-13T22:30:58.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE= 2024-04-13T22:30:58.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j+tKC/jTXo7C+Y4GM3WZqH0BwNE=', 'Content-Length': '0'} 2024-04-13T22:30:58.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da23e35debbc436a-00661b07a2-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.010 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.011 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.011 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.011 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.011 DEBUG:boto:Method: GET 2024-04-13T22:30:59.011 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.011 DEBUG:boto:Data: 2024-04-13T22:30:59.011 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.011 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.011 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.012 DEBUG:boto:Params: {} 2024-04-13T22:30:59.012 DEBUG:boto:Token: None 2024-04-13T22:30:59.012 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.012 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e188f7bfc8625554-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.015 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.015 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.015 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.016 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.016 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.016 DEBUG:boto:Method: GET 2024-04-13T22:30:59.016 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.016 DEBUG:boto:Data: 2024-04-13T22:30:59.016 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.017 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.017 DEBUG:boto:Params: {} 2024-04-13T22:30:59.017 DEBUG:boto:Token: None 2024-04-13T22:30:59.017 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e0e8c2b21ef278e-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.021 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.021 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:59.021 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:59.021 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.021 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:59.021 DEBUG:boto:Data: 2024-04-13T22:30:59.022 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.022 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.022 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.022 DEBUG:boto:Params: {} 2024-04-13T22:30:59.022 DEBUG:boto:Token: None 2024-04-13T22:30:59.022 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj92 2024-04-13T22:30:59.022 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/qTY6WF0agQLDRwybs8UzLEbFfY= 2024-04-13T22:30:59.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/qTY6WF0agQLDRwybs8UzLEbFfY=', 'Content-Length': '0'} 2024-04-13T22:30:59.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f6459968cf6d23-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.024 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:59.025 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:59.025 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.025 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:59.025 DEBUG:boto:Data: 2024-04-13T22:30:59.026 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.026 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.026 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.026 DEBUG:boto:Params: {} 2024-04-13T22:30:59.026 DEBUG:boto:Token: None 2024-04-13T22:30:59.026 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj92 2024-04-13T22:30:59.027 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/qTY6WF0agQLDRwybs8UzLEbFfY= 2024-04-13T22:30:59.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/qTY6WF0agQLDRwybs8UzLEbFfY=', 'Content-Length': '0'} 2024-04-13T22:30:59.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fbbac3ed564bb78-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-13T22:30:59.028 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:59.029 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:59.029 DEBUG:boto:Method: GET 2024-04-13T22:30:59.029 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:59.029 DEBUG:boto:Data: 2024-04-13T22:30:59.029 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.029 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.029 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.029 DEBUG:boto:Params: {} 2024-04-13T22:30:59.029 DEBUG:boto:Token: None 2024-04-13T22:30:59.030 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj92 2024-04-13T22:30:59.030 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nSXz7oT14C9U0gzHm3vWggbN4as= 2024-04-13T22:30:59.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nSXz7oT14C9U0gzHm3vWggbN4as=', 'Content-Length': '0'} 2024-04-13T22:30:59.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b95bbb9f05896447-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.074 DEBUG:boto:path=/vwnfns-61/obj92 2024-04-13T22:30:59.075 DEBUG:boto:auth_path=/vwnfns-61/obj92 2024-04-13T22:30:59.075 DEBUG:boto:Method: GET 2024-04-13T22:30:59.075 DEBUG:boto:Path: /vwnfns-61/obj92 2024-04-13T22:30:59.075 DEBUG:boto:Data: 2024-04-13T22:30:59.075 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.076 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.076 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.076 DEBUG:boto:Params: {} 2024-04-13T22:30:59.076 DEBUG:boto:Token: None 2024-04-13T22:30:59.076 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj92 2024-04-13T22:30:59.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nSXz7oT14C9U0gzHm3vWggbN4as= 2024-04-13T22:30:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nSXz7oT14C9U0gzHm3vWggbN4as=', 'Content-Length': '0'} 2024-04-13T22:30:59.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:22:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9684363d2a22037-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.080 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.080 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.080 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.080 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.080 DEBUG:boto:Method: GET 2024-04-13T22:30:59.081 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.081 DEBUG:boto:Data: 2024-04-13T22:30:59.081 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.081 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.082 DEBUG:boto:Params: {} 2024-04-13T22:30:59.082 DEBUG:boto:Token: None 2024-04-13T22:30:59.082 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b523b85fa3c2b00-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.085 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.086 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.086 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.087 DEBUG:boto:path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.087 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.087 DEBUG:boto:Method: GET 2024-04-13T22:30:59.087 DEBUG:boto:Path: /vwnfns-61/?prefix=obj92 2024-04-13T22:30:59.087 DEBUG:boto:Data: 2024-04-13T22:30:59.087 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.088 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.088 DEBUG:boto:Params: {} 2024-04-13T22:30:59.088 DEBUG:boto:Token: None 2024-04-13T22:30:59.088 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f231bb11751dd21f-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.138 DEBUG:boto:b'vwnfns-61obj921000falseobj922024-04-13T22:22:50.647Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:30:59.139 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.139 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.139 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.139 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.139 DEBUG:boto:Method: GET 2024-04-13T22:30:59.140 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.140 DEBUG:boto:Data: 2024-04-13T22:30:59.140 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.140 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.140 DEBUG:boto:Params: {} 2024-04-13T22:30:59.140 DEBUG:boto:Token: None 2024-04-13T22:30:59.140 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE= 2024-04-13T22:30:59.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE=', 'Content-Length': '0'} 2024-04-13T22:30:59.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b666c01d4e4b6b-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.143 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.143 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.144 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.144 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.144 DEBUG:boto:Method: GET 2024-04-13T22:30:59.144 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.144 DEBUG:boto:Data: 2024-04-13T22:30:59.144 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.145 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.145 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.145 DEBUG:boto:Params: {} 2024-04-13T22:30:59.145 DEBUG:boto:Token: None 2024-04-13T22:30:59.146 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.146 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE= 2024-04-13T22:30:59.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE=', 'Content-Length': '0'} 2024-04-13T22:30:59.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048494cace76e01c0-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.148 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.148 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.149 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.149 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.149 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.149 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.149 DEBUG:boto:Data: 2024-04-13T22:30:59.149 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.150 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.150 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.150 DEBUG:boto:Params: {} 2024-04-13T22:30:59.150 DEBUG:boto:Token: None 2024-04-13T22:30:59.151 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.151 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y6qnGXR4uLlfBNylVcH9yY0lAac= 2024-04-13T22:30:59.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y6qnGXR4uLlfBNylVcH9yY0lAac=', 'Content-Length': '0'} 2024-04-13T22:30:59.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ff19ea7e0133dd-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.153 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.153 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.153 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.153 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.154 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.154 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.154 DEBUG:boto:Data: 2024-04-13T22:30:59.154 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.154 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.154 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.154 DEBUG:boto:Params: {} 2024-04-13T22:30:59.154 DEBUG:boto:Token: None 2024-04-13T22:30:59.154 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.155 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:y6qnGXR4uLlfBNylVcH9yY0lAac= 2024-04-13T22:30:59.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:y6qnGXR4uLlfBNylVcH9yY0lAac=', 'Content-Length': '0'} 2024-04-13T22:30:59.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b808b1012519b8e5-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:30:59.157 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.157 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.157 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.157 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.157 DEBUG:boto:Method: GET 2024-04-13T22:30:59.157 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.157 DEBUG:boto:Data: 2024-04-13T22:30:59.157 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.157 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.157 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.158 DEBUG:boto:Params: {} 2024-04-13T22:30:59.158 DEBUG:boto:Token: None 2024-04-13T22:30:59.158 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.158 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE= 2024-04-13T22:30:59.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE=', 'Content-Length': '0'} 2024-04-13T22:30:59.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002631639e50415024-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.160 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.160 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.160 DEBUG:boto:path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.160 DEBUG:boto:auth_path=/vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.160 DEBUG:boto:Method: GET 2024-04-13T22:30:59.161 DEBUG:boto:Path: /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.161 DEBUG:boto:Data: 2024-04-13T22:30:59.161 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.161 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.161 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.161 DEBUG:boto:Params: {} 2024-04-13T22:30:59.161 DEBUG:boto:Token: None 2024-04-13T22:30:59.161 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93?versionId=null 2024-04-13T22:30:59.161 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE= 2024-04-13T22:30:59.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z19U5JOEVOu0iO5s9ykYttp1XqE=', 'Content-Length': '0'} 2024-04-13T22:30:59.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25ea2f85cc3be04-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.163 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.163 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.163 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.163 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.164 DEBUG:boto:Method: GET 2024-04-13T22:30:59.164 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.164 DEBUG:boto:Data: 2024-04-13T22:30:59.165 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.165 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.165 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.165 DEBUG:boto:Params: {} 2024-04-13T22:30:59.165 DEBUG:boto:Token: None 2024-04-13T22:30:59.165 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.165 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba67cf8f015520d-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.168 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.168 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.168 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.168 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.168 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.169 DEBUG:boto:Method: GET 2024-04-13T22:30:59.169 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.169 DEBUG:boto:Data: 2024-04-13T22:30:59.169 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.169 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.169 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.169 DEBUG:boto:Params: {} 2024-04-13T22:30:59.169 DEBUG:boto:Token: None 2024-04-13T22:30:59.169 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.169 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0b4b8d3aed4fdca-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.173 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.173 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.173 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.174 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.174 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:59.174 DEBUG:boto:Data: 2024-04-13T22:30:59.174 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.174 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.174 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.174 DEBUG:boto:Params: {} 2024-04-13T22:30:59.174 DEBUG:boto:Token: None 2024-04-13T22:30:59.174 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93 2024-04-13T22:30:59.175 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rfROz4QAyZ3Gi9U56neYEks+pLI= 2024-04-13T22:30:59.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rfROz4QAyZ3Gi9U56neYEks+pLI=', 'Content-Length': '0'} 2024-04-13T22:30:59.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66d5631eeb9ece3-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.178 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.178 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.178 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.178 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:59.178 DEBUG:boto:Data: 2024-04-13T22:30:59.178 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.178 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.179 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.179 DEBUG:boto:Params: {} 2024-04-13T22:30:59.179 DEBUG:boto:Token: None 2024-04-13T22:30:59.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93 2024-04-13T22:30:59.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rfROz4QAyZ3Gi9U56neYEks+pLI= 2024-04-13T22:30:59.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rfROz4QAyZ3Gi9U56neYEks+pLI=', 'Content-Length': '0'} 2024-04-13T22:30:59.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbbbfb1c75d8f97e-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-13T22:30:59.181 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.181 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.181 DEBUG:boto:Method: GET 2024-04-13T22:30:59.181 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:59.182 DEBUG:boto:Data: 2024-04-13T22:30:59.182 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.182 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.182 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.182 DEBUG:boto:Params: {} 2024-04-13T22:30:59.182 DEBUG:boto:Token: None 2024-04-13T22:30:59.182 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93 2024-04-13T22:30:59.182 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MZFLzHLaO/6RHgqg21ezRTRqF44= 2024-04-13T22:30:59.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MZFLzHLaO/6RHgqg21ezRTRqF44=', 'Content-Length': '0'} 2024-04-13T22:30:59.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d72fd4c3af409a-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.226 DEBUG:boto:path=/vwnfns-61/obj93 2024-04-13T22:30:59.227 DEBUG:boto:auth_path=/vwnfns-61/obj93 2024-04-13T22:30:59.227 DEBUG:boto:Method: GET 2024-04-13T22:30:59.227 DEBUG:boto:Path: /vwnfns-61/obj93 2024-04-13T22:30:59.227 DEBUG:boto:Data: 2024-04-13T22:30:59.227 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.227 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.227 DEBUG:boto:Params: {} 2024-04-13T22:30:59.227 DEBUG:boto:Token: None 2024-04-13T22:30:59.228 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj93 2024-04-13T22:30:59.228 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MZFLzHLaO/6RHgqg21ezRTRqF44= 2024-04-13T22:30:59.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MZFLzHLaO/6RHgqg21ezRTRqF44=', 'Content-Length': '0'} 2024-04-13T22:30:59.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:27:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001fb48f8a8fa8503-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.230 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.230 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.230 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.230 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.230 DEBUG:boto:Method: GET 2024-04-13T22:30:59.231 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.231 DEBUG:boto:Data: 2024-04-13T22:30:59.231 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.231 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.231 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.231 DEBUG:boto:Params: {} 2024-04-13T22:30:59.231 DEBUG:boto:Token: None 2024-04-13T22:30:59.231 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.231 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7d33bbd9aad739-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.234 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.234 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.234 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.235 DEBUG:boto:path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.235 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.235 DEBUG:boto:Method: GET 2024-04-13T22:30:59.235 DEBUG:boto:Path: /vwnfns-61/?prefix=obj93 2024-04-13T22:30:59.235 DEBUG:boto:Data: 2024-04-13T22:30:59.235 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.235 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.235 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.235 DEBUG:boto:Params: {} 2024-04-13T22:30:59.235 DEBUG:boto:Token: None 2024-04-13T22:30:59.236 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.236 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ed7d7575dc2829-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.239 DEBUG:boto:b'vwnfns-61obj931000falseobj932024-04-13T22:27:52.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:30:59.239 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.239 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.240 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.240 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.240 DEBUG:boto:Method: GET 2024-04-13T22:30:59.240 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.240 DEBUG:boto:Data: 2024-04-13T22:30:59.240 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.240 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.240 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.240 DEBUG:boto:Params: {} 2024-04-13T22:30:59.240 DEBUG:boto:Token: None 2024-04-13T22:30:59.240 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.241 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA= 2024-04-13T22:30:59.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA=', 'Content-Length': '0'} 2024-04-13T22:30:59.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc81eb43f8f5a2cc-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.243 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.243 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.243 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.243 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.243 DEBUG:boto:Method: GET 2024-04-13T22:30:59.243 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.243 DEBUG:boto:Data: 2024-04-13T22:30:59.243 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.243 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.244 DEBUG:boto:Params: {} 2024-04-13T22:30:59.244 DEBUG:boto:Token: None 2024-04-13T22:30:59.244 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA= 2024-04-13T22:30:59.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA=', 'Content-Length': '0'} 2024-04-13T22:30:59.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc2f675d47a61bb-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.246 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.246 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.246 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.246 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.246 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.246 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.247 DEBUG:boto:Data: 2024-04-13T22:30:59.247 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.247 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.247 DEBUG:boto:Params: {} 2024-04-13T22:30:59.247 DEBUG:boto:Token: None 2024-04-13T22:30:59.247 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BPgfgSEiVtOEaUU96eTneTTervg= 2024-04-13T22:30:59.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BPgfgSEiVtOEaUU96eTneTTervg=', 'Content-Length': '0'} 2024-04-13T22:30:59.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e6007d30bc63db-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.249 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.250 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.250 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.250 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.250 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.250 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.250 DEBUG:boto:Data: 2024-04-13T22:30:59.250 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.251 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.251 DEBUG:boto:Params: {} 2024-04-13T22:30:59.251 DEBUG:boto:Token: None 2024-04-13T22:30:59.251 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BPgfgSEiVtOEaUU96eTneTTervg= 2024-04-13T22:30:59.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BPgfgSEiVtOEaUU96eTneTTervg=', 'Content-Length': '0'} 2024-04-13T22:30:59.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000589f88d0c121e0ea-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:30:59.253 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.253 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.253 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.254 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.254 DEBUG:boto:Method: GET 2024-04-13T22:30:59.254 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.254 DEBUG:boto:Data: 2024-04-13T22:30:59.254 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.254 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.254 DEBUG:boto:Params: {} 2024-04-13T22:30:59.254 DEBUG:boto:Token: None 2024-04-13T22:30:59.255 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA= 2024-04-13T22:30:59.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA=', 'Content-Length': '0'} 2024-04-13T22:30:59.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9a73fc6c0394ce-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.257 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.257 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.257 DEBUG:boto:path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.257 DEBUG:boto:auth_path=/vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.258 DEBUG:boto:Method: GET 2024-04-13T22:30:59.258 DEBUG:boto:Path: /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.258 DEBUG:boto:Data: 2024-04-13T22:30:59.258 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.258 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.259 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.259 DEBUG:boto:Params: {} 2024-04-13T22:30:59.259 DEBUG:boto:Token: None 2024-04-13T22:30:59.259 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95?versionId=null 2024-04-13T22:30:59.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA= 2024-04-13T22:30:59.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rtOPg2EQ4HXSXx2IeJWVytf31AA=', 'Content-Length': '0'} 2024-04-13T22:30:59.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea3fbf3e55563ab-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.261 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.261 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.261 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.261 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.262 DEBUG:boto:Method: GET 2024-04-13T22:30:59.262 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.262 DEBUG:boto:Data: 2024-04-13T22:30:59.262 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.262 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.262 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.262 DEBUG:boto:Params: {} 2024-04-13T22:30:59.262 DEBUG:boto:Token: None 2024-04-13T22:30:59.262 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007ace427b0ba7370-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.265 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.265 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.265 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.265 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.265 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.266 DEBUG:boto:Method: GET 2024-04-13T22:30:59.266 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.266 DEBUG:boto:Data: 2024-04-13T22:30:59.266 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.266 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.266 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.266 DEBUG:boto:Params: {} 2024-04-13T22:30:59.266 DEBUG:boto:Token: None 2024-04-13T22:30:59.266 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.267 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc7d2a4b0912b32-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.270 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.270 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.271 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.271 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.271 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:59.271 DEBUG:boto:Data: 2024-04-13T22:30:59.271 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.271 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.271 DEBUG:boto:Params: {} 2024-04-13T22:30:59.271 DEBUG:boto:Token: None 2024-04-13T22:30:59.272 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95 2024-04-13T22:30:59.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:boehZTbiShJnazS+jnOAt3NdBR4= 2024-04-13T22:30:59.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:boehZTbiShJnazS+jnOAt3NdBR4=', 'Content-Length': '0'} 2024-04-13T22:30:59.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2aa2b21f957333-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.274 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.274 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.274 DEBUG:boto:Method: HEAD 2024-04-13T22:30:59.274 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:59.274 DEBUG:boto:Data: 2024-04-13T22:30:59.274 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.274 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.274 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.275 DEBUG:boto:Params: {} 2024-04-13T22:30:59.275 DEBUG:boto:Token: None 2024-04-13T22:30:59.275 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95 2024-04-13T22:30:59.275 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:boehZTbiShJnazS+jnOAt3NdBR4= 2024-04-13T22:30:59.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:boehZTbiShJnazS+jnOAt3NdBR4=', 'Content-Length': '0'} 2024-04-13T22:30:59.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059d163379c4db8ea-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-13T22:30:59.277 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.277 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.277 DEBUG:boto:Method: GET 2024-04-13T22:30:59.278 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:59.278 DEBUG:boto:Data: 2024-04-13T22:30:59.278 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.278 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.278 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.278 DEBUG:boto:Params: {} 2024-04-13T22:30:59.278 DEBUG:boto:Token: None 2024-04-13T22:30:59.278 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95 2024-04-13T22:30:59.279 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sGnmXkyQRN7PJG2E0OXz4l5mbLo= 2024-04-13T22:30:59.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sGnmXkyQRN7PJG2E0OXz4l5mbLo=', 'Content-Length': '0'} 2024-04-13T22:30:59.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-61:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c1220cac43a3ba-00661b07a3-21de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.280 DEBUG:boto:path=/vwnfns-61/obj95 2024-04-13T22:30:59.280 DEBUG:boto:auth_path=/vwnfns-61/obj95 2024-04-13T22:30:59.281 DEBUG:boto:Method: GET 2024-04-13T22:30:59.281 DEBUG:boto:Path: /vwnfns-61/obj95 2024-04-13T22:30:59.281 DEBUG:boto:Data: 2024-04-13T22:30:59.281 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.281 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.281 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.281 DEBUG:boto:Params: {} 2024-04-13T22:30:59.281 DEBUG:boto:Token: None 2024-04-13T22:30:59.281 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/obj95 2024-04-13T22:30:59.281 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sGnmXkyQRN7PJG2E0OXz4l5mbLo= 2024-04-13T22:30:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sGnmXkyQRN7PJG2E0OXz4l5mbLo=', 'Content-Length': '0'} 2024-04-13T22:30:59.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:21:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f69d594bd216cd-00661b07a3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.284 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.284 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.284 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.284 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.284 DEBUG:boto:Method: GET 2024-04-13T22:30:59.284 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.284 DEBUG:boto:Data: 2024-04-13T22:30:59.284 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.285 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:30:59.285 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.285 DEBUG:boto:Params: {} 2024-04-13T22:30:59.285 DEBUG:boto:Token: None 2024-04-13T22:30:59.285 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.285 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698a7efe6106a591-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.287 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.288 DEBUG:boto:path=/vwnfns-61/ 2024-04-13T22:30:59.288 DEBUG:boto:auth_path=/vwnfns-61/ 2024-04-13T22:30:59.288 DEBUG:boto:path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.288 DEBUG:boto:auth_path=/vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.288 DEBUG:boto:Method: GET 2024-04-13T22:30:59.288 DEBUG:boto:Path: /vwnfns-61/?prefix=obj95 2024-04-13T22:30:59.288 DEBUG:boto:Data: 2024-04-13T22:30:59.288 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.288 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:30:59.288 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.289 DEBUG:boto:Params: {} 2024-04-13T22:30:59.289 DEBUG:boto:Token: None 2024-04-13T22:30:59.289 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:30:59 GMT /vwnfns-61/ 2024-04-13T22:30:59.289 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM= 2024-04-13T22:30:59.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:30:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GPlJcrb/LcTEg9DikRSa5U8kAaM=', 'Content-Length': '0'} 2024-04-13T22:30:59.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad38f0b835a98fd-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.292 DEBUG:boto:b'vwnfns-61obj951000falseobj952024-04-13T22:21:13.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:30:59.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-61 zones={a2, a1} 2024-04-13T22:30:59.293 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2024-04-13T22:30:59.298 INFO:rgw_multi.tests:create role zone=a1 name=roles-vwnfns-1 2024-04-13T22:30:59.298 DEBUG:boto:Method: POST 2024-04-13T22:30:59.298 DEBUG:boto:Path: / 2024-04-13T22:30:59.298 DEBUG:boto:Data: 2024-04-13T22:30:59.299 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com 2024-04-13T22:30:59.299 DEBUG:boto:Port: 8000 2024-04-13T22:30:59.299 DEBUG:boto:Params: {'RoleName': 'roles-vwnfns-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-13T22:30:59.299 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:30:59.299 DEBUG:boto:Token: None 2024-04-13T22:30:59.299 DEBUG:boto:CanonicalRequest: POST / host:smithi132.front.sepia.ceph.com:8000 x-amz-date:20240413T223059Z host;x-amz-date bbe87db952a0c975e6300db84a84a32cba23e14331a2515be18252190060aa86 2024-04-13T22:30:59.300 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240413T223059Z 20240413/front/smithi132/aws4_request 0dd3f923b1ddd7ec080289a107c0b666f9993c5fa3f5f2557b975538fd4a40d6 2024-04-13T22:30:59.300 DEBUG:boto:Signature: bbb7292b84ef9d24e1ef9a533ee9c275f4ed77979aa13f002ad2dfe9b395173c 2024-04-13T22:30:59.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240413T223059Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=C81CVW06BOFYI3GBQ4YX/20240413/front/smithi132/aws4_request,SignedHeaders=host;x-amz-date,Signature=bbb7292b84ef9d24e1ef9a533ee9c275f4ed77979aa13f002ad2dfe9b395173c', 'Host': 'smithi132.front.sepia.ceph.com:8000'} 2024-04-13T22:30:59.310 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1c832c6d5de4caa-00661b07a3-1a22-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.310 DEBUG:boto:b'db1d3c65-d8f6-4e8b-ae35-17d0e4fea0b9roles-vwnfns-1/arn:aws:iam:::role/roles-vwnfns-12024-04-13T22:30:59.302Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000f1c832c6d5de4caa-00661b07a3-1a22-a1' 2024-04-13T22:30:59.311 INFO:rgw_multi.tests:create role zone=a2 name=roles-vwnfns-2 2024-04-13T22:30:59.311 DEBUG:boto:Method: POST 2024-04-13T22:30:59.311 DEBUG:boto:Path: / 2024-04-13T22:30:59.311 DEBUG:boto:Data: 2024-04-13T22:30:59.312 DEBUG:boto:Headers: {} 2024-04-13T22:30:59.312 DEBUG:boto:Host: smithi132.front.sepia.ceph.com 2024-04-13T22:30:59.312 DEBUG:boto:Port: 8001 2024-04-13T22:30:59.312 DEBUG:boto:Params: {'RoleName': 'roles-vwnfns-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-13T22:30:59.312 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:30:59.312 DEBUG:boto:Token: None 2024-04-13T22:30:59.312 DEBUG:boto:CanonicalRequest: POST / host:smithi132.front.sepia.ceph.com:8001 x-amz-date:20240413T223059Z host;x-amz-date a7106da7db51da7e791128f2d7b53ea1bf1a6d68e6495ab2c1aee82a5c324e37 2024-04-13T22:30:59.312 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240413T223059Z 20240413/front/smithi132/aws4_request a8eee4bd6643e5ec6574228de4f3c6b46a89d8f78f60f4d02ca1e9ff8e8949e8 2024-04-13T22:30:59.312 DEBUG:boto:Signature: 39b1dc93a139bb9d58c4c1ccc15719a40525992ca10edce5b90e6ede8a6f4da5 2024-04-13T22:30:59.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240413T223059Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=C81CVW06BOFYI3GBQ4YX/20240413/front/smithi132/aws4_request,SignedHeaders=host;x-amz-date,Signature=39b1dc93a139bb9d58c4c1ccc15719a40525992ca10edce5b90e6ede8a6f4da5', 'Host': 'smithi132.front.sepia.ceph.com:8001'} 2024-04-13T22:30:59.328 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ca4fc6314d9f96c-00661b07a3-21de-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Sat, 13 Apr 2024 22:30:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:30:59.328 DEBUG:boto:b'cdc9262f-37bd-4371-aa10-e834d0d24911roles-vwnfns-2/arn:aws:iam:::role/roles-vwnfns-22024-04-13T22:30:59.322Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000008ca4fc6314d9f96c-00661b07a3-21de-a2' 2024-04-13T22:30:59.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:59.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:59.329 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:30:59.377 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:59.447 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:59.448 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:28:56.075922Z" 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.449 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:30:59.450 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:59.451 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:30:59.451 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:30:59.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:30:59.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:59.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:30:59.451 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:30:59.500 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:30:59.500 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:30:59.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:59.526+0000 7f40ec758780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:30:59.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:30:59.527+0000 7f40ec758780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:30:59.544 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:30:59.544 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:30:59.545 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.546 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:30:59.548 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:30:59.549 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:30:59Z" 2024-04-13T22:30:59.550 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:30:59.551 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:30:59Z" } 2024-04-13T22:30:59.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:30:59.552 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:30:59.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:30:59.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:04.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:04.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:04.554 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:04.609 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:04.609 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:04.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:04.641+0000 7f61fd68d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:04.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:04.642+0000 7f61fd68d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:04.658 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:04.659 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:04.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:04.662 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:04.663 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:04.664 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:04.665 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:04.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:04.665 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:04Z" 2024-04-13T22:31:04.665 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:04.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:04Z" } 2024-04-13T22:31:04.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:04.666 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:04.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:04.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:09.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:09.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:09.668 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:09.724 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:09.724 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:09.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:09.755+0000 7f980ba49780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:09.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:09.756+0000 7f980ba49780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:09.772 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:09.772 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:09.772 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:09.773 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:09.774 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:09.775 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:09.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:09.777 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:09Z" 2024-04-13T22:31:09.778 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:09.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:09Z" } 2024-04-13T22:31:09.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:09.780 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:09.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:09.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:14.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:14.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:14.782 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:14.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:14.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:14.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:14.867+0000 7f64af7dc780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:14.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:14.868+0000 7f64af7dc780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:14.885 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:14.886 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:14.888 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:14.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:14.890 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:14Z" 2024-04-13T22:31:14.891 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:14.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:14Z" } 2024-04-13T22:31:14.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:14.892 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:14.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:14.893 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:19.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:19.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:19.894 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:19.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:19.949 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:19.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:19.977+0000 7fa55b388780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:19.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:19.978+0000 7fa55b388780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:19.996 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:19.997 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:19.998 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:19.999 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:20.000 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:19Z" 2024-04-13T22:31:20.001 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:20.002 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:19Z" } 2024-04-13T22:31:20.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:20.003 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:20.003 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:20.003 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:25.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:25.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:25.013 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:25.073 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:25.073 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:25.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:25.103+0000 7f4e50f23780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:25.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:25.104+0000 7f4e50f23780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:25.121 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:25.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:25.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:25.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:25.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:25.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:25Z" 2024-04-13T22:31:25.127 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:25.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:25Z" } 2024-04-13T22:31:25.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:25.128 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:25.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:25.129 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:30.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:30.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:30.130 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:30.185 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:30.185 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:30.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:30.214+0000 7f6406334780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:30.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:30.215+0000 7f6406334780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:30.232 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:30.232 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:30.233 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:30.234 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:30.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:30.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:30.237 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:30Z" 2024-04-13T22:31:30.238 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:30.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:30Z" } 2024-04-13T22:31:30.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:30.240 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:30.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:30.240 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:35.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:35.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:35.241 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:35.297 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:35.297 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:35.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:35.327+0000 7f3a5219e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:35.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:35.328+0000 7f3a5219e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:35.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:35.346 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:35.347 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:35.348 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:35.349 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:35.350 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:35.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:35.351 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:35Z" 2024-04-13T22:31:35.351 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:35.352 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:35Z" } 2024-04-13T22:31:35.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:35.352 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:35.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:35.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:40.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:40.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:40.353 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:40.409 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:40.410 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:40.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:40.440+0000 7fad02e72780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:40.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:40.441+0000 7fad02e72780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:40.457 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:40.457 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:40.457 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:40.458 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:40.459 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:40.460 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:40.461 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:40.462 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:40Z" 2024-04-13T22:31:40.463 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:40.464 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:40Z" } 2024-04-13T22:31:40.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:40.464 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:40.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:40.465 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:45.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:45.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:45.466 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:45.522 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:45.522 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:45.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:45.554+0000 7ff343d8e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:45.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:45.555+0000 7ff343d8e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:45Z" 2024-04-13T22:31:45.577 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:45.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:45Z" } 2024-04-13T22:31:45.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:45.578 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:45.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:45.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:50.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:50.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:50.580 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:50.637 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:50.637 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:50.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:50.674+0000 7f65e9348780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:50.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:50.675+0000 7f65e9348780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:50.691 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:50.692 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:50.693 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:50.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:50.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:50.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:50Z" 2024-04-13T22:31:50.697 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:50.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:50Z" } 2024-04-13T22:31:50.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:50.698 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:50.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:50.699 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:31:55.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:55.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:31:55.717 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:31:55.777 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:31:55.777 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:31:55.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:55.807+0000 7f39f3e7d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:55.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:31:55.808+0000 7f39f3e7d780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:31:55.825 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:31:55.825 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:31:55.825 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:31:55.825 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:31:55.825 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:55.826 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:31:55.827 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:55.828 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:31:55.829 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:31:55.830 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:31:55.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:31:55.831 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:31:55Z" 2024-04-13T22:31:55.831 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:31:55.832 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:31:55Z" } 2024-04-13T22:31:55.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:31:55.832 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:31:55.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:31:55.832 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:32:00.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:00.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:00.833 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:00.889 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:00.889 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:00.922 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:00.919+0000 7fcbcddd7780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:00.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:00.920+0000 7fcbcddd7780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:00.937 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:00.937 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:00.937 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:00.938 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:00.939 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:00.940 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:00.941 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "", 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "0.000000", 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:00.942 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:00.943 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:00.943 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:00.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:00.943 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:00Z" 2024-04-13T22:32:00.943 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:00.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:00Z" } 2024-04-13T22:32:00.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T22:32:00.944 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:00.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T22:32:00.944 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1713047459.318155_197.1 target=, shard=3 master=1_1713047459.305268_202.1 target= 2024-04-13T22:32:05.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:05.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:05.945 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:06.002 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:06.002 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:06.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:06.031+0000 7f8c795cb780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:06.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:06.032+0000 7f8c795cb780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:06.049 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:06.049 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:06.049 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.050 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.052 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.053 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:06.054 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:06.055 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:06.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.055 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:06Z" 2024-04-13T22:32:06.055 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:06.056 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:06Z" } 2024-04-13T22:32:06.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:32:06.056 DEBUG:rgw_multi.tests:log_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:06.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:06.056 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:32:06.056 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-vwnfns-1 zones={a2, a1} 2024-04-13T22:32:06.056 DEBUG:boto:Method: POST 2024-04-13T22:32:06.057 DEBUG:boto:Path: / 2024-04-13T22:32:06.057 DEBUG:boto:Data: 2024-04-13T22:32:06.057 DEBUG:boto:Headers: {} 2024-04-13T22:32:06.057 DEBUG:boto:Host: smithi132.front.sepia.ceph.com 2024-04-13T22:32:06.057 DEBUG:boto:Port: 8001 2024-04-13T22:32:06.057 DEBUG:boto:Params: {'RoleName': 'roles-vwnfns-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-13T22:32:06.057 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:32:06.058 DEBUG:boto:Token: None 2024-04-13T22:32:06.058 DEBUG:boto:CanonicalRequest: POST / host:smithi132.front.sepia.ceph.com:8001 x-amz-date:20240413T223206Z host;x-amz-date 47b42a1551b47e95f504c3cb6f94b7bc48e659dc0092366861742ed8cdfaad95 2024-04-13T22:32:06.058 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240413T223206Z 20240413/front/smithi132/aws4_request 77550cb622beae80f00130461efd387f4ea4cf0e67aa8adcdc5aaa3a768f0e89 2024-04-13T22:32:06.058 DEBUG:boto:Signature: 84287c64997150f7ba6628d45a1a9a6234d2371b8e5de6647b5b4d294a63fc19 2024-04-13T22:32:06.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240413T223206Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=C81CVW06BOFYI3GBQ4YX/20240413/front/smithi132/aws4_request,SignedHeaders=host;x-amz-date,Signature=84287c64997150f7ba6628d45a1a9a6234d2371b8e5de6647b5b4d294a63fc19', 'Host': 'smithi132.front.sepia.ceph.com:8001'} 2024-04-13T22:32:06.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cc91c3e63964b4ce-00661b07e6-21de-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 13 Apr 2024 22:32:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:06.063 DEBUG:boto:b'tx00000cc91c3e63964b4ce-00661b07e6-21de-a2db1d3c65-d8f6-4e8b-ae35-17d0e4fea0b9roles-vwnfns-1/arn:aws:iam:::role/roles-vwnfns-12024-04-13T22:32:05.645Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-13T22:32:06.064 DEBUG:boto:Method: POST 2024-04-13T22:32:06.064 DEBUG:boto:Path: / 2024-04-13T22:32:06.064 DEBUG:boto:Data: 2024-04-13T22:32:06.064 DEBUG:boto:Headers: {} 2024-04-13T22:32:06.064 DEBUG:boto:Host: smithi132.front.sepia.ceph.com 2024-04-13T22:32:06.064 DEBUG:boto:Port: 8000 2024-04-13T22:32:06.064 DEBUG:boto:Params: {'RoleName': 'roles-vwnfns-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-13T22:32:06.064 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:32:06.065 DEBUG:boto:Token: None 2024-04-13T22:32:06.065 DEBUG:boto:CanonicalRequest: POST / host:smithi132.front.sepia.ceph.com:8000 x-amz-date:20240413T223206Z host;x-amz-date 47b42a1551b47e95f504c3cb6f94b7bc48e659dc0092366861742ed8cdfaad95 2024-04-13T22:32:06.065 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240413T223206Z 20240413/front/smithi132/aws4_request a76c0b1655dbacc182f600217041995515284168209b1783a116f45863ade576 2024-04-13T22:32:06.065 DEBUG:boto:Signature: 45c91de68e2900787a049e5c1df4719cbfab2a554b316e51a480c8839c6e705a 2024-04-13T22:32:06.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240413T223206Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=C81CVW06BOFYI3GBQ4YX/20240413/front/smithi132/aws4_request,SignedHeaders=host;x-amz-date,Signature=45c91de68e2900787a049e5c1df4719cbfab2a554b316e51a480c8839c6e705a', 'Host': 'smithi132.front.sepia.ceph.com:8000'} 2024-04-13T22:32:06.068 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8a5767bcb79228e-00661b07e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 13 Apr 2024 22:32:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:06.068 DEBUG:boto:b'tx00000d8a5767bcb79228e-00661b07e6-1a22-a1db1d3c65-d8f6-4e8b-ae35-17d0e4fea0b9roles-vwnfns-1/arn:aws:iam:::role/roles-vwnfns-12024-04-13T22:30:59.302Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-13T22:32:06.068 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-vwnfns-1 2024-04-13T22:32:06.068 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-vwnfns-1 zones={a2, a1} 2024-04-13T22:32:06.069 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-vwnfns-2 zones={a1, a2} 2024-04-13T22:32:06.069 DEBUG:boto:Method: POST 2024-04-13T22:32:06.069 DEBUG:boto:Path: / 2024-04-13T22:32:06.069 DEBUG:boto:Data: 2024-04-13T22:32:06.069 DEBUG:boto:Headers: {} 2024-04-13T22:32:06.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com 2024-04-13T22:32:06.069 DEBUG:boto:Port: 8000 2024-04-13T22:32:06.069 DEBUG:boto:Params: {'RoleName': 'roles-vwnfns-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-13T22:32:06.069 DEBUG:boto:Token: None 2024-04-13T22:32:06.069 DEBUG:boto:CanonicalRequest: POST / host:smithi132.front.sepia.ceph.com:8000 x-amz-date:20240413T223206Z host;x-amz-date f9b3f4d50a2025a465b7976f7f7f55c3d4041086001b582bae684c2b4519ffbd 2024-04-13T22:32:06.070 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240413T223206Z 20240413/front/smithi132/aws4_request 9642cc0193fdf41d104a403b9edd1d588888737057ea93d90e71c3429345e5a2 2024-04-13T22:32:06.070 DEBUG:boto:Signature: 26e265359c380bd0fe0914f9627646224e3d0930ae4a2c735584aac4118ca863 2024-04-13T22:32:06.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240413T223206Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=C81CVW06BOFYI3GBQ4YX/20240413/front/smithi132/aws4_request,SignedHeaders=host;x-amz-date,Signature=26e265359c380bd0fe0914f9627646224e3d0930ae4a2c735584aac4118ca863', 'Host': 'smithi132.front.sepia.ceph.com:8000'} 2024-04-13T22:32:06.071 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ca62a812fc9a181-00661b07e6-1a22-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 13 Apr 2024 22:32:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:06.071 DEBUG:boto:b'tx000004ca62a812fc9a181-00661b07e6-1a22-a1cdc9262f-37bd-4371-aa10-e834d0d24911roles-vwnfns-2/arn:aws:iam:::role/roles-vwnfns-22024-04-13T22:30:59.315Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-13T22:32:06.071 DEBUG:boto:Method: POST 2024-04-13T22:32:06.072 DEBUG:boto:Path: / 2024-04-13T22:32:06.072 DEBUG:boto:Data: 2024-04-13T22:32:06.072 DEBUG:boto:Headers: {} 2024-04-13T22:32:06.072 DEBUG:boto:Host: smithi132.front.sepia.ceph.com 2024-04-13T22:32:06.072 DEBUG:boto:Port: 8001 2024-04-13T22:32:06.072 DEBUG:boto:Params: {'RoleName': 'roles-vwnfns-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-13T22:32:06.072 DEBUG:boto:Token: None 2024-04-13T22:32:06.072 DEBUG:boto:CanonicalRequest: POST / host:smithi132.front.sepia.ceph.com:8001 x-amz-date:20240413T223206Z host;x-amz-date f9b3f4d50a2025a465b7976f7f7f55c3d4041086001b582bae684c2b4519ffbd 2024-04-13T22:32:06.072 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240413T223206Z 20240413/front/smithi132/aws4_request 0a215a30b87aa5e1c958cc7f7e3839475fb3c71b6aa43d6a52e159a5163ebecb 2024-04-13T22:32:06.073 DEBUG:boto:Signature: ec99c9b7e73e4b0d6e20247b5470093f9bc7d391a00dba379dc183c282d820c4 2024-04-13T22:32:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240413T223206Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=C81CVW06BOFYI3GBQ4YX/20240413/front/smithi132/aws4_request,SignedHeaders=host;x-amz-date,Signature=ec99c9b7e73e4b0d6e20247b5470093f9bc7d391a00dba379dc183c282d820c4', 'Host': 'smithi132.front.sepia.ceph.com:8001'} 2024-04-13T22:32:06.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006998b3eb2a36f77c-00661b07e6-21de-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 13 Apr 2024 22:32:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:06.074 DEBUG:boto:b'tx000006998b3eb2a36f77c-00661b07e6-21de-a2cdc9262f-37bd-4371-aa10-e834d0d24911roles-vwnfns-2/arn:aws:iam:::role/roles-vwnfns-22024-04-13T22:30:59.315Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-13T22:32:06.075 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-vwnfns-2 2024-04-13T22:32:06.075 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-vwnfns-2 zones={a1, a2} 2024-04-13T22:32:06.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-04-13T22:32:06.076 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:32:06.076 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:32:06.076 DEBUG:boto:Method: PUT 2024-04-13T22:32:06.076 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:32:06.077 DEBUG:boto:Data: 2024-04-13T22:32:06.077 DEBUG:boto:Headers: {} 2024-04-13T22:32:06.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:06.077 DEBUG:boto:Port: 8000 2024-04-13T22:32:06.077 DEBUG:boto:Params: {} 2024-04-13T22:32:06.077 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:32:06.077 DEBUG:boto:Token: None 2024-04-13T22:32:06.077 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:32:06 GMT /vwnfns-62/ 2024-04-13T22:32:06.078 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HRDOy4T80IDfBe+/XS7bKcENiP4= 2024-04-13T22:32:06.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:32:06 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HRDOy4T80IDfBe+/XS7bKcENiP4=', 'Content-Length': '0'} 2024-04-13T22:32:06.094 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005010b95b33b07e06-00661b07e6-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:32:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:06.094 DEBUG:rgw_multi.tests:created bucket=vwnfns-62 2024-04-13T22:32:06.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:06.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:06.095 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:06.142 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:06.142 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:06.172 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:32:06.185 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:32:06.185 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.185 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047526.090978_882.1", 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:32:06.090978Z" 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:32:06.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.187 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:06.187 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:32:06.187 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.187 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:32:06.188 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:06.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:32:06.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:06.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:06.188 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:06.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:06.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:06.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:06.262+0000 7fe29b229780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:06.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:06.263+0000 7fe29b229780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:06.280 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:06.280 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:06.280 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:06.280 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:06.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:06.280 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.281 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.282 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:06.285 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:06.286 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:06Z" 2024-04-13T22:32:06.286 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:06.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:06Z" } 2024-04-13T22:32:06.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:32:06.287 DEBUG:rgw_multi.tests:log_status={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:06.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:06.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713047526.090978_882.1 target=1_1713047336.075922_866.1 2024-04-13T22:32:11.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:11.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:11.288 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:11.344 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:11.344 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:11.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:11.375+0000 7f6690a5a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:11.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:11.376+0000 7f6690a5a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:11.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:32:11.394 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:11.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:11.396 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:11.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:11Z" 2024-04-13T22:32:11.398 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:11.399 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:11Z" } 2024-04-13T22:32:11.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:32:11.399 DEBUG:rgw_multi.tests:log_status={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:11.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:11.400 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713047526.090978_882.1 target=1_1713047336.075922_866.1 2024-04-13T22:32:16.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:16.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:16.401 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:16.458 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:16.459 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:16.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:16.490+0000 7fa48d029780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:16.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:16.492+0000 7fa48d029780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:16.508 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:16.508 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:16.508 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047336.075922_866.1", 2024-04-13T22:32:16.509 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.075922Z", 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:16.510 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:16.512 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:16.513 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:16.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:16.514 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:16Z" 2024-04-13T22:32:16.514 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:16.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047336.075922_866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:28:56.075922Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:16Z" } 2024-04-13T22:32:16.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047336.075922_866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.075922Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:32:16.515 DEBUG:rgw_multi.tests:log_status={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:16.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047336.075922_866.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:16.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713047526.090978_882.1 target=1_1713047336.075922_866.1 2024-04-13T22:32:21.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:21.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:21.517 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:21.572 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:21.573 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:21.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:21.602+0000 7f4c50d83780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:21.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:21.603+0000 7f4c50d83780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:21.619 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:21.619 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:21.619 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:21.619 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:21.619 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:21.619 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047526.085129_879.1", 2024-04-13T22:32:21.620 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:06.085129Z", 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:21.621 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:21.623 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:21.624 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:21.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:21.626 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:21Z" 2024-04-13T22:32:21.626 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:21.627 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047526.085129_879.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:32:06.085129Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:21Z" } 2024-04-13T22:32:21.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047526.085129_879.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:32:06.085129Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:32:21.628 DEBUG:rgw_multi.tests:log_status={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:21.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047526.085129_879.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:21.628 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713047526.090978_882.1 target=1_1713047526.085129_879.1 2024-04-13T22:32:26.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:26.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:32:26.629 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:32:26.685 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:26.685 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:26.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:26.715+0000 7f288446f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:26.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:26.716+0000 7f288446f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:32:26.732 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:32:26.732 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:32:26.732 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:32:26.732 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047526.090978_882.1", 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:06.090978Z", 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:26.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:26.735 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:26.736 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:32:26.737 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:32:26.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:32:26.739 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:32:26Z" 2024-04-13T22:32:26.739 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:32:26.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047526.090978_882.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:32:06.090978Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:32:26Z" } 2024-04-13T22:32:26.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047526.090978_882.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:32:06.090978Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:32:26.740 DEBUG:rgw_multi.tests:log_status={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:26.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047526.090978_882.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:32:26.740 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:32:26.741 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:32:26.741 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:32:26.741 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:32:26.739+0000 7f511b1e1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 109058) UID: 0 2024-04-13T22:32:26.742 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:32:26.739+0000 7f51203b67c0 -1 shutting down 2024-04-13T22:32:30.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:32:32.842 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:32:32.842 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:32:32.843 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:32:32.843 DEBUG:boto:Method: HEAD 2024-04-13T22:32:32.843 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:32:32.843 DEBUG:boto:Data: 2024-04-13T22:32:32.843 DEBUG:boto:Headers: {} 2024-04-13T22:32:32.843 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.843 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.843 DEBUG:boto:Params: {} 2024-04-13T22:32:32.844 DEBUG:boto:Token: None 2024-04-13T22:32:32.844 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/ 2024-04-13T22:32:32.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4= 2024-04-13T22:32:32.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4=', 'Content-Length': '0'} 2024-04-13T22:32:32.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3fe05516429b868-00661b0800-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.847 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:32:32.847 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:32:32.847 DEBUG:boto:Method: PUT 2024-04-13T22:32:32.847 DEBUG:boto:Path: /vwnfns-62/obj11 2024-04-13T22:32:32.847 DEBUG:boto:Data: 2024-04-13T22:32:32.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:32:32.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.847 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.848 DEBUG:boto:Params: {} 2024-04-13T22:32:32.848 DEBUG:boto:Token: None 2024-04-13T22:32:32.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/obj11 2024-04-13T22:32:32.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I8F7AEKZX8krlv/IMoq8/XQQz70= 2024-04-13T22:32:32.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I8F7AEKZX8krlv/IMoq8/XQQz70='} 2024-04-13T22:32:32.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000162f8b713e19bbb7-00661b0800-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.856 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:32:32.856 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:32:32.856 DEBUG:boto:Method: HEAD 2024-04-13T22:32:32.856 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:32:32.856 DEBUG:boto:Data: 2024-04-13T22:32:32.856 DEBUG:boto:Headers: {} 2024-04-13T22:32:32.856 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.856 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.856 DEBUG:boto:Params: {} 2024-04-13T22:32:32.857 DEBUG:boto:Token: None 2024-04-13T22:32:32.857 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/ 2024-04-13T22:32:32.857 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4= 2024-04-13T22:32:32.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4=', 'Content-Length': '0'} 2024-04-13T22:32:32.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007567852f69158ee6-00661b0800-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.860 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:32:32.860 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:32:32.860 DEBUG:boto:Method: PUT 2024-04-13T22:32:32.860 DEBUG:boto:Path: /vwnfns-62/obj22 2024-04-13T22:32:32.860 DEBUG:boto:Data: 2024-04-13T22:32:32.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:32:32.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.860 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.860 DEBUG:boto:Params: {} 2024-04-13T22:32:32.860 DEBUG:boto:Token: None 2024-04-13T22:32:32.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/obj22 2024-04-13T22:32:32.861 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mowOtBYFkUpBmje6xyn1MfFT0jo= 2024-04-13T22:32:32.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mowOtBYFkUpBmje6xyn1MfFT0jo='} 2024-04-13T22:32:32.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c6cb58ab854b51d-00661b0800-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.907 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:32:32.907 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:32:32.907 DEBUG:boto:Method: HEAD 2024-04-13T22:32:32.907 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:32:32.908 DEBUG:boto:Data: 2024-04-13T22:32:32.908 DEBUG:boto:Headers: {} 2024-04-13T22:32:32.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.908 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.908 DEBUG:boto:Params: {} 2024-04-13T22:32:32.908 DEBUG:boto:Token: None 2024-04-13T22:32:32.908 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/ 2024-04-13T22:32:32.908 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4= 2024-04-13T22:32:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4=', 'Content-Length': '0'} 2024-04-13T22:32:32.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2b219638a7ca9de-00661b0800-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.911 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:32:32.911 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:32:32.911 DEBUG:boto:Method: PUT 2024-04-13T22:32:32.911 DEBUG:boto:Path: /vwnfns-62/obj33 2024-04-13T22:32:32.912 DEBUG:boto:Data: 2024-04-13T22:32:32.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:32:32.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.912 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.912 DEBUG:boto:Params: {} 2024-04-13T22:32:32.912 DEBUG:boto:Token: None 2024-04-13T22:32:32.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/obj33 2024-04-13T22:32:32.912 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IN/2FviWhX6pMgmMCnWVAHfXZ1Y= 2024-04-13T22:32:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IN/2FviWhX6pMgmMCnWVAHfXZ1Y='} 2024-04-13T22:32:32.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7a601e968fac9be-00661b0800-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.918 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:32:32.921 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:32:32.921 DEBUG:boto:Method: HEAD 2024-04-13T22:32:32.921 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:32:32.921 DEBUG:boto:Data: 2024-04-13T22:32:32.922 DEBUG:boto:Headers: {} 2024-04-13T22:32:32.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.922 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.922 DEBUG:boto:Params: {} 2024-04-13T22:32:32.922 DEBUG:boto:Token: None 2024-04-13T22:32:32.922 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/ 2024-04-13T22:32:32.922 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4= 2024-04-13T22:32:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4=', 'Content-Length': '0'} 2024-04-13T22:32:32.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024d079a8d2850d43-00661b0800-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.925 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:32:32.925 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:32:32.925 DEBUG:boto:Method: PUT 2024-04-13T22:32:32.925 DEBUG:boto:Path: /vwnfns-62/obj44 2024-04-13T22:32:32.925 DEBUG:boto:Data: 2024-04-13T22:32:32.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:32:32.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.926 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.926 DEBUG:boto:Params: {} 2024-04-13T22:32:32.926 DEBUG:boto:Token: None 2024-04-13T22:32:32.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/obj44 2024-04-13T22:32:32.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MsfsgtWYxsRd4OYZTD8VWW8rS0k= 2024-04-13T22:32:32.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MsfsgtWYxsRd4OYZTD8VWW8rS0k='} 2024-04-13T22:32:32.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000843f024678630f12-00661b0800-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.975 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:32:32.975 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:32:32.975 DEBUG:boto:Method: HEAD 2024-04-13T22:32:32.975 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:32:32.975 DEBUG:boto:Data: 2024-04-13T22:32:32.975 DEBUG:boto:Headers: {} 2024-04-13T22:32:32.975 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.975 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.976 DEBUG:boto:Params: {} 2024-04-13T22:32:32.976 DEBUG:boto:Token: None 2024-04-13T22:32:32.976 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/ 2024-04-13T22:32:32.976 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4= 2024-04-13T22:32:32.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pm5uEnBVqnSNw+H9XD/vF2qpfj4=', 'Content-Length': '0'} 2024-04-13T22:32:32.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070928ce70878ab7c-00661b0800-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.978 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:32:32.979 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:32:32.979 DEBUG:boto:Method: PUT 2024-04-13T22:32:32.979 DEBUG:boto:Path: /vwnfns-62/obj55 2024-04-13T22:32:32.979 DEBUG:boto:Data: 2024-04-13T22:32:32.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:32:32.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:32:32.979 DEBUG:boto:Port: 8000 2024-04-13T22:32:32.979 DEBUG:boto:Params: {} 2024-04-13T22:32:32.979 DEBUG:boto:Token: None 2024-04-13T22:32:32.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:32:32 GMT /vwnfns-62/obj55 2024-04-13T22:32:32.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rzvc55+YeBw/l0D0TN/siMuqOdA= 2024-04-13T22:32:32.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:32:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rzvc55+YeBw/l0D0TN/siMuqOdA='} 2024-04-13T22:32:32.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f2b563ecfa9dee4-00661b0800-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:32:32.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:32:32.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:32:32.985 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:32:33.037 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:33.037 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:33.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:33.067+0000 7fd3dfc9d780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-13T22:32:33.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:33.068+0000 7fd3dfc9d780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-13T22:32:33.093 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:32:33.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:32:33.095 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:32:33.096 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:32:33.115 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:32:33.115 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:32:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:32:33.117 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:32:34.118 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:32:34.140 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:32:34.140 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:32:34.140 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:32:34.141 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:32:34.141 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-62 2024-04-13T22:32:34.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:32:34.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:32:34.142 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:32:34.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:32:34.236 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:32:34.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:34.268+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:32:36.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:32:39.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:39.269+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:32:44.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:44.269+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:32:49.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:49.271+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:32:54.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:54.271+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:32:59.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:32:59.272+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:04.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:04.272+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:09.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:09.272+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:14.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:14.273+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:19.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:19.274+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:24.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:24.274+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:29.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:29.275+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:34.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:34.275+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:39.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:39.275+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:44.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:44.276+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:49.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:49.276+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:54.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:54.276+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:33:59.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:33:59.277+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:04.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:04.277+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:09.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:09.279+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:14.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:14.280+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:19.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:19.280+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:24.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:24.280+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:29.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:29.281+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:34.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:34.281+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:39.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:39.282+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:44.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:44.283+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:49.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:49.284+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:54.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:54.285+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:34:59.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:34:59.285+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:04.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:04.285+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:09.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:09.286+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:14.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:14.286+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:19.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:19.286+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:24.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:24.288+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:29.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:29.288+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:34.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:34.289+0000 7f5c40788780 1 waiting to reach incremental sync.. 2024-04-13T22:35:39.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:39.291+0000 7f5c40788780 1 bucket sync caught up with source: 2024-04-13T22:35:39.293 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.88.6, 00000000002.81.6, 00000000002.106.6] 2024-04-13T22:35:39.293 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.88.6, 00000000002.81.6, 00000000002.106.6] 2024-04-13T22:35:39.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:39.291+0000 7f5c40788780 0 bucket checkpoint complete 2024-04-13T22:35:39.311 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-62 2024-04-13T22:35:39.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:35:39.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:35:39.313 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:35:39.373 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:39.373 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:39.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:39.408+0000 7f721c8d4780 1 bucket sync caught up with source: 2024-04-13T22:35:39.411 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.1012.6, 00000000002.1950.6, 00000000002.1472.6] 2024-04-13T22:35:39.411 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.1012.6, 00000000002.1950.6, 00000000002.1472.6] 2024-04-13T22:35:39.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:39.408+0000 7f721c8d4780 0 bucket checkpoint complete 2024-04-13T22:35:39.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-62 zones={a2, a1} 2024-04-13T22:35:39.425 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.425 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.425 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.426 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:35:39.426 DEBUG:boto:Data: 2024-04-13T22:35:39.426 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.426 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.426 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.426 DEBUG:boto:Params: {} 2024-04-13T22:35:39.427 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:35:39.427 DEBUG:boto:Token: None 2024-04-13T22:35:39.427 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.427 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EFotBSvhN8G+DWB+VG9GazbbpRk= 2024-04-13T22:35:39.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EFotBSvhN8G+DWB+VG9GazbbpRk=', 'Content-Length': '0'} 2024-04-13T22:35:39.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7ce996c3a3d5688-00661b08bb-22a6-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.435 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.435 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.435 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.435 DEBUG:boto:Path: /vwnfns-62/ 2024-04-13T22:35:39.435 DEBUG:boto:Data: 2024-04-13T22:35:39.435 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.435 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.435 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.435 DEBUG:boto:Params: {} 2024-04-13T22:35:39.435 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:35:39.436 DEBUG:boto:Token: None 2024-04-13T22:35:39.436 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.436 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EFotBSvhN8G+DWB+VG9GazbbpRk= 2024-04-13T22:35:39.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EFotBSvhN8G+DWB+VG9GazbbpRk=', 'Content-Length': '0'} 2024-04-13T22:35:39.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002a86e4cf5214bfc-00661b08bb-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.441 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:35:39.441 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.442 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.442 DEBUG:boto:path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.442 DEBUG:boto:auth_path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.442 DEBUG:boto:Method: GET 2024-04-13T22:35:39.442 DEBUG:boto:Path: /vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.442 DEBUG:boto:Data: 2024-04-13T22:35:39.442 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.442 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.442 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.442 DEBUG:boto:Params: {} 2024-04-13T22:35:39.443 DEBUG:boto:Token: None 2024-04-13T22:35:39.443 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/?versions 2024-04-13T22:35:39.443 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs= 2024-04-13T22:35:39.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs=', 'Content-Length': '0'} 2024-04-13T22:35:39.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fab735c6397df22f-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.445 DEBUG:boto:b'vwnfns-62999falseobj11nulltrue2024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:35:39.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:35:39.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:35:39.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:35:39.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:35:39.446 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:35:39.447 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.447 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.447 DEBUG:boto:path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.447 DEBUG:boto:auth_path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.447 DEBUG:boto:Method: GET 2024-04-13T22:35:39.447 DEBUG:boto:Path: /vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.447 DEBUG:boto:Data: 2024-04-13T22:35:39.447 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.447 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.447 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.447 DEBUG:boto:Params: {} 2024-04-13T22:35:39.448 DEBUG:boto:Token: None 2024-04-13T22:35:39.448 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/?versions 2024-04-13T22:35:39.448 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs= 2024-04-13T22:35:39.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs=', 'Content-Length': '0'} 2024-04-13T22:35:39.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4304cc82d4b742-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.450 DEBUG:boto:b'vwnfns-62999falseobj11nulltrue2024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:35:39.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:35:39.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:35:39.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:35:39.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:35:39.451 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.451 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.451 DEBUG:boto:path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.452 DEBUG:boto:auth_path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.452 DEBUG:boto:Method: GET 2024-04-13T22:35:39.452 DEBUG:boto:Path: /vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.452 DEBUG:boto:Data: 2024-04-13T22:35:39.452 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.452 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.452 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.452 DEBUG:boto:Params: {} 2024-04-13T22:35:39.452 DEBUG:boto:Token: None 2024-04-13T22:35:39.452 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/?versions 2024-04-13T22:35:39.453 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs= 2024-04-13T22:35:39.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs=', 'Content-Length': '0'} 2024-04-13T22:35:39.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008024bf79589c92f5-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.455 DEBUG:boto:b'vwnfns-62999falseobj11nulltrue2024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.455 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.455 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.456 DEBUG:boto:path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.456 DEBUG:boto:auth_path=/vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.456 DEBUG:boto:Method: GET 2024-04-13T22:35:39.456 DEBUG:boto:Path: /vwnfns-62/?versions&max-keys=999 2024-04-13T22:35:39.456 DEBUG:boto:Data: 2024-04-13T22:35:39.456 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.456 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.456 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.456 DEBUG:boto:Params: {} 2024-04-13T22:35:39.456 DEBUG:boto:Token: None 2024-04-13T22:35:39.457 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/?versions 2024-04-13T22:35:39.457 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs= 2024-04-13T22:35:39.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pXo+O24KUJD+SgxUoMlKMSrXtzs=', 'Content-Length': '0'} 2024-04-13T22:35:39.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002810bbd7a4f63f2f-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.459 DEBUG:boto:b'vwnfns-62999falseobj11nulltrue2024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:35:39.460 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.460 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.460 DEBUG:boto:path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.460 DEBUG:boto:auth_path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.460 DEBUG:boto:Method: GET 2024-04-13T22:35:39.460 DEBUG:boto:Path: /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.460 DEBUG:boto:Data: 2024-04-13T22:35:39.460 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.461 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.461 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.461 DEBUG:boto:Params: {} 2024-04-13T22:35:39.461 DEBUG:boto:Token: None 2024-04-13T22:35:39.461 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4= 2024-04-13T22:35:39.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4=', 'Content-Length': '0'} 2024-04-13T22:35:39.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54e96c19fe06630-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.463 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.463 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.463 DEBUG:boto:path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.463 DEBUG:boto:auth_path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.464 DEBUG:boto:Method: GET 2024-04-13T22:35:39.464 DEBUG:boto:Path: /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.464 DEBUG:boto:Data: 2024-04-13T22:35:39.464 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.464 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.464 DEBUG:boto:Params: {} 2024-04-13T22:35:39.464 DEBUG:boto:Token: None 2024-04-13T22:35:39.464 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4= 2024-04-13T22:35:39.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4=', 'Content-Length': '0'} 2024-04-13T22:35:39.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a14c09168ae829-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.466 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.466 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.466 DEBUG:boto:path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.466 DEBUG:boto:auth_path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.466 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.467 DEBUG:boto:Path: /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.467 DEBUG:boto:Data: 2024-04-13T22:35:39.467 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.467 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.467 DEBUG:boto:Params: {} 2024-04-13T22:35:39.467 DEBUG:boto:Token: None 2024-04-13T22:35:39.467 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qVLNTNYfWm+/sEhKzalNANL3obQ= 2024-04-13T22:35:39.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qVLNTNYfWm+/sEhKzalNANL3obQ=', 'Content-Length': '0'} 2024-04-13T22:35:39.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008451d20d1e25b206-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.469 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.469 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.469 DEBUG:boto:path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.469 DEBUG:boto:auth_path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.469 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.469 DEBUG:boto:Path: /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.469 DEBUG:boto:Data: 2024-04-13T22:35:39.470 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.470 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.470 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.470 DEBUG:boto:Params: {} 2024-04-13T22:35:39.470 DEBUG:boto:Token: None 2024-04-13T22:35:39.470 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.470 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qVLNTNYfWm+/sEhKzalNANL3obQ= 2024-04-13T22:35:39.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qVLNTNYfWm+/sEhKzalNANL3obQ=', 'Content-Length': '0'} 2024-04-13T22:35:39.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004405ef42b3277d17-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:35:39.472 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.472 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.472 DEBUG:boto:path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.472 DEBUG:boto:auth_path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.472 DEBUG:boto:Method: GET 2024-04-13T22:35:39.472 DEBUG:boto:Path: /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.473 DEBUG:boto:Data: 2024-04-13T22:35:39.473 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.473 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.473 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.473 DEBUG:boto:Params: {} 2024-04-13T22:35:39.473 DEBUG:boto:Token: None 2024-04-13T22:35:39.473 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.474 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4= 2024-04-13T22:35:39.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4=', 'Content-Length': '0'} 2024-04-13T22:35:39.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaac95e8a69d118e-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.476 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.477 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.477 DEBUG:boto:path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.477 DEBUG:boto:auth_path=/vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.477 DEBUG:boto:Method: GET 2024-04-13T22:35:39.477 DEBUG:boto:Path: /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.477 DEBUG:boto:Data: 2024-04-13T22:35:39.477 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.477 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.477 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.477 DEBUG:boto:Params: {} 2024-04-13T22:35:39.478 DEBUG:boto:Token: None 2024-04-13T22:35:39.478 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11?versionId=null 2024-04-13T22:35:39.478 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4= 2024-04-13T22:35:39.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qrj8H1yl1cAQRuoY9PH/K65wwk4=', 'Content-Length': '0'} 2024-04-13T22:35:39.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000328931b7d01a3e52-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.480 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.480 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.480 DEBUG:boto:path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.480 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.480 DEBUG:boto:Method: GET 2024-04-13T22:35:39.480 DEBUG:boto:Path: /vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.480 DEBUG:boto:Data: 2024-04-13T22:35:39.480 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.480 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.481 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.481 DEBUG:boto:Params: {} 2024-04-13T22:35:39.481 DEBUG:boto:Token: None 2024-04-13T22:35:39.481 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.481 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e448588703f12c-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.483 DEBUG:boto:b'vwnfns-62obj111000falseobj112024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.484 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.484 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.484 DEBUG:boto:path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.484 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.484 DEBUG:boto:Method: GET 2024-04-13T22:35:39.484 DEBUG:boto:Path: /vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.484 DEBUG:boto:Data: 2024-04-13T22:35:39.484 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.484 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.484 DEBUG:boto:Params: {} 2024-04-13T22:35:39.485 DEBUG:boto:Token: None 2024-04-13T22:35:39.485 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7f30ed1cd01a3f-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.487 DEBUG:boto:b'vwnfns-62obj111000falseobj112024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.487 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.487 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.487 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.488 DEBUG:boto:Path: /vwnfns-62/obj11 2024-04-13T22:35:39.488 DEBUG:boto:Data: 2024-04-13T22:35:39.488 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.488 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.488 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.488 DEBUG:boto:Params: {} 2024-04-13T22:35:39.488 DEBUG:boto:Token: None 2024-04-13T22:35:39.488 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11 2024-04-13T22:35:39.488 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:facgECgPcAcdWflfRfdGPrTFH7E= 2024-04-13T22:35:39.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:facgECgPcAcdWflfRfdGPrTFH7E=', 'Content-Length': '0'} 2024-04-13T22:35:39.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5caac4342bf5f5-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.490 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.490 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.490 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.490 DEBUG:boto:Path: /vwnfns-62/obj11 2024-04-13T22:35:39.490 DEBUG:boto:Data: 2024-04-13T22:35:39.491 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.491 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.491 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.491 DEBUG:boto:Params: {} 2024-04-13T22:35:39.491 DEBUG:boto:Token: None 2024-04-13T22:35:39.491 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11 2024-04-13T22:35:39.491 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:facgECgPcAcdWflfRfdGPrTFH7E= 2024-04-13T22:35:39.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:facgECgPcAcdWflfRfdGPrTFH7E=', 'Content-Length': '0'} 2024-04-13T22:35:39.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000696cf845bc8d08f8-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:35:39.493 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.493 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.493 DEBUG:boto:Method: GET 2024-04-13T22:35:39.493 DEBUG:boto:Path: /vwnfns-62/obj11 2024-04-13T22:35:39.494 DEBUG:boto:Data: 2024-04-13T22:35:39.494 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.494 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.494 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.494 DEBUG:boto:Params: {} 2024-04-13T22:35:39.494 DEBUG:boto:Token: None 2024-04-13T22:35:39.494 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11 2024-04-13T22:35:39.494 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8gknFgpKMPGj2lO1VeuQmEmMpuc= 2024-04-13T22:35:39.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8gknFgpKMPGj2lO1VeuQmEmMpuc=', 'Content-Length': '0'} 2024-04-13T22:35:39.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931f1f21f3efc847-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.538 DEBUG:boto:path=/vwnfns-62/obj11 2024-04-13T22:35:39.539 DEBUG:boto:auth_path=/vwnfns-62/obj11 2024-04-13T22:35:39.539 DEBUG:boto:Method: GET 2024-04-13T22:35:39.539 DEBUG:boto:Path: /vwnfns-62/obj11 2024-04-13T22:35:39.539 DEBUG:boto:Data: 2024-04-13T22:35:39.539 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.539 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.539 DEBUG:boto:Params: {} 2024-04-13T22:35:39.539 DEBUG:boto:Token: None 2024-04-13T22:35:39.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj11 2024-04-13T22:35:39.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8gknFgpKMPGj2lO1VeuQmEmMpuc= 2024-04-13T22:35:39.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8gknFgpKMPGj2lO1VeuQmEmMpuc=', 'Content-Length': '0'} 2024-04-13T22:35:39.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5642c4b11a1e584-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.542 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.542 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.542 DEBUG:boto:path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.542 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.542 DEBUG:boto:Method: GET 2024-04-13T22:35:39.543 DEBUG:boto:Path: /vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.543 DEBUG:boto:Data: 2024-04-13T22:35:39.543 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.543 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.543 DEBUG:boto:Params: {} 2024-04-13T22:35:39.543 DEBUG:boto:Token: None 2024-04-13T22:35:39.543 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0a4a2a8aed5f02c-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.546 DEBUG:boto:b'vwnfns-62obj111000falseobj112024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.546 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.546 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.546 DEBUG:boto:path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.546 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.546 DEBUG:boto:Method: GET 2024-04-13T22:35:39.546 DEBUG:boto:Path: /vwnfns-62/?prefix=obj11 2024-04-13T22:35:39.547 DEBUG:boto:Data: 2024-04-13T22:35:39.547 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.547 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.547 DEBUG:boto:Params: {} 2024-04-13T22:35:39.547 DEBUG:boto:Token: None 2024-04-13T22:35:39.547 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000046ea8e5e86b8fa-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.550 DEBUG:boto:b'vwnfns-62obj111000falseobj112024-04-13T22:32:32.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:35:39.550 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.550 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.550 DEBUG:boto:path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.550 DEBUG:boto:auth_path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.551 DEBUG:boto:Method: GET 2024-04-13T22:35:39.551 DEBUG:boto:Path: /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.551 DEBUG:boto:Data: 2024-04-13T22:35:39.551 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.551 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.551 DEBUG:boto:Params: {} 2024-04-13T22:35:39.551 DEBUG:boto:Token: None 2024-04-13T22:35:39.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4= 2024-04-13T22:35:39.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4=', 'Content-Length': '0'} 2024-04-13T22:35:39.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76131f63eac5e0b-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.554 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.554 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.554 DEBUG:boto:path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.554 DEBUG:boto:auth_path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.554 DEBUG:boto:Method: GET 2024-04-13T22:35:39.554 DEBUG:boto:Path: /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.554 DEBUG:boto:Data: 2024-04-13T22:35:39.554 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.555 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.555 DEBUG:boto:Params: {} 2024-04-13T22:35:39.555 DEBUG:boto:Token: None 2024-04-13T22:35:39.555 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4= 2024-04-13T22:35:39.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4=', 'Content-Length': '0'} 2024-04-13T22:35:39.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6f9f2534b6a06c-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.557 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.557 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.558 DEBUG:boto:path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.558 DEBUG:boto:auth_path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.558 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.558 DEBUG:boto:Path: /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.558 DEBUG:boto:Data: 2024-04-13T22:35:39.558 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.558 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.558 DEBUG:boto:Params: {} 2024-04-13T22:35:39.558 DEBUG:boto:Token: None 2024-04-13T22:35:39.559 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.559 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fR9WgtZAQSN+wg6N9RzGOtH64l4= 2024-04-13T22:35:39.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fR9WgtZAQSN+wg6N9RzGOtH64l4=', 'Content-Length': '0'} 2024-04-13T22:35:39.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf85b512c857866-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.561 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.561 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.561 DEBUG:boto:path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.561 DEBUG:boto:auth_path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.561 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.561 DEBUG:boto:Path: /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.561 DEBUG:boto:Data: 2024-04-13T22:35:39.561 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.562 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.562 DEBUG:boto:Params: {} 2024-04-13T22:35:39.562 DEBUG:boto:Token: None 2024-04-13T22:35:39.562 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fR9WgtZAQSN+wg6N9RzGOtH64l4= 2024-04-13T22:35:39.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fR9WgtZAQSN+wg6N9RzGOtH64l4=', 'Content-Length': '0'} 2024-04-13T22:35:39.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071456eae45b7065f-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:35:39.564 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.564 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.564 DEBUG:boto:path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.564 DEBUG:boto:auth_path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.565 DEBUG:boto:Method: GET 2024-04-13T22:35:39.565 DEBUG:boto:Path: /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.565 DEBUG:boto:Data: 2024-04-13T22:35:39.565 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.565 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.565 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.565 DEBUG:boto:Params: {} 2024-04-13T22:35:39.565 DEBUG:boto:Token: None 2024-04-13T22:35:39.565 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4= 2024-04-13T22:35:39.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4=', 'Content-Length': '0'} 2024-04-13T22:35:39.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6aa212ebc2b1b15-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.567 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.567 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.567 DEBUG:boto:path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.568 DEBUG:boto:auth_path=/vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.568 DEBUG:boto:Method: GET 2024-04-13T22:35:39.568 DEBUG:boto:Path: /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.568 DEBUG:boto:Data: 2024-04-13T22:35:39.568 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.568 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.568 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.568 DEBUG:boto:Params: {} 2024-04-13T22:35:39.568 DEBUG:boto:Token: None 2024-04-13T22:35:39.568 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22?versionId=null 2024-04-13T22:35:39.569 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4= 2024-04-13T22:35:39.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wf8dvBcfVGy/rK34HE3m/4LEpW4=', 'Content-Length': '0'} 2024-04-13T22:35:39.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d28ef36983e4c176-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.570 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.571 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.571 DEBUG:boto:path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.571 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.571 DEBUG:boto:Method: GET 2024-04-13T22:35:39.571 DEBUG:boto:Path: /vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.571 DEBUG:boto:Data: 2024-04-13T22:35:39.571 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.571 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.571 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.571 DEBUG:boto:Params: {} 2024-04-13T22:35:39.571 DEBUG:boto:Token: None 2024-04-13T22:35:39.572 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.572 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050b79bda49d5360d-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.574 DEBUG:boto:b'vwnfns-62obj221000falseobj222024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.574 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.575 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.575 DEBUG:boto:path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.575 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.575 DEBUG:boto:Method: GET 2024-04-13T22:35:39.575 DEBUG:boto:Path: /vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.575 DEBUG:boto:Data: 2024-04-13T22:35:39.575 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.575 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.575 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.575 DEBUG:boto:Params: {} 2024-04-13T22:35:39.575 DEBUG:boto:Token: None 2024-04-13T22:35:39.575 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.576 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eba70ec41adc5482-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.578 DEBUG:boto:b'vwnfns-62obj221000falseobj222024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.579 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.579 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.579 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.579 DEBUG:boto:Path: /vwnfns-62/obj22 2024-04-13T22:35:39.579 DEBUG:boto:Data: 2024-04-13T22:35:39.579 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.579 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.579 DEBUG:boto:Params: {} 2024-04-13T22:35:39.579 DEBUG:boto:Token: None 2024-04-13T22:35:39.580 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22 2024-04-13T22:35:39.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aVNg/ZzVKLotce6/RPvFENpdncY= 2024-04-13T22:35:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aVNg/ZzVKLotce6/RPvFENpdncY=', 'Content-Length': '0'} 2024-04-13T22:35:39.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b807c37d2f1837-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.582 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.582 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.582 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.582 DEBUG:boto:Path: /vwnfns-62/obj22 2024-04-13T22:35:39.582 DEBUG:boto:Data: 2024-04-13T22:35:39.582 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.582 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.582 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.582 DEBUG:boto:Params: {} 2024-04-13T22:35:39.583 DEBUG:boto:Token: None 2024-04-13T22:35:39.583 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22 2024-04-13T22:35:39.583 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aVNg/ZzVKLotce6/RPvFENpdncY= 2024-04-13T22:35:39.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aVNg/ZzVKLotce6/RPvFENpdncY=', 'Content-Length': '0'} 2024-04-13T22:35:39.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebfb27babbf1af99-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:35:39.585 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.585 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.585 DEBUG:boto:Method: GET 2024-04-13T22:35:39.585 DEBUG:boto:Path: /vwnfns-62/obj22 2024-04-13T22:35:39.585 DEBUG:boto:Data: 2024-04-13T22:35:39.585 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.585 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.585 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.585 DEBUG:boto:Params: {} 2024-04-13T22:35:39.586 DEBUG:boto:Token: None 2024-04-13T22:35:39.586 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22 2024-04-13T22:35:39.586 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5wYUJLUiCmnQo5xv7tDXYwV9Jtw= 2024-04-13T22:35:39.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5wYUJLUiCmnQo5xv7tDXYwV9Jtw=', 'Content-Length': '0'} 2024-04-13T22:35:39.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef842aba5e15913a-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.588 DEBUG:boto:path=/vwnfns-62/obj22 2024-04-13T22:35:39.588 DEBUG:boto:auth_path=/vwnfns-62/obj22 2024-04-13T22:35:39.588 DEBUG:boto:Method: GET 2024-04-13T22:35:39.588 DEBUG:boto:Path: /vwnfns-62/obj22 2024-04-13T22:35:39.588 DEBUG:boto:Data: 2024-04-13T22:35:39.588 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.588 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.588 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.588 DEBUG:boto:Params: {} 2024-04-13T22:35:39.589 DEBUG:boto:Token: None 2024-04-13T22:35:39.589 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj22 2024-04-13T22:35:39.589 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5wYUJLUiCmnQo5xv7tDXYwV9Jtw= 2024-04-13T22:35:39.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5wYUJLUiCmnQo5xv7tDXYwV9Jtw=', 'Content-Length': '0'} 2024-04-13T22:35:39.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003367c372f5031044-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.591 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.591 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.591 DEBUG:boto:path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.591 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.591 DEBUG:boto:Method: GET 2024-04-13T22:35:39.591 DEBUG:boto:Path: /vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.591 DEBUG:boto:Data: 2024-04-13T22:35:39.591 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.591 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.591 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.592 DEBUG:boto:Params: {} 2024-04-13T22:35:39.592 DEBUG:boto:Token: None 2024-04-13T22:35:39.592 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.592 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026446855912a93c3-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.595 DEBUG:boto:b'vwnfns-62obj221000falseobj222024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.595 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.595 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.595 DEBUG:boto:path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.595 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.595 DEBUG:boto:Method: GET 2024-04-13T22:35:39.595 DEBUG:boto:Path: /vwnfns-62/?prefix=obj22 2024-04-13T22:35:39.595 DEBUG:boto:Data: 2024-04-13T22:35:39.595 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.596 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.596 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.596 DEBUG:boto:Params: {} 2024-04-13T22:35:39.596 DEBUG:boto:Token: None 2024-04-13T22:35:39.596 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.596 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024929a01172fb019-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.642 DEBUG:boto:b'vwnfns-62obj221000falseobj222024-04-13T22:32:32.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:35:39.643 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.643 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.643 DEBUG:boto:path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.643 DEBUG:boto:auth_path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.643 DEBUG:boto:Method: GET 2024-04-13T22:35:39.643 DEBUG:boto:Path: /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.643 DEBUG:boto:Data: 2024-04-13T22:35:39.643 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.643 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.644 DEBUG:boto:Params: {} 2024-04-13T22:35:39.644 DEBUG:boto:Token: None 2024-04-13T22:35:39.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U= 2024-04-13T22:35:39.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U=', 'Content-Length': '0'} 2024-04-13T22:35:39.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa665ff1a4b30c48-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.646 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.646 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.646 DEBUG:boto:path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.646 DEBUG:boto:auth_path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.646 DEBUG:boto:Method: GET 2024-04-13T22:35:39.647 DEBUG:boto:Path: /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.647 DEBUG:boto:Data: 2024-04-13T22:35:39.647 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.647 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.647 DEBUG:boto:Params: {} 2024-04-13T22:35:39.647 DEBUG:boto:Token: None 2024-04-13T22:35:39.647 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U= 2024-04-13T22:35:39.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U=', 'Content-Length': '0'} 2024-04-13T22:35:39.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000328ec62cacaec584-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.649 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.649 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.649 DEBUG:boto:path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.650 DEBUG:boto:auth_path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.650 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.650 DEBUG:boto:Path: /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.650 DEBUG:boto:Data: 2024-04-13T22:35:39.650 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.650 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.650 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.650 DEBUG:boto:Params: {} 2024-04-13T22:35:39.650 DEBUG:boto:Token: None 2024-04-13T22:35:39.651 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oRmB+RJ+eoxkXwBWRSNiMGDhcO8= 2024-04-13T22:35:39.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oRmB+RJ+eoxkXwBWRSNiMGDhcO8=', 'Content-Length': '0'} 2024-04-13T22:35:39.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9330a8d672ae4ae-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.652 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.653 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.653 DEBUG:boto:path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.653 DEBUG:boto:auth_path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.653 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.653 DEBUG:boto:Path: /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.653 DEBUG:boto:Data: 2024-04-13T22:35:39.653 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.653 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.653 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.653 DEBUG:boto:Params: {} 2024-04-13T22:35:39.653 DEBUG:boto:Token: None 2024-04-13T22:35:39.654 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.654 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oRmB+RJ+eoxkXwBWRSNiMGDhcO8= 2024-04-13T22:35:39.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oRmB+RJ+eoxkXwBWRSNiMGDhcO8=', 'Content-Length': '0'} 2024-04-13T22:35:39.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007871f4e421437b57-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:35:39.656 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.656 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.656 DEBUG:boto:path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.656 DEBUG:boto:auth_path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.656 DEBUG:boto:Method: GET 2024-04-13T22:35:39.656 DEBUG:boto:Path: /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.656 DEBUG:boto:Data: 2024-04-13T22:35:39.656 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.656 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.656 DEBUG:boto:Params: {} 2024-04-13T22:35:39.657 DEBUG:boto:Token: None 2024-04-13T22:35:39.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U= 2024-04-13T22:35:39.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U=', 'Content-Length': '0'} 2024-04-13T22:35:39.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae41324ebb99761e-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.659 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.659 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.659 DEBUG:boto:path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.659 DEBUG:boto:auth_path=/vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.659 DEBUG:boto:Method: GET 2024-04-13T22:35:39.659 DEBUG:boto:Path: /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.659 DEBUG:boto:Data: 2024-04-13T22:35:39.659 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.660 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.660 DEBUG:boto:Params: {} 2024-04-13T22:35:39.660 DEBUG:boto:Token: None 2024-04-13T22:35:39.660 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33?versionId=null 2024-04-13T22:35:39.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U= 2024-04-13T22:35:39.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:27QgpSuwumtbnko2xeHLAhmaZ6U=', 'Content-Length': '0'} 2024-04-13T22:35:39.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13fc6845f07f2cd-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.662 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.662 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.662 DEBUG:boto:path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.662 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.662 DEBUG:boto:Method: GET 2024-04-13T22:35:39.663 DEBUG:boto:Path: /vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.663 DEBUG:boto:Data: 2024-04-13T22:35:39.663 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.663 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.663 DEBUG:boto:Params: {} 2024-04-13T22:35:39.663 DEBUG:boto:Token: None 2024-04-13T22:35:39.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.663 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000821a44b58cc30df5-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.706 DEBUG:boto:b'vwnfns-62obj331000falseobj332024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.707 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.707 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.707 DEBUG:boto:path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.707 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.707 DEBUG:boto:Method: GET 2024-04-13T22:35:39.707 DEBUG:boto:Path: /vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.707 DEBUG:boto:Data: 2024-04-13T22:35:39.707 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.708 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.708 DEBUG:boto:Params: {} 2024-04-13T22:35:39.708 DEBUG:boto:Token: None 2024-04-13T22:35:39.708 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4578dadced27901-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.710 DEBUG:boto:b'vwnfns-62obj331000falseobj332024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.711 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.711 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.711 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.711 DEBUG:boto:Path: /vwnfns-62/obj33 2024-04-13T22:35:39.711 DEBUG:boto:Data: 2024-04-13T22:35:39.711 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.711 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.712 DEBUG:boto:Params: {} 2024-04-13T22:35:39.712 DEBUG:boto:Token: None 2024-04-13T22:35:39.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33 2024-04-13T22:35:39.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TQT7DZMw8Un/6mWX+XFvK/Hp5Wc= 2024-04-13T22:35:39.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TQT7DZMw8Un/6mWX+XFvK/Hp5Wc=', 'Content-Length': '0'} 2024-04-13T22:35:39.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dea26863766b92f-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.714 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.714 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.714 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.714 DEBUG:boto:Path: /vwnfns-62/obj33 2024-04-13T22:35:39.714 DEBUG:boto:Data: 2024-04-13T22:35:39.714 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.715 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.715 DEBUG:boto:Params: {} 2024-04-13T22:35:39.715 DEBUG:boto:Token: None 2024-04-13T22:35:39.715 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33 2024-04-13T22:35:39.715 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TQT7DZMw8Un/6mWX+XFvK/Hp5Wc= 2024-04-13T22:35:39.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TQT7DZMw8Un/6mWX+XFvK/Hp5Wc=', 'Content-Length': '0'} 2024-04-13T22:35:39.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a963946eee809e-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:35:39.717 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.717 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.717 DEBUG:boto:Method: GET 2024-04-13T22:35:39.717 DEBUG:boto:Path: /vwnfns-62/obj33 2024-04-13T22:35:39.717 DEBUG:boto:Data: 2024-04-13T22:35:39.717 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.717 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.717 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.717 DEBUG:boto:Params: {} 2024-04-13T22:35:39.717 DEBUG:boto:Token: None 2024-04-13T22:35:39.718 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33 2024-04-13T22:35:39.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Bdpn1m2XbLrOWtROd++KUEhuK4= 2024-04-13T22:35:39.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Bdpn1m2XbLrOWtROd++KUEhuK4=', 'Content-Length': '0'} 2024-04-13T22:35:39.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd96c8d3968463e-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.719 DEBUG:boto:path=/vwnfns-62/obj33 2024-04-13T22:35:39.720 DEBUG:boto:auth_path=/vwnfns-62/obj33 2024-04-13T22:35:39.720 DEBUG:boto:Method: GET 2024-04-13T22:35:39.720 DEBUG:boto:Path: /vwnfns-62/obj33 2024-04-13T22:35:39.720 DEBUG:boto:Data: 2024-04-13T22:35:39.720 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.720 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.720 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.720 DEBUG:boto:Params: {} 2024-04-13T22:35:39.720 DEBUG:boto:Token: None 2024-04-13T22:35:39.720 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj33 2024-04-13T22:35:39.721 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Bdpn1m2XbLrOWtROd++KUEhuK4= 2024-04-13T22:35:39.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Bdpn1m2XbLrOWtROd++KUEhuK4=', 'Content-Length': '0'} 2024-04-13T22:35:39.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdd845010bd78d73-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.723 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.723 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.723 DEBUG:boto:path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.723 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.723 DEBUG:boto:Method: GET 2024-04-13T22:35:39.723 DEBUG:boto:Path: /vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.723 DEBUG:boto:Data: 2024-04-13T22:35:39.723 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.724 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.724 DEBUG:boto:Params: {} 2024-04-13T22:35:39.724 DEBUG:boto:Token: None 2024-04-13T22:35:39.724 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.724 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143903240516aacc-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.727 DEBUG:boto:b'vwnfns-62obj331000falseobj332024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.727 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.727 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.728 DEBUG:boto:path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.728 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.728 DEBUG:boto:Method: GET 2024-04-13T22:35:39.728 DEBUG:boto:Path: /vwnfns-62/?prefix=obj33 2024-04-13T22:35:39.728 DEBUG:boto:Data: 2024-04-13T22:35:39.728 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.728 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.728 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.728 DEBUG:boto:Params: {} 2024-04-13T22:35:39.728 DEBUG:boto:Token: None 2024-04-13T22:35:39.728 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.729 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc394eb9bbe2e59-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.774 DEBUG:boto:b'vwnfns-62obj331000falseobj332024-04-13T22:32:32.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:35:39.775 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.775 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.775 DEBUG:boto:path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.775 DEBUG:boto:auth_path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.775 DEBUG:boto:Method: GET 2024-04-13T22:35:39.775 DEBUG:boto:Path: /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.775 DEBUG:boto:Data: 2024-04-13T22:35:39.776 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.776 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.776 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.776 DEBUG:boto:Params: {} 2024-04-13T22:35:39.776 DEBUG:boto:Token: None 2024-04-13T22:35:39.776 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo= 2024-04-13T22:35:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo=', 'Content-Length': '0'} 2024-04-13T22:35:39.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9a8b6b3004bdf3-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.778 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.778 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.778 DEBUG:boto:path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.778 DEBUG:boto:auth_path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.779 DEBUG:boto:Method: GET 2024-04-13T22:35:39.779 DEBUG:boto:Path: /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.779 DEBUG:boto:Data: 2024-04-13T22:35:39.779 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.779 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.779 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.779 DEBUG:boto:Params: {} 2024-04-13T22:35:39.779 DEBUG:boto:Token: None 2024-04-13T22:35:39.779 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.779 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo= 2024-04-13T22:35:39.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo=', 'Content-Length': '0'} 2024-04-13T22:35:39.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f24f9418c56a84-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.781 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.782 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.782 DEBUG:boto:path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.782 DEBUG:boto:auth_path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.782 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.782 DEBUG:boto:Path: /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.782 DEBUG:boto:Data: 2024-04-13T22:35:39.782 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.782 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.782 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.782 DEBUG:boto:Params: {} 2024-04-13T22:35:39.783 DEBUG:boto:Token: None 2024-04-13T22:35:39.783 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.783 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dbLHmKSPheMr4apZMCfC0UQTdjo= 2024-04-13T22:35:39.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dbLHmKSPheMr4apZMCfC0UQTdjo=', 'Content-Length': '0'} 2024-04-13T22:35:39.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e33212884c001e4-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.785 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.785 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.785 DEBUG:boto:path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.785 DEBUG:boto:auth_path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.785 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.785 DEBUG:boto:Path: /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.785 DEBUG:boto:Data: 2024-04-13T22:35:39.785 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.785 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.785 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.786 DEBUG:boto:Params: {} 2024-04-13T22:35:39.786 DEBUG:boto:Token: None 2024-04-13T22:35:39.786 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.786 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dbLHmKSPheMr4apZMCfC0UQTdjo= 2024-04-13T22:35:39.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dbLHmKSPheMr4apZMCfC0UQTdjo=', 'Content-Length': '0'} 2024-04-13T22:35:39.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ba59ecb1e56a73-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:35:39.788 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.788 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.788 DEBUG:boto:path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.788 DEBUG:boto:auth_path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.788 DEBUG:boto:Method: GET 2024-04-13T22:35:39.788 DEBUG:boto:Path: /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.788 DEBUG:boto:Data: 2024-04-13T22:35:39.788 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.789 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.789 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.789 DEBUG:boto:Params: {} 2024-04-13T22:35:39.789 DEBUG:boto:Token: None 2024-04-13T22:35:39.789 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo= 2024-04-13T22:35:39.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo=', 'Content-Length': '0'} 2024-04-13T22:35:39.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f65ffb22d5454ff8-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.791 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.791 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.791 DEBUG:boto:path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.791 DEBUG:boto:auth_path=/vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.791 DEBUG:boto:Method: GET 2024-04-13T22:35:39.791 DEBUG:boto:Path: /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.792 DEBUG:boto:Data: 2024-04-13T22:35:39.792 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.792 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.792 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.792 DEBUG:boto:Params: {} 2024-04-13T22:35:39.792 DEBUG:boto:Token: None 2024-04-13T22:35:39.792 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44?versionId=null 2024-04-13T22:35:39.792 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo= 2024-04-13T22:35:39.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aAH0kcyvKJ4+G4X8j4tO1/E9IOo=', 'Content-Length': '0'} 2024-04-13T22:35:39.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d7d9b9fad28893-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.794 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.794 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.794 DEBUG:boto:path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.795 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.795 DEBUG:boto:Method: GET 2024-04-13T22:35:39.795 DEBUG:boto:Path: /vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.795 DEBUG:boto:Data: 2024-04-13T22:35:39.795 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.795 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.795 DEBUG:boto:Params: {} 2024-04-13T22:35:39.795 DEBUG:boto:Token: None 2024-04-13T22:35:39.795 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.795 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d423987509515ae8-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.798 DEBUG:boto:b'vwnfns-62obj441000falseobj442024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.798 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.798 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.798 DEBUG:boto:path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.798 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.798 DEBUG:boto:Method: GET 2024-04-13T22:35:39.798 DEBUG:boto:Path: /vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.798 DEBUG:boto:Data: 2024-04-13T22:35:39.799 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.799 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.799 DEBUG:boto:Params: {} 2024-04-13T22:35:39.799 DEBUG:boto:Token: None 2024-04-13T22:35:39.799 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.799 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053eb550b2e8c92da-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.801 DEBUG:boto:b'vwnfns-62obj441000falseobj442024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.802 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.802 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.802 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.802 DEBUG:boto:Path: /vwnfns-62/obj44 2024-04-13T22:35:39.802 DEBUG:boto:Data: 2024-04-13T22:35:39.802 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.802 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.802 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.803 DEBUG:boto:Params: {} 2024-04-13T22:35:39.803 DEBUG:boto:Token: None 2024-04-13T22:35:39.803 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44 2024-04-13T22:35:39.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UFQPCAj6SRR4k9umD/zNGgnqFXM= 2024-04-13T22:35:39.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UFQPCAj6SRR4k9umD/zNGgnqFXM=', 'Content-Length': '0'} 2024-04-13T22:35:39.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1875118cfc8a4c-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.805 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.805 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.805 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.805 DEBUG:boto:Path: /vwnfns-62/obj44 2024-04-13T22:35:39.805 DEBUG:boto:Data: 2024-04-13T22:35:39.805 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.805 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.805 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.805 DEBUG:boto:Params: {} 2024-04-13T22:35:39.805 DEBUG:boto:Token: None 2024-04-13T22:35:39.806 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44 2024-04-13T22:35:39.806 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UFQPCAj6SRR4k9umD/zNGgnqFXM= 2024-04-13T22:35:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UFQPCAj6SRR4k9umD/zNGgnqFXM=', 'Content-Length': '0'} 2024-04-13T22:35:39.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81552d77f90f264-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:35:39.807 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.808 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.808 DEBUG:boto:Method: GET 2024-04-13T22:35:39.808 DEBUG:boto:Path: /vwnfns-62/obj44 2024-04-13T22:35:39.808 DEBUG:boto:Data: 2024-04-13T22:35:39.808 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.808 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.808 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.808 DEBUG:boto:Params: {} 2024-04-13T22:35:39.808 DEBUG:boto:Token: None 2024-04-13T22:35:39.808 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44 2024-04-13T22:35:39.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JEwc9PpZtMcBF2v3aonIAQiYeOc= 2024-04-13T22:35:39.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JEwc9PpZtMcBF2v3aonIAQiYeOc=', 'Content-Length': '0'} 2024-04-13T22:35:39.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4f7ee5bced80e5-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.810 DEBUG:boto:path=/vwnfns-62/obj44 2024-04-13T22:35:39.810 DEBUG:boto:auth_path=/vwnfns-62/obj44 2024-04-13T22:35:39.810 DEBUG:boto:Method: GET 2024-04-13T22:35:39.811 DEBUG:boto:Path: /vwnfns-62/obj44 2024-04-13T22:35:39.811 DEBUG:boto:Data: 2024-04-13T22:35:39.811 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.811 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.811 DEBUG:boto:Params: {} 2024-04-13T22:35:39.811 DEBUG:boto:Token: None 2024-04-13T22:35:39.811 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj44 2024-04-13T22:35:39.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JEwc9PpZtMcBF2v3aonIAQiYeOc= 2024-04-13T22:35:39.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JEwc9PpZtMcBF2v3aonIAQiYeOc=', 'Content-Length': '0'} 2024-04-13T22:35:39.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e6577f135b8f69-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.813 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.813 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.813 DEBUG:boto:path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.813 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.813 DEBUG:boto:Method: GET 2024-04-13T22:35:39.814 DEBUG:boto:Path: /vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.814 DEBUG:boto:Data: 2024-04-13T22:35:39.814 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.814 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.814 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.814 DEBUG:boto:Params: {} 2024-04-13T22:35:39.814 DEBUG:boto:Token: None 2024-04-13T22:35:39.814 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.814 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a9895dae4ee332-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.816 DEBUG:boto:b'vwnfns-62obj441000falseobj442024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.817 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.817 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.817 DEBUG:boto:path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.817 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.817 DEBUG:boto:Method: GET 2024-04-13T22:35:39.817 DEBUG:boto:Path: /vwnfns-62/?prefix=obj44 2024-04-13T22:35:39.817 DEBUG:boto:Data: 2024-04-13T22:35:39.817 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.818 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.818 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.818 DEBUG:boto:Params: {} 2024-04-13T22:35:39.818 DEBUG:boto:Token: None 2024-04-13T22:35:39.818 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.818 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f136253c6f90f1-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.820 DEBUG:boto:b'vwnfns-62obj441000falseobj442024-04-13T22:32:32.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:35:39.821 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.821 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.821 DEBUG:boto:path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.821 DEBUG:boto:auth_path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.821 DEBUG:boto:Method: GET 2024-04-13T22:35:39.821 DEBUG:boto:Path: /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.821 DEBUG:boto:Data: 2024-04-13T22:35:39.822 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.822 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.822 DEBUG:boto:Params: {} 2024-04-13T22:35:39.822 DEBUG:boto:Token: None 2024-04-13T22:35:39.822 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.822 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk= 2024-04-13T22:35:39.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk=', 'Content-Length': '0'} 2024-04-13T22:35:39.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce1b249b779749b-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.824 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.824 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.824 DEBUG:boto:path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.824 DEBUG:boto:auth_path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.824 DEBUG:boto:Method: GET 2024-04-13T22:35:39.824 DEBUG:boto:Path: /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.825 DEBUG:boto:Data: 2024-04-13T22:35:39.825 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.825 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.825 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.825 DEBUG:boto:Params: {} 2024-04-13T22:35:39.825 DEBUG:boto:Token: None 2024-04-13T22:35:39.825 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.825 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk= 2024-04-13T22:35:39.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk=', 'Content-Length': '0'} 2024-04-13T22:35:39.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e8a041073ff203-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.827 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.827 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.827 DEBUG:boto:path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.827 DEBUG:boto:auth_path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.827 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.828 DEBUG:boto:Path: /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.828 DEBUG:boto:Data: 2024-04-13T22:35:39.828 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.828 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.828 DEBUG:boto:Params: {} 2024-04-13T22:35:39.828 DEBUG:boto:Token: None 2024-04-13T22:35:39.828 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.828 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fHxDc4OozG4W3mxq9IJqLayQcNY= 2024-04-13T22:35:39.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fHxDc4OozG4W3mxq9IJqLayQcNY=', 'Content-Length': '0'} 2024-04-13T22:35:39.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebff895b7b2b4b28-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.830 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.830 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.830 DEBUG:boto:path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.831 DEBUG:boto:auth_path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.831 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.831 DEBUG:boto:Path: /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.831 DEBUG:boto:Data: 2024-04-13T22:35:39.831 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.831 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.831 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.831 DEBUG:boto:Params: {} 2024-04-13T22:35:39.831 DEBUG:boto:Token: None 2024-04-13T22:35:39.831 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.832 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fHxDc4OozG4W3mxq9IJqLayQcNY= 2024-04-13T22:35:39.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fHxDc4OozG4W3mxq9IJqLayQcNY=', 'Content-Length': '0'} 2024-04-13T22:35:39.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1fa32fafa6bf14d-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:35:39.833 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.833 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.833 DEBUG:boto:path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.834 DEBUG:boto:auth_path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.834 DEBUG:boto:Method: GET 2024-04-13T22:35:39.834 DEBUG:boto:Path: /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.834 DEBUG:boto:Data: 2024-04-13T22:35:39.834 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.834 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.834 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.834 DEBUG:boto:Params: {} 2024-04-13T22:35:39.834 DEBUG:boto:Token: None 2024-04-13T22:35:39.834 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.835 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk= 2024-04-13T22:35:39.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk=', 'Content-Length': '0'} 2024-04-13T22:35:39.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857a30639c173bb7-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.836 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.836 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.837 DEBUG:boto:path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.837 DEBUG:boto:auth_path=/vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.837 DEBUG:boto:Method: GET 2024-04-13T22:35:39.837 DEBUG:boto:Path: /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.837 DEBUG:boto:Data: 2024-04-13T22:35:39.837 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.837 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.837 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.837 DEBUG:boto:Params: {} 2024-04-13T22:35:39.837 DEBUG:boto:Token: None 2024-04-13T22:35:39.838 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55?versionId=null 2024-04-13T22:35:39.838 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk= 2024-04-13T22:35:39.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1wIyPdHOqyGACqmEwsejRCaHRsk=', 'Content-Length': '0'} 2024-04-13T22:35:39.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afcc73cbf05d9e7b-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.840 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.840 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.840 DEBUG:boto:path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.840 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.840 DEBUG:boto:Method: GET 2024-04-13T22:35:39.840 DEBUG:boto:Path: /vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.840 DEBUG:boto:Data: 2024-04-13T22:35:39.840 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.840 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.840 DEBUG:boto:Params: {} 2024-04-13T22:35:39.841 DEBUG:boto:Token: None 2024-04-13T22:35:39.841 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.841 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9c722173a0d33d-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.843 DEBUG:boto:b'vwnfns-62obj551000falseobj552024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.843 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.844 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.844 DEBUG:boto:path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.844 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.844 DEBUG:boto:Method: GET 2024-04-13T22:35:39.844 DEBUG:boto:Path: /vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.844 DEBUG:boto:Data: 2024-04-13T22:35:39.844 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.844 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.844 DEBUG:boto:Params: {} 2024-04-13T22:35:39.844 DEBUG:boto:Token: None 2024-04-13T22:35:39.845 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.845 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce2441f6244915a-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.847 DEBUG:boto:b'vwnfns-62obj551000falseobj552024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.847 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.847 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.847 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.847 DEBUG:boto:Path: /vwnfns-62/obj55 2024-04-13T22:35:39.848 DEBUG:boto:Data: 2024-04-13T22:35:39.848 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.848 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.848 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.848 DEBUG:boto:Params: {} 2024-04-13T22:35:39.848 DEBUG:boto:Token: None 2024-04-13T22:35:39.848 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55 2024-04-13T22:35:39.848 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nLeSEDHEHVhEuQYSnBixfhCU7qk= 2024-04-13T22:35:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nLeSEDHEHVhEuQYSnBixfhCU7qk=', 'Content-Length': '0'} 2024-04-13T22:35:39.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9fdfb993b2e196f-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.850 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.850 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.850 DEBUG:boto:Method: HEAD 2024-04-13T22:35:39.850 DEBUG:boto:Path: /vwnfns-62/obj55 2024-04-13T22:35:39.850 DEBUG:boto:Data: 2024-04-13T22:35:39.850 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.851 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.851 DEBUG:boto:Params: {} 2024-04-13T22:35:39.851 DEBUG:boto:Token: None 2024-04-13T22:35:39.851 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55 2024-04-13T22:35:39.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nLeSEDHEHVhEuQYSnBixfhCU7qk= 2024-04-13T22:35:39.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nLeSEDHEHVhEuQYSnBixfhCU7qk=', 'Content-Length': '0'} 2024-04-13T22:35:39.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5e929c565dcebf-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:35:39.853 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.853 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.853 DEBUG:boto:Method: GET 2024-04-13T22:35:39.853 DEBUG:boto:Path: /vwnfns-62/obj55 2024-04-13T22:35:39.853 DEBUG:boto:Data: 2024-04-13T22:35:39.853 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.853 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.853 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.853 DEBUG:boto:Params: {} 2024-04-13T22:35:39.854 DEBUG:boto:Token: None 2024-04-13T22:35:39.854 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55 2024-04-13T22:35:39.854 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DxP1Kb1ujMiXdraUsIS1R25G+o8= 2024-04-13T22:35:39.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DxP1Kb1ujMiXdraUsIS1R25G+o8=', 'Content-Length': '0'} 2024-04-13T22:35:39.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-62:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af53a0e879863dbc-00661b08bb-22a6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.898 DEBUG:boto:path=/vwnfns-62/obj55 2024-04-13T22:35:39.898 DEBUG:boto:auth_path=/vwnfns-62/obj55 2024-04-13T22:35:39.898 DEBUG:boto:Method: GET 2024-04-13T22:35:39.899 DEBUG:boto:Path: /vwnfns-62/obj55 2024-04-13T22:35:39.899 DEBUG:boto:Data: 2024-04-13T22:35:39.899 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.899 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.899 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.899 DEBUG:boto:Params: {} 2024-04-13T22:35:39.899 DEBUG:boto:Token: None 2024-04-13T22:35:39.899 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/obj55 2024-04-13T22:35:39.899 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DxP1Kb1ujMiXdraUsIS1R25G+o8= 2024-04-13T22:35:39.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DxP1Kb1ujMiXdraUsIS1R25G+o8=', 'Content-Length': '0'} 2024-04-13T22:35:39.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000650f9f1534ed6af9-00661b08bb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.901 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.902 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.902 DEBUG:boto:path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.902 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.902 DEBUG:boto:Method: GET 2024-04-13T22:35:39.902 DEBUG:boto:Path: /vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.902 DEBUG:boto:Data: 2024-04-13T22:35:39.902 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:39.902 DEBUG:boto:Port: 8001 2024-04-13T22:35:39.902 DEBUG:boto:Params: {} 2024-04-13T22:35:39.903 DEBUG:boto:Token: None 2024-04-13T22:35:39.903 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.903 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab5be3158df0b1a-00661b08bb-22a6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.905 DEBUG:boto:b'vwnfns-62obj551000falseobj552024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.905 DEBUG:boto:path=/vwnfns-62/ 2024-04-13T22:35:39.905 DEBUG:boto:auth_path=/vwnfns-62/ 2024-04-13T22:35:39.905 DEBUG:boto:path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.906 DEBUG:boto:auth_path=/vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.906 DEBUG:boto:Method: GET 2024-04-13T22:35:39.906 DEBUG:boto:Path: /vwnfns-62/?prefix=obj55 2024-04-13T22:35:39.906 DEBUG:boto:Data: 2024-04-13T22:35:39.906 DEBUG:boto:Headers: {} 2024-04-13T22:35:39.906 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:39.906 DEBUG:boto:Port: 8000 2024-04-13T22:35:39.906 DEBUG:boto:Params: {} 2024-04-13T22:35:39.906 DEBUG:boto:Token: None 2024-04-13T22:35:39.906 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:35:39 GMT /vwnfns-62/ 2024-04-13T22:35:39.907 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w= 2024-04-13T22:35:39.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:39 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6qGA51mc8+SCSlJS4htazBvBp2w=', 'Content-Length': '0'} 2024-04-13T22:35:39.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434a9f43a312dfa1-00661b08bb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:39.909 DEBUG:boto:b'vwnfns-62obj551000falseobj552024-04-13T22:32:32.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:35:39.909 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-62 zones={a2, a1} 2024-04-13T22:35:39.909 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-13T22:35:39.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:39.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:39.910 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:35:39.956 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:39.956 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:40.006 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:35:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000028757", 2024-04-13T22:35:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:32:39.247243Z" 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000027529", 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:32:39.248984Z" 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000020303", 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:28:56.110637Z" 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000032993", 2024-04-13T22:35:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:32:32.929859Z" 2024-04-13T22:35:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.008 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:35:40.009 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000028757', 1: '00000000000000000000:00000000000000027529', 2: '00000000000000000000:00000000000000020303', 3: '00000000000000000000:00000000000000032993'} 2024-04-13T22:35:40.009 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:35:40.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.009 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:35:40.058 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:40.058 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:40.103 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:35:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:35:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:35:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:35:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:35:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 7263913088590904495 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000028757", 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:39.247243Z" 2024-04-13T22:35:40.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000027529", 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:39.248984Z" 2024-04-13T22:35:40.105 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000020303", 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:28:56.110637Z" 2024-04-13T22:35:40.106 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000032993", 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 53, 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:32.929859Z" 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 62, 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 62 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:35:40Z" 2024-04-13T22:35:40.108 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:35:40.109 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7263913088590904495 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028757", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:32:39.247243Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027529", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:32:39.248984Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020303", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:28:56.110637Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032993", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2024-04-13T22:32:32.929859Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2024-04-13T22:35:40Z" } 2024-04-13T22:35:40.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028757', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:32:39.247243Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027529', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:32:39.248984Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020303', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:28:56.110637Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032993', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2024-04-13T22:32:32.929859Z'}}] 2024-04-13T22:35:40.109 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028757', 1: '00000000000000000000:00000000000000027529', 2: '00000000000000000000:00000000000000020303', 3: '00000000000000000000:00000000000000032993'} 2024-04-13T22:35:40.109 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028757', 1: '00000000000000000000:00000000000000027529', 2: '00000000000000000000:00000000000000020303', 3: '00000000000000000000:00000000000000032993'} 2024-04-13T22:35:40.109 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:35:40.109 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-13T22:35:40.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.110 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:35:40.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:40.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:40.206 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000015313", 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:32:55.200789Z" 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000012382", 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:34:33.376893Z" 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.207 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006851", 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:48.019983Z" 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000017280", 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:32:36.579759Z" 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.208 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:35:40.209 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015313', 1: '00000000000000000000:00000000000000012382', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000017280'} 2024-04-13T22:35:40.209 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:35:40.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.210 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:35:40.259 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:40.259 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:40.302 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:35:40.303 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:35:40.303 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:35:40.303 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:35:40.303 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:35:40.303 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:35:40.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000015313", 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:55.200789Z" 2024-04-13T22:35:40.304 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000012382", 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.305 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:34:33.376893Z" 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000006851", 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.306 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:48.019983Z" 2024-04-13T22:35:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000017280", 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:32:36.579759Z" 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.308 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:35:40Z" 2024-04-13T22:35:40.309 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:35:40.311 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015313", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:32:55.200789Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012382", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:34:33.376893Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006851", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:48.019983Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017280", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:32:36.579759Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:35:40Z" } 2024-04-13T22:35:40.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015313', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:32:55.200789Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012382', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:34:33.376893Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006851', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:48.019983Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017280', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:32:36.579759Z'}}] 2024-04-13T22:35:40.311 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015313', 1: '00000000000000000000:00000000000000012382', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000017280'} 2024-04-13T22:35:40.311 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015313', 1: '00000000000000000000:00000000000000012382', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000017280'} 2024-04-13T22:35:40.311 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:35:40.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2024-04-13T22:35:40.313 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:40.313 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:40.313 DEBUG:boto:Method: PUT 2024-04-13T22:35:40.313 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:40.313 DEBUG:boto:Data: 2024-04-13T22:35:40.314 DEBUG:boto:Headers: {} 2024-04-13T22:35:40.314 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:40.314 DEBUG:boto:Port: 8000 2024-04-13T22:35:40.314 DEBUG:boto:Params: {} 2024-04-13T22:35:40.314 DEBUG:boto:Token: None 2024-04-13T22:35:40.314 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:35:40 GMT /vwnfns-63/ 2024-04-13T22:35:40.314 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+sCKc6S1+Aon+Pp2d0yT2I+3Evk= 2024-04-13T22:35:40.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+sCKc6S1+Aon+Pp2d0yT2I+3Evk=', 'Content-Length': '0'} 2024-04-13T22:35:40.328 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000233c73b11eadab9a-00661b08bc-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:40.328 DEBUG:rgw_multi.tests:created bucket=vwnfns-63 2024-04-13T22:35:40.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.329 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:35:40.377 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:40.377 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047740.325648_889.1", 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:35:40.325648Z" 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.424 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:35:40.425 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713047740.325648_889.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:35:40.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:35:40.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:35:40.426 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:35:40.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:40.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:40.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:40.499+0000 7f4b82a0b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:35:40.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:40.500+0000 7f4b82a0b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:35:40.516 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:35:40.516 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:35:40.516 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:35:40.516 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:35:40.516 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047740.325648_889.1", 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.517 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:35:40.325648Z", 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:35:40.518 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.519 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:35:40.520 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:35:40.521 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:35:40Z" 2024-04-13T22:35:40.522 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:35:40.523 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047740.325648_889.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:35:40.325648Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:35:40Z" } 2024-04-13T22:35:40.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047740.325648_889.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:35:40.325648Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:35:40.523 DEBUG:rgw_multi.tests:log_status={0: '1_1713047740.325648_889.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:35:40.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047740.325648_889.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:35:40.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:35:40.524 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:35:40.524 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:35:40.524 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:35:40.522+0000 7f05630ed700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 112291) UID: 0 2024-04-13T22:35:40.525 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:35:40.522+0000 7f05682c27c0 -1 shutting down 2024-04-13T22:35:44.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:35:46.626 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:35:46.626 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.626 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.627 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.627 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.627 DEBUG:boto:Data: 2024-04-13T22:35:46.627 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.628 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.628 DEBUG:boto:Params: {} 2024-04-13T22:35:46.628 DEBUG:boto:Token: None 2024-04-13T22:35:46.628 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003371a8a788292efb-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.632 DEBUG:boto:path=/vwnfns-63/obj11 2024-04-13T22:35:46.632 DEBUG:boto:auth_path=/vwnfns-63/obj11 2024-04-13T22:35:46.632 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.632 DEBUG:boto:Path: /vwnfns-63/obj11 2024-04-13T22:35:46.632 DEBUG:boto:Data: 2024-04-13T22:35:46.632 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.633 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.633 DEBUG:boto:Params: {} 2024-04-13T22:35:46.633 DEBUG:boto:Token: None 2024-04-13T22:35:46.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj11 2024-04-13T22:35:46.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mmtq0dYeXY1ps5YZpDTY/nboPhk= 2024-04-13T22:35:46.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mmtq0dYeXY1ps5YZpDTY/nboPhk='} 2024-04-13T22:35:46.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac2f3c06f161acd4-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.641 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.641 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.641 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.641 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.641 DEBUG:boto:Data: 2024-04-13T22:35:46.641 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.641 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.641 DEBUG:boto:Params: {} 2024-04-13T22:35:46.641 DEBUG:boto:Token: None 2024-04-13T22:35:46.641 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c22c3c9ac3db75f-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.645 DEBUG:boto:path=/vwnfns-63/obj22 2024-04-13T22:35:46.645 DEBUG:boto:auth_path=/vwnfns-63/obj22 2024-04-13T22:35:46.645 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.645 DEBUG:boto:Path: /vwnfns-63/obj22 2024-04-13T22:35:46.645 DEBUG:boto:Data: 2024-04-13T22:35:46.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.645 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.645 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.645 DEBUG:boto:Params: {} 2024-04-13T22:35:46.645 DEBUG:boto:Token: None 2024-04-13T22:35:46.646 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj22 2024-04-13T22:35:46.646 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mtmz1bypdMuJwfzmIrDXc4P9fZM= 2024-04-13T22:35:46.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mtmz1bypdMuJwfzmIrDXc4P9fZM='} 2024-04-13T22:35:46.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000616a0452e8dd2382-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.650 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.650 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.651 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.651 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.651 DEBUG:boto:Data: 2024-04-13T22:35:46.651 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.651 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.651 DEBUG:boto:Params: {} 2024-04-13T22:35:46.651 DEBUG:boto:Token: None 2024-04-13T22:35:46.651 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065ea72056e1f76df-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.654 DEBUG:boto:path=/vwnfns-63/obj33 2024-04-13T22:35:46.654 DEBUG:boto:auth_path=/vwnfns-63/obj33 2024-04-13T22:35:46.655 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.655 DEBUG:boto:Path: /vwnfns-63/obj33 2024-04-13T22:35:46.655 DEBUG:boto:Data: 2024-04-13T22:35:46.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.655 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.655 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.655 DEBUG:boto:Params: {} 2024-04-13T22:35:46.655 DEBUG:boto:Token: None 2024-04-13T22:35:46.655 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj33 2024-04-13T22:35:46.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:gva/bH4PI4Lcz0jzs9gVO0KkZUY= 2024-04-13T22:35:46.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gva/bH4PI4Lcz0jzs9gVO0KkZUY='} 2024-04-13T22:35:46.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b88811ecf8a9b0a-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.661 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.661 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.662 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.662 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.662 DEBUG:boto:Data: 2024-04-13T22:35:46.662 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.662 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.662 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.662 DEBUG:boto:Params: {} 2024-04-13T22:35:46.662 DEBUG:boto:Token: None 2024-04-13T22:35:46.662 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b5215dfe6ee5d9c-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.665 DEBUG:boto:path=/vwnfns-63/obj44 2024-04-13T22:35:46.665 DEBUG:boto:auth_path=/vwnfns-63/obj44 2024-04-13T22:35:46.665 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.665 DEBUG:boto:Path: /vwnfns-63/obj44 2024-04-13T22:35:46.665 DEBUG:boto:Data: 2024-04-13T22:35:46.665 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.666 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.666 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.666 DEBUG:boto:Params: {} 2024-04-13T22:35:46.666 DEBUG:boto:Token: None 2024-04-13T22:35:46.666 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj44 2024-04-13T22:35:46.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fj5xUQ438XDidh08Qov24bjjt2E= 2024-04-13T22:35:46.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fj5xUQ438XDidh08Qov24bjjt2E='} 2024-04-13T22:35:46.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb8d4f6f068190a7-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.711 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.711 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.711 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.711 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.711 DEBUG:boto:Data: 2024-04-13T22:35:46.711 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.711 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.712 DEBUG:boto:Params: {} 2024-04-13T22:35:46.712 DEBUG:boto:Token: None 2024-04-13T22:35:46.712 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.712 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000153a10c6595ec44d-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.715 DEBUG:boto:path=/vwnfns-63/obj55 2024-04-13T22:35:46.715 DEBUG:boto:auth_path=/vwnfns-63/obj55 2024-04-13T22:35:46.715 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.715 DEBUG:boto:Path: /vwnfns-63/obj55 2024-04-13T22:35:46.715 DEBUG:boto:Data: 2024-04-13T22:35:46.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.715 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.715 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.715 DEBUG:boto:Params: {} 2024-04-13T22:35:46.716 DEBUG:boto:Token: None 2024-04-13T22:35:46.716 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj55 2024-04-13T22:35:46.716 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:di/SuBc3smUX2mps4uwiv7TlV9g= 2024-04-13T22:35:46.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:di/SuBc3smUX2mps4uwiv7TlV9g='} 2024-04-13T22:35:46.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a547b43b7818571-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:46.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:46.721 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:46.776 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:46.777 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:46.820 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:35:46.820 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-63 2024-04-13T22:35:46.824 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 5 2024-04-13T22:35:46.862 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.862 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.862 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.862 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.862 DEBUG:boto:Data: 2024-04-13T22:35:46.863 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.863 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.863 DEBUG:boto:Params: {} 2024-04-13T22:35:46.863 DEBUG:boto:Token: None 2024-04-13T22:35:46.863 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e7f1c87bc59f35f-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.866 DEBUG:boto:path=/vwnfns-63/obj17 2024-04-13T22:35:46.866 DEBUG:boto:auth_path=/vwnfns-63/obj17 2024-04-13T22:35:46.866 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.866 DEBUG:boto:Path: /vwnfns-63/obj17 2024-04-13T22:35:46.867 DEBUG:boto:Data: 2024-04-13T22:35:46.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.867 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.867 DEBUG:boto:Params: {} 2024-04-13T22:35:46.867 DEBUG:boto:Token: None 2024-04-13T22:35:46.867 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj17 2024-04-13T22:35:46.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XZsB1NZzCx39274IvLkuus8oKuQ= 2024-04-13T22:35:46.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XZsB1NZzCx39274IvLkuus8oKuQ='} 2024-04-13T22:35:46.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056d32a3927a700c7-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.873 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.873 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.873 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.873 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.873 DEBUG:boto:Data: 2024-04-13T22:35:46.873 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.873 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.873 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.873 DEBUG:boto:Params: {} 2024-04-13T22:35:46.873 DEBUG:boto:Token: None 2024-04-13T22:35:46.874 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.874 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ace3f9b0a2b736b-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.876 DEBUG:boto:path=/vwnfns-63/obj34 2024-04-13T22:35:46.876 DEBUG:boto:auth_path=/vwnfns-63/obj34 2024-04-13T22:35:46.877 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.877 DEBUG:boto:Path: /vwnfns-63/obj34 2024-04-13T22:35:46.877 DEBUG:boto:Data: 2024-04-13T22:35:46.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.877 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.877 DEBUG:boto:Params: {} 2024-04-13T22:35:46.877 DEBUG:boto:Token: None 2024-04-13T22:35:46.877 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj34 2024-04-13T22:35:46.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oNfSXSsANEFhGBukAcCzn6n3AKk= 2024-04-13T22:35:46.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oNfSXSsANEFhGBukAcCzn6n3AKk='} 2024-04-13T22:35:46.923 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9b1a943f6fb74e8-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.923 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.923 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.923 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.924 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.924 DEBUG:boto:Data: 2024-04-13T22:35:46.924 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.924 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.924 DEBUG:boto:Params: {} 2024-04-13T22:35:46.924 DEBUG:boto:Token: None 2024-04-13T22:35:46.924 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ee13baf0ea61e7f-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.928 DEBUG:boto:path=/vwnfns-63/obj51 2024-04-13T22:35:46.928 DEBUG:boto:auth_path=/vwnfns-63/obj51 2024-04-13T22:35:46.928 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.928 DEBUG:boto:Path: /vwnfns-63/obj51 2024-04-13T22:35:46.928 DEBUG:boto:Data: 2024-04-13T22:35:46.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.928 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.928 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.928 DEBUG:boto:Params: {} 2024-04-13T22:35:46.928 DEBUG:boto:Token: None 2024-04-13T22:35:46.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj51 2024-04-13T22:35:46.929 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SzWCXu/qwvN5Fo2QPYN4EysZ/eI= 2024-04-13T22:35:46.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SzWCXu/qwvN5Fo2QPYN4EysZ/eI='} 2024-04-13T22:35:46.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000742e72a6da1a58b0-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.934 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.934 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.934 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.934 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.934 DEBUG:boto:Data: 2024-04-13T22:35:46.934 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.935 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.935 DEBUG:boto:Params: {} 2024-04-13T22:35:46.935 DEBUG:boto:Token: None 2024-04-13T22:35:46.935 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097410f8c83c5cacd-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.938 DEBUG:boto:path=/vwnfns-63/obj68 2024-04-13T22:35:46.938 DEBUG:boto:auth_path=/vwnfns-63/obj68 2024-04-13T22:35:46.938 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.938 DEBUG:boto:Path: /vwnfns-63/obj68 2024-04-13T22:35:46.938 DEBUG:boto:Data: 2024-04-13T22:35:46.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.938 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.939 DEBUG:boto:Params: {} 2024-04-13T22:35:46.939 DEBUG:boto:Token: None 2024-04-13T22:35:46.939 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj68 2024-04-13T22:35:46.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A8sHLZt6VP23M7b9XY4ISYu7ri0= 2024-04-13T22:35:46.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A8sHLZt6VP23M7b9XY4ISYu7ri0='} 2024-04-13T22:35:46.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000472aa395e40a5af4-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.983 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:46.983 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:46.983 DEBUG:boto:Method: HEAD 2024-04-13T22:35:46.983 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:46.983 DEBUG:boto:Data: 2024-04-13T22:35:46.983 DEBUG:boto:Headers: {} 2024-04-13T22:35:46.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.983 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.984 DEBUG:boto:Params: {} 2024-04-13T22:35:46.984 DEBUG:boto:Token: None 2024-04-13T22:35:46.984 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/ 2024-04-13T22:35:46.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T22:35:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T22:35:46.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b0e7f2a3d45ec60-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.987 DEBUG:boto:path=/vwnfns-63/obj85 2024-04-13T22:35:46.987 DEBUG:boto:auth_path=/vwnfns-63/obj85 2024-04-13T22:35:46.987 DEBUG:boto:Method: PUT 2024-04-13T22:35:46.987 DEBUG:boto:Path: /vwnfns-63/obj85 2024-04-13T22:35:46.987 DEBUG:boto:Data: 2024-04-13T22:35:46.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:46.987 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:46.987 DEBUG:boto:Port: 8000 2024-04-13T22:35:46.987 DEBUG:boto:Params: {} 2024-04-13T22:35:46.987 DEBUG:boto:Token: None 2024-04-13T22:35:46.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:46 GMT /vwnfns-63/obj85 2024-04-13T22:35:46.988 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oCT5y4ylBOhkDFkTMXyr8SfddC4= 2024-04-13T22:35:46.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oCT5y4ylBOhkDFkTMXyr8SfddC4='} 2024-04-13T22:35:46.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a86410e9619719e9-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:46.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:46.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:46.993 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:47.043 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:47.043 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:47.088 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:35:47.088 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-63 2024-04-13T22:35:47.094 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 10 2024-04-13T22:35:47.139 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.139 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.139 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.139 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.139 DEBUG:boto:Data: 2024-04-13T22:35:47.139 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.139 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.140 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.140 DEBUG:boto:Params: {} 2024-04-13T22:35:47.140 DEBUG:boto:Token: None 2024-04-13T22:35:47.140 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.140 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2f9e4a33c3ed899-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.143 DEBUG:boto:path=/vwnfns-63/obj19 2024-04-13T22:35:47.143 DEBUG:boto:auth_path=/vwnfns-63/obj19 2024-04-13T22:35:47.143 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.143 DEBUG:boto:Path: /vwnfns-63/obj19 2024-04-13T22:35:47.143 DEBUG:boto:Data: 2024-04-13T22:35:47.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.144 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.144 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.144 DEBUG:boto:Params: {} 2024-04-13T22:35:47.144 DEBUG:boto:Token: None 2024-04-13T22:35:47.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj19 2024-04-13T22:35:47.144 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9Hovjhmb+adWuw8+jzykFmKWsU4= 2024-04-13T22:35:47.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Hovjhmb+adWuw8+jzykFmKWsU4='} 2024-04-13T22:35:47.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000502d96fb01716e64-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.187 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.187 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.187 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.187 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.187 DEBUG:boto:Data: 2024-04-13T22:35:47.187 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.187 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.187 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.187 DEBUG:boto:Params: {} 2024-04-13T22:35:47.188 DEBUG:boto:Token: None 2024-04-13T22:35:47.188 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.188 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092ef0682c96f4d20-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.191 DEBUG:boto:path=/vwnfns-63/obj38 2024-04-13T22:35:47.191 DEBUG:boto:auth_path=/vwnfns-63/obj38 2024-04-13T22:35:47.191 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.191 DEBUG:boto:Path: /vwnfns-63/obj38 2024-04-13T22:35:47.191 DEBUG:boto:Data: 2024-04-13T22:35:47.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.191 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.191 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.191 DEBUG:boto:Params: {} 2024-04-13T22:35:47.191 DEBUG:boto:Token: None 2024-04-13T22:35:47.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj38 2024-04-13T22:35:47.192 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:72WCDr9f86mMNlO+2Tzs2gDSKsE= 2024-04-13T22:35:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:72WCDr9f86mMNlO+2Tzs2gDSKsE='} 2024-04-13T22:35:47.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe9f1f8c0727db6d-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.196 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.196 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.196 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.196 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.196 DEBUG:boto:Data: 2024-04-13T22:35:47.196 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.197 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.197 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.197 DEBUG:boto:Params: {} 2024-04-13T22:35:47.197 DEBUG:boto:Token: None 2024-04-13T22:35:47.197 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.197 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057bd96ede10acb64-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.200 DEBUG:boto:path=/vwnfns-63/obj57 2024-04-13T22:35:47.200 DEBUG:boto:auth_path=/vwnfns-63/obj57 2024-04-13T22:35:47.200 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.201 DEBUG:boto:Path: /vwnfns-63/obj57 2024-04-13T22:35:47.201 DEBUG:boto:Data: 2024-04-13T22:35:47.201 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.201 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.201 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.201 DEBUG:boto:Params: {} 2024-04-13T22:35:47.201 DEBUG:boto:Token: None 2024-04-13T22:35:47.201 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj57 2024-04-13T22:35:47.202 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QULUYiSEKraGYhDYv0d+eS2dNIw= 2024-04-13T22:35:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QULUYiSEKraGYhDYv0d+eS2dNIw='} 2024-04-13T22:35:47.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1bda648acec9215-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.247 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.247 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.247 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.247 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.247 DEBUG:boto:Data: 2024-04-13T22:35:47.247 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.248 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.248 DEBUG:boto:Params: {} 2024-04-13T22:35:47.248 DEBUG:boto:Token: None 2024-04-13T22:35:47.248 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.248 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddc90481796d05ea-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.251 DEBUG:boto:path=/vwnfns-63/obj76 2024-04-13T22:35:47.251 DEBUG:boto:auth_path=/vwnfns-63/obj76 2024-04-13T22:35:47.251 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.251 DEBUG:boto:Path: /vwnfns-63/obj76 2024-04-13T22:35:47.251 DEBUG:boto:Data: 2024-04-13T22:35:47.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.252 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.252 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.252 DEBUG:boto:Params: {} 2024-04-13T22:35:47.252 DEBUG:boto:Token: None 2024-04-13T22:35:47.252 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj76 2024-04-13T22:35:47.253 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pWPCH7IeL7mMZ5o3sItJRKslHwE= 2024-04-13T22:35:47.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pWPCH7IeL7mMZ5o3sItJRKslHwE='} 2024-04-13T22:35:47.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6c35469a8ac21d5-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.258 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.258 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.258 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.258 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.258 DEBUG:boto:Data: 2024-04-13T22:35:47.259 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.259 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.259 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.259 DEBUG:boto:Params: {} 2024-04-13T22:35:47.259 DEBUG:boto:Token: None 2024-04-13T22:35:47.259 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.259 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fbbe021500156fa-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.262 DEBUG:boto:path=/vwnfns-63/obj95 2024-04-13T22:35:47.262 DEBUG:boto:auth_path=/vwnfns-63/obj95 2024-04-13T22:35:47.262 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.262 DEBUG:boto:Path: /vwnfns-63/obj95 2024-04-13T22:35:47.262 DEBUG:boto:Data: 2024-04-13T22:35:47.262 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.263 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.263 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.263 DEBUG:boto:Params: {} 2024-04-13T22:35:47.263 DEBUG:boto:Token: None 2024-04-13T22:35:47.263 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj95 2024-04-13T22:35:47.263 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Y4x8TL2mOSbx2jlDhh+WhiF+Q4Q= 2024-04-13T22:35:47.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y4x8TL2mOSbx2jlDhh+WhiF+Q4Q='} 2024-04-13T22:35:47.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008129c375b519f07e-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:47.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:47.307 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:47.404 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:35:47.404 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-63 2024-04-13T22:35:47.411 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 15 2024-04-13T22:35:47.459 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.459 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.459 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.459 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.460 DEBUG:boto:Data: 2024-04-13T22:35:47.460 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.460 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.460 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.460 DEBUG:boto:Params: {} 2024-04-13T22:35:47.460 DEBUG:boto:Token: None 2024-04-13T22:35:47.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.461 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efea4c40bec2d45e-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.464 DEBUG:boto:path=/vwnfns-63/obj23 2024-04-13T22:35:47.464 DEBUG:boto:auth_path=/vwnfns-63/obj23 2024-04-13T22:35:47.464 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.464 DEBUG:boto:Path: /vwnfns-63/obj23 2024-04-13T22:35:47.464 DEBUG:boto:Data: 2024-04-13T22:35:47.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.464 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.464 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.464 DEBUG:boto:Params: {} 2024-04-13T22:35:47.465 DEBUG:boto:Token: None 2024-04-13T22:35:47.465 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj23 2024-04-13T22:35:47.465 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:huVK6boPbRggJBUJT86kLIVHItA= 2024-04-13T22:35:47.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:huVK6boPbRggJBUJT86kLIVHItA='} 2024-04-13T22:35:47.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c92eec90aaa1b21e-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.470 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.470 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.471 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.471 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.471 DEBUG:boto:Data: 2024-04-13T22:35:47.471 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.471 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.471 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.471 DEBUG:boto:Params: {} 2024-04-13T22:35:47.471 DEBUG:boto:Token: None 2024-04-13T22:35:47.471 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.471 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c63274d61c62a3a5-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.474 DEBUG:boto:path=/vwnfns-63/obj46 2024-04-13T22:35:47.474 DEBUG:boto:auth_path=/vwnfns-63/obj46 2024-04-13T22:35:47.474 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.475 DEBUG:boto:Path: /vwnfns-63/obj46 2024-04-13T22:35:47.475 DEBUG:boto:Data: 2024-04-13T22:35:47.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.475 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.475 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.475 DEBUG:boto:Params: {} 2024-04-13T22:35:47.475 DEBUG:boto:Token: None 2024-04-13T22:35:47.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj46 2024-04-13T22:35:47.475 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:BRgk9yo/Hl91FTuo63mymN4reZg= 2024-04-13T22:35:47.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BRgk9yo/Hl91FTuo63mymN4reZg='} 2024-04-13T22:35:47.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093209ddab83714c3-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.480 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.480 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.481 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.481 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.481 DEBUG:boto:Data: 2024-04-13T22:35:47.481 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.481 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.481 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.481 DEBUG:boto:Params: {} 2024-04-13T22:35:47.481 DEBUG:boto:Token: None 2024-04-13T22:35:47.481 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.481 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1610a478e52e603-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.484 DEBUG:boto:path=/vwnfns-63/obj69 2024-04-13T22:35:47.484 DEBUG:boto:auth_path=/vwnfns-63/obj69 2024-04-13T22:35:47.484 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.484 DEBUG:boto:Path: /vwnfns-63/obj69 2024-04-13T22:35:47.484 DEBUG:boto:Data: 2024-04-13T22:35:47.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.485 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.485 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.485 DEBUG:boto:Params: {} 2024-04-13T22:35:47.485 DEBUG:boto:Token: None 2024-04-13T22:35:47.485 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj69 2024-04-13T22:35:47.485 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K4fVxpiQ1FGbVJGWRhF3tpi8A4Q= 2024-04-13T22:35:47.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K4fVxpiQ1FGbVJGWRhF3tpi8A4Q='} 2024-04-13T22:35:47.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000071d2e71fd340be6-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.531 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.531 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.531 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.531 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.531 DEBUG:boto:Data: 2024-04-13T22:35:47.531 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.531 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.531 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.531 DEBUG:boto:Params: {} 2024-04-13T22:35:47.532 DEBUG:boto:Token: None 2024-04-13T22:35:47.532 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.532 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2abb2f68238a336-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.535 DEBUG:boto:path=/vwnfns-63/obj92 2024-04-13T22:35:47.535 DEBUG:boto:auth_path=/vwnfns-63/obj92 2024-04-13T22:35:47.535 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.535 DEBUG:boto:Path: /vwnfns-63/obj92 2024-04-13T22:35:47.535 DEBUG:boto:Data: 2024-04-13T22:35:47.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.535 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.535 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.536 DEBUG:boto:Params: {} 2024-04-13T22:35:47.536 DEBUG:boto:Token: None 2024-04-13T22:35:47.536 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj92 2024-04-13T22:35:47.536 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WKC4HbQ4tYp6E2VBOl6mK/bVePA= 2024-04-13T22:35:47.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WKC4HbQ4tYp6E2VBOl6mK/bVePA='} 2024-04-13T22:35:47.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d7ecdc7c9f6eed8-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.541 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.541 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.541 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.541 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.541 DEBUG:boto:Data: 2024-04-13T22:35:47.541 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.541 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.542 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.542 DEBUG:boto:Params: {} 2024-04-13T22:35:47.542 DEBUG:boto:Token: None 2024-04-13T22:35:47.542 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.542 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afa3b185092b20cb-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.548 DEBUG:boto:path=/vwnfns-63/obj115 2024-04-13T22:35:47.548 DEBUG:boto:auth_path=/vwnfns-63/obj115 2024-04-13T22:35:47.548 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.548 DEBUG:boto:Path: /vwnfns-63/obj115 2024-04-13T22:35:47.548 DEBUG:boto:Data: 2024-04-13T22:35:47.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.548 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.548 DEBUG:boto:Params: {} 2024-04-13T22:35:47.548 DEBUG:boto:Token: None 2024-04-13T22:35:47.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj115 2024-04-13T22:35:47.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jtONIxt8VrKL5xcyrgI7D45GSYI= 2024-04-13T22:35:47.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jtONIxt8VrKL5xcyrgI7D45GSYI='} 2024-04-13T22:35:47.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078e9b4848aefc011-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:47.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:47.591 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:47.643 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:47.643 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:47.691 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:35:47.691 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-63 2024-04-13T22:35:47.699 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 20 2024-04-13T22:35:47.750 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.750 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.750 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.750 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.750 DEBUG:boto:Data: 2024-04-13T22:35:47.750 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.751 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.751 DEBUG:boto:Params: {} 2024-04-13T22:35:47.751 DEBUG:boto:Token: None 2024-04-13T22:35:47.751 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.751 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eafe0b61ed463381-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.755 DEBUG:boto:path=/vwnfns-63/obj29 2024-04-13T22:35:47.755 DEBUG:boto:auth_path=/vwnfns-63/obj29 2024-04-13T22:35:47.755 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.755 DEBUG:boto:Path: /vwnfns-63/obj29 2024-04-13T22:35:47.755 DEBUG:boto:Data: 2024-04-13T22:35:47.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.755 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.755 DEBUG:boto:Params: {} 2024-04-13T22:35:47.756 DEBUG:boto:Token: None 2024-04-13T22:35:47.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj29 2024-04-13T22:35:47.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:moJz0QikIqg44Z6RngAqPEADj1A= 2024-04-13T22:35:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:moJz0QikIqg44Z6RngAqPEADj1A='} 2024-04-13T22:35:47.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9d844b8c3b1023c-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.761 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.761 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.761 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.762 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.762 DEBUG:boto:Data: 2024-04-13T22:35:47.762 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.762 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.762 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.762 DEBUG:boto:Params: {} 2024-04-13T22:35:47.763 DEBUG:boto:Token: None 2024-04-13T22:35:47.763 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.763 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e5b0994bc3a1ffc-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.767 DEBUG:boto:path=/vwnfns-63/obj58 2024-04-13T22:35:47.767 DEBUG:boto:auth_path=/vwnfns-63/obj58 2024-04-13T22:35:47.767 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.767 DEBUG:boto:Path: /vwnfns-63/obj58 2024-04-13T22:35:47.767 DEBUG:boto:Data: 2024-04-13T22:35:47.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.767 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.767 DEBUG:boto:Params: {} 2024-04-13T22:35:47.768 DEBUG:boto:Token: None 2024-04-13T22:35:47.768 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj58 2024-04-13T22:35:47.768 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ComYD1EXot2DS4Ug+1xMUzuiM0Q= 2024-04-13T22:35:47.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ComYD1EXot2DS4Ug+1xMUzuiM0Q='} 2024-04-13T22:35:47.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b07dcd246459bc46-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.811 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.811 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.811 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.811 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.811 DEBUG:boto:Data: 2024-04-13T22:35:47.811 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.812 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.812 DEBUG:boto:Params: {} 2024-04-13T22:35:47.812 DEBUG:boto:Token: None 2024-04-13T22:35:47.812 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d098eb1a8c85eb9d-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.815 DEBUG:boto:path=/vwnfns-63/obj87 2024-04-13T22:35:47.815 DEBUG:boto:auth_path=/vwnfns-63/obj87 2024-04-13T22:35:47.816 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.816 DEBUG:boto:Path: /vwnfns-63/obj87 2024-04-13T22:35:47.816 DEBUG:boto:Data: 2024-04-13T22:35:47.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.816 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.816 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.816 DEBUG:boto:Params: {} 2024-04-13T22:35:47.816 DEBUG:boto:Token: None 2024-04-13T22:35:47.816 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj87 2024-04-13T22:35:47.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iGpwRQRuv9/t65mpMZaqjlELQbA= 2024-04-13T22:35:47.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iGpwRQRuv9/t65mpMZaqjlELQbA='} 2024-04-13T22:35:47.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f53da6777007a33-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.821 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.822 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.822 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.822 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.822 DEBUG:boto:Data: 2024-04-13T22:35:47.822 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.822 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.822 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.822 DEBUG:boto:Params: {} 2024-04-13T22:35:47.822 DEBUG:boto:Token: None 2024-04-13T22:35:47.823 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.823 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c84e6dc008d85773-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.826 DEBUG:boto:path=/vwnfns-63/obj116 2024-04-13T22:35:47.826 DEBUG:boto:auth_path=/vwnfns-63/obj116 2024-04-13T22:35:47.826 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.826 DEBUG:boto:Path: /vwnfns-63/obj116 2024-04-13T22:35:47.826 DEBUG:boto:Data: 2024-04-13T22:35:47.826 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.826 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.826 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.826 DEBUG:boto:Params: {} 2024-04-13T22:35:47.827 DEBUG:boto:Token: None 2024-04-13T22:35:47.827 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj116 2024-04-13T22:35:47.827 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PKj/fhkbPSaCrCYJsUSIsVonJPs= 2024-04-13T22:35:47.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PKj/fhkbPSaCrCYJsUSIsVonJPs='} 2024-04-13T22:35:47.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093aa19bdba5a0dee-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.871 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:47.871 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:47.871 DEBUG:boto:Method: HEAD 2024-04-13T22:35:47.871 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:47.871 DEBUG:boto:Data: 2024-04-13T22:35:47.871 DEBUG:boto:Headers: {} 2024-04-13T22:35:47.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.871 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.871 DEBUG:boto:Params: {} 2024-04-13T22:35:47.872 DEBUG:boto:Token: None 2024-04-13T22:35:47.872 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/ 2024-04-13T22:35:47.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T22:35:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T22:35:47.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000970ff83e56a8240f-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.875 DEBUG:boto:path=/vwnfns-63/obj145 2024-04-13T22:35:47.875 DEBUG:boto:auth_path=/vwnfns-63/obj145 2024-04-13T22:35:47.875 DEBUG:boto:Method: PUT 2024-04-13T22:35:47.875 DEBUG:boto:Path: /vwnfns-63/obj145 2024-04-13T22:35:47.875 DEBUG:boto:Data: 2024-04-13T22:35:47.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:47.875 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:47.876 DEBUG:boto:Port: 8000 2024-04-13T22:35:47.876 DEBUG:boto:Params: {} 2024-04-13T22:35:47.876 DEBUG:boto:Token: None 2024-04-13T22:35:47.876 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:47 GMT /vwnfns-63/obj145 2024-04-13T22:35:47.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j1wJ13s/UecB6MVNh0C+GSBo1Pk= 2024-04-13T22:35:47.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j1wJ13s/UecB6MVNh0C+GSBo1Pk='} 2024-04-13T22:35:47.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d55d1d34d2dffbaa-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:47.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:47.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:47.882 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:47.933 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:47.934 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:47.984 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:35:47.985 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-63 2024-04-13T22:35:47.995 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 25 2024-04-13T22:35:48.055 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.055 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.055 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.055 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.055 DEBUG:boto:Data: 2024-04-13T22:35:48.055 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.055 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.056 DEBUG:boto:Params: {} 2024-04-13T22:35:48.056 DEBUG:boto:Token: None 2024-04-13T22:35:48.056 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.056 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e58ca65f53d76bb-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.060 DEBUG:boto:path=/vwnfns-63/obj31 2024-04-13T22:35:48.060 DEBUG:boto:auth_path=/vwnfns-63/obj31 2024-04-13T22:35:48.060 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.060 DEBUG:boto:Path: /vwnfns-63/obj31 2024-04-13T22:35:48.060 DEBUG:boto:Data: 2024-04-13T22:35:48.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.061 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.061 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.061 DEBUG:boto:Params: {} 2024-04-13T22:35:48.061 DEBUG:boto:Token: None 2024-04-13T22:35:48.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj31 2024-04-13T22:35:48.061 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:D/NTW873XkF63kY8bfuPsboNYfY= 2024-04-13T22:35:48.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D/NTW873XkF63kY8bfuPsboNYfY='} 2024-04-13T22:35:48.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009891b8cf2bb40285-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.107 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.107 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.107 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.107 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.107 DEBUG:boto:Data: 2024-04-13T22:35:48.107 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.107 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.108 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.108 DEBUG:boto:Params: {} 2024-04-13T22:35:48.108 DEBUG:boto:Token: None 2024-04-13T22:35:48.108 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.108 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a569628cbde6d75-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.111 DEBUG:boto:path=/vwnfns-63/obj62 2024-04-13T22:35:48.111 DEBUG:boto:auth_path=/vwnfns-63/obj62 2024-04-13T22:35:48.111 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.112 DEBUG:boto:Path: /vwnfns-63/obj62 2024-04-13T22:35:48.112 DEBUG:boto:Data: 2024-04-13T22:35:48.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.112 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.112 DEBUG:boto:Params: {} 2024-04-13T22:35:48.112 DEBUG:boto:Token: None 2024-04-13T22:35:48.112 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj62 2024-04-13T22:35:48.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JUTUWmXjQH19hunLUkOqBtP2n9Q= 2024-04-13T22:35:48.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JUTUWmXjQH19hunLUkOqBtP2n9Q='} 2024-04-13T22:35:48.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000944954d6e39eeecb-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.118 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.118 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.118 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.118 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.118 DEBUG:boto:Data: 2024-04-13T22:35:48.118 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.118 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.118 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.118 DEBUG:boto:Params: {} 2024-04-13T22:35:48.118 DEBUG:boto:Token: None 2024-04-13T22:35:48.119 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.119 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082841af653fdfe86-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.122 DEBUG:boto:path=/vwnfns-63/obj93 2024-04-13T22:35:48.122 DEBUG:boto:auth_path=/vwnfns-63/obj93 2024-04-13T22:35:48.122 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.122 DEBUG:boto:Path: /vwnfns-63/obj93 2024-04-13T22:35:48.122 DEBUG:boto:Data: 2024-04-13T22:35:48.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.123 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.123 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.123 DEBUG:boto:Params: {} 2024-04-13T22:35:48.123 DEBUG:boto:Token: None 2024-04-13T22:35:48.123 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj93 2024-04-13T22:35:48.123 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ftURUK9Xqap8KG0nued1QIW8Sxw= 2024-04-13T22:35:48.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ftURUK9Xqap8KG0nued1QIW8Sxw='} 2024-04-13T22:35:48.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9121d15c4227bfa-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.167 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.167 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.167 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.167 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.167 DEBUG:boto:Data: 2024-04-13T22:35:48.167 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.167 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.167 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.167 DEBUG:boto:Params: {} 2024-04-13T22:35:48.168 DEBUG:boto:Token: None 2024-04-13T22:35:48.168 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.168 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0e7f1bd53c29bed-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.172 DEBUG:boto:path=/vwnfns-63/obj124 2024-04-13T22:35:48.172 DEBUG:boto:auth_path=/vwnfns-63/obj124 2024-04-13T22:35:48.172 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.172 DEBUG:boto:Path: /vwnfns-63/obj124 2024-04-13T22:35:48.172 DEBUG:boto:Data: 2024-04-13T22:35:48.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.172 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.172 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.172 DEBUG:boto:Params: {} 2024-04-13T22:35:48.173 DEBUG:boto:Token: None 2024-04-13T22:35:48.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj124 2024-04-13T22:35:48.173 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hzgsY7GSx8Mmirvwk/9D6ggNWFw= 2024-04-13T22:35:48.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hzgsY7GSx8Mmirvwk/9D6ggNWFw='} 2024-04-13T22:35:48.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071adb6d440c753c6-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.178 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.178 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.178 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.178 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.179 DEBUG:boto:Data: 2024-04-13T22:35:48.179 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.179 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.179 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.179 DEBUG:boto:Params: {} 2024-04-13T22:35:48.179 DEBUG:boto:Token: None 2024-04-13T22:35:48.179 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.179 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004354f600707267a8-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.182 DEBUG:boto:path=/vwnfns-63/obj155 2024-04-13T22:35:48.183 DEBUG:boto:auth_path=/vwnfns-63/obj155 2024-04-13T22:35:48.183 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.183 DEBUG:boto:Path: /vwnfns-63/obj155 2024-04-13T22:35:48.183 DEBUG:boto:Data: 2024-04-13T22:35:48.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.183 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.183 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.183 DEBUG:boto:Params: {} 2024-04-13T22:35:48.183 DEBUG:boto:Token: None 2024-04-13T22:35:48.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj155 2024-04-13T22:35:48.184 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HeYf37bqO7UMpilyhOH3twLB9Tc= 2024-04-13T22:35:48.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HeYf37bqO7UMpilyhOH3twLB9Tc='} 2024-04-13T22:35:48.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9a506b3fc24c73b-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:48.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:48.227 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:48.278 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:48.279 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:48.329 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:35:48.329 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-63 2024-04-13T22:35:48.340 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 30 2024-04-13T22:35:48.402 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.403 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.403 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.403 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.403 DEBUG:boto:Data: 2024-04-13T22:35:48.403 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.403 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.403 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.403 DEBUG:boto:Params: {} 2024-04-13T22:35:48.404 DEBUG:boto:Token: None 2024-04-13T22:35:48.404 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe362e10b53fdace-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.408 DEBUG:boto:path=/vwnfns-63/obj37 2024-04-13T22:35:48.408 DEBUG:boto:auth_path=/vwnfns-63/obj37 2024-04-13T22:35:48.408 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.408 DEBUG:boto:Path: /vwnfns-63/obj37 2024-04-13T22:35:48.408 DEBUG:boto:Data: 2024-04-13T22:35:48.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.409 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.409 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.409 DEBUG:boto:Params: {} 2024-04-13T22:35:48.409 DEBUG:boto:Token: None 2024-04-13T22:35:48.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj37 2024-04-13T22:35:48.409 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:n5VX74Hllq7qiIKdvw7abfbkhzs= 2024-04-13T22:35:48.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n5VX74Hllq7qiIKdvw7abfbkhzs='} 2024-04-13T22:35:48.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b012a2e86edb2f1-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.415 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.415 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.415 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.415 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.415 DEBUG:boto:Data: 2024-04-13T22:35:48.415 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.415 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.416 DEBUG:boto:Params: {} 2024-04-13T22:35:48.416 DEBUG:boto:Token: None 2024-04-13T22:35:48.416 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046eb9d7d11b3cc79-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.420 DEBUG:boto:path=/vwnfns-63/obj74 2024-04-13T22:35:48.420 DEBUG:boto:auth_path=/vwnfns-63/obj74 2024-04-13T22:35:48.420 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.420 DEBUG:boto:Path: /vwnfns-63/obj74 2024-04-13T22:35:48.420 DEBUG:boto:Data: 2024-04-13T22:35:48.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.420 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.421 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.421 DEBUG:boto:Params: {} 2024-04-13T22:35:48.421 DEBUG:boto:Token: None 2024-04-13T22:35:48.421 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj74 2024-04-13T22:35:48.421 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jXKDUcIh15J78ahthpAylFRZxzI= 2024-04-13T22:35:48.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jXKDUcIh15J78ahthpAylFRZxzI='} 2024-04-13T22:35:48.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006311d4a342f88dfa-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.467 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.467 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.467 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.467 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.467 DEBUG:boto:Data: 2024-04-13T22:35:48.467 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.467 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.467 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.467 DEBUG:boto:Params: {} 2024-04-13T22:35:48.468 DEBUG:boto:Token: None 2024-04-13T22:35:48.468 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.468 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fabeba93149efad3-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.472 DEBUG:boto:path=/vwnfns-63/obj111 2024-04-13T22:35:48.472 DEBUG:boto:auth_path=/vwnfns-63/obj111 2024-04-13T22:35:48.472 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.472 DEBUG:boto:Path: /vwnfns-63/obj111 2024-04-13T22:35:48.472 DEBUG:boto:Data: 2024-04-13T22:35:48.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.472 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.472 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.472 DEBUG:boto:Params: {} 2024-04-13T22:35:48.473 DEBUG:boto:Token: None 2024-04-13T22:35:48.473 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj111 2024-04-13T22:35:48.473 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kgLsUOSop71HuWcrIUk8sqEToOQ= 2024-04-13T22:35:48.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kgLsUOSop71HuWcrIUk8sqEToOQ='} 2024-04-13T22:35:48.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a61f2bf1f8e9db-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.478 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.478 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.478 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.478 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.478 DEBUG:boto:Data: 2024-04-13T22:35:48.479 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.479 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.479 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.479 DEBUG:boto:Params: {} 2024-04-13T22:35:48.479 DEBUG:boto:Token: None 2024-04-13T22:35:48.479 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.479 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000586461aeeb022ad9-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.483 DEBUG:boto:path=/vwnfns-63/obj148 2024-04-13T22:35:48.483 DEBUG:boto:auth_path=/vwnfns-63/obj148 2024-04-13T22:35:48.483 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.483 DEBUG:boto:Path: /vwnfns-63/obj148 2024-04-13T22:35:48.483 DEBUG:boto:Data: 2024-04-13T22:35:48.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.483 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.483 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.484 DEBUG:boto:Params: {} 2024-04-13T22:35:48.484 DEBUG:boto:Token: None 2024-04-13T22:35:48.484 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj148 2024-04-13T22:35:48.484 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OAyH5FBZY6eiZ4qbppHqiDGvqWI= 2024-04-13T22:35:48.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OAyH5FBZY6eiZ4qbppHqiDGvqWI='} 2024-04-13T22:35:48.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6b6301bcb4ba8d2-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.527 DEBUG:boto:path=/vwnfns-63/ 2024-04-13T22:35:48.527 DEBUG:boto:auth_path=/vwnfns-63/ 2024-04-13T22:35:48.527 DEBUG:boto:Method: HEAD 2024-04-13T22:35:48.527 DEBUG:boto:Path: /vwnfns-63/ 2024-04-13T22:35:48.527 DEBUG:boto:Data: 2024-04-13T22:35:48.527 DEBUG:boto:Headers: {} 2024-04-13T22:35:48.527 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.527 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.527 DEBUG:boto:Params: {} 2024-04-13T22:35:48.528 DEBUG:boto:Token: None 2024-04-13T22:35:48.528 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/ 2024-04-13T22:35:48.528 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T22:35:48.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T22:35:48.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005da6686bb98ad9e0-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.532 DEBUG:boto:path=/vwnfns-63/obj185 2024-04-13T22:35:48.532 DEBUG:boto:auth_path=/vwnfns-63/obj185 2024-04-13T22:35:48.532 DEBUG:boto:Method: PUT 2024-04-13T22:35:48.532 DEBUG:boto:Path: /vwnfns-63/obj185 2024-04-13T22:35:48.532 DEBUG:boto:Data: 2024-04-13T22:35:48.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:35:48.532 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:35:48.532 DEBUG:boto:Port: 8000 2024-04-13T22:35:48.533 DEBUG:boto:Params: {} 2024-04-13T22:35:48.533 DEBUG:boto:Token: None 2024-04-13T22:35:48.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:35:48 GMT /vwnfns-63/obj185 2024-04-13T22:35:48.533 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CHg3hqPO3GyRlB4kA4msY3d/6Ts= 2024-04-13T22:35:48.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CHg3hqPO3GyRlB4kA4msY3d/6Ts='} 2024-04-13T22:35:48.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce42496f0ac06b24-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:35:48.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:48.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:35:48.538 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:35:48.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:48.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:48.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:48.616+0000 7f38ad606780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-13T22:35:48.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:48.617+0000 7f38ad606780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-13T22:35:48.644 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:35:48.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:35:48.646 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:35:48.646 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:48.664 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:35:48.664 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:35:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:35:48.667 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:35:49.668 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:35:49.689 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:35:49.689 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:35:49.690 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:35:49.691 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:35:49.691 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-63 2024-04-13T22:35:49.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:35:49.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:35:49.692 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:35:49.786 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:35:49.786 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:35:49.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:49.820+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:35:50.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:35:54.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:54.821+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:35:59.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:35:59.822+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:04.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:04.823+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:09.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:09.823+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:14.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:14.824+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:19.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:19.824+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:24.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:24.824+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:29.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:29.825+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:34.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:34.826+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:39.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:39.827+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:44.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:44.828+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:49.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:49.828+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:54.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:54.829+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:36:59.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:36:59.828+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:04.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:04.829+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:09.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:09.829+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:14.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:14.830+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:19.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:19.831+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:24.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:24.832+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:29.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:29.832+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:34.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:34.833+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:39.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:39.832+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:44.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:44.833+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:49.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:49.834+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:54.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:54.835+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:37:59.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:37:59.836+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:04.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:04.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:09.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:09.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:14.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:14.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:19.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:19.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:24.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:24.838+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:29.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:29.839+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:34.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:34.840+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:39.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:39.841+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:44.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:44.842+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:49.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:49.842+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:54.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:54.842+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:38:59.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:38:59.843+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:04.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:04.843+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:09.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:09.844+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:14.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:14.845+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:19.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:19.846+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:24.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:24.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:29.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:29.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:34.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:34.846+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:39.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:44.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:44.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:49.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:49.848+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T22:39:54.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:54.853+0000 7f339a15b780 1 bucket sync caught up with source: 2024-04-13T22:39:54.855 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.66.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.111.6, , , , 00000000001.88.6] 2024-04-13T22:39:54.855 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.66.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.111.6, , , , 00000000001.88.6] 2024-04-13T22:39:54.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:54.853+0000 7f339a15b780 0 bucket checkpoint complete 2024-04-13T22:39:54.873 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-63 2024-04-13T22:39:54.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:39:54.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:39:54.874 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:39:54.936 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:39:54.936 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:39:54.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:54.972+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:39:59.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:39:59.972+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:04.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:04.973+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:09.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:09.973+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:14.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:14.973+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:19.974+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:24.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:24.975+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:29.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:29.976+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:34.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:34.977+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:39.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:39.977+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:44.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:44.978+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:49.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:49.977+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:54.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:54.978+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:40:59.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:40:59.979+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:04.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:04.980+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:09.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:09.981+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:14.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:14.981+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:19.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:19.982+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:24.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:24.982+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:29.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:29.982+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:34.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:34.983+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:39.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:39.984+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:44.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:44.984+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:49.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:49.986+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:54.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:54.986+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:41:59.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:41:59.987+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:04.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:04.987+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:09.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:09.987+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:14.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:14.988+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:19.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:19.988+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:24.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:24.990+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:29.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:29.990+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:34.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:34.991+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:39.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:39.992+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:44.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:44.991+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:49.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:49.992+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:54.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:54.993+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:42:59.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:42:59.993+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:04.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:04.995+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:09.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:09.996+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:14.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:14.996+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:19.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:19.997+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:24.996+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:30.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:29.997+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:35.001 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:34.998+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:40.001 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:39.999+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:45.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:45.000+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:50.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:50.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:43:55.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:43:55.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:00.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:00.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:05.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:05.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:10.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:10.002+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:15.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:15.002+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:20.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:20.004+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:25.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:25.004+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:30.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:30.005+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:35.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:35.006+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:40.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:40.006+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:45.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:45.006+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T22:44:50.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:50.007+0000 7f0c67bc8780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-13T22:44:50.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:50.007+0000 7f0c67bc8780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T22:44:50.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:50.007+0000 7f0c67bc8780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T22:44:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-13T22:44:50.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-04-13T22:44:50.032 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:44:50.032 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:44:50.032 DEBUG:boto:Method: PUT 2024-04-13T22:44:50.032 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:44:50.032 DEBUG:boto:Data: 2024-04-13T22:44:50.032 DEBUG:boto:Headers: {} 2024-04-13T22:44:50.033 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:44:50.033 DEBUG:boto:Port: 8000 2024-04-13T22:44:50.033 DEBUG:boto:Params: {} 2024-04-13T22:44:50.033 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:44:50.033 DEBUG:boto:Token: None 2024-04-13T22:44:50.034 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:44:50 GMT /vwnfns-64/ 2024-04-13T22:44:50.034 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iJXV/S6cBU46hMvHM4dR7Nnf1p0= 2024-04-13T22:44:50.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:44:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iJXV/S6cBU46hMvHM4dR7Nnf1p0=', 'Content-Length': '0'} 2024-04-13T22:44:50.055 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2438c6318610840-00661b0ae2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:44:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:44:50.055 DEBUG:rgw_multi.tests:created bucket=vwnfns-64 2024-04-13T22:44:50.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:44:50.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:44:50.055 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:44:50.106 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:44:50.106 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:44:50.137 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:44:50.150 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:44:50.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.150 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048290.051569_931.1", 2024-04-13T22:44:50.150 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:44:50.051569Z" 2024-04-13T22:44:50.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:44:50.152 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:44:50.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:50.152 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:44:50.153 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713048290.051569_931.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:44:50.153 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:44:50.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:44:50.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:44:50.153 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:44:50.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:44:50.202 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:44:50.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:50.229+0000 7f97365b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:44:50.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:50.231+0000 7f97365b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:44:50.248 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:44:50.248 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:44:50.248 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.249 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047748.347361_913.1", 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:35:48.347361Z", 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:50.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:44:50.251 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:50.252 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:50.253 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:50.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:44:50Z" 2024-04-13T22:44:50.255 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:44:50.256 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713047748.347361_913.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:35:48.347361Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:44:50Z" } 2024-04-13T22:44:50.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047748.347361_913.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:35:48.347361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:44:50.257 DEBUG:rgw_multi.tests:log_status={0: '1_1713048290.051569_931.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:44:50.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1713047748.347361_913.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:44:50.257 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713048290.051569_931.1 target=1_1713047748.347361_913.1 2024-04-13T22:44:55.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:44:55.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:44:55.258 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:44:55.313 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:44:55.313 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:44:55.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:55.342+0000 7fb5bc8e2780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:44:55.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:55.343+0000 7fb5bc8e2780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:44:55.360 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048290.051569_931.1", 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:55.361 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:44:50.051569Z", 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:44:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:55.363 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:44:55.364 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:44:55.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:55.366 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:44:55.366 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:44:55.366 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:44:55.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:44:55.366 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:44:55Z" 2024-04-13T22:44:55.366 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:44:55.367 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048290.051569_931.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:44:50.051569Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:44:55Z" } 2024-04-13T22:44:55.367 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048290.051569_931.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:44:50.051569Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:44:55.367 DEBUG:rgw_multi.tests:log_status={0: '1_1713048290.051569_931.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:44:55.368 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048290.051569_931.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:44:55.368 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:44:55.368 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:44:55.368 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:44:55.368 DEBUG:boto:Method: HEAD 2024-04-13T22:44:55.368 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:44:55.368 DEBUG:boto:Data: 2024-04-13T22:44:55.368 DEBUG:boto:Headers: {} 2024-04-13T22:44:55.368 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:44:55.368 DEBUG:boto:Port: 8000 2024-04-13T22:44:55.369 DEBUG:boto:Params: {} 2024-04-13T22:44:55.369 DEBUG:boto:Token: None 2024-04-13T22:44:55.369 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:44:55 GMT /vwnfns-64/ 2024-04-13T22:44:55.369 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9XyrUUdWl7THYM8d5ZdJws07crA= 2024-04-13T22:44:55.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:44:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9XyrUUdWl7THYM8d5ZdJws07crA=', 'Content-Length': '0'} 2024-04-13T22:44:55.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031b1dd14618c1bb2-00661b0ae7-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:44:55.372 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:44:55.372 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:44:55.372 DEBUG:boto:Method: PUT 2024-04-13T22:44:55.372 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:44:55.373 DEBUG:boto:Data: 2024-04-13T22:44:55.373 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:44:55.373 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:44:55.373 DEBUG:boto:Port: 8000 2024-04-13T22:44:55.373 DEBUG:boto:Params: {} 2024-04-13T22:44:55.373 DEBUG:boto:Token: None 2024-04-13T22:44:55.373 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:44:55 GMT /vwnfns-64/dummy 2024-04-13T22:44:55.373 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:p+eVxjcTXo9fm6lQLSXaUkiUoaQ= 2024-04-13T22:44:55.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:44:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:p+eVxjcTXo9fm6lQLSXaUkiUoaQ='} 2024-04-13T22:44:55.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d47ec6ea4c72e9bd-00661b0ae7-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:44:55.380 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-64 2024-04-13T22:44:55.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:44:55.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:44:55.380 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:44:55.436 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:44:55.436 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:44:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:44:55.469+0000 7fb65a98e780 1 waiting to reach incremental sync.. 2024-04-13T22:45:00.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:00.471+0000 7fb65a98e780 1 waiting to reach incremental sync.. 2024-04-13T22:45:05.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:05.471+0000 7fb65a98e780 1 waiting to reach incremental sync.. 2024-04-13T22:45:10.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:10.473+0000 7fb65a98e780 1 bucket sync caught up with source: 2024-04-13T22:45:10.475 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:10.475 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:10.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:10.473+0000 7fb65a98e780 0 bucket checkpoint complete 2024-04-13T22:45:10.493 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-64 2024-04-13T22:45:10.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:45:10.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:45:10.494 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:45:10.553 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:45:10.553 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:45:10.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:10.589+0000 7fcf54e52780 1 waiting to reach incremental sync.. 2024-04-13T22:45:15.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:15.591+0000 7fcf54e52780 1 bucket sync caught up with source: 2024-04-13T22:45:15.593 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1787.6, , , , , ] 2024-04-13T22:45:15.593 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1787.6, , , , , ] 2024-04-13T22:45:15.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:15.591+0000 7fcf54e52780 0 bucket checkpoint complete 2024-04-13T22:45:15.609 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-64 zones={a2, a1} 2024-04-13T22:45:15.609 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.609 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.609 DEBUG:boto:Method: HEAD 2024-04-13T22:45:15.609 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:15.609 DEBUG:boto:Data: 2024-04-13T22:45:15.609 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.609 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.610 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.610 DEBUG:boto:Params: {} 2024-04-13T22:45:15.610 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:45:15.610 DEBUG:boto:Token: None 2024-04-13T22:45:15.610 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/ 2024-04-13T22:45:15.611 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QByZdRMHY33DQFsU878VfaW7voA= 2024-04-13T22:45:15.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QByZdRMHY33DQFsU878VfaW7voA=', 'Content-Length': '0'} 2024-04-13T22:45:15.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa1ddce3f4ad4197-00661b0afb-2326-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.616 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.616 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.616 DEBUG:boto:Method: HEAD 2024-04-13T22:45:15.616 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:15.617 DEBUG:boto:Data: 2024-04-13T22:45:15.617 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.617 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.617 DEBUG:boto:Params: {} 2024-04-13T22:45:15.617 DEBUG:boto:Token: None 2024-04-13T22:45:15.617 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/ 2024-04-13T22:45:15.617 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QByZdRMHY33DQFsU878VfaW7voA= 2024-04-13T22:45:15.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QByZdRMHY33DQFsU878VfaW7voA=', 'Content-Length': '0'} 2024-04-13T22:45:15.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbc0ae47dee50d2e-00661b0afb-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.619 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:45:15.620 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.620 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.620 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.620 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.620 DEBUG:boto:Method: GET 2024-04-13T22:45:15.620 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.620 DEBUG:boto:Data: 2024-04-13T22:45:15.620 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.620 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.620 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.620 DEBUG:boto:Params: {} 2024-04-13T22:45:15.621 DEBUG:boto:Token: None 2024-04-13T22:45:15.621 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/?versions 2024-04-13T22:45:15.621 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU= 2024-04-13T22:45:15.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU=', 'Content-Length': '0'} 2024-04-13T22:45:15.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a52e9f989b4b875-00661b0afb-2326-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.623 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.624 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T22:45:15.624 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:45:15.624 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.624 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.624 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.624 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.624 DEBUG:boto:Method: GET 2024-04-13T22:45:15.625 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.625 DEBUG:boto:Data: 2024-04-13T22:45:15.625 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.625 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.625 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.625 DEBUG:boto:Params: {} 2024-04-13T22:45:15.625 DEBUG:boto:Token: None 2024-04-13T22:45:15.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/?versions 2024-04-13T22:45:15.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU= 2024-04-13T22:45:15.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU=', 'Content-Length': '0'} 2024-04-13T22:45:15.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ab18e6d33577b3-00661b0afb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.627 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.627 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T22:45:15.628 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.628 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.628 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.628 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.628 DEBUG:boto:Method: GET 2024-04-13T22:45:15.628 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.628 DEBUG:boto:Data: 2024-04-13T22:45:15.628 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.628 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.629 DEBUG:boto:Params: {} 2024-04-13T22:45:15.629 DEBUG:boto:Token: None 2024-04-13T22:45:15.629 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/?versions 2024-04-13T22:45:15.629 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU= 2024-04-13T22:45:15.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU=', 'Content-Length': '0'} 2024-04-13T22:45:15.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084de1f70924dfbd9-00661b0afb-2326-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.631 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.631 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.631 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.631 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.632 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.632 DEBUG:boto:Method: GET 2024-04-13T22:45:15.632 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:45:15.632 DEBUG:boto:Data: 2024-04-13T22:45:15.632 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.632 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.632 DEBUG:boto:Params: {} 2024-04-13T22:45:15.632 DEBUG:boto:Token: None 2024-04-13T22:45:15.632 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/?versions 2024-04-13T22:45:15.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU= 2024-04-13T22:45:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HINPPiRRABCUneK3f7e+p1XstZU=', 'Content-Length': '0'} 2024-04-13T22:45:15.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000583196ab5e70b122-00661b0afb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.678 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T22:45:15.679 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.679 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.679 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.679 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.679 DEBUG:boto:Method: GET 2024-04-13T22:45:15.680 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.680 DEBUG:boto:Data: 2024-04-13T22:45:15.680 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.680 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.680 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.680 DEBUG:boto:Params: {} 2024-04-13T22:45:15.680 DEBUG:boto:Token: None 2024-04-13T22:45:15.681 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.681 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I= 2024-04-13T22:45:15.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I=', 'Content-Length': '0'} 2024-04-13T22:45:15.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e5402b821d3772-00661b0afb-2326-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.684 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.684 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.684 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.684 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.684 DEBUG:boto:Method: GET 2024-04-13T22:45:15.684 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.684 DEBUG:boto:Data: 2024-04-13T22:45:15.684 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.684 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.684 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.685 DEBUG:boto:Params: {} 2024-04-13T22:45:15.685 DEBUG:boto:Token: None 2024-04-13T22:45:15.685 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I= 2024-04-13T22:45:15.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I=', 'Content-Length': '0'} 2024-04-13T22:45:15.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001075db95116f0a9a-00661b0afb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.687 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.687 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.687 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.687 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.688 DEBUG:boto:Method: HEAD 2024-04-13T22:45:15.688 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.688 DEBUG:boto:Data: 2024-04-13T22:45:15.688 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.688 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.688 DEBUG:boto:Params: {} 2024-04-13T22:45:15.688 DEBUG:boto:Token: None 2024-04-13T22:45:15.688 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.689 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lwqUN0DtfBbUyGeyVbYYg+3kRJE= 2024-04-13T22:45:15.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lwqUN0DtfBbUyGeyVbYYg+3kRJE=', 'Content-Length': '0'} 2024-04-13T22:45:15.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a6a9c6690f0868-00661b0afb-2326-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.691 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.691 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.692 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.692 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.692 DEBUG:boto:Method: HEAD 2024-04-13T22:45:15.692 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.692 DEBUG:boto:Data: 2024-04-13T22:45:15.692 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.692 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.692 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.692 DEBUG:boto:Params: {} 2024-04-13T22:45:15.693 DEBUG:boto:Token: None 2024-04-13T22:45:15.693 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.693 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lwqUN0DtfBbUyGeyVbYYg+3kRJE= 2024-04-13T22:45:15.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lwqUN0DtfBbUyGeyVbYYg+3kRJE=', 'Content-Length': '0'} 2024-04-13T22:45:15.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e43c6098defd497-00661b0afb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T22:45:15.695 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.695 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.695 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.695 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.695 DEBUG:boto:Method: GET 2024-04-13T22:45:15.695 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.695 DEBUG:boto:Data: 2024-04-13T22:45:15.696 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.696 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.696 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.696 DEBUG:boto:Params: {} 2024-04-13T22:45:15.696 DEBUG:boto:Token: None 2024-04-13T22:45:15.696 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.696 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I= 2024-04-13T22:45:15.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I=', 'Content-Length': '0'} 2024-04-13T22:45:15.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb4ed5c0fc2f60b3-00661b0afb-2326-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.742 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.743 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.743 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.743 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.743 DEBUG:boto:Method: GET 2024-04-13T22:45:15.743 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.743 DEBUG:boto:Data: 2024-04-13T22:45:15.743 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.743 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.743 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.743 DEBUG:boto:Params: {} 2024-04-13T22:45:15.744 DEBUG:boto:Token: None 2024-04-13T22:45:15.744 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:45:15.744 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I= 2024-04-13T22:45:15.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+uePs+xi20La/eXtXr7Wnk6333I=', 'Content-Length': '0'} 2024-04-13T22:45:15.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6092307ab94b4c2-00661b0afb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.746 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.746 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.747 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.747 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.747 DEBUG:boto:Method: GET 2024-04-13T22:45:15.747 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.747 DEBUG:boto:Data: 2024-04-13T22:45:15.747 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.747 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.747 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.747 DEBUG:boto:Params: {} 2024-04-13T22:45:15.747 DEBUG:boto:Token: None 2024-04-13T22:45:15.748 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/ 2024-04-13T22:45:15.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc= 2024-04-13T22:45:15.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc=', 'Content-Length': '0'} 2024-04-13T22:45:15.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d0a28d617d7fe21-00661b0afb-2326-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.750 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.750 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.751 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.751 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.751 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.751 DEBUG:boto:Method: GET 2024-04-13T22:45:15.751 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.751 DEBUG:boto:Data: 2024-04-13T22:45:15.751 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.752 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.752 DEBUG:boto:Params: {} 2024-04-13T22:45:15.752 DEBUG:boto:Token: None 2024-04-13T22:45:15.752 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/ 2024-04-13T22:45:15.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc= 2024-04-13T22:45:15.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc=', 'Content-Length': '0'} 2024-04-13T22:45:15.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf45d593867d4a2b-00661b0afb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.754 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.755 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.755 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.755 DEBUG:boto:Method: HEAD 2024-04-13T22:45:15.755 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:45:15.755 DEBUG:boto:Data: 2024-04-13T22:45:15.755 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.755 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.755 DEBUG:boto:Params: {} 2024-04-13T22:45:15.756 DEBUG:boto:Token: None 2024-04-13T22:45:15.756 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy 2024-04-13T22:45:15.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scUcWepa1pbDgywVEsMGPEbyIBo= 2024-04-13T22:45:15.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scUcWepa1pbDgywVEsMGPEbyIBo=', 'Content-Length': '0'} 2024-04-13T22:45:15.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007deb087a2b48ec50-00661b0afb-2326-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.758 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.758 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.758 DEBUG:boto:Method: HEAD 2024-04-13T22:45:15.758 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:45:15.758 DEBUG:boto:Data: 2024-04-13T22:45:15.758 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.758 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.758 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.758 DEBUG:boto:Params: {} 2024-04-13T22:45:15.759 DEBUG:boto:Token: None 2024-04-13T22:45:15.759 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy 2024-04-13T22:45:15.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:scUcWepa1pbDgywVEsMGPEbyIBo= 2024-04-13T22:45:15.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:scUcWepa1pbDgywVEsMGPEbyIBo=', 'Content-Length': '0'} 2024-04-13T22:45:15.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2c6e3ae14279ca-00661b0afb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T22:45:15.761 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.761 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.761 DEBUG:boto:Method: GET 2024-04-13T22:45:15.762 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:45:15.762 DEBUG:boto:Data: 2024-04-13T22:45:15.762 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.762 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.762 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.762 DEBUG:boto:Params: {} 2024-04-13T22:45:15.762 DEBUG:boto:Token: None 2024-04-13T22:45:15.762 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy 2024-04-13T22:45:15.762 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dLzOUAJhD0o+VoVxve/Jy/iAGmM= 2024-04-13T22:45:15.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dLzOUAJhD0o+VoVxve/Jy/iAGmM=', 'Content-Length': '0'} 2024-04-13T22:45:15.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc225c89f74d32a6-00661b0afb-2326-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.764 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:45:15.764 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:45:15.764 DEBUG:boto:Method: GET 2024-04-13T22:45:15.765 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:45:15.765 DEBUG:boto:Data: 2024-04-13T22:45:15.765 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.765 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.765 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.765 DEBUG:boto:Params: {} 2024-04-13T22:45:15.765 DEBUG:boto:Token: None 2024-04-13T22:45:15.765 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/dummy 2024-04-13T22:45:15.765 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dLzOUAJhD0o+VoVxve/Jy/iAGmM= 2024-04-13T22:45:15.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dLzOUAJhD0o+VoVxve/Jy/iAGmM=', 'Content-Length': '0'} 2024-04-13T22:45:15.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb48a0fb4d62087e-00661b0afb-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.810 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.810 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.811 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.811 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.811 DEBUG:boto:Method: GET 2024-04-13T22:45:15.811 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.811 DEBUG:boto:Data: 2024-04-13T22:45:15.811 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.811 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:15.811 DEBUG:boto:Port: 8001 2024-04-13T22:45:15.811 DEBUG:boto:Params: {} 2024-04-13T22:45:15.811 DEBUG:boto:Token: None 2024-04-13T22:45:15.811 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/ 2024-04-13T22:45:15.812 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc= 2024-04-13T22:45:15.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc=', 'Content-Length': '0'} 2024-04-13T22:45:15.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b6ec1b4bb4c8ff-00661b0afb-2326-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.814 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.814 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:15.814 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:15.814 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.815 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.815 DEBUG:boto:Method: GET 2024-04-13T22:45:15.815 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:45:15.815 DEBUG:boto:Data: 2024-04-13T22:45:15.815 DEBUG:boto:Headers: {} 2024-04-13T22:45:15.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:15.815 DEBUG:boto:Port: 8000 2024-04-13T22:45:15.815 DEBUG:boto:Params: {} 2024-04-13T22:45:15.815 DEBUG:boto:Token: None 2024-04-13T22:45:15.815 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:45:15 GMT /vwnfns-64/ 2024-04-13T22:45:15.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc= 2024-04-13T22:45:15.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:15 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+QAGQqZW8E+HjYlj3HHsQ0nYRYc=', 'Content-Length': '0'} 2024-04-13T22:45:15.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081249efa11f23246-00661b0afb-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:45:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:15.818 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:45:15.818 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-64 zones={a2, a1} 2024-04-13T22:45:15.819 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:45:15.819 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:45:15.820 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:45:15.818+0000 7f0208c53700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 114571) UID: 0 2024-04-13T22:45:15.820 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:45:15.818+0000 7f020de287c0 -1 shutting down 2024-04-13T22:45:20.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:45:21.919 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:45:21.920 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:21.920 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:21.920 DEBUG:boto:Method: HEAD 2024-04-13T22:45:21.920 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:21.920 DEBUG:boto:Data: 2024-04-13T22:45:21.920 DEBUG:boto:Headers: {} 2024-04-13T22:45:21.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.920 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.920 DEBUG:boto:Params: {} 2024-04-13T22:45:21.921 DEBUG:boto:Token: None 2024-04-13T22:45:21.921 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/ 2024-04-13T22:45:21.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc= 2024-04-13T22:45:21.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc=', 'Content-Length': '0'} 2024-04-13T22:45:21.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d9125b1523f19c9-00661b0b01-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.924 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:45:21.924 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:45:21.924 DEBUG:boto:Method: PUT 2024-04-13T22:45:21.924 DEBUG:boto:Path: /vwnfns-64/obj11 2024-04-13T22:45:21.924 DEBUG:boto:Data: 2024-04-13T22:45:21.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:45:21.925 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.925 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.925 DEBUG:boto:Params: {} 2024-04-13T22:45:21.925 DEBUG:boto:Token: None 2024-04-13T22:45:21.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/obj11 2024-04-13T22:45:21.925 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wxGCtuMHNaCR5k3xm+MBSfGFeQk= 2024-04-13T22:45:21.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wxGCtuMHNaCR5k3xm+MBSfGFeQk='} 2024-04-13T22:45:21.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d00f21346260dc94-00661b0b01-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.932 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:21.932 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:21.932 DEBUG:boto:Method: HEAD 2024-04-13T22:45:21.932 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:21.932 DEBUG:boto:Data: 2024-04-13T22:45:21.932 DEBUG:boto:Headers: {} 2024-04-13T22:45:21.932 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.932 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.932 DEBUG:boto:Params: {} 2024-04-13T22:45:21.932 DEBUG:boto:Token: None 2024-04-13T22:45:21.933 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/ 2024-04-13T22:45:21.933 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc= 2024-04-13T22:45:21.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc=', 'Content-Length': '0'} 2024-04-13T22:45:21.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039801791ab17447b-00661b0b01-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.935 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:45:21.935 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:45:21.935 DEBUG:boto:Method: PUT 2024-04-13T22:45:21.935 DEBUG:boto:Path: /vwnfns-64/obj22 2024-04-13T22:45:21.935 DEBUG:boto:Data: 2024-04-13T22:45:21.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:45:21.936 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.936 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.936 DEBUG:boto:Params: {} 2024-04-13T22:45:21.936 DEBUG:boto:Token: None 2024-04-13T22:45:21.936 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/obj22 2024-04-13T22:45:21.936 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x9MMP9S7JpgsvCEVpKcmI80/Ke8= 2024-04-13T22:45:21.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x9MMP9S7JpgsvCEVpKcmI80/Ke8='} 2024-04-13T22:45:21.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ab5e8868e93d07d-00661b0b01-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.979 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:21.979 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:21.979 DEBUG:boto:Method: HEAD 2024-04-13T22:45:21.979 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:21.979 DEBUG:boto:Data: 2024-04-13T22:45:21.979 DEBUG:boto:Headers: {} 2024-04-13T22:45:21.979 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.979 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.979 DEBUG:boto:Params: {} 2024-04-13T22:45:21.980 DEBUG:boto:Token: None 2024-04-13T22:45:21.980 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/ 2024-04-13T22:45:21.980 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc= 2024-04-13T22:45:21.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc=', 'Content-Length': '0'} 2024-04-13T22:45:21.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d52967c44e14f770-00661b0b01-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.983 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:45:21.983 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:45:21.983 DEBUG:boto:Method: PUT 2024-04-13T22:45:21.983 DEBUG:boto:Path: /vwnfns-64/obj33 2024-04-13T22:45:21.983 DEBUG:boto:Data: 2024-04-13T22:45:21.983 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:45:21.983 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.983 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.983 DEBUG:boto:Params: {} 2024-04-13T22:45:21.983 DEBUG:boto:Token: None 2024-04-13T22:45:21.984 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/obj33 2024-04-13T22:45:21.984 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:25/chEFmvp05P88nSktZSomyIEo= 2024-04-13T22:45:21.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:25/chEFmvp05P88nSktZSomyIEo='} 2024-04-13T22:45:21.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d2887b80b2cbf85-00661b0b01-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.990 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:21.990 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:21.990 DEBUG:boto:Method: HEAD 2024-04-13T22:45:21.990 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:21.990 DEBUG:boto:Data: 2024-04-13T22:45:21.990 DEBUG:boto:Headers: {} 2024-04-13T22:45:21.990 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.991 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.991 DEBUG:boto:Params: {} 2024-04-13T22:45:21.991 DEBUG:boto:Token: None 2024-04-13T22:45:21.991 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/ 2024-04-13T22:45:21.991 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc= 2024-04-13T22:45:21.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UK2XNN4BjS0Bpo0+VshcFciAxmc=', 'Content-Length': '0'} 2024-04-13T22:45:21.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d3d7033642d2907-00661b0b01-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:21.994 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:45:21.994 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:45:21.994 DEBUG:boto:Method: PUT 2024-04-13T22:45:21.994 DEBUG:boto:Path: /vwnfns-64/obj44 2024-04-13T22:45:21.994 DEBUG:boto:Data: 2024-04-13T22:45:21.994 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:45:21.994 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:21.994 DEBUG:boto:Port: 8000 2024-04-13T22:45:21.994 DEBUG:boto:Params: {} 2024-04-13T22:45:21.994 DEBUG:boto:Token: None 2024-04-13T22:45:21.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:45:21 GMT /vwnfns-64/obj44 2024-04-13T22:45:21.995 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:do8CoCQxRTdsarb5F2hjT6CoVWY= 2024-04-13T22:45:21.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:45:21 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:do8CoCQxRTdsarb5F2hjT6CoVWY='} 2024-04-13T22:45:22.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d7dc1cea087b535-00661b0b01-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:22.039 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:45:22.039 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:45:22.039 DEBUG:boto:Method: HEAD 2024-04-13T22:45:22.039 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:45:22.039 DEBUG:boto:Data: 2024-04-13T22:45:22.039 DEBUG:boto:Headers: {} 2024-04-13T22:45:22.039 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:22.040 DEBUG:boto:Port: 8000 2024-04-13T22:45:22.040 DEBUG:boto:Params: {} 2024-04-13T22:45:22.040 DEBUG:boto:Token: None 2024-04-13T22:45:22.040 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:45:22 GMT /vwnfns-64/ 2024-04-13T22:45:22.040 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0gMISIsMLe8wun1B+KO8A/9c+W0= 2024-04-13T22:45:22.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:45:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0gMISIsMLe8wun1B+KO8A/9c+W0=', 'Content-Length': '0'} 2024-04-13T22:45:22.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e7e33c724fbd829-00661b0b02-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:22.043 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:45:22.043 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:45:22.043 DEBUG:boto:Method: PUT 2024-04-13T22:45:22.043 DEBUG:boto:Path: /vwnfns-64/obj55 2024-04-13T22:45:22.043 DEBUG:boto:Data: 2024-04-13T22:45:22.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:45:22.044 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:45:22.044 DEBUG:boto:Port: 8000 2024-04-13T22:45:22.044 DEBUG:boto:Params: {} 2024-04-13T22:45:22.044 DEBUG:boto:Token: None 2024-04-13T22:45:22.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:45:22 GMT /vwnfns-64/obj55 2024-04-13T22:45:22.044 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UPIdPnQYuEzsyzSA6Ufran476gA= 2024-04-13T22:45:22.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:45:22 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UPIdPnQYuEzsyzSA6Ufran476gA='} 2024-04-13T22:45:22.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bede914310cf6da-00661b0b02-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:45:22.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:45:22.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:45:22.050 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:45:22.105 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:45:22.121 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:45:22.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:22.136+0000 7fe86a71d780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-13T22:45:22.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:22.137+0000 7fe86a71d780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-13T22:45:22.163 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:45:22.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:45:22.165 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:45:22.166 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:45:22.187 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:45:22.187 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:45:22.187 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:45:23.188 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:45:23.209 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:45:23.209 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:45:23.210 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:45:23.211 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:45:23.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-64 2024-04-13T22:45:23.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:45:23.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:45:23.212 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:45:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:45:23.307 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:45:23.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:23.340+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:23.343 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:23.343 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:26.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:45:28.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:28.341+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:28.344 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:28.345 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:33.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:33.343+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:33.346 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:33.346 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:38.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:38.345+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:38.347 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:38.347 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:43.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:43.346+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:43.348 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:43.349 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:48.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:48.348+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:48.350 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:48.350 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:53.349+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:53.351 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:53.351 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:45:58.352 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:45:58.349+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:45:58.353 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:45:58.353 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:03.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:03.351+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:03.354 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:03.354 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:08.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:08.352+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:08.355 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:08.355 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:13.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:13.353+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:13.357 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:13.357 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:18.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:18.356+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:18.358 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:18.358 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:23.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:23.357+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:23.359 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:23.359 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:28.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:28.358+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:28.360 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:28.360 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:33.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:33.359+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:33.362 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:33.362 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:38.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:38.360+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:38.363 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:38.363 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:43.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:43.361+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:43.365 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:43.365 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:48.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:48.363+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:48.366 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , , ] 2024-04-13T22:46:48.366 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:53.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:53.365+0000 7f6e47ec2780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:53.368 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , , 00000000001.113.6, ] 2024-04-13T22:46:53.368 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:58.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:58.367+0000 7f6e47ec2780 1 bucket sync caught up with source: 2024-04-13T22:46:58.369 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:58.369 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.114.6, , , 00000000001.111.6, 00000000002.115.6, 00000000002.93.6] 2024-04-13T22:46:58.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:58.367+0000 7f6e47ec2780 0 bucket checkpoint complete 2024-04-13T22:46:58.388 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-64 2024-04-13T22:46:58.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:46:58.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:46:58.388 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:46:58.448 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:46:58.448 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:46:58.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:46:58.484+0000 7fdc8fa69780 1 waiting for incremental sync to catch up: 2024-04-13T22:46:58.486 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1787.6, , , , 00000000001.1475.6, ] 2024-04-13T22:46:58.486 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1787.6, , , 00000000001.1815.6, 00000000002.1477.6, 00000000002.1017.6] 2024-04-13T22:47:03.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:03.485+0000 7fdc8fa69780 1 waiting for incremental sync to catch up: 2024-04-13T22:47:03.487 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1787.6, , , , 00000000001.1475.6, ] 2024-04-13T22:47:03.487 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1787.6, , , 00000000001.1815.6, 00000000002.1477.6, 00000000002.1017.6] 2024-04-13T22:47:08.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:08.487+0000 7fdc8fa69780 1 waiting for incremental sync to catch up: 2024-04-13T22:47:08.489 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1787.6, , , , 00000000001.1475.6, ] 2024-04-13T22:47:08.489 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1787.6, , , 00000000001.1815.6, 00000000002.1477.6, 00000000002.1017.6] 2024-04-13T22:47:13.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:13.487+0000 7fdc8fa69780 1 waiting for incremental sync to catch up: 2024-04-13T22:47:13.490 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1787.6, , , , 00000000001.1475.6, ] 2024-04-13T22:47:13.490 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1787.6, , , 00000000001.1815.6, 00000000002.1477.6, 00000000002.1017.6] 2024-04-13T22:47:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:18.489+0000 7fdc8fa69780 1 bucket sync caught up with source: 2024-04-13T22:47:18.492 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , 00000000001.1787.6, , , 00000000001.1815.6, 00000000002.1477.6, 00000000002.1017.6] 2024-04-13T22:47:18.492 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , 00000000001.1787.6, , , 00000000001.1815.6, 00000000002.1477.6, 00000000002.1017.6] 2024-04-13T22:47:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:18.489+0000 7fdc8fa69780 0 bucket checkpoint complete 2024-04-13T22:47:18.509 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-64 zones={a2, a1} 2024-04-13T22:47:18.509 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.509 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.510 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.510 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:47:18.510 DEBUG:boto:Data: 2024-04-13T22:47:18.510 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.510 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.510 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.510 DEBUG:boto:Params: {} 2024-04-13T22:47:18.511 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:47:18.511 DEBUG:boto:Token: None 2024-04-13T22:47:18.511 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.511 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q1cl/6hAgrSKs/kWCJ2PSoT+HTk= 2024-04-13T22:47:18.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q1cl/6hAgrSKs/kWCJ2PSoT+HTk=', 'Content-Length': '0'} 2024-04-13T22:47:18.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3cc9a76ac3315f6-00661b0b76-236e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.518 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.518 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.518 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.519 DEBUG:boto:Path: /vwnfns-64/ 2024-04-13T22:47:18.519 DEBUG:boto:Data: 2024-04-13T22:47:18.519 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.519 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.519 DEBUG:boto:Params: {} 2024-04-13T22:47:18.519 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:47:18.519 DEBUG:boto:Token: None 2024-04-13T22:47:18.519 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q1cl/6hAgrSKs/kWCJ2PSoT+HTk= 2024-04-13T22:47:18.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q1cl/6hAgrSKs/kWCJ2PSoT+HTk=', 'Content-Length': '0'} 2024-04-13T22:47:18.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000443720e1edc4323f-00661b0b76-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:47:18.526 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.526 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.526 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.526 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.526 DEBUG:boto:Method: GET 2024-04-13T22:47:18.526 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.526 DEBUG:boto:Data: 2024-04-13T22:47:18.526 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.526 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.527 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.527 DEBUG:boto:Params: {} 2024-04-13T22:47:18.527 DEBUG:boto:Token: None 2024-04-13T22:47:18.527 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/?versions 2024-04-13T22:47:18.527 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c= 2024-04-13T22:47:18.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c=', 'Content-Length': '0'} 2024-04-13T22:47:18.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002254776ba5872e15-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.530 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.531 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T22:47:18.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:47:18.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:47:18.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:47:18.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:47:18.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:47:18.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:47:18.532 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.532 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.532 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.533 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.533 DEBUG:boto:Method: GET 2024-04-13T22:47:18.533 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.533 DEBUG:boto:Data: 2024-04-13T22:47:18.533 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.533 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.533 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.533 DEBUG:boto:Params: {} 2024-04-13T22:47:18.533 DEBUG:boto:Token: None 2024-04-13T22:47:18.533 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/?versions 2024-04-13T22:47:18.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c= 2024-04-13T22:47:18.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c=', 'Content-Length': '0'} 2024-04-13T22:47:18.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6911dea484511dd-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.536 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.537 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-13T22:47:18.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:47:18.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:47:18.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:47:18.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:47:18.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:47:18.537 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.537 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.537 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.538 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.538 DEBUG:boto:Method: GET 2024-04-13T22:47:18.538 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.538 DEBUG:boto:Data: 2024-04-13T22:47:18.538 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.538 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.538 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.538 DEBUG:boto:Params: {} 2024-04-13T22:47:18.538 DEBUG:boto:Token: None 2024-04-13T22:47:18.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/?versions 2024-04-13T22:47:18.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c= 2024-04-13T22:47:18.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c=', 'Content-Length': '0'} 2024-04-13T22:47:18.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000040b687301c19ee-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.541 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.542 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.542 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.542 DEBUG:boto:path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.542 DEBUG:boto:auth_path=/vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.542 DEBUG:boto:Method: GET 2024-04-13T22:47:18.542 DEBUG:boto:Path: /vwnfns-64/?versions&max-keys=999 2024-04-13T22:47:18.542 DEBUG:boto:Data: 2024-04-13T22:47:18.542 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.543 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.543 DEBUG:boto:Params: {} 2024-04-13T22:47:18.543 DEBUG:boto:Token: None 2024-04-13T22:47:18.543 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/?versions 2024-04-13T22:47:18.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c= 2024-04-13T22:47:18.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cM43drOJ/xLmJuWg9ok2WZJlz/c=', 'Content-Length': '0'} 2024-04-13T22:47:18.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b271a704bd549aac-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.546 DEBUG:boto:b'vwnfns-64999falsedummynulltrue2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T22:47:18.547 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.547 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.547 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.547 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.547 DEBUG:boto:Method: GET 2024-04-13T22:47:18.547 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.547 DEBUG:boto:Data: 2024-04-13T22:47:18.547 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.548 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.548 DEBUG:boto:Params: {} 2024-04-13T22:47:18.548 DEBUG:boto:Token: None 2024-04-13T22:47:18.548 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.548 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM= 2024-04-13T22:47:18.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM=', 'Content-Length': '0'} 2024-04-13T22:47:18.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4acda38cda5580e-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.551 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.551 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.551 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.551 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.551 DEBUG:boto:Method: GET 2024-04-13T22:47:18.551 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.551 DEBUG:boto:Data: 2024-04-13T22:47:18.551 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.551 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.551 DEBUG:boto:Params: {} 2024-04-13T22:47:18.552 DEBUG:boto:Token: None 2024-04-13T22:47:18.552 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM= 2024-04-13T22:47:18.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM=', 'Content-Length': '0'} 2024-04-13T22:47:18.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d063f7216485b35-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.554 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.554 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.554 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.554 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.554 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.554 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.554 DEBUG:boto:Data: 2024-04-13T22:47:18.554 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.555 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.555 DEBUG:boto:Params: {} 2024-04-13T22:47:18.555 DEBUG:boto:Token: None 2024-04-13T22:47:18.555 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yZm76OKvTdnRU0LFEwRMZqK7UHg= 2024-04-13T22:47:18.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yZm76OKvTdnRU0LFEwRMZqK7UHg=', 'Content-Length': '0'} 2024-04-13T22:47:18.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003edd081a470aaaf0-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.557 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.557 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.557 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.557 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.557 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.558 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.558 DEBUG:boto:Data: 2024-04-13T22:47:18.558 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.558 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.558 DEBUG:boto:Params: {} 2024-04-13T22:47:18.558 DEBUG:boto:Token: None 2024-04-13T22:47:18.558 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.558 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yZm76OKvTdnRU0LFEwRMZqK7UHg= 2024-04-13T22:47:18.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yZm76OKvTdnRU0LFEwRMZqK7UHg=', 'Content-Length': '0'} 2024-04-13T22:47:18.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c235fcd5c51fa9-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T22:47:18.561 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.561 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.561 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.561 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.561 DEBUG:boto:Method: GET 2024-04-13T22:47:18.561 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.561 DEBUG:boto:Data: 2024-04-13T22:47:18.561 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.561 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.561 DEBUG:boto:Params: {} 2024-04-13T22:47:18.562 DEBUG:boto:Token: None 2024-04-13T22:47:18.562 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM= 2024-04-13T22:47:18.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM=', 'Content-Length': '0'} 2024-04-13T22:47:18.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f4661374e43180-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.564 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.564 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.564 DEBUG:boto:path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.564 DEBUG:boto:auth_path=/vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.564 DEBUG:boto:Method: GET 2024-04-13T22:47:18.565 DEBUG:boto:Path: /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.565 DEBUG:boto:Data: 2024-04-13T22:47:18.565 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.565 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.565 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.565 DEBUG:boto:Params: {} 2024-04-13T22:47:18.565 DEBUG:boto:Token: None 2024-04-13T22:47:18.565 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy?versionId=null 2024-04-13T22:47:18.565 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM= 2024-04-13T22:47:18.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Alo7/f37iAPrclEEw7Eau659tSM=', 'Content-Length': '0'} 2024-04-13T22:47:18.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009e32bdf1b1e1740-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.567 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.567 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.567 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.568 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.568 DEBUG:boto:Method: GET 2024-04-13T22:47:18.568 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.568 DEBUG:boto:Data: 2024-04-13T22:47:18.568 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.568 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.568 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.568 DEBUG:boto:Params: {} 2024-04-13T22:47:18.568 DEBUG:boto:Token: None 2024-04-13T22:47:18.568 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.569 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001599d327f584d95e-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.571 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.571 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.571 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.572 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.572 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.572 DEBUG:boto:Method: GET 2024-04-13T22:47:18.572 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.572 DEBUG:boto:Data: 2024-04-13T22:47:18.572 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.572 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.572 DEBUG:boto:Params: {} 2024-04-13T22:47:18.572 DEBUG:boto:Token: None 2024-04-13T22:47:18.573 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.573 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f24ea7444d87ac-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.575 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.576 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.576 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.576 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.576 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:47:18.576 DEBUG:boto:Data: 2024-04-13T22:47:18.576 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.576 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.576 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.577 DEBUG:boto:Params: {} 2024-04-13T22:47:18.577 DEBUG:boto:Token: None 2024-04-13T22:47:18.577 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy 2024-04-13T22:47:18.577 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iSmq0t2YAi89i2ahbwrkT+0frik= 2024-04-13T22:47:18.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iSmq0t2YAi89i2ahbwrkT+0frik=', 'Content-Length': '0'} 2024-04-13T22:47:18.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000711c023087392fc8-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.579 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.579 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.579 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.579 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:47:18.579 DEBUG:boto:Data: 2024-04-13T22:47:18.579 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.579 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.579 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.579 DEBUG:boto:Params: {} 2024-04-13T22:47:18.580 DEBUG:boto:Token: None 2024-04-13T22:47:18.580 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy 2024-04-13T22:47:18.580 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iSmq0t2YAi89i2ahbwrkT+0frik= 2024-04-13T22:47:18.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iSmq0t2YAi89i2ahbwrkT+0frik=', 'Content-Length': '0'} 2024-04-13T22:47:18.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e56223a2869e31e5-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-13T22:47:18.582 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.582 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.582 DEBUG:boto:Method: GET 2024-04-13T22:47:18.582 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:47:18.582 DEBUG:boto:Data: 2024-04-13T22:47:18.582 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.582 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.582 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.583 DEBUG:boto:Params: {} 2024-04-13T22:47:18.583 DEBUG:boto:Token: None 2024-04-13T22:47:18.583 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy 2024-04-13T22:47:18.583 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tBTihxD8Ck9sVQ4uj13ZWcQPnQc= 2024-04-13T22:47:18.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tBTihxD8Ck9sVQ4uj13ZWcQPnQc=', 'Content-Length': '0'} 2024-04-13T22:47:18.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072843fc0c6dbee7f-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.626 DEBUG:boto:path=/vwnfns-64/dummy 2024-04-13T22:47:18.627 DEBUG:boto:auth_path=/vwnfns-64/dummy 2024-04-13T22:47:18.627 DEBUG:boto:Method: GET 2024-04-13T22:47:18.627 DEBUG:boto:Path: /vwnfns-64/dummy 2024-04-13T22:47:18.627 DEBUG:boto:Data: 2024-04-13T22:47:18.627 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.627 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.627 DEBUG:boto:Params: {} 2024-04-13T22:47:18.627 DEBUG:boto:Token: None 2024-04-13T22:47:18.628 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/dummy 2024-04-13T22:47:18.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tBTihxD8Ck9sVQ4uj13ZWcQPnQc= 2024-04-13T22:47:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tBTihxD8Ck9sVQ4uj13ZWcQPnQc=', 'Content-Length': '0'} 2024-04-13T22:47:18.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:44:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7393d2a40dc3d82-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.630 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.630 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.630 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.630 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.630 DEBUG:boto:Method: GET 2024-04-13T22:47:18.630 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.630 DEBUG:boto:Data: 2024-04-13T22:47:18.631 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.631 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.631 DEBUG:boto:Params: {} 2024-04-13T22:47:18.631 DEBUG:boto:Token: None 2024-04-13T22:47:18.631 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.631 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dfbefd4f68da290-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.633 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.634 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.634 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.634 DEBUG:boto:path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.634 DEBUG:boto:auth_path=/vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.634 DEBUG:boto:Method: GET 2024-04-13T22:47:18.635 DEBUG:boto:Path: /vwnfns-64/?prefix=dummy 2024-04-13T22:47:18.635 DEBUG:boto:Data: 2024-04-13T22:47:18.635 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.635 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.635 DEBUG:boto:Params: {} 2024-04-13T22:47:18.635 DEBUG:boto:Token: None 2024-04-13T22:47:18.635 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98c8a910d8b0c6d-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.638 DEBUG:boto:b'vwnfns-64dummy1000falsedummy2024-04-13T22:44:55.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:47:18.638 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.639 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.639 DEBUG:boto:path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.639 DEBUG:boto:auth_path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.639 DEBUG:boto:Method: GET 2024-04-13T22:47:18.639 DEBUG:boto:Path: /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.639 DEBUG:boto:Data: 2024-04-13T22:47:18.639 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.639 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.639 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.639 DEBUG:boto:Params: {} 2024-04-13T22:47:18.639 DEBUG:boto:Token: None 2024-04-13T22:47:18.640 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.640 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w= 2024-04-13T22:47:18.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w=', 'Content-Length': '0'} 2024-04-13T22:47:18.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004894bc4117578f5-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.642 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.642 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.642 DEBUG:boto:path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.642 DEBUG:boto:auth_path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.642 DEBUG:boto:Method: GET 2024-04-13T22:47:18.642 DEBUG:boto:Path: /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.643 DEBUG:boto:Data: 2024-04-13T22:47:18.643 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.643 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.643 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.643 DEBUG:boto:Params: {} 2024-04-13T22:47:18.643 DEBUG:boto:Token: None 2024-04-13T22:47:18.643 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w= 2024-04-13T22:47:18.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w=', 'Content-Length': '0'} 2024-04-13T22:47:18.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f8be788f5eccd14-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.645 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.646 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.646 DEBUG:boto:path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.646 DEBUG:boto:auth_path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.646 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.646 DEBUG:boto:Path: /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.646 DEBUG:boto:Data: 2024-04-13T22:47:18.646 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.646 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.647 DEBUG:boto:Params: {} 2024-04-13T22:47:18.647 DEBUG:boto:Token: None 2024-04-13T22:47:18.647 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yDd7SzBEIMgCsTPPmmsGPqCGJmw= 2024-04-13T22:47:18.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yDd7SzBEIMgCsTPPmmsGPqCGJmw=', 'Content-Length': '0'} 2024-04-13T22:47:18.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e490b3de1f50c8-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.649 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.649 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.649 DEBUG:boto:path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.649 DEBUG:boto:auth_path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.649 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.650 DEBUG:boto:Path: /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.650 DEBUG:boto:Data: 2024-04-13T22:47:18.650 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.650 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.650 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.650 DEBUG:boto:Params: {} 2024-04-13T22:47:18.650 DEBUG:boto:Token: None 2024-04-13T22:47:18.650 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yDd7SzBEIMgCsTPPmmsGPqCGJmw= 2024-04-13T22:47:18.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yDd7SzBEIMgCsTPPmmsGPqCGJmw=', 'Content-Length': '0'} 2024-04-13T22:47:18.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b4b78be229ad0a0-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:47:18.653 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.653 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.653 DEBUG:boto:path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.653 DEBUG:boto:auth_path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.653 DEBUG:boto:Method: GET 2024-04-13T22:47:18.653 DEBUG:boto:Path: /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.654 DEBUG:boto:Data: 2024-04-13T22:47:18.654 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.654 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.654 DEBUG:boto:Params: {} 2024-04-13T22:47:18.654 DEBUG:boto:Token: None 2024-04-13T22:47:18.654 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w= 2024-04-13T22:47:18.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w=', 'Content-Length': '0'} 2024-04-13T22:47:18.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c880bfcddb49d9-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.657 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.657 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.657 DEBUG:boto:path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.657 DEBUG:boto:auth_path=/vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.657 DEBUG:boto:Method: GET 2024-04-13T22:47:18.657 DEBUG:boto:Path: /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.657 DEBUG:boto:Data: 2024-04-13T22:47:18.657 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.658 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.658 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.658 DEBUG:boto:Params: {} 2024-04-13T22:47:18.658 DEBUG:boto:Token: None 2024-04-13T22:47:18.658 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11?versionId=null 2024-04-13T22:47:18.658 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w= 2024-04-13T22:47:18.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S+yCwBEQjROjgTCZOg+eqOl739w=', 'Content-Length': '0'} 2024-04-13T22:47:18.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092544666ec657069-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.660 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.660 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.660 DEBUG:boto:path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.660 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.661 DEBUG:boto:Method: GET 2024-04-13T22:47:18.661 DEBUG:boto:Path: /vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.661 DEBUG:boto:Data: 2024-04-13T22:47:18.661 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.661 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.661 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.661 DEBUG:boto:Params: {} 2024-04-13T22:47:18.661 DEBUG:boto:Token: None 2024-04-13T22:47:18.661 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.662 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e7cebc13cc191f3-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.664 DEBUG:boto:b'vwnfns-64obj111000falseobj112024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.664 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.664 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.665 DEBUG:boto:path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.665 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.665 DEBUG:boto:Method: GET 2024-04-13T22:47:18.665 DEBUG:boto:Path: /vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.665 DEBUG:boto:Data: 2024-04-13T22:47:18.665 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.665 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.665 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.665 DEBUG:boto:Params: {} 2024-04-13T22:47:18.665 DEBUG:boto:Token: None 2024-04-13T22:47:18.666 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.666 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075532f1e7b29fae7-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.669 DEBUG:boto:b'vwnfns-64obj111000falseobj112024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.669 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.669 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.669 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.669 DEBUG:boto:Path: /vwnfns-64/obj11 2024-04-13T22:47:18.669 DEBUG:boto:Data: 2024-04-13T22:47:18.669 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.669 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.670 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.670 DEBUG:boto:Params: {} 2024-04-13T22:47:18.670 DEBUG:boto:Token: None 2024-04-13T22:47:18.670 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11 2024-04-13T22:47:18.670 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pTomMTQXS7ScTwA+ZVTP8/EKeQw= 2024-04-13T22:47:18.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pTomMTQXS7ScTwA+ZVTP8/EKeQw=', 'Content-Length': '0'} 2024-04-13T22:47:18.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e031606d92a1a040-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.672 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.672 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.672 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.672 DEBUG:boto:Path: /vwnfns-64/obj11 2024-04-13T22:47:18.672 DEBUG:boto:Data: 2024-04-13T22:47:18.672 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.672 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.673 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.673 DEBUG:boto:Params: {} 2024-04-13T22:47:18.673 DEBUG:boto:Token: None 2024-04-13T22:47:18.673 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11 2024-04-13T22:47:18.673 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:pTomMTQXS7ScTwA+ZVTP8/EKeQw= 2024-04-13T22:47:18.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pTomMTQXS7ScTwA+ZVTP8/EKeQw=', 'Content-Length': '0'} 2024-04-13T22:47:18.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba549e1d29619958-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:47:18.675 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.675 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.675 DEBUG:boto:Method: GET 2024-04-13T22:47:18.675 DEBUG:boto:Path: /vwnfns-64/obj11 2024-04-13T22:47:18.675 DEBUG:boto:Data: 2024-04-13T22:47:18.675 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.676 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.676 DEBUG:boto:Params: {} 2024-04-13T22:47:18.676 DEBUG:boto:Token: None 2024-04-13T22:47:18.676 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11 2024-04-13T22:47:18.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Etvr5tah2rvYyFNwaJrxbt9NUE= 2024-04-13T22:47:18.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Etvr5tah2rvYyFNwaJrxbt9NUE=', 'Content-Length': '0'} 2024-04-13T22:47:18.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82194afa8ce790f-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.678 DEBUG:boto:path=/vwnfns-64/obj11 2024-04-13T22:47:18.678 DEBUG:boto:auth_path=/vwnfns-64/obj11 2024-04-13T22:47:18.678 DEBUG:boto:Method: GET 2024-04-13T22:47:18.678 DEBUG:boto:Path: /vwnfns-64/obj11 2024-04-13T22:47:18.678 DEBUG:boto:Data: 2024-04-13T22:47:18.679 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.679 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.679 DEBUG:boto:Params: {} 2024-04-13T22:47:18.679 DEBUG:boto:Token: None 2024-04-13T22:47:18.679 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj11 2024-04-13T22:47:18.679 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3Etvr5tah2rvYyFNwaJrxbt9NUE= 2024-04-13T22:47:18.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3Etvr5tah2rvYyFNwaJrxbt9NUE=', 'Content-Length': '0'} 2024-04-13T22:47:18.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdfcd370cb4bc853-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.681 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.681 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.681 DEBUG:boto:path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.681 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.681 DEBUG:boto:Method: GET 2024-04-13T22:47:18.682 DEBUG:boto:Path: /vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.682 DEBUG:boto:Data: 2024-04-13T22:47:18.682 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.682 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.682 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.682 DEBUG:boto:Params: {} 2024-04-13T22:47:18.682 DEBUG:boto:Token: None 2024-04-13T22:47:18.682 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.682 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052fa0a556f9218ec-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.685 DEBUG:boto:b'vwnfns-64obj111000falseobj112024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.685 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.685 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.685 DEBUG:boto:path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.685 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.685 DEBUG:boto:Method: GET 2024-04-13T22:47:18.685 DEBUG:boto:Path: /vwnfns-64/?prefix=obj11 2024-04-13T22:47:18.685 DEBUG:boto:Data: 2024-04-13T22:47:18.686 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.686 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.686 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.686 DEBUG:boto:Params: {} 2024-04-13T22:47:18.686 DEBUG:boto:Token: None 2024-04-13T22:47:18.686 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.686 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3dc3c8890a714d1-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.689 DEBUG:boto:b'vwnfns-64obj111000falseobj112024-04-13T22:45:21.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:47:18.689 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.689 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.689 DEBUG:boto:path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.689 DEBUG:boto:auth_path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.690 DEBUG:boto:Method: GET 2024-04-13T22:47:18.690 DEBUG:boto:Path: /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.690 DEBUG:boto:Data: 2024-04-13T22:47:18.690 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.690 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.690 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.690 DEBUG:boto:Params: {} 2024-04-13T22:47:18.690 DEBUG:boto:Token: None 2024-04-13T22:47:18.690 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.690 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw= 2024-04-13T22:47:18.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw=', 'Content-Length': '0'} 2024-04-13T22:47:18.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9d109b2ee8288a-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.693 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.693 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.693 DEBUG:boto:path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.693 DEBUG:boto:auth_path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.693 DEBUG:boto:Method: GET 2024-04-13T22:47:18.693 DEBUG:boto:Path: /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.693 DEBUG:boto:Data: 2024-04-13T22:47:18.693 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.693 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.693 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.694 DEBUG:boto:Params: {} 2024-04-13T22:47:18.694 DEBUG:boto:Token: None 2024-04-13T22:47:18.694 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.694 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw= 2024-04-13T22:47:18.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw=', 'Content-Length': '0'} 2024-04-13T22:47:18.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9be697801059ffd-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.696 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.696 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.696 DEBUG:boto:path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.696 DEBUG:boto:auth_path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.696 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.696 DEBUG:boto:Path: /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.697 DEBUG:boto:Data: 2024-04-13T22:47:18.697 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.697 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.697 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.697 DEBUG:boto:Params: {} 2024-04-13T22:47:18.697 DEBUG:boto:Token: None 2024-04-13T22:47:18.697 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.697 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yD+WJGB/ofbct9/JitF26SqCQUo= 2024-04-13T22:47:18.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yD+WJGB/ofbct9/JitF26SqCQUo=', 'Content-Length': '0'} 2024-04-13T22:47:18.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072adccf4161fd867-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.699 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.699 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.700 DEBUG:boto:path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.700 DEBUG:boto:auth_path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.700 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.700 DEBUG:boto:Path: /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.700 DEBUG:boto:Data: 2024-04-13T22:47:18.700 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.700 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.700 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.700 DEBUG:boto:Params: {} 2024-04-13T22:47:18.700 DEBUG:boto:Token: None 2024-04-13T22:47:18.701 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.701 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:yD+WJGB/ofbct9/JitF26SqCQUo= 2024-04-13T22:47:18.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:yD+WJGB/ofbct9/JitF26SqCQUo=', 'Content-Length': '0'} 2024-04-13T22:47:18.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bdf0fcd73917a9c-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:47:18.703 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.703 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.703 DEBUG:boto:path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.703 DEBUG:boto:auth_path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.703 DEBUG:boto:Method: GET 2024-04-13T22:47:18.703 DEBUG:boto:Path: /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.703 DEBUG:boto:Data: 2024-04-13T22:47:18.703 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.704 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.704 DEBUG:boto:Params: {} 2024-04-13T22:47:18.704 DEBUG:boto:Token: None 2024-04-13T22:47:18.704 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw= 2024-04-13T22:47:18.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw=', 'Content-Length': '0'} 2024-04-13T22:47:18.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005218b77861090d23-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.706 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.706 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.706 DEBUG:boto:path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.707 DEBUG:boto:auth_path=/vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.707 DEBUG:boto:Method: GET 2024-04-13T22:47:18.707 DEBUG:boto:Path: /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.707 DEBUG:boto:Data: 2024-04-13T22:47:18.707 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.707 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.707 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.707 DEBUG:boto:Params: {} 2024-04-13T22:47:18.707 DEBUG:boto:Token: None 2024-04-13T22:47:18.707 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22?versionId=null 2024-04-13T22:47:18.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw= 2024-04-13T22:47:18.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+lpe3GjwtePJR3aPmXONaDmP9Mw=', 'Content-Length': '0'} 2024-04-13T22:47:18.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a9735ca103e5480-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.709 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.710 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.710 DEBUG:boto:path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.710 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.710 DEBUG:boto:Method: GET 2024-04-13T22:47:18.710 DEBUG:boto:Path: /vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.710 DEBUG:boto:Data: 2024-04-13T22:47:18.710 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.710 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.710 DEBUG:boto:Params: {} 2024-04-13T22:47:18.711 DEBUG:boto:Token: None 2024-04-13T22:47:18.711 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b2de292d6480f0-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.713 DEBUG:boto:b'vwnfns-64obj221000falseobj222024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.713 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.713 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.713 DEBUG:boto:path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.713 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.714 DEBUG:boto:Method: GET 2024-04-13T22:47:18.714 DEBUG:boto:Path: /vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.714 DEBUG:boto:Data: 2024-04-13T22:47:18.714 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.714 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.714 DEBUG:boto:Params: {} 2024-04-13T22:47:18.714 DEBUG:boto:Token: None 2024-04-13T22:47:18.714 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad52aeb569547bf-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.717 DEBUG:boto:b'vwnfns-64obj221000falseobj222024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.717 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.717 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.717 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.717 DEBUG:boto:Path: /vwnfns-64/obj22 2024-04-13T22:47:18.718 DEBUG:boto:Data: 2024-04-13T22:47:18.718 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.718 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.718 DEBUG:boto:Params: {} 2024-04-13T22:47:18.718 DEBUG:boto:Token: None 2024-04-13T22:47:18.718 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22 2024-04-13T22:47:18.718 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7bARzXWmJAuX2iocOlJshDOvY/k= 2024-04-13T22:47:18.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7bARzXWmJAuX2iocOlJshDOvY/k=', 'Content-Length': '0'} 2024-04-13T22:47:18.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a801f789123f621f-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.722 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.722 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.722 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.722 DEBUG:boto:Path: /vwnfns-64/obj22 2024-04-13T22:47:18.723 DEBUG:boto:Data: 2024-04-13T22:47:18.723 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.723 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.723 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.723 DEBUG:boto:Params: {} 2024-04-13T22:47:18.723 DEBUG:boto:Token: None 2024-04-13T22:47:18.723 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22 2024-04-13T22:47:18.723 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7bARzXWmJAuX2iocOlJshDOvY/k= 2024-04-13T22:47:18.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7bARzXWmJAuX2iocOlJshDOvY/k=', 'Content-Length': '0'} 2024-04-13T22:47:18.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f1c59537fe9fb62-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:47:18.725 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.725 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.726 DEBUG:boto:Method: GET 2024-04-13T22:47:18.726 DEBUG:boto:Path: /vwnfns-64/obj22 2024-04-13T22:47:18.726 DEBUG:boto:Data: 2024-04-13T22:47:18.726 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.726 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.726 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.726 DEBUG:boto:Params: {} 2024-04-13T22:47:18.726 DEBUG:boto:Token: None 2024-04-13T22:47:18.726 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22 2024-04-13T22:47:18.727 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wqh9R3hFHnn8tk9uPgtmo0Svglk= 2024-04-13T22:47:18.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wqh9R3hFHnn8tk9uPgtmo0Svglk=', 'Content-Length': '0'} 2024-04-13T22:47:18.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea1685e331099edd-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.728 DEBUG:boto:path=/vwnfns-64/obj22 2024-04-13T22:47:18.729 DEBUG:boto:auth_path=/vwnfns-64/obj22 2024-04-13T22:47:18.729 DEBUG:boto:Method: GET 2024-04-13T22:47:18.729 DEBUG:boto:Path: /vwnfns-64/obj22 2024-04-13T22:47:18.729 DEBUG:boto:Data: 2024-04-13T22:47:18.729 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.729 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.729 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.729 DEBUG:boto:Params: {} 2024-04-13T22:47:18.730 DEBUG:boto:Token: None 2024-04-13T22:47:18.730 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj22 2024-04-13T22:47:18.730 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Wqh9R3hFHnn8tk9uPgtmo0Svglk= 2024-04-13T22:47:18.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Wqh9R3hFHnn8tk9uPgtmo0Svglk=', 'Content-Length': '0'} 2024-04-13T22:47:18.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009771159ca4c1fd44-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.732 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.732 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.733 DEBUG:boto:path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.733 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.733 DEBUG:boto:Method: GET 2024-04-13T22:47:18.733 DEBUG:boto:Path: /vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.733 DEBUG:boto:Data: 2024-04-13T22:47:18.733 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.733 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.733 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.733 DEBUG:boto:Params: {} 2024-04-13T22:47:18.733 DEBUG:boto:Token: None 2024-04-13T22:47:18.734 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.734 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cdffb3b964ec306-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.736 DEBUG:boto:b'vwnfns-64obj221000falseobj222024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.736 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.737 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.737 DEBUG:boto:path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.737 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.737 DEBUG:boto:Method: GET 2024-04-13T22:47:18.737 DEBUG:boto:Path: /vwnfns-64/?prefix=obj22 2024-04-13T22:47:18.737 DEBUG:boto:Data: 2024-04-13T22:47:18.737 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.737 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.737 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.737 DEBUG:boto:Params: {} 2024-04-13T22:47:18.738 DEBUG:boto:Token: None 2024-04-13T22:47:18.738 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.738 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dd4a3336efae7a2-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.740 DEBUG:boto:b'vwnfns-64obj221000falseobj222024-04-13T22:45:21.936Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:47:18.741 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.741 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.741 DEBUG:boto:path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.741 DEBUG:boto:auth_path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.741 DEBUG:boto:Method: GET 2024-04-13T22:47:18.741 DEBUG:boto:Path: /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.741 DEBUG:boto:Data: 2024-04-13T22:47:18.741 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.741 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.742 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.742 DEBUG:boto:Params: {} 2024-04-13T22:47:18.742 DEBUG:boto:Token: None 2024-04-13T22:47:18.742 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.742 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y= 2024-04-13T22:47:18.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y=', 'Content-Length': '0'} 2024-04-13T22:47:18.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000694534e4d813ebc5-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.744 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.744 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.744 DEBUG:boto:path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.744 DEBUG:boto:auth_path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.745 DEBUG:boto:Method: GET 2024-04-13T22:47:18.745 DEBUG:boto:Path: /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.745 DEBUG:boto:Data: 2024-04-13T22:47:18.745 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.745 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.745 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.745 DEBUG:boto:Params: {} 2024-04-13T22:47:18.745 DEBUG:boto:Token: None 2024-04-13T22:47:18.745 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.745 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y= 2024-04-13T22:47:18.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y=', 'Content-Length': '0'} 2024-04-13T22:47:18.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009994f23a1d045d7d-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.747 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.747 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.747 DEBUG:boto:path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.748 DEBUG:boto:auth_path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.748 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.748 DEBUG:boto:Path: /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.748 DEBUG:boto:Data: 2024-04-13T22:47:18.748 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.748 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.748 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.748 DEBUG:boto:Params: {} 2024-04-13T22:47:18.748 DEBUG:boto:Token: None 2024-04-13T22:47:18.748 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.748 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VAWqck3RHt2vv1V9PXUdFxnTjKI= 2024-04-13T22:47:18.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VAWqck3RHt2vv1V9PXUdFxnTjKI=', 'Content-Length': '0'} 2024-04-13T22:47:18.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c86dccea230ae7-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.750 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.751 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.751 DEBUG:boto:path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.751 DEBUG:boto:auth_path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.751 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.751 DEBUG:boto:Path: /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.751 DEBUG:boto:Data: 2024-04-13T22:47:18.751 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.751 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.751 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.751 DEBUG:boto:Params: {} 2024-04-13T22:47:18.752 DEBUG:boto:Token: None 2024-04-13T22:47:18.752 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.752 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VAWqck3RHt2vv1V9PXUdFxnTjKI= 2024-04-13T22:47:18.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VAWqck3RHt2vv1V9PXUdFxnTjKI=', 'Content-Length': '0'} 2024-04-13T22:47:18.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a51fd663064c47-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:47:18.754 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.754 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.754 DEBUG:boto:path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.754 DEBUG:boto:auth_path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.754 DEBUG:boto:Method: GET 2024-04-13T22:47:18.754 DEBUG:boto:Path: /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.754 DEBUG:boto:Data: 2024-04-13T22:47:18.755 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.755 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.755 DEBUG:boto:Params: {} 2024-04-13T22:47:18.755 DEBUG:boto:Token: None 2024-04-13T22:47:18.755 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.755 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y= 2024-04-13T22:47:18.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y=', 'Content-Length': '0'} 2024-04-13T22:47:18.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c894dd12282b7b6f-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.757 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.757 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.757 DEBUG:boto:path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.757 DEBUG:boto:auth_path=/vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.757 DEBUG:boto:Method: GET 2024-04-13T22:47:18.758 DEBUG:boto:Path: /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.758 DEBUG:boto:Data: 2024-04-13T22:47:18.758 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.758 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.758 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.758 DEBUG:boto:Params: {} 2024-04-13T22:47:18.758 DEBUG:boto:Token: None 2024-04-13T22:47:18.758 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33?versionId=null 2024-04-13T22:47:18.758 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y= 2024-04-13T22:47:18.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vB5bK+Iqje/1obwACCyVWvHyD/Y=', 'Content-Length': '0'} 2024-04-13T22:47:18.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240a693e3eddce2f-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.804 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.804 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.804 DEBUG:boto:path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.804 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.804 DEBUG:boto:Method: GET 2024-04-13T22:47:18.804 DEBUG:boto:Path: /vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.804 DEBUG:boto:Data: 2024-04-13T22:47:18.804 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.804 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.805 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.805 DEBUG:boto:Params: {} 2024-04-13T22:47:18.805 DEBUG:boto:Token: None 2024-04-13T22:47:18.805 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.805 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007591ed6428aa8bbe-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.807 DEBUG:boto:b'vwnfns-64obj331000falseobj332024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.808 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.808 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.808 DEBUG:boto:path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.808 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.808 DEBUG:boto:Method: GET 2024-04-13T22:47:18.808 DEBUG:boto:Path: /vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.809 DEBUG:boto:Data: 2024-04-13T22:47:18.809 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.809 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.809 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.809 DEBUG:boto:Params: {} 2024-04-13T22:47:18.809 DEBUG:boto:Token: None 2024-04-13T22:47:18.809 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.809 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063545157dc00f7eb-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.812 DEBUG:boto:b'vwnfns-64obj331000falseobj332024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.812 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.812 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.812 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.812 DEBUG:boto:Path: /vwnfns-64/obj33 2024-04-13T22:47:18.812 DEBUG:boto:Data: 2024-04-13T22:47:18.812 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.812 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.813 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.813 DEBUG:boto:Params: {} 2024-04-13T22:47:18.813 DEBUG:boto:Token: None 2024-04-13T22:47:18.813 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33 2024-04-13T22:47:18.813 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NMLE0FJif1W2YEgpuJCkMXABA60= 2024-04-13T22:47:18.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NMLE0FJif1W2YEgpuJCkMXABA60=', 'Content-Length': '0'} 2024-04-13T22:47:18.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd4e749fde824c6-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.815 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.815 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.815 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.815 DEBUG:boto:Path: /vwnfns-64/obj33 2024-04-13T22:47:18.815 DEBUG:boto:Data: 2024-04-13T22:47:18.815 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.815 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.815 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.816 DEBUG:boto:Params: {} 2024-04-13T22:47:18.816 DEBUG:boto:Token: None 2024-04-13T22:47:18.816 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33 2024-04-13T22:47:18.816 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NMLE0FJif1W2YEgpuJCkMXABA60= 2024-04-13T22:47:18.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NMLE0FJif1W2YEgpuJCkMXABA60=', 'Content-Length': '0'} 2024-04-13T22:47:18.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000967a95fc3a31f66-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:47:18.818 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.818 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.818 DEBUG:boto:Method: GET 2024-04-13T22:47:18.818 DEBUG:boto:Path: /vwnfns-64/obj33 2024-04-13T22:47:18.818 DEBUG:boto:Data: 2024-04-13T22:47:18.818 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.818 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.819 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.819 DEBUG:boto:Params: {} 2024-04-13T22:47:18.819 DEBUG:boto:Token: None 2024-04-13T22:47:18.819 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33 2024-04-13T22:47:18.819 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JDbniFvyjoYkdfbjoU03eRefVpc= 2024-04-13T22:47:18.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JDbniFvyjoYkdfbjoU03eRefVpc=', 'Content-Length': '0'} 2024-04-13T22:47:18.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e012d28668ea86-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.863 DEBUG:boto:path=/vwnfns-64/obj33 2024-04-13T22:47:18.863 DEBUG:boto:auth_path=/vwnfns-64/obj33 2024-04-13T22:47:18.863 DEBUG:boto:Method: GET 2024-04-13T22:47:18.864 DEBUG:boto:Path: /vwnfns-64/obj33 2024-04-13T22:47:18.864 DEBUG:boto:Data: 2024-04-13T22:47:18.864 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.864 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.864 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.864 DEBUG:boto:Params: {} 2024-04-13T22:47:18.864 DEBUG:boto:Token: None 2024-04-13T22:47:18.864 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj33 2024-04-13T22:47:18.864 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JDbniFvyjoYkdfbjoU03eRefVpc= 2024-04-13T22:47:18.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JDbniFvyjoYkdfbjoU03eRefVpc=', 'Content-Length': '0'} 2024-04-13T22:47:18.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b9f600bea45491-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.867 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.867 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.867 DEBUG:boto:path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.867 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.867 DEBUG:boto:Method: GET 2024-04-13T22:47:18.867 DEBUG:boto:Path: /vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.867 DEBUG:boto:Data: 2024-04-13T22:47:18.867 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.867 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.867 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.868 DEBUG:boto:Params: {} 2024-04-13T22:47:18.868 DEBUG:boto:Token: None 2024-04-13T22:47:18.868 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.868 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736fed9fa4f50d41-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.870 DEBUG:boto:b'vwnfns-64obj331000falseobj332024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.870 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.871 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.871 DEBUG:boto:path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.871 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.871 DEBUG:boto:Method: GET 2024-04-13T22:47:18.871 DEBUG:boto:Path: /vwnfns-64/?prefix=obj33 2024-04-13T22:47:18.871 DEBUG:boto:Data: 2024-04-13T22:47:18.871 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.871 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.871 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.871 DEBUG:boto:Params: {} 2024-04-13T22:47:18.872 DEBUG:boto:Token: None 2024-04-13T22:47:18.872 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.872 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aa9fb9830e3272d-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.918 DEBUG:boto:b'vwnfns-64obj331000falseobj332024-04-13T22:45:21.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:47:18.919 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.919 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.919 DEBUG:boto:path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.919 DEBUG:boto:auth_path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.920 DEBUG:boto:Method: GET 2024-04-13T22:47:18.920 DEBUG:boto:Path: /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.920 DEBUG:boto:Data: 2024-04-13T22:47:18.920 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.920 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.920 DEBUG:boto:Params: {} 2024-04-13T22:47:18.920 DEBUG:boto:Token: None 2024-04-13T22:47:18.920 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.921 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE= 2024-04-13T22:47:18.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE=', 'Content-Length': '0'} 2024-04-13T22:47:18.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8caa2827bc4617-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.923 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.923 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.923 DEBUG:boto:path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.923 DEBUG:boto:auth_path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.923 DEBUG:boto:Method: GET 2024-04-13T22:47:18.923 DEBUG:boto:Path: /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.924 DEBUG:boto:Data: 2024-04-13T22:47:18.924 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.924 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.924 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.924 DEBUG:boto:Params: {} 2024-04-13T22:47:18.924 DEBUG:boto:Token: None 2024-04-13T22:47:18.924 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.924 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE= 2024-04-13T22:47:18.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE=', 'Content-Length': '0'} 2024-04-13T22:47:18.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2294c4641fb35e1-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.926 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.926 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.927 DEBUG:boto:path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.927 DEBUG:boto:auth_path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.927 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.927 DEBUG:boto:Path: /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.927 DEBUG:boto:Data: 2024-04-13T22:47:18.927 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.927 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.927 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.927 DEBUG:boto:Params: {} 2024-04-13T22:47:18.927 DEBUG:boto:Token: None 2024-04-13T22:47:18.927 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.928 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DMCMUSHt+E/xdl6F0h+YjYbAfXg= 2024-04-13T22:47:18.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DMCMUSHt+E/xdl6F0h+YjYbAfXg=', 'Content-Length': '0'} 2024-04-13T22:47:18.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfa6c5e85da1b60-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.930 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.930 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.930 DEBUG:boto:path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.930 DEBUG:boto:auth_path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.930 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.930 DEBUG:boto:Path: /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.930 DEBUG:boto:Data: 2024-04-13T22:47:18.930 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.931 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.931 DEBUG:boto:Params: {} 2024-04-13T22:47:18.931 DEBUG:boto:Token: None 2024-04-13T22:47:18.931 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.931 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DMCMUSHt+E/xdl6F0h+YjYbAfXg= 2024-04-13T22:47:18.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DMCMUSHt+E/xdl6F0h+YjYbAfXg=', 'Content-Length': '0'} 2024-04-13T22:47:18.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d193c985368153-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:47:18.933 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.933 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.933 DEBUG:boto:path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.934 DEBUG:boto:auth_path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.934 DEBUG:boto:Method: GET 2024-04-13T22:47:18.934 DEBUG:boto:Path: /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.934 DEBUG:boto:Data: 2024-04-13T22:47:18.934 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.934 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.934 DEBUG:boto:Params: {} 2024-04-13T22:47:18.934 DEBUG:boto:Token: None 2024-04-13T22:47:18.934 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE= 2024-04-13T22:47:18.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE=', 'Content-Length': '0'} 2024-04-13T22:47:18.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eaddda2b6ad8fc5-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.936 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.937 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.937 DEBUG:boto:path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.937 DEBUG:boto:auth_path=/vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.937 DEBUG:boto:Method: GET 2024-04-13T22:47:18.937 DEBUG:boto:Path: /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.937 DEBUG:boto:Data: 2024-04-13T22:47:18.937 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.937 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.937 DEBUG:boto:Params: {} 2024-04-13T22:47:18.938 DEBUG:boto:Token: None 2024-04-13T22:47:18.938 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44?versionId=null 2024-04-13T22:47:18.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE= 2024-04-13T22:47:18.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CojlmfDZXv/QikoLxfV+1kUW4jE=', 'Content-Length': '0'} 2024-04-13T22:47:18.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a8648e3e21f215b-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.940 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.940 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.940 DEBUG:boto:path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:18.940 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:18.940 DEBUG:boto:Method: GET 2024-04-13T22:47:18.940 DEBUG:boto:Path: /vwnfns-64/?prefix=obj44 2024-04-13T22:47:18.940 DEBUG:boto:Data: 2024-04-13T22:47:18.940 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.941 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.941 DEBUG:boto:Params: {} 2024-04-13T22:47:18.941 DEBUG:boto:Token: None 2024-04-13T22:47:18.941 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.941 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010314e776968274d-00661b0b76-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.943 DEBUG:boto:b'vwnfns-64obj441000falseobj442024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.944 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:18.944 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:18.944 DEBUG:boto:path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:18.944 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:18.944 DEBUG:boto:Method: GET 2024-04-13T22:47:18.944 DEBUG:boto:Path: /vwnfns-64/?prefix=obj44 2024-04-13T22:47:18.944 DEBUG:boto:Data: 2024-04-13T22:47:18.944 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.944 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.945 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.945 DEBUG:boto:Params: {} 2024-04-13T22:47:18.945 DEBUG:boto:Token: None 2024-04-13T22:47:18.945 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/ 2024-04-13T22:47:18.945 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc= 2024-04-13T22:47:18.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vw9e7Kh2O65BUg8ar8MOAEQQ5wc=', 'Content-Length': '0'} 2024-04-13T22:47:18.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e852ac30760657f5-00661b0b76-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.947 DEBUG:boto:b'vwnfns-64obj441000falseobj442024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:18.948 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.948 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.948 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.948 DEBUG:boto:Path: /vwnfns-64/obj44 2024-04-13T22:47:18.948 DEBUG:boto:Data: 2024-04-13T22:47:18.948 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.948 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.948 DEBUG:boto:Params: {} 2024-04-13T22:47:18.949 DEBUG:boto:Token: None 2024-04-13T22:47:18.949 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44 2024-04-13T22:47:18.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TeKdr74MbnHPXUQa+DhE9BtFyzk= 2024-04-13T22:47:18.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TeKdr74MbnHPXUQa+DhE9BtFyzk=', 'Content-Length': '0'} 2024-04-13T22:47:18.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004afa288aaf0feb09-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.951 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.951 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.951 DEBUG:boto:Method: HEAD 2024-04-13T22:47:18.951 DEBUG:boto:Path: /vwnfns-64/obj44 2024-04-13T22:47:18.952 DEBUG:boto:Data: 2024-04-13T22:47:18.952 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.952 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.952 DEBUG:boto:Params: {} 2024-04-13T22:47:18.952 DEBUG:boto:Token: None 2024-04-13T22:47:18.952 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44 2024-04-13T22:47:18.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TeKdr74MbnHPXUQa+DhE9BtFyzk= 2024-04-13T22:47:18.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TeKdr74MbnHPXUQa+DhE9BtFyzk=', 'Content-Length': '0'} 2024-04-13T22:47:18.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e39d602d53dd3b5-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:47:18.954 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.955 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.955 DEBUG:boto:Method: GET 2024-04-13T22:47:18.955 DEBUG:boto:Path: /vwnfns-64/obj44 2024-04-13T22:47:18.955 DEBUG:boto:Data: 2024-04-13T22:47:18.955 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:18.955 DEBUG:boto:Port: 8001 2024-04-13T22:47:18.955 DEBUG:boto:Params: {} 2024-04-13T22:47:18.955 DEBUG:boto:Token: None 2024-04-13T22:47:18.955 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44 2024-04-13T22:47:18.955 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/T86NqfYaO01sfOQdRkDxB2LzMo= 2024-04-13T22:47:18.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/T86NqfYaO01sfOQdRkDxB2LzMo=', 'Content-Length': '0'} 2024-04-13T22:47:18.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000779d60f61c1dac0b-00661b0b76-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:18.998 DEBUG:boto:path=/vwnfns-64/obj44 2024-04-13T22:47:18.999 DEBUG:boto:auth_path=/vwnfns-64/obj44 2024-04-13T22:47:18.999 DEBUG:boto:Method: GET 2024-04-13T22:47:18.999 DEBUG:boto:Path: /vwnfns-64/obj44 2024-04-13T22:47:18.999 DEBUG:boto:Data: 2024-04-13T22:47:18.999 DEBUG:boto:Headers: {} 2024-04-13T22:47:18.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:18.999 DEBUG:boto:Port: 8000 2024-04-13T22:47:18.999 DEBUG:boto:Params: {} 2024-04-13T22:47:18.999 DEBUG:boto:Token: None 2024-04-13T22:47:18.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:18 GMT /vwnfns-64/obj44 2024-04-13T22:47:19.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/T86NqfYaO01sfOQdRkDxB2LzMo= 2024-04-13T22:47:19.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:18 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/T86NqfYaO01sfOQdRkDxB2LzMo=', 'Content-Length': '0'} 2024-04-13T22:47:19.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003090a3578d65b2e7-00661b0b76-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.002 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:19.002 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:19.002 DEBUG:boto:path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:19.002 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:19.002 DEBUG:boto:Method: GET 2024-04-13T22:47:19.002 DEBUG:boto:Path: /vwnfns-64/?prefix=obj44 2024-04-13T22:47:19.002 DEBUG:boto:Data: 2024-04-13T22:47:19.002 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.002 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.003 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.003 DEBUG:boto:Params: {} 2024-04-13T22:47:19.003 DEBUG:boto:Token: None 2024-04-13T22:47:19.003 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/ 2024-04-13T22:47:19.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs= 2024-04-13T22:47:19.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs=', 'Content-Length': '0'} 2024-04-13T22:47:19.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0bcbbe2eac227b-00661b0b77-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.005 DEBUG:boto:b'vwnfns-64obj441000falseobj442024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:19.005 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:19.006 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:19.006 DEBUG:boto:path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:19.006 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj44 2024-04-13T22:47:19.006 DEBUG:boto:Method: GET 2024-04-13T22:47:19.006 DEBUG:boto:Path: /vwnfns-64/?prefix=obj44 2024-04-13T22:47:19.006 DEBUG:boto:Data: 2024-04-13T22:47:19.006 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.006 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.006 DEBUG:boto:Params: {} 2024-04-13T22:47:19.007 DEBUG:boto:Token: None 2024-04-13T22:47:19.007 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/ 2024-04-13T22:47:19.007 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs= 2024-04-13T22:47:19.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs=', 'Content-Length': '0'} 2024-04-13T22:47:19.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5e61ab8b433e0c8-00661b0b77-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.009 DEBUG:boto:b'vwnfns-64obj441000falseobj442024-04-13T22:45:21.995Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:19.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:47:19.010 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.010 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.010 DEBUG:boto:path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.010 DEBUG:boto:auth_path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.010 DEBUG:boto:Method: GET 2024-04-13T22:47:19.010 DEBUG:boto:Path: /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.010 DEBUG:boto:Data: 2024-04-13T22:47:19.010 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.010 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.010 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.011 DEBUG:boto:Params: {} 2024-04-13T22:47:19.011 DEBUG:boto:Token: None 2024-04-13T22:47:19.011 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.011 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE= 2024-04-13T22:47:19.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE=', 'Content-Length': '0'} 2024-04-13T22:47:19.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d270ee869ecaeda-00661b0b77-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.013 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.013 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.013 DEBUG:boto:path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.014 DEBUG:boto:auth_path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.014 DEBUG:boto:Method: GET 2024-04-13T22:47:19.014 DEBUG:boto:Path: /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.014 DEBUG:boto:Data: 2024-04-13T22:47:19.014 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.014 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.014 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.014 DEBUG:boto:Params: {} 2024-04-13T22:47:19.014 DEBUG:boto:Token: None 2024-04-13T22:47:19.015 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.015 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE= 2024-04-13T22:47:19.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE=', 'Content-Length': '0'} 2024-04-13T22:47:19.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d43bbbabebe450-00661b0b77-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.017 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.017 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.017 DEBUG:boto:path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.017 DEBUG:boto:auth_path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.017 DEBUG:boto:Method: HEAD 2024-04-13T22:47:19.017 DEBUG:boto:Path: /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.017 DEBUG:boto:Data: 2024-04-13T22:47:19.017 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.017 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.017 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.017 DEBUG:boto:Params: {} 2024-04-13T22:47:19.017 DEBUG:boto:Token: None 2024-04-13T22:47:19.018 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.018 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:maU6vBumjLHc7blr+D8Ri7Xp010= 2024-04-13T22:47:19.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:maU6vBumjLHc7blr+D8Ri7Xp010=', 'Content-Length': '0'} 2024-04-13T22:47:19.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed6e482010e4d83-00661b0b77-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.020 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.020 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.020 DEBUG:boto:path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.020 DEBUG:boto:auth_path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.020 DEBUG:boto:Method: HEAD 2024-04-13T22:47:19.020 DEBUG:boto:Path: /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.020 DEBUG:boto:Data: 2024-04-13T22:47:19.020 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.021 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.021 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.021 DEBUG:boto:Params: {} 2024-04-13T22:47:19.021 DEBUG:boto:Token: None 2024-04-13T22:47:19.021 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:maU6vBumjLHc7blr+D8Ri7Xp010= 2024-04-13T22:47:19.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:maU6vBumjLHc7blr+D8Ri7Xp010=', 'Content-Length': '0'} 2024-04-13T22:47:19.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa1ffd300bd17b91-00661b0b77-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:47:19.023 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.023 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.023 DEBUG:boto:path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.023 DEBUG:boto:auth_path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.023 DEBUG:boto:Method: GET 2024-04-13T22:47:19.023 DEBUG:boto:Path: /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.024 DEBUG:boto:Data: 2024-04-13T22:47:19.024 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.024 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.024 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.024 DEBUG:boto:Params: {} 2024-04-13T22:47:19.024 DEBUG:boto:Token: None 2024-04-13T22:47:19.024 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.024 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE= 2024-04-13T22:47:19.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE=', 'Content-Length': '0'} 2024-04-13T22:47:19.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b94e07b5fe8327df-00661b0b77-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.026 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.026 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.026 DEBUG:boto:path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.027 DEBUG:boto:auth_path=/vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.027 DEBUG:boto:Method: GET 2024-04-13T22:47:19.027 DEBUG:boto:Path: /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.027 DEBUG:boto:Data: 2024-04-13T22:47:19.027 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.027 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.027 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.027 DEBUG:boto:Params: {} 2024-04-13T22:47:19.027 DEBUG:boto:Token: None 2024-04-13T22:47:19.028 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55?versionId=null 2024-04-13T22:47:19.028 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE= 2024-04-13T22:47:19.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2jCk5zG46XPpHsvArDOZ/5Vs6gE=', 'Content-Length': '0'} 2024-04-13T22:47:19.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cba2bed724d970b-00661b0b77-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.030 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:19.030 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:19.030 DEBUG:boto:path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.030 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.030 DEBUG:boto:Method: GET 2024-04-13T22:47:19.030 DEBUG:boto:Path: /vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.030 DEBUG:boto:Data: 2024-04-13T22:47:19.030 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.031 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.031 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.031 DEBUG:boto:Params: {} 2024-04-13T22:47:19.031 DEBUG:boto:Token: None 2024-04-13T22:47:19.031 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/ 2024-04-13T22:47:19.031 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs= 2024-04-13T22:47:19.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs=', 'Content-Length': '0'} 2024-04-13T22:47:19.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73ee8a8894032f2-00661b0b77-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.033 DEBUG:boto:b'vwnfns-64obj551000falseobj552024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:19.034 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:19.034 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:19.034 DEBUG:boto:path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.034 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.034 DEBUG:boto:Method: GET 2024-04-13T22:47:19.034 DEBUG:boto:Path: /vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.034 DEBUG:boto:Data: 2024-04-13T22:47:19.034 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.034 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.034 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.035 DEBUG:boto:Params: {} 2024-04-13T22:47:19.035 DEBUG:boto:Token: None 2024-04-13T22:47:19.035 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/ 2024-04-13T22:47:19.035 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs= 2024-04-13T22:47:19.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs=', 'Content-Length': '0'} 2024-04-13T22:47:19.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2259ababd3bcdc-00661b0b77-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.037 DEBUG:boto:b'vwnfns-64obj551000falseobj552024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:19.038 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.038 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.038 DEBUG:boto:Method: HEAD 2024-04-13T22:47:19.038 DEBUG:boto:Path: /vwnfns-64/obj55 2024-04-13T22:47:19.038 DEBUG:boto:Data: 2024-04-13T22:47:19.038 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.038 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.039 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.039 DEBUG:boto:Params: {} 2024-04-13T22:47:19.039 DEBUG:boto:Token: None 2024-04-13T22:47:19.039 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55 2024-04-13T22:47:19.039 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6vnusqKMvb5v6tjvOylH1rNX27o= 2024-04-13T22:47:19.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6vnusqKMvb5v6tjvOylH1rNX27o=', 'Content-Length': '0'} 2024-04-13T22:47:19.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ba43f944a41ec2-00661b0b77-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.041 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.041 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.041 DEBUG:boto:Method: HEAD 2024-04-13T22:47:19.041 DEBUG:boto:Path: /vwnfns-64/obj55 2024-04-13T22:47:19.041 DEBUG:boto:Data: 2024-04-13T22:47:19.041 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.041 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.041 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.041 DEBUG:boto:Params: {} 2024-04-13T22:47:19.042 DEBUG:boto:Token: None 2024-04-13T22:47:19.042 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55 2024-04-13T22:47:19.042 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:6vnusqKMvb5v6tjvOylH1rNX27o= 2024-04-13T22:47:19.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6vnusqKMvb5v6tjvOylH1rNX27o=', 'Content-Length': '0'} 2024-04-13T22:47:19.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050dcc943dad15332-00661b0b77-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:47:19.044 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.044 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.044 DEBUG:boto:Method: GET 2024-04-13T22:47:19.044 DEBUG:boto:Path: /vwnfns-64/obj55 2024-04-13T22:47:19.044 DEBUG:boto:Data: 2024-04-13T22:47:19.044 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.044 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.044 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.045 DEBUG:boto:Params: {} 2024-04-13T22:47:19.045 DEBUG:boto:Token: None 2024-04-13T22:47:19.045 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55 2024-04-13T22:47:19.045 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L82m2Zl9deYPR3e0KmXIwlcbf0w= 2024-04-13T22:47:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L82m2Zl9deYPR3e0KmXIwlcbf0w=', 'Content-Length': '0'} 2024-04-13T22:47:19.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-64:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e448f4a352499ac-00661b0b77-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.047 DEBUG:boto:path=/vwnfns-64/obj55 2024-04-13T22:47:19.047 DEBUG:boto:auth_path=/vwnfns-64/obj55 2024-04-13T22:47:19.047 DEBUG:boto:Method: GET 2024-04-13T22:47:19.047 DEBUG:boto:Path: /vwnfns-64/obj55 2024-04-13T22:47:19.047 DEBUG:boto:Data: 2024-04-13T22:47:19.048 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.048 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.048 DEBUG:boto:Params: {} 2024-04-13T22:47:19.048 DEBUG:boto:Token: None 2024-04-13T22:47:19.048 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/obj55 2024-04-13T22:47:19.048 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:L82m2Zl9deYPR3e0KmXIwlcbf0w= 2024-04-13T22:47:19.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L82m2Zl9deYPR3e0KmXIwlcbf0w=', 'Content-Length': '0'} 2024-04-13T22:47:19.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:45:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697bebd1e7805525-00661b0b77-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.050 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:19.050 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:19.050 DEBUG:boto:path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.050 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.050 DEBUG:boto:Method: GET 2024-04-13T22:47:19.051 DEBUG:boto:Path: /vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.051 DEBUG:boto:Data: 2024-04-13T22:47:19.051 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.051 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:47:19.051 DEBUG:boto:Port: 8001 2024-04-13T22:47:19.051 DEBUG:boto:Params: {} 2024-04-13T22:47:19.051 DEBUG:boto:Token: None 2024-04-13T22:47:19.051 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/ 2024-04-13T22:47:19.051 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs= 2024-04-13T22:47:19.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs=', 'Content-Length': '0'} 2024-04-13T22:47:19.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dec6112dc0291e6-00661b0b77-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.054 DEBUG:boto:b'vwnfns-64obj551000falseobj552024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:19.054 DEBUG:boto:path=/vwnfns-64/ 2024-04-13T22:47:19.054 DEBUG:boto:auth_path=/vwnfns-64/ 2024-04-13T22:47:19.054 DEBUG:boto:path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.054 DEBUG:boto:auth_path=/vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.055 DEBUG:boto:Method: GET 2024-04-13T22:47:19.055 DEBUG:boto:Path: /vwnfns-64/?prefix=obj55 2024-04-13T22:47:19.055 DEBUG:boto:Data: 2024-04-13T22:47:19.055 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.055 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.055 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.055 DEBUG:boto:Params: {} 2024-04-13T22:47:19.055 DEBUG:boto:Token: None 2024-04-13T22:47:19.055 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-64/ 2024-04-13T22:47:19.055 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs= 2024-04-13T22:47:19.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WvRH5Y3qhf28SlHgzqqejA5bZNs=', 'Content-Length': '0'} 2024-04-13T22:47:19.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e731d2a0d332c32c-00661b0b77-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.058 DEBUG:boto:b'vwnfns-64obj551000falseobj552024-04-13T22:45:22.045Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:47:19.058 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-64 zones={a2, a1} 2024-04-13T22:47:19.058 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-13T22:47:19.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.059 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:19.106 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:19.106 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:19.154 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000034838", 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:45:29.260957Z" 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000034185", 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:45:29.262836Z" 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000024692", 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:35:57.255274Z" 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000039784", 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:45:21.998941Z" 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.156 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:47:19.157 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000034838', 1: '00000000000000000000:00000000000000034185', 2: '00000000000000000000:00000000000000024692', 3: '00000000000000000000:00000000000000039784'} 2024-04-13T22:47:19.157 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:47:19.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.158 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:19.207 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:19.207 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:19.252 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:47:19.252 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 2588147585097420363 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.253 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000034838", 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:45:29.260957Z" 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.254 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000034185", 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:45:29.262836Z" 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:47:19.255 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000024692", 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:35:57.255274Z" 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:47:19.256 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000039784", 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 55, 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:45:21.998941Z" 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 64, 2024-04-13T22:47:19.257 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 64 2024-04-13T22:47:19.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.258 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:47:19Z" 2024-04-13T22:47:19.258 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:47:19.259 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2588147585097420363 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034838", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:45:29.260957Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:45:29.262836Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024692", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:35:57.255274Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000039784", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2024-04-13T22:45:21.998941Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2024-04-13T22:47:19Z" } 2024-04-13T22:47:19.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034838', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:45:29.260957Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:45:29.262836Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024692', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:35:57.255274Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000039784', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2024-04-13T22:45:21.998941Z'}}] 2024-04-13T22:47:19.259 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000034838', 1: '00000000000000000000:00000000000000034185', 2: '00000000000000000000:00000000000000024692', 3: '00000000000000000000:00000000000000039784'} 2024-04-13T22:47:19.259 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000034838', 1: '00000000000000000000:00000000000000034185', 2: '00000000000000000000:00000000000000024692', 3: '00000000000000000000:00000000000000039784'} 2024-04-13T22:47:19.259 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T22:47:19.259 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-13T22:47:19.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.260 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:19.306 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:19.307 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:19.355 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:47:19.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.355 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000016723", 2024-04-13T22:47:19.355 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:47:12.296027Z" 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000013373", 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:47:12.297797Z" 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:38:00.753739Z" 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.356 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000017985", 2024-04-13T22:47:19.357 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:46:56.654726Z" 2024-04-13T22:47:19.357 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.357 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:47:19.358 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000016723', 1: '00000000000000000000:00000000000000013373', 2: '00000000000000000000:00000000000000007274', 3: '00000000000000000000:00000000000000017985'} 2024-04-13T22:47:19.358 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:47:19.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.359 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:19.456 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.457 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000016723", 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:47:12.296027Z" 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.458 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000013373", 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:47:12.297797Z" 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:38:00.753739Z" 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000017985", 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:46:56.654726Z" 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:47:19Z" 2024-04-13T22:47:19.462 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:47:19.463 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:47:12.296027Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013373", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:47:12.297797Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:38:00.753739Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017985", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:46:56.654726Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T22:47:19Z" } 2024-04-13T22:47:19.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:47:12.296027Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013373', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:47:12.297797Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:38:00.753739Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017985', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:46:56.654726Z'}}] 2024-04-13T22:47:19.464 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000016723', 1: '00000000000000000000:00000000000000013373', 2: '00000000000000000000:00000000000000007274', 3: '00000000000000000000:00000000000000017985'} 2024-04-13T22:47:19.464 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000016723', 1: '00000000000000000000:00000000000000013373', 2: '00000000000000000000:00000000000000007274', 3: '00000000000000000000:00000000000000017985'} 2024-04-13T22:47:19.464 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T22:47:19.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2024-04-13T22:47:19.466 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:19.466 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:19.466 DEBUG:boto:Method: PUT 2024-04-13T22:47:19.466 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:19.466 DEBUG:boto:Data: 2024-04-13T22:47:19.466 DEBUG:boto:Headers: {} 2024-04-13T22:47:19.466 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:19.466 DEBUG:boto:Port: 8000 2024-04-13T22:47:19.466 DEBUG:boto:Params: {} 2024-04-13T22:47:19.467 DEBUG:boto:Token: None 2024-04-13T22:47:19.467 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:47:19 GMT /vwnfns-65/ 2024-04-13T22:47:19.467 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z+Ni6ZLpMarhjizv+G9mHRlnL6U= 2024-04-13T22:47:19.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z+Ni6ZLpMarhjizv+G9mHRlnL6U=', 'Content-Length': '0'} 2024-04-13T22:47:19.482 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7610b9eab028240-00661b0b77-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:19.482 DEBUG:rgw_multi.tests:created bucket=vwnfns-65 2024-04-13T22:47:19.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.483 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:19.532 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:19.532 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:19.564 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:47:19.577 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048439.479509_937.1", 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:47:19.479509Z" 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.579 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:47:19.580 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:47:19.580 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:47:19.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:19.581 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:19.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:19.654+0000 7f439f56e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:47:19.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:19.655+0000 7f439f56e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:47:19.671 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:47:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:47:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:47:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:47:19.671 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048439.473212_934.1", 2024-04-13T22:47:19.672 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:47:19.473212Z", 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:19.673 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:47:19.675 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:19.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:19.676 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:47:19.677 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:19.677 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:19.677 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:47:19.677 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:19.677 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:47:19.678 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:47:19.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:19.679 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:47:19Z" 2024-04-13T22:47:19.679 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:47:19.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048439.473212_934.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:47:19.473212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:47:19Z" } 2024-04-13T22:47:19.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048439.473212_934.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:47:19.473212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:47:19.680 DEBUG:rgw_multi.tests:log_status={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:47:19.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048439.473212_934.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:47:19.680 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713048439.479509_937.1 target=1_1713048439.473212_934.1 2024-04-13T22:47:24.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:24.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:47:24.682 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:47:24.737 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:24.737 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:24.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:24.769+0000 7f3ad801e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:47:24.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:24.770+0000 7f3ad801e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:47:24.787 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:47:24.787 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:47:24.787 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:24.788 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048439.479509_937.1", 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:47:19.479509Z", 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:24.790 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:24.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:47:24.792 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:47:24Z" 2024-04-13T22:47:24.793 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:47:24.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048439.479509_937.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:47:19.479509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:47:24Z" } 2024-04-13T22:47:24.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048439.479509_937.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:47:19.479509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:47:24.794 DEBUG:rgw_multi.tests:log_status={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:47:24.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:47:24.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:47:24.795 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:24.795 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:24.795 DEBUG:boto:Method: HEAD 2024-04-13T22:47:24.795 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:24.795 DEBUG:boto:Data: 2024-04-13T22:47:24.795 DEBUG:boto:Headers: {} 2024-04-13T22:47:24.795 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.795 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.796 DEBUG:boto:Params: {} 2024-04-13T22:47:24.796 DEBUG:boto:Token: None 2024-04-13T22:47:24.796 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/ 2024-04-13T22:47:24.796 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T22:47:24.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T22:47:24.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7afbc1a0610771e-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.799 DEBUG:boto:path=/vwnfns-65/obj11 2024-04-13T22:47:24.799 DEBUG:boto:auth_path=/vwnfns-65/obj11 2024-04-13T22:47:24.799 DEBUG:boto:Method: PUT 2024-04-13T22:47:24.799 DEBUG:boto:Path: /vwnfns-65/obj11 2024-04-13T22:47:24.799 DEBUG:boto:Data: 2024-04-13T22:47:24.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:24.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.799 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.799 DEBUG:boto:Params: {} 2024-04-13T22:47:24.800 DEBUG:boto:Token: None 2024-04-13T22:47:24.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/obj11 2024-04-13T22:47:24.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jMp7Mq0w+FXy1VJhE1WZiyidmVk= 2024-04-13T22:47:24.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jMp7Mq0w+FXy1VJhE1WZiyidmVk='} 2024-04-13T22:47:24.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a28e513050b1bbcd-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.806 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:24.806 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:24.806 DEBUG:boto:Method: HEAD 2024-04-13T22:47:24.806 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:24.806 DEBUG:boto:Data: 2024-04-13T22:47:24.807 DEBUG:boto:Headers: {} 2024-04-13T22:47:24.807 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.807 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.807 DEBUG:boto:Params: {} 2024-04-13T22:47:24.807 DEBUG:boto:Token: None 2024-04-13T22:47:24.807 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/ 2024-04-13T22:47:24.807 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T22:47:24.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T22:47:24.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f956683f4a68662-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.810 DEBUG:boto:path=/vwnfns-65/obj22 2024-04-13T22:47:24.810 DEBUG:boto:auth_path=/vwnfns-65/obj22 2024-04-13T22:47:24.810 DEBUG:boto:Method: PUT 2024-04-13T22:47:24.810 DEBUG:boto:Path: /vwnfns-65/obj22 2024-04-13T22:47:24.810 DEBUG:boto:Data: 2024-04-13T22:47:24.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:24.810 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.811 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.811 DEBUG:boto:Params: {} 2024-04-13T22:47:24.811 DEBUG:boto:Token: None 2024-04-13T22:47:24.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/obj22 2024-04-13T22:47:24.811 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5bO5dttzskQOiyCx1n1cljhG6vQ= 2024-04-13T22:47:24.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5bO5dttzskQOiyCx1n1cljhG6vQ='} 2024-04-13T22:47:24.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000274b85af013ddd3-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.855 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:24.855 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:24.855 DEBUG:boto:Method: HEAD 2024-04-13T22:47:24.855 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:24.855 DEBUG:boto:Data: 2024-04-13T22:47:24.855 DEBUG:boto:Headers: {} 2024-04-13T22:47:24.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.855 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.856 DEBUG:boto:Params: {} 2024-04-13T22:47:24.856 DEBUG:boto:Token: None 2024-04-13T22:47:24.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/ 2024-04-13T22:47:24.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T22:47:24.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T22:47:24.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c074b2d259407c99-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.858 DEBUG:boto:path=/vwnfns-65/obj33 2024-04-13T22:47:24.858 DEBUG:boto:auth_path=/vwnfns-65/obj33 2024-04-13T22:47:24.858 DEBUG:boto:Method: PUT 2024-04-13T22:47:24.859 DEBUG:boto:Path: /vwnfns-65/obj33 2024-04-13T22:47:24.859 DEBUG:boto:Data: 2024-04-13T22:47:24.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:24.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.859 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.859 DEBUG:boto:Params: {} 2024-04-13T22:47:24.859 DEBUG:boto:Token: None 2024-04-13T22:47:24.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/obj33 2024-04-13T22:47:24.859 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:j3VxkW0mEV3Sym37J4DMdO5lvmQ= 2024-04-13T22:47:24.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j3VxkW0mEV3Sym37J4DMdO5lvmQ='} 2024-04-13T22:47:24.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0fa41aa77f4a1a1-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.866 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:24.866 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:24.866 DEBUG:boto:Method: HEAD 2024-04-13T22:47:24.866 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:24.866 DEBUG:boto:Data: 2024-04-13T22:47:24.866 DEBUG:boto:Headers: {} 2024-04-13T22:47:24.866 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.866 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.867 DEBUG:boto:Params: {} 2024-04-13T22:47:24.867 DEBUG:boto:Token: None 2024-04-13T22:47:24.867 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/ 2024-04-13T22:47:24.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T22:47:24.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T22:47:24.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9738efadd9524ae-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.870 DEBUG:boto:path=/vwnfns-65/obj44 2024-04-13T22:47:24.870 DEBUG:boto:auth_path=/vwnfns-65/obj44 2024-04-13T22:47:24.870 DEBUG:boto:Method: PUT 2024-04-13T22:47:24.870 DEBUG:boto:Path: /vwnfns-65/obj44 2024-04-13T22:47:24.870 DEBUG:boto:Data: 2024-04-13T22:47:24.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:24.870 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.870 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.870 DEBUG:boto:Params: {} 2024-04-13T22:47:24.871 DEBUG:boto:Token: None 2024-04-13T22:47:24.871 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/obj44 2024-04-13T22:47:24.871 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kIJNMiGDqhyBmovGRtKCI1ksLuk= 2024-04-13T22:47:24.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kIJNMiGDqhyBmovGRtKCI1ksLuk='} 2024-04-13T22:47:24.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eaf590b7940d7eaf-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.876 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:24.876 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:24.876 DEBUG:boto:Method: HEAD 2024-04-13T22:47:24.876 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:24.877 DEBUG:boto:Data: 2024-04-13T22:47:24.877 DEBUG:boto:Headers: {} 2024-04-13T22:47:24.877 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.877 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.877 DEBUG:boto:Params: {} 2024-04-13T22:47:24.877 DEBUG:boto:Token: None 2024-04-13T22:47:24.877 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/ 2024-04-13T22:47:24.877 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T22:47:24.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T22:47:24.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bab2683542e84ca8-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.880 DEBUG:boto:path=/vwnfns-65/obj55 2024-04-13T22:47:24.880 DEBUG:boto:auth_path=/vwnfns-65/obj55 2024-04-13T22:47:24.880 DEBUG:boto:Method: PUT 2024-04-13T22:47:24.880 DEBUG:boto:Path: /vwnfns-65/obj55 2024-04-13T22:47:24.880 DEBUG:boto:Data: 2024-04-13T22:47:24.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:24.880 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:24.880 DEBUG:boto:Port: 8000 2024-04-13T22:47:24.880 DEBUG:boto:Params: {} 2024-04-13T22:47:24.881 DEBUG:boto:Token: None 2024-04-13T22:47:24.881 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:24 GMT /vwnfns-65/obj55 2024-04-13T22:47:24.881 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R/rHSHQRvNElrNDiauC9U7pQLPc= 2024-04-13T22:47:24.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R/rHSHQRvNElrNDiauC9U7pQLPc='} 2024-04-13T22:47:24.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9e1d5c56baf3b28-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:24.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:24.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:24.927 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:47:24.978 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:24.979 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:25.021 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:47:25.021 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-65 2024-04-13T22:47:25.024 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 5 2024-04-13T22:47:25.063 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.065 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.065 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.065 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.066 DEBUG:boto:Data: 2024-04-13T22:47:25.066 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.066 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.066 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.066 DEBUG:boto:Params: {} 2024-04-13T22:47:25.066 DEBUG:boto:Token: None 2024-04-13T22:47:25.066 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.066 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc0e07ae0a0016f9-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.070 DEBUG:boto:path=/vwnfns-65/obj17 2024-04-13T22:47:25.070 DEBUG:boto:auth_path=/vwnfns-65/obj17 2024-04-13T22:47:25.070 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.070 DEBUG:boto:Path: /vwnfns-65/obj17 2024-04-13T22:47:25.070 DEBUG:boto:Data: 2024-04-13T22:47:25.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.070 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.070 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.070 DEBUG:boto:Params: {} 2024-04-13T22:47:25.071 DEBUG:boto:Token: None 2024-04-13T22:47:25.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj17 2024-04-13T22:47:25.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DavugBFTPROR8GC318g+baBerrU= 2024-04-13T22:47:25.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DavugBFTPROR8GC318g+baBerrU='} 2024-04-13T22:47:25.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6a20774d4aa2168-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.077 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.077 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.077 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.077 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.077 DEBUG:boto:Data: 2024-04-13T22:47:25.077 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.077 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.077 DEBUG:boto:Params: {} 2024-04-13T22:47:25.078 DEBUG:boto:Token: None 2024-04-13T22:47:25.078 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.078 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9ceb0a0da242ce8-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.080 DEBUG:boto:path=/vwnfns-65/obj34 2024-04-13T22:47:25.081 DEBUG:boto:auth_path=/vwnfns-65/obj34 2024-04-13T22:47:25.081 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.081 DEBUG:boto:Path: /vwnfns-65/obj34 2024-04-13T22:47:25.081 DEBUG:boto:Data: 2024-04-13T22:47:25.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.081 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.081 DEBUG:boto:Params: {} 2024-04-13T22:47:25.081 DEBUG:boto:Token: None 2024-04-13T22:47:25.082 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj34 2024-04-13T22:47:25.082 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:h2Op0B01V8lZsafuTyO0Y0UoP+E= 2024-04-13T22:47:25.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h2Op0B01V8lZsafuTyO0Y0UoP+E='} 2024-04-13T22:47:25.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c7afffe2c8d99ee-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.087 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.087 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.087 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.087 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.087 DEBUG:boto:Data: 2024-04-13T22:47:25.087 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.087 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.087 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.087 DEBUG:boto:Params: {} 2024-04-13T22:47:25.087 DEBUG:boto:Token: None 2024-04-13T22:47:25.088 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.088 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059972ed5c2ac6e5a-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.090 DEBUG:boto:path=/vwnfns-65/obj51 2024-04-13T22:47:25.090 DEBUG:boto:auth_path=/vwnfns-65/obj51 2024-04-13T22:47:25.090 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.091 DEBUG:boto:Path: /vwnfns-65/obj51 2024-04-13T22:47:25.091 DEBUG:boto:Data: 2024-04-13T22:47:25.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.091 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.091 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.091 DEBUG:boto:Params: {} 2024-04-13T22:47:25.091 DEBUG:boto:Token: None 2024-04-13T22:47:25.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj51 2024-04-13T22:47:25.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TuYZSa2AXtWD76x5K3H9fYPEZcw= 2024-04-13T22:47:25.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TuYZSa2AXtWD76x5K3H9fYPEZcw='} 2024-04-13T22:47:25.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088745bf37cb28665-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.097 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.097 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.097 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.097 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.097 DEBUG:boto:Data: 2024-04-13T22:47:25.097 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.097 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.098 DEBUG:boto:Params: {} 2024-04-13T22:47:25.098 DEBUG:boto:Token: None 2024-04-13T22:47:25.098 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.098 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b33f3159268f453-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.101 DEBUG:boto:path=/vwnfns-65/obj68 2024-04-13T22:47:25.101 DEBUG:boto:auth_path=/vwnfns-65/obj68 2024-04-13T22:47:25.101 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.101 DEBUG:boto:Path: /vwnfns-65/obj68 2024-04-13T22:47:25.101 DEBUG:boto:Data: 2024-04-13T22:47:25.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.101 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.102 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.102 DEBUG:boto:Params: {} 2024-04-13T22:47:25.102 DEBUG:boto:Token: None 2024-04-13T22:47:25.102 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj68 2024-04-13T22:47:25.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oxYHFMI65mqlbLKw5RQP5hN+NBU= 2024-04-13T22:47:25.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oxYHFMI65mqlbLKw5RQP5hN+NBU='} 2024-04-13T22:47:25.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048ee41ba00827f2b-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.147 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.147 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.147 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.147 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.147 DEBUG:boto:Data: 2024-04-13T22:47:25.147 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.148 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.148 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.148 DEBUG:boto:Params: {} 2024-04-13T22:47:25.148 DEBUG:boto:Token: None 2024-04-13T22:47:25.148 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.148 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f0e9a12b3120d10-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.151 DEBUG:boto:path=/vwnfns-65/obj85 2024-04-13T22:47:25.151 DEBUG:boto:auth_path=/vwnfns-65/obj85 2024-04-13T22:47:25.151 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.151 DEBUG:boto:Path: /vwnfns-65/obj85 2024-04-13T22:47:25.151 DEBUG:boto:Data: 2024-04-13T22:47:25.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.152 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.152 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.152 DEBUG:boto:Params: {} 2024-04-13T22:47:25.152 DEBUG:boto:Token: None 2024-04-13T22:47:25.152 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj85 2024-04-13T22:47:25.152 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uf9qd4B8ikbbYlABnXzGObE59/o= 2024-04-13T22:47:25.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uf9qd4B8ikbbYlABnXzGObE59/o='} 2024-04-13T22:47:25.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f4004a20ae2493e-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:25.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:25.157 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:47:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:25.252 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:47:25.252 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-65 2024-04-13T22:47:25.258 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 10 2024-04-13T22:47:25.304 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.304 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.304 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.304 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.304 DEBUG:boto:Data: 2024-04-13T22:47:25.305 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.305 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.305 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.305 DEBUG:boto:Params: {} 2024-04-13T22:47:25.305 DEBUG:boto:Token: None 2024-04-13T22:47:25.305 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000552f630f5f64eef4-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.309 DEBUG:boto:path=/vwnfns-65/obj19 2024-04-13T22:47:25.309 DEBUG:boto:auth_path=/vwnfns-65/obj19 2024-04-13T22:47:25.309 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.309 DEBUG:boto:Path: /vwnfns-65/obj19 2024-04-13T22:47:25.309 DEBUG:boto:Data: 2024-04-13T22:47:25.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.309 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.309 DEBUG:boto:Params: {} 2024-04-13T22:47:25.310 DEBUG:boto:Token: None 2024-04-13T22:47:25.310 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj19 2024-04-13T22:47:25.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Q2OWFHfr9fd8J4ekb8vkdP+LGto= 2024-04-13T22:47:25.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q2OWFHfr9fd8J4ekb8vkdP+LGto='} 2024-04-13T22:47:25.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036fe4754040155fc-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.355 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.355 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.355 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.355 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.355 DEBUG:boto:Data: 2024-04-13T22:47:25.355 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.355 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.355 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.356 DEBUG:boto:Params: {} 2024-04-13T22:47:25.356 DEBUG:boto:Token: None 2024-04-13T22:47:25.356 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.356 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6c0b0c06e2861ca-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.359 DEBUG:boto:path=/vwnfns-65/obj38 2024-04-13T22:47:25.359 DEBUG:boto:auth_path=/vwnfns-65/obj38 2024-04-13T22:47:25.359 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.359 DEBUG:boto:Path: /vwnfns-65/obj38 2024-04-13T22:47:25.359 DEBUG:boto:Data: 2024-04-13T22:47:25.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.359 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.360 DEBUG:boto:Params: {} 2024-04-13T22:47:25.360 DEBUG:boto:Token: None 2024-04-13T22:47:25.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj38 2024-04-13T22:47:25.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lHQ1sk3+7t4tqLDWRj/sB7aiIBc= 2024-04-13T22:47:25.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lHQ1sk3+7t4tqLDWRj/sB7aiIBc='} 2024-04-13T22:47:25.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce18a5762fce99ff-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.364 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.364 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.364 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.365 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.365 DEBUG:boto:Data: 2024-04-13T22:47:25.365 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.365 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.365 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.365 DEBUG:boto:Params: {} 2024-04-13T22:47:25.365 DEBUG:boto:Token: None 2024-04-13T22:47:25.365 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.366 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000805f033dba21343-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.368 DEBUG:boto:path=/vwnfns-65/obj57 2024-04-13T22:47:25.369 DEBUG:boto:auth_path=/vwnfns-65/obj57 2024-04-13T22:47:25.369 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.369 DEBUG:boto:Path: /vwnfns-65/obj57 2024-04-13T22:47:25.369 DEBUG:boto:Data: 2024-04-13T22:47:25.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.369 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.369 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.369 DEBUG:boto:Params: {} 2024-04-13T22:47:25.369 DEBUG:boto:Token: None 2024-04-13T22:47:25.370 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj57 2024-04-13T22:47:25.370 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vKBIKKmHRYpkjh0Nufwa7NI6H4w= 2024-04-13T22:47:25.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vKBIKKmHRYpkjh0Nufwa7NI6H4w='} 2024-04-13T22:47:25.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed56aeb7767a2d20-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.415 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.415 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.415 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.415 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.415 DEBUG:boto:Data: 2024-04-13T22:47:25.415 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.415 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.415 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.415 DEBUG:boto:Params: {} 2024-04-13T22:47:25.416 DEBUG:boto:Token: None 2024-04-13T22:47:25.416 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003aa97d25b009e1c-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.418 DEBUG:boto:path=/vwnfns-65/obj76 2024-04-13T22:47:25.419 DEBUG:boto:auth_path=/vwnfns-65/obj76 2024-04-13T22:47:25.419 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.419 DEBUG:boto:Path: /vwnfns-65/obj76 2024-04-13T22:47:25.419 DEBUG:boto:Data: 2024-04-13T22:47:25.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.419 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.419 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.419 DEBUG:boto:Params: {} 2024-04-13T22:47:25.419 DEBUG:boto:Token: None 2024-04-13T22:47:25.419 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj76 2024-04-13T22:47:25.420 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jCCofFF6MR5PfJRJp4tEFIGh9Jk= 2024-04-13T22:47:25.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jCCofFF6MR5PfJRJp4tEFIGh9Jk='} 2024-04-13T22:47:25.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec3da2cb5422f5c8-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.425 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.425 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.425 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.425 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.425 DEBUG:boto:Data: 2024-04-13T22:47:25.425 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.425 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.426 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.426 DEBUG:boto:Params: {} 2024-04-13T22:47:25.426 DEBUG:boto:Token: None 2024-04-13T22:47:25.426 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.426 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd746666740f705c-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.429 DEBUG:boto:path=/vwnfns-65/obj95 2024-04-13T22:47:25.429 DEBUG:boto:auth_path=/vwnfns-65/obj95 2024-04-13T22:47:25.429 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.429 DEBUG:boto:Path: /vwnfns-65/obj95 2024-04-13T22:47:25.429 DEBUG:boto:Data: 2024-04-13T22:47:25.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.429 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.430 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.430 DEBUG:boto:Params: {} 2024-04-13T22:47:25.430 DEBUG:boto:Token: None 2024-04-13T22:47:25.430 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj95 2024-04-13T22:47:25.430 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3JwaHSSQ2UeCroLpa2C9nGVwUf4= 2024-04-13T22:47:25.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3JwaHSSQ2UeCroLpa2C9nGVwUf4='} 2024-04-13T22:47:25.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000216a73e9a216a391-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:25.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:25.476 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:47:25.528 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:25.528 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:25.575 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:47:25.576 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-65 2024-04-13T22:47:25.583 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 15 2024-04-13T22:47:25.631 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.631 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.631 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.631 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.631 DEBUG:boto:Data: 2024-04-13T22:47:25.631 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.631 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.632 DEBUG:boto:Params: {} 2024-04-13T22:47:25.632 DEBUG:boto:Token: None 2024-04-13T22:47:25.632 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000d4e912b0d01959-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.635 DEBUG:boto:path=/vwnfns-65/obj23 2024-04-13T22:47:25.635 DEBUG:boto:auth_path=/vwnfns-65/obj23 2024-04-13T22:47:25.635 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.636 DEBUG:boto:Path: /vwnfns-65/obj23 2024-04-13T22:47:25.636 DEBUG:boto:Data: 2024-04-13T22:47:25.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.636 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.636 DEBUG:boto:Params: {} 2024-04-13T22:47:25.636 DEBUG:boto:Token: None 2024-04-13T22:47:25.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj23 2024-04-13T22:47:25.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bFWEL4yKsOuIlNXu5qe1YFqp8vE= 2024-04-13T22:47:25.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bFWEL4yKsOuIlNXu5qe1YFqp8vE='} 2024-04-13T22:47:25.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000118946a050d2df88-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.642 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.642 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.642 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.642 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.642 DEBUG:boto:Data: 2024-04-13T22:47:25.642 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.642 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.642 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.642 DEBUG:boto:Params: {} 2024-04-13T22:47:25.642 DEBUG:boto:Token: None 2024-04-13T22:47:25.643 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.643 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042427b0d2c308351-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.646 DEBUG:boto:path=/vwnfns-65/obj46 2024-04-13T22:47:25.646 DEBUG:boto:auth_path=/vwnfns-65/obj46 2024-04-13T22:47:25.646 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.646 DEBUG:boto:Path: /vwnfns-65/obj46 2024-04-13T22:47:25.646 DEBUG:boto:Data: 2024-04-13T22:47:25.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.646 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.646 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.646 DEBUG:boto:Params: {} 2024-04-13T22:47:25.647 DEBUG:boto:Token: None 2024-04-13T22:47:25.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj46 2024-04-13T22:47:25.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iOIWYrFs9czBx0PxfcBtvoVVk3A= 2024-04-13T22:47:25.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iOIWYrFs9czBx0PxfcBtvoVVk3A='} 2024-04-13T22:47:25.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e94ea39b2e775d40-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.652 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.652 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.652 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.652 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.652 DEBUG:boto:Data: 2024-04-13T22:47:25.652 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.652 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.653 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.653 DEBUG:boto:Params: {} 2024-04-13T22:47:25.653 DEBUG:boto:Token: None 2024-04-13T22:47:25.653 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.653 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfdf274fc7a2b2d6-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.656 DEBUG:boto:path=/vwnfns-65/obj69 2024-04-13T22:47:25.656 DEBUG:boto:auth_path=/vwnfns-65/obj69 2024-04-13T22:47:25.656 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.656 DEBUG:boto:Path: /vwnfns-65/obj69 2024-04-13T22:47:25.656 DEBUG:boto:Data: 2024-04-13T22:47:25.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.656 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.657 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.657 DEBUG:boto:Params: {} 2024-04-13T22:47:25.657 DEBUG:boto:Token: None 2024-04-13T22:47:25.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj69 2024-04-13T22:47:25.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ivhoxUrZj00YkukeKx5w/glLW6g= 2024-04-13T22:47:25.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ivhoxUrZj00YkukeKx5w/glLW6g='} 2024-04-13T22:47:25.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1a04a009e370d27-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.703 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.703 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.703 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.703 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.703 DEBUG:boto:Data: 2024-04-13T22:47:25.703 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.703 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.704 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.704 DEBUG:boto:Params: {} 2024-04-13T22:47:25.704 DEBUG:boto:Token: None 2024-04-13T22:47:25.704 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.704 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012740a2a36bf045d-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.707 DEBUG:boto:path=/vwnfns-65/obj92 2024-04-13T22:47:25.707 DEBUG:boto:auth_path=/vwnfns-65/obj92 2024-04-13T22:47:25.707 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.707 DEBUG:boto:Path: /vwnfns-65/obj92 2024-04-13T22:47:25.707 DEBUG:boto:Data: 2024-04-13T22:47:25.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.708 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.708 DEBUG:boto:Params: {} 2024-04-13T22:47:25.708 DEBUG:boto:Token: None 2024-04-13T22:47:25.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj92 2024-04-13T22:47:25.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Aav2sj9R73hOmGm5Av/7/4RyHZ8= 2024-04-13T22:47:25.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Aav2sj9R73hOmGm5Av/7/4RyHZ8='} 2024-04-13T22:47:25.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b68d0525ee68b76c-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.713 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.713 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.713 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.714 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.714 DEBUG:boto:Data: 2024-04-13T22:47:25.714 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.714 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.714 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.714 DEBUG:boto:Params: {} 2024-04-13T22:47:25.714 DEBUG:boto:Token: None 2024-04-13T22:47:25.714 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.714 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.717 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ac187c1248df93-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.717 DEBUG:boto:path=/vwnfns-65/obj115 2024-04-13T22:47:25.718 DEBUG:boto:auth_path=/vwnfns-65/obj115 2024-04-13T22:47:25.718 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.718 DEBUG:boto:Path: /vwnfns-65/obj115 2024-04-13T22:47:25.718 DEBUG:boto:Data: 2024-04-13T22:47:25.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.718 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.718 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.718 DEBUG:boto:Params: {} 2024-04-13T22:47:25.718 DEBUG:boto:Token: None 2024-04-13T22:47:25.719 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj115 2024-04-13T22:47:25.719 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tq3FSwNYMm1yIR17LmY6Gf81uMk= 2024-04-13T22:47:25.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tq3FSwNYMm1yIR17LmY6Gf81uMk='} 2024-04-13T22:47:25.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000973d31c0882849d1-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:25.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:25.763 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:47:25.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:25.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:25.860 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:47:25.861 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-65 2024-04-13T22:47:25.869 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 20 2024-04-13T22:47:25.920 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.920 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.920 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.921 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.921 DEBUG:boto:Data: 2024-04-13T22:47:25.921 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.921 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.921 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.922 DEBUG:boto:Params: {} 2024-04-13T22:47:25.922 DEBUG:boto:Token: None 2024-04-13T22:47:25.923 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa3e994b3e8c6c7e-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.927 DEBUG:boto:path=/vwnfns-65/obj29 2024-04-13T22:47:25.928 DEBUG:boto:auth_path=/vwnfns-65/obj29 2024-04-13T22:47:25.928 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.928 DEBUG:boto:Path: /vwnfns-65/obj29 2024-04-13T22:47:25.928 DEBUG:boto:Data: 2024-04-13T22:47:25.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.929 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.929 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.929 DEBUG:boto:Params: {} 2024-04-13T22:47:25.929 DEBUG:boto:Token: None 2024-04-13T22:47:25.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj29 2024-04-13T22:47:25.930 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FJBRLUqA7AuO3KvH6No+g0suDhc= 2024-04-13T22:47:25.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FJBRLUqA7AuO3KvH6No+g0suDhc='} 2024-04-13T22:47:25.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000647176f118ab63ef-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.936 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.936 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.936 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.936 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.936 DEBUG:boto:Data: 2024-04-13T22:47:25.937 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.937 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.937 DEBUG:boto:Params: {} 2024-04-13T22:47:25.937 DEBUG:boto:Token: None 2024-04-13T22:47:25.938 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000612a702ecdade4d9-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.941 DEBUG:boto:path=/vwnfns-65/obj58 2024-04-13T22:47:25.941 DEBUG:boto:auth_path=/vwnfns-65/obj58 2024-04-13T22:47:25.941 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.941 DEBUG:boto:Path: /vwnfns-65/obj58 2024-04-13T22:47:25.942 DEBUG:boto:Data: 2024-04-13T22:47:25.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.942 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.942 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.942 DEBUG:boto:Params: {} 2024-04-13T22:47:25.942 DEBUG:boto:Token: None 2024-04-13T22:47:25.942 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj58 2024-04-13T22:47:25.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EUwyBq0Ff2lo+UfV59rgksY7yjI= 2024-04-13T22:47:25.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EUwyBq0Ff2lo+UfV59rgksY7yjI='} 2024-04-13T22:47:25.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfc1261e045a1652-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.948 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.948 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.948 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.948 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.948 DEBUG:boto:Data: 2024-04-13T22:47:25.948 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.949 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.949 DEBUG:boto:Params: {} 2024-04-13T22:47:25.949 DEBUG:boto:Token: None 2024-04-13T22:47:25.949 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/ 2024-04-13T22:47:25.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T22:47:25.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T22:47:25.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae089809344b4d4d-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.952 DEBUG:boto:path=/vwnfns-65/obj87 2024-04-13T22:47:25.952 DEBUG:boto:auth_path=/vwnfns-65/obj87 2024-04-13T22:47:25.952 DEBUG:boto:Method: PUT 2024-04-13T22:47:25.952 DEBUG:boto:Path: /vwnfns-65/obj87 2024-04-13T22:47:25.952 DEBUG:boto:Data: 2024-04-13T22:47:25.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:25.952 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.953 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.953 DEBUG:boto:Params: {} 2024-04-13T22:47:25.953 DEBUG:boto:Token: None 2024-04-13T22:47:25.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:25 GMT /vwnfns-65/obj87 2024-04-13T22:47:25.953 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+HQfG/0bepLQXjABp5czaDbdVQw= 2024-04-13T22:47:25.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+HQfG/0bepLQXjABp5czaDbdVQw='} 2024-04-13T22:47:25.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bea773cbccabadb8-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:25.999 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:25.999 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:25.999 DEBUG:boto:Method: HEAD 2024-04-13T22:47:25.999 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:25.999 DEBUG:boto:Data: 2024-04-13T22:47:25.999 DEBUG:boto:Headers: {} 2024-04-13T22:47:25.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:25.999 DEBUG:boto:Port: 8000 2024-04-13T22:47:25.999 DEBUG:boto:Params: {} 2024-04-13T22:47:26.000 DEBUG:boto:Token: None 2024-04-13T22:47:26.000 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113f1c19479f7f36-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.003 DEBUG:boto:path=/vwnfns-65/obj116 2024-04-13T22:47:26.004 DEBUG:boto:auth_path=/vwnfns-65/obj116 2024-04-13T22:47:26.004 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.004 DEBUG:boto:Path: /vwnfns-65/obj116 2024-04-13T22:47:26.004 DEBUG:boto:Data: 2024-04-13T22:47:26.004 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.005 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.005 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.005 DEBUG:boto:Params: {} 2024-04-13T22:47:26.005 DEBUG:boto:Token: None 2024-04-13T22:47:26.005 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj116 2024-04-13T22:47:26.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r+jpvl2S4JyfsB9vDMBi1ZjcDbI= 2024-04-13T22:47:26.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r+jpvl2S4JyfsB9vDMBi1ZjcDbI='} 2024-04-13T22:47:26.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db3ad9f37c8e0c3f-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.011 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.011 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.011 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.012 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.012 DEBUG:boto:Data: 2024-04-13T22:47:26.012 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.012 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.012 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.013 DEBUG:boto:Params: {} 2024-04-13T22:47:26.013 DEBUG:boto:Token: None 2024-04-13T22:47:26.013 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.013 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e620d1cf50f08ca-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.016 DEBUG:boto:path=/vwnfns-65/obj145 2024-04-13T22:47:26.016 DEBUG:boto:auth_path=/vwnfns-65/obj145 2024-04-13T22:47:26.017 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.017 DEBUG:boto:Path: /vwnfns-65/obj145 2024-04-13T22:47:26.018 DEBUG:boto:Data: 2024-04-13T22:47:26.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.018 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.018 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.019 DEBUG:boto:Params: {} 2024-04-13T22:47:26.019 DEBUG:boto:Token: None 2024-04-13T22:47:26.019 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj145 2024-04-13T22:47:26.019 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+H+7ilSz6qoJt4eEmhdkHS4ppaw= 2024-04-13T22:47:26.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+H+7ilSz6qoJt4eEmhdkHS4ppaw='} 2024-04-13T22:47:26.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000546324f22091e11b-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:26.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:26.063 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:47:26.115 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:26.115 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:26.163 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:47:26.164 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-65 2024-04-13T22:47:26.173 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 25 2024-04-13T22:47:26.233 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.233 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.234 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.234 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.234 DEBUG:boto:Data: 2024-04-13T22:47:26.234 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.234 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.234 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.234 DEBUG:boto:Params: {} 2024-04-13T22:47:26.234 DEBUG:boto:Token: None 2024-04-13T22:47:26.234 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.235 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d75461d86cde47-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.239 DEBUG:boto:path=/vwnfns-65/obj31 2024-04-13T22:47:26.239 DEBUG:boto:auth_path=/vwnfns-65/obj31 2024-04-13T22:47:26.239 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.239 DEBUG:boto:Path: /vwnfns-65/obj31 2024-04-13T22:47:26.239 DEBUG:boto:Data: 2024-04-13T22:47:26.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.239 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.239 DEBUG:boto:Params: {} 2024-04-13T22:47:26.239 DEBUG:boto:Token: None 2024-04-13T22:47:26.240 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj31 2024-04-13T22:47:26.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hHx3ISGJgqxAtmtPdQWfxCk5YJ4= 2024-04-13T22:47:26.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHx3ISGJgqxAtmtPdQWfxCk5YJ4='} 2024-04-13T22:47:26.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077f4efa58370b9a0-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.245 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.245 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.245 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.245 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.245 DEBUG:boto:Data: 2024-04-13T22:47:26.245 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.245 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.246 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.246 DEBUG:boto:Params: {} 2024-04-13T22:47:26.246 DEBUG:boto:Token: None 2024-04-13T22:47:26.246 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.246 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdfb7c439b4b87b8-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.249 DEBUG:boto:path=/vwnfns-65/obj62 2024-04-13T22:47:26.249 DEBUG:boto:auth_path=/vwnfns-65/obj62 2024-04-13T22:47:26.250 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.250 DEBUG:boto:Path: /vwnfns-65/obj62 2024-04-13T22:47:26.250 DEBUG:boto:Data: 2024-04-13T22:47:26.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.250 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.250 DEBUG:boto:Params: {} 2024-04-13T22:47:26.250 DEBUG:boto:Token: None 2024-04-13T22:47:26.250 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj62 2024-04-13T22:47:26.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:33sU5DtQPwATrlFPpSf7P1s8y+E= 2024-04-13T22:47:26.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:33sU5DtQPwATrlFPpSf7P1s8y+E='} 2024-04-13T22:47:26.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000556f7ae6be1f1a01-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.256 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.256 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.256 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.256 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.256 DEBUG:boto:Data: 2024-04-13T22:47:26.256 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.256 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.256 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.256 DEBUG:boto:Params: {} 2024-04-13T22:47:26.256 DEBUG:boto:Token: None 2024-04-13T22:47:26.257 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.257 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073d8988252a89c4b-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.261 DEBUG:boto:path=/vwnfns-65/obj93 2024-04-13T22:47:26.261 DEBUG:boto:auth_path=/vwnfns-65/obj93 2024-04-13T22:47:26.261 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.261 DEBUG:boto:Path: /vwnfns-65/obj93 2024-04-13T22:47:26.261 DEBUG:boto:Data: 2024-04-13T22:47:26.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.261 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.261 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.261 DEBUG:boto:Params: {} 2024-04-13T22:47:26.261 DEBUG:boto:Token: None 2024-04-13T22:47:26.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj93 2024-04-13T22:47:26.262 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8ZX80yhh1PwcQlMvJqLeUBBWxks= 2024-04-13T22:47:26.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8ZX80yhh1PwcQlMvJqLeUBBWxks='} 2024-04-13T22:47:26.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000698c1a07b3d3112c-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.307 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.307 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.307 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.307 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.307 DEBUG:boto:Data: 2024-04-13T22:47:26.307 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.308 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.308 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.308 DEBUG:boto:Params: {} 2024-04-13T22:47:26.308 DEBUG:boto:Token: None 2024-04-13T22:47:26.308 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb794c16c3029123-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.312 DEBUG:boto:path=/vwnfns-65/obj124 2024-04-13T22:47:26.312 DEBUG:boto:auth_path=/vwnfns-65/obj124 2024-04-13T22:47:26.312 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.312 DEBUG:boto:Path: /vwnfns-65/obj124 2024-04-13T22:47:26.312 DEBUG:boto:Data: 2024-04-13T22:47:26.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.313 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.313 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.313 DEBUG:boto:Params: {} 2024-04-13T22:47:26.313 DEBUG:boto:Token: None 2024-04-13T22:47:26.313 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj124 2024-04-13T22:47:26.313 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:T6L64Kvql+Kiuald6R3voPar1RY= 2024-04-13T22:47:26.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T6L64Kvql+Kiuald6R3voPar1RY='} 2024-04-13T22:47:26.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c2251cc31c7a91f-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.318 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.318 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.319 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.319 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.319 DEBUG:boto:Data: 2024-04-13T22:47:26.319 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.319 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.319 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.319 DEBUG:boto:Params: {} 2024-04-13T22:47:26.319 DEBUG:boto:Token: None 2024-04-13T22:47:26.319 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.319 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d53d1261e41ead5-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.323 DEBUG:boto:path=/vwnfns-65/obj155 2024-04-13T22:47:26.323 DEBUG:boto:auth_path=/vwnfns-65/obj155 2024-04-13T22:47:26.323 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.323 DEBUG:boto:Path: /vwnfns-65/obj155 2024-04-13T22:47:26.323 DEBUG:boto:Data: 2024-04-13T22:47:26.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.323 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.323 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.324 DEBUG:boto:Params: {} 2024-04-13T22:47:26.324 DEBUG:boto:Token: None 2024-04-13T22:47:26.324 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj155 2024-04-13T22:47:26.324 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UvE+HrLy+zaWPzUE90E3vbU/ukM= 2024-04-13T22:47:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UvE+HrLy+zaWPzUE90E3vbU/ukM='} 2024-04-13T22:47:26.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000faef6a208c87226c-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:26.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:47:26.367 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:47:26.418 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:26.419 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:26.468 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:47:26.469 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-65 2024-04-13T22:47:26.480 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 30 2024-04-13T22:47:26.542 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.542 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.542 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.543 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.543 DEBUG:boto:Data: 2024-04-13T22:47:26.543 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.543 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.543 DEBUG:boto:Params: {} 2024-04-13T22:47:26.543 DEBUG:boto:Token: None 2024-04-13T22:47:26.543 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.543 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000671f02ac2d567bbc-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.547 DEBUG:boto:path=/vwnfns-65/obj37 2024-04-13T22:47:26.547 DEBUG:boto:auth_path=/vwnfns-65/obj37 2024-04-13T22:47:26.548 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.548 DEBUG:boto:Path: /vwnfns-65/obj37 2024-04-13T22:47:26.548 DEBUG:boto:Data: 2024-04-13T22:47:26.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.548 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.548 DEBUG:boto:Params: {} 2024-04-13T22:47:26.548 DEBUG:boto:Token: None 2024-04-13T22:47:26.548 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj37 2024-04-13T22:47:26.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:LbOpIUExzC+RctP0acdq3br5xHE= 2024-04-13T22:47:26.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LbOpIUExzC+RctP0acdq3br5xHE='} 2024-04-13T22:47:26.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e969143e3dc2aae-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.554 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.554 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.554 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.554 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.554 DEBUG:boto:Data: 2024-04-13T22:47:26.554 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.555 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.555 DEBUG:boto:Params: {} 2024-04-13T22:47:26.555 DEBUG:boto:Token: None 2024-04-13T22:47:26.555 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b948692e1cc3fe2f-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.559 DEBUG:boto:path=/vwnfns-65/obj74 2024-04-13T22:47:26.559 DEBUG:boto:auth_path=/vwnfns-65/obj74 2024-04-13T22:47:26.559 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.559 DEBUG:boto:Path: /vwnfns-65/obj74 2024-04-13T22:47:26.559 DEBUG:boto:Data: 2024-04-13T22:47:26.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.559 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.560 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.560 DEBUG:boto:Params: {} 2024-04-13T22:47:26.560 DEBUG:boto:Token: None 2024-04-13T22:47:26.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj74 2024-04-13T22:47:26.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2gjk/pQaDOF+wAawlYDi9FUpqFU= 2024-04-13T22:47:26.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2gjk/pQaDOF+wAawlYDi9FUpqFU='} 2024-04-13T22:47:26.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d98b528f2d88bb9d-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.603 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.603 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.603 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.603 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.603 DEBUG:boto:Data: 2024-04-13T22:47:26.603 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.603 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.604 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.604 DEBUG:boto:Params: {} 2024-04-13T22:47:26.604 DEBUG:boto:Token: None 2024-04-13T22:47:26.604 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.604 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dadba39da83a96a7-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.607 DEBUG:boto:path=/vwnfns-65/obj111 2024-04-13T22:47:26.607 DEBUG:boto:auth_path=/vwnfns-65/obj111 2024-04-13T22:47:26.608 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.608 DEBUG:boto:Path: /vwnfns-65/obj111 2024-04-13T22:47:26.608 DEBUG:boto:Data: 2024-04-13T22:47:26.608 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.608 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.608 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.608 DEBUG:boto:Params: {} 2024-04-13T22:47:26.608 DEBUG:boto:Token: None 2024-04-13T22:47:26.608 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj111 2024-04-13T22:47:26.609 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qeePr5mh0RJV9RXPk+EYR8ugpvM= 2024-04-13T22:47:26.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qeePr5mh0RJV9RXPk+EYR8ugpvM='} 2024-04-13T22:47:26.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006254f77967342114-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.614 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.614 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.614 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.614 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.614 DEBUG:boto:Data: 2024-04-13T22:47:26.614 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.614 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.614 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.614 DEBUG:boto:Params: {} 2024-04-13T22:47:26.615 DEBUG:boto:Token: None 2024-04-13T22:47:26.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.615 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da280f3a0f628506-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.618 DEBUG:boto:path=/vwnfns-65/obj148 2024-04-13T22:47:26.618 DEBUG:boto:auth_path=/vwnfns-65/obj148 2024-04-13T22:47:26.618 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.619 DEBUG:boto:Path: /vwnfns-65/obj148 2024-04-13T22:47:26.619 DEBUG:boto:Data: 2024-04-13T22:47:26.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.619 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.619 DEBUG:boto:Params: {} 2024-04-13T22:47:26.619 DEBUG:boto:Token: None 2024-04-13T22:47:26.619 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj148 2024-04-13T22:47:26.620 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YUiBC/MioaF+OEvjmaJWaq2tlM4= 2024-04-13T22:47:26.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YUiBC/MioaF+OEvjmaJWaq2tlM4='} 2024-04-13T22:47:26.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062ea926d20bdc836-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.663 DEBUG:boto:path=/vwnfns-65/ 2024-04-13T22:47:26.663 DEBUG:boto:auth_path=/vwnfns-65/ 2024-04-13T22:47:26.663 DEBUG:boto:Method: HEAD 2024-04-13T22:47:26.663 DEBUG:boto:Path: /vwnfns-65/ 2024-04-13T22:47:26.663 DEBUG:boto:Data: 2024-04-13T22:47:26.663 DEBUG:boto:Headers: {} 2024-04-13T22:47:26.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.663 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.664 DEBUG:boto:Params: {} 2024-04-13T22:47:26.664 DEBUG:boto:Token: None 2024-04-13T22:47:26.664 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/ 2024-04-13T22:47:26.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T22:47:26.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T22:47:26.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f96241915df83a7f-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.669 DEBUG:boto:path=/vwnfns-65/obj185 2024-04-13T22:47:26.669 DEBUG:boto:auth_path=/vwnfns-65/obj185 2024-04-13T22:47:26.669 DEBUG:boto:Method: PUT 2024-04-13T22:47:26.669 DEBUG:boto:Path: /vwnfns-65/obj185 2024-04-13T22:47:26.669 DEBUG:boto:Data: 2024-04-13T22:47:26.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:47:26.670 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:47:26.670 DEBUG:boto:Port: 8000 2024-04-13T22:47:26.670 DEBUG:boto:Params: {} 2024-04-13T22:47:26.670 DEBUG:boto:Token: None 2024-04-13T22:47:26.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:47:26 GMT /vwnfns-65/obj185 2024-04-13T22:47:26.670 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:r3xWywo6I+OLAdU1iF4ts4HJfK0= 2024-04-13T22:47:26.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r3xWywo6I+OLAdU1iF4ts4HJfK0='} 2024-04-13T22:47:26.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005abd1b57ffa9531b-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:47:26.676 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-65 2024-04-13T22:47:26.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:47:26.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:47:26.676 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:47:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:26.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:26.768+0000 7f27da411780 1 waiting to reach incremental sync.. 2024-04-13T22:47:31.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:31.769+0000 7f27da411780 1 waiting to reach incremental sync.. 2024-04-13T22:47:36.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:36.773+0000 7f27da411780 1 bucket sync caught up with source: 2024-04-13T22:47:36.775 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , 00000000002.121.6, , , , , , 00000000002.186.6, , , , , 00000000002.170.6, , , , , , , , , , , , , , , 00000000001.101.6, , , , 00000000001.99.6] 2024-04-13T22:47:36.776 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , 00000000002.121.6, , , , , , 00000000002.186.6, , , , , 00000000002.170.6, , , , , , , , , , , , , , , 00000000001.101.6, , , , 00000000001.99.6] 2024-04-13T22:47:36.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:36.773+0000 7f27da411780 0 bucket checkpoint complete 2024-04-13T22:47:36.793 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-65 2024-04-13T22:47:36.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:47:36.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:47:36.794 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:47:36.857 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:47:36.857 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:47:36.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:36.903+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:47:41.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:41.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:47:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:46.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:47:51.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:51.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:47:56.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:47:56.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:01.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:01.905+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:06.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:06.906+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:11.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:11.907+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:16.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:16.908+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:21.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:21.909+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:26.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:26.909+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:31.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:31.909+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:36.910+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:41.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:41.910+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:46.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:46.911+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:51.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:51.913+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:48:56.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:48:56.914+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:01.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:01.915+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:06.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:06.914+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:11.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:11.915+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:16.915+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:21.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:21.916+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:26.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:26.917+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:31.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:31.918+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:36.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:36.919+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:41.922 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:41.920+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:46.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:46.920+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:51.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:51.920+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:49:56.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:49:56.921+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:01.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:01.922+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:06.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:06.923+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:11.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:11.924+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:16.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:16.924+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:21.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:21.925+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:26.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:26.925+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:31.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:31.925+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:36.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:36.926+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:41.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:41.927+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:46.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:46.929+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:51.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:51.929+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:50:56.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:50:56.930+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:01.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:01.929+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:06.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:06.930+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:11.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:11.931+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:16.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:16.931+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:21.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:21.932+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:26.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:26.934+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:31.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:31.934+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:36.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:36.935+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:41.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:41.935+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:46.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:46.935+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:51.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:51.936+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:51:56.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:51:56.936+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:01.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:01.938+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:06.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:06.939+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:11.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:11.939+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:16.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:16.940+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:21.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:21.940+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:26.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:26.940+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T22:52:31.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:31.941+0000 7f4300684780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-13T22:52:31.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:31.941+0000 7f4300684780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T22:52:31.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:31.941+0000 7f4300684780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T22:52:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-13T22:52:31.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-04-13T22:52:31.964 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:31.965 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:31.965 DEBUG:boto:Method: PUT 2024-04-13T22:52:31.965 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:31.965 DEBUG:boto:Data: 2024-04-13T22:52:31.965 DEBUG:boto:Headers: {} 2024-04-13T22:52:31.965 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:31.965 DEBUG:boto:Port: 8000 2024-04-13T22:52:31.965 DEBUG:boto:Params: {} 2024-04-13T22:52:31.966 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:52:31.966 DEBUG:boto:Token: None 2024-04-13T22:52:31.966 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:52:31 GMT /vwnfns-66/ 2024-04-13T22:52:31.966 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X+J7rKzo3/TAbfoI8dEw9U2QJHo= 2024-04-13T22:52:31.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X+J7rKzo3/TAbfoI8dEw9U2QJHo=', 'Content-Length': '0'} 2024-04-13T22:52:31.986 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000063dc36c78db19ad7-00661b0caf-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:31.987 DEBUG:rgw_multi.tests:created bucket=vwnfns-66 2024-04-13T22:52:31.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:52:31.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:52:31.987 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:52:32.040 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:52:32.040 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:52:32.076 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048751.983401_980.1", 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:52:31.983401Z" 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:52:32.092 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:52:32.093 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713048751.983401_980.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:52:32.093 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:52:32.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:52:32.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:52:32.094 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:52:32.143 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:52:32.143 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:52:32.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:32.171+0000 7f4bf9b9e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:52:32.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:32.172+0000 7f4bf9b9e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:52:32.188 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:52:32.188 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:52:32.188 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:52:32.188 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048751.983401_980.1", 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:52:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:52:31.983401Z", 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:52:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:52:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:52:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:52:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:52:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:52:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:52:32Z" 2024-04-13T22:52:32.194 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:52:32.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048751.983401_980.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:52:31.983401Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:52:32Z" } 2024-04-13T22:52:32.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048751.983401_980.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:52:31.983401Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:52:32.195 DEBUG:rgw_multi.tests:log_status={0: '1_1713048751.983401_980.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:52:32.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048751.983401_980.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:52:32.195 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:52:32.195 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:32.195 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:32.195 DEBUG:boto:Method: HEAD 2024-04-13T22:52:32.195 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:32.196 DEBUG:boto:Data: 2024-04-13T22:52:32.196 DEBUG:boto:Headers: {} 2024-04-13T22:52:32.196 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.196 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.196 DEBUG:boto:Params: {} 2024-04-13T22:52:32.196 DEBUG:boto:Token: None 2024-04-13T22:52:32.196 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/ 2024-04-13T22:52:32.196 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T22:52:32.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T22:52:32.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be163686d055a831-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.199 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:32.199 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:32.199 DEBUG:boto:Method: PUT 2024-04-13T22:52:32.199 DEBUG:boto:Path: /vwnfns-66/obj11 2024-04-13T22:52:32.199 DEBUG:boto:Data: 2024-04-13T22:52:32.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:52:32.199 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.199 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.199 DEBUG:boto:Params: {} 2024-04-13T22:52:32.200 DEBUG:boto:Token: None 2024-04-13T22:52:32.200 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/obj11 2024-04-13T22:52:32.200 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:thvZOGq1TEREQsWO222jEKaslOw= 2024-04-13T22:52:32.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:thvZOGq1TEREQsWO222jEKaslOw='} 2024-04-13T22:52:32.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085423f256b78c592-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.206 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:32.206 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:32.206 DEBUG:boto:Method: HEAD 2024-04-13T22:52:32.206 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:32.206 DEBUG:boto:Data: 2024-04-13T22:52:32.206 DEBUG:boto:Headers: {} 2024-04-13T22:52:32.206 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.206 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.206 DEBUG:boto:Params: {} 2024-04-13T22:52:32.207 DEBUG:boto:Token: None 2024-04-13T22:52:32.207 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/ 2024-04-13T22:52:32.207 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T22:52:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T22:52:32.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaace3364e421275-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.209 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:32.209 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:32.209 DEBUG:boto:Method: PUT 2024-04-13T22:52:32.210 DEBUG:boto:Path: /vwnfns-66/obj22 2024-04-13T22:52:32.210 DEBUG:boto:Data: 2024-04-13T22:52:32.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:52:32.210 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.210 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.210 DEBUG:boto:Params: {} 2024-04-13T22:52:32.210 DEBUG:boto:Token: None 2024-04-13T22:52:32.210 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/obj22 2024-04-13T22:52:32.210 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MSMZqeXmFmcU+kgFjdAvKeqf1Xw= 2024-04-13T22:52:32.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MSMZqeXmFmcU+kgFjdAvKeqf1Xw='} 2024-04-13T22:52:32.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a10d95f281c1716-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.214 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:32.214 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:32.214 DEBUG:boto:Method: HEAD 2024-04-13T22:52:32.214 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:32.214 DEBUG:boto:Data: 2024-04-13T22:52:32.214 DEBUG:boto:Headers: {} 2024-04-13T22:52:32.214 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.215 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.215 DEBUG:boto:Params: {} 2024-04-13T22:52:32.215 DEBUG:boto:Token: None 2024-04-13T22:52:32.215 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/ 2024-04-13T22:52:32.215 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T22:52:32.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T22:52:32.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000939551de1210a718-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.217 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:32.217 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:32.217 DEBUG:boto:Method: PUT 2024-04-13T22:52:32.218 DEBUG:boto:Path: /vwnfns-66/obj33 2024-04-13T22:52:32.218 DEBUG:boto:Data: 2024-04-13T22:52:32.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:52:32.218 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.218 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.218 DEBUG:boto:Params: {} 2024-04-13T22:52:32.218 DEBUG:boto:Token: None 2024-04-13T22:52:32.218 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/obj33 2024-04-13T22:52:32.218 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YJ3O+p7xy36T+yc8KELunvLYwQw= 2024-04-13T22:52:32.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YJ3O+p7xy36T+yc8KELunvLYwQw='} 2024-04-13T22:52:32.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025f2a4e7a286cba3-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.223 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:32.223 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:32.223 DEBUG:boto:Method: HEAD 2024-04-13T22:52:32.223 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:32.223 DEBUG:boto:Data: 2024-04-13T22:52:32.224 DEBUG:boto:Headers: {} 2024-04-13T22:52:32.224 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.224 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.224 DEBUG:boto:Params: {} 2024-04-13T22:52:32.224 DEBUG:boto:Token: None 2024-04-13T22:52:32.224 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/ 2024-04-13T22:52:32.224 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T22:52:32.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T22:52:32.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000775024217fa958cf-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.226 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:32.226 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:32.226 DEBUG:boto:Method: PUT 2024-04-13T22:52:32.227 DEBUG:boto:Path: /vwnfns-66/obj44 2024-04-13T22:52:32.227 DEBUG:boto:Data: 2024-04-13T22:52:32.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:52:32.227 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.227 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.227 DEBUG:boto:Params: {} 2024-04-13T22:52:32.227 DEBUG:boto:Token: None 2024-04-13T22:52:32.227 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/obj44 2024-04-13T22:52:32.227 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Jd1bEbnWYu4Wf5NGvShgmaGh6iY= 2024-04-13T22:52:32.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jd1bEbnWYu4Wf5NGvShgmaGh6iY='} 2024-04-13T22:52:32.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c92c0096ab088dd7-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.271 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:32.271 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:32.271 DEBUG:boto:Method: HEAD 2024-04-13T22:52:32.271 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:32.271 DEBUG:boto:Data: 2024-04-13T22:52:32.271 DEBUG:boto:Headers: {} 2024-04-13T22:52:32.271 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.271 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.271 DEBUG:boto:Params: {} 2024-04-13T22:52:32.271 DEBUG:boto:Token: None 2024-04-13T22:52:32.271 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/ 2024-04-13T22:52:32.272 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T22:52:32.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T22:52:32.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003289e36dab3d7d49-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.274 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:32.274 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:32.275 DEBUG:boto:Method: PUT 2024-04-13T22:52:32.275 DEBUG:boto:Path: /vwnfns-66/obj55 2024-04-13T22:52:32.275 DEBUG:boto:Data: 2024-04-13T22:52:32.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:52:32.275 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:32.275 DEBUG:boto:Port: 8000 2024-04-13T22:52:32.275 DEBUG:boto:Params: {} 2024-04-13T22:52:32.275 DEBUG:boto:Token: None 2024-04-13T22:52:32.275 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:52:32 GMT /vwnfns-66/obj55 2024-04-13T22:52:32.276 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:K5zGLNLaDDJvYRDN1PzfNABpHjU= 2024-04-13T22:52:32.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K5zGLNLaDDJvYRDN1PzfNABpHjU='} 2024-04-13T22:52:32.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ea6d4251ac8925f-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:32.279 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-66 2024-04-13T22:52:32.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:52:32.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:52:32.280 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:52:32.334 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:52:32.334 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:52:32.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:32.366+0000 7f7d296c9780 1 waiting to reach incremental sync.. 2024-04-13T22:52:37.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:37.368+0000 7f7d296c9780 1 bucket sync caught up with source: 2024-04-13T22:52:37.371 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.190.6, 00000000002.113.6, 00000000002.81.6] 2024-04-13T22:52:37.371 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.190.6, 00000000002.113.6, 00000000002.81.6] 2024-04-13T22:52:37.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:37.368+0000 7f7d296c9780 0 bucket checkpoint complete 2024-04-13T22:52:37.389 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=vwnfns-66 2024-04-13T22:52:37.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:52:37.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:52:37.389 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-66 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:52:37.450 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:52:37.450 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:52:37.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:37.486+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T22:52:42.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:42.488+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T22:52:47.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:47.488+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T22:52:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:52.489+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T22:52:57.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:57.490+0000 7f821f887780 1 bucket sync caught up with source: 2024-04-13T22:52:57.493 INFO:teuthology.orchestra.run.smithi132.stderr: local status: [, , , , , , , , 00000000001.784.6, 00000000002.530.6, 00000000002.1162.6] 2024-04-13T22:52:57.493 INFO:teuthology.orchestra.run.smithi132.stderr: remote markers: [, , , , , , , , 00000000001.784.6, 00000000002.530.6, 00000000002.1162.6] 2024-04-13T22:52:57.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:52:57.490+0000 7f821f887780 0 bucket checkpoint complete 2024-04-13T22:52:57.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vwnfns-66 zones={a2, a1} 2024-04-13T22:52:57.512 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.512 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.512 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.512 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:57.512 DEBUG:boto:Data: 2024-04-13T22:52:57.512 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.512 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.512 DEBUG:boto:Params: {} 2024-04-13T22:52:57.513 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:52:57.513 DEBUG:boto:Token: None 2024-04-13T22:52:57.513 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.513 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA= 2024-04-13T22:52:57.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA=', 'Content-Length': '0'} 2024-04-13T22:52:57.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d37ab93aee516df-00661b0cc9-236e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.518 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.519 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.519 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.519 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:52:57.519 DEBUG:boto:Data: 2024-04-13T22:52:57.519 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.519 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.519 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.519 DEBUG:boto:Params: {} 2024-04-13T22:52:57.519 DEBUG:boto:Token: None 2024-04-13T22:52:57.519 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.520 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA= 2024-04-13T22:52:57.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA=', 'Content-Length': '0'} 2024-04-13T22:52:57.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e484ed6213fa2c0-00661b0cc9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-13T22:52:57.522 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.523 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.523 DEBUG:boto:path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.523 DEBUG:boto:auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.523 DEBUG:boto:Method: GET 2024-04-13T22:52:57.523 DEBUG:boto:Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.523 DEBUG:boto:Data: 2024-04-13T22:52:57.523 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.523 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.523 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.523 DEBUG:boto:Params: {} 2024-04-13T22:52:57.523 DEBUG:boto:Token: None 2024-04-13T22:52:57.524 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/?versions 2024-04-13T22:52:57.524 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T22:52:57.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T22:52:57.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b8dcd84a761d3b-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.526 DEBUG:boto:b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:52:57.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:52:57.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:52:57.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:52:57.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:52:57.528 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-13T22:52:57.528 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.528 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.528 DEBUG:boto:path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.528 DEBUG:boto:auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.528 DEBUG:boto:Method: GET 2024-04-13T22:52:57.528 DEBUG:boto:Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.528 DEBUG:boto:Data: 2024-04-13T22:52:57.529 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.529 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.529 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.529 DEBUG:boto:Params: {} 2024-04-13T22:52:57.529 DEBUG:boto:Token: None 2024-04-13T22:52:57.529 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/?versions 2024-04-13T22:52:57.529 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T22:52:57.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T22:52:57.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf375c713a51f0b3-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.532 DEBUG:boto:b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-13T22:52:57.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-13T22:52:57.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-13T22:52:57.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-13T22:52:57.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-13T22:52:57.533 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.533 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.533 DEBUG:boto:path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.533 DEBUG:boto:auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.533 DEBUG:boto:Method: GET 2024-04-13T22:52:57.533 DEBUG:boto:Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.533 DEBUG:boto:Data: 2024-04-13T22:52:57.533 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.534 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.534 DEBUG:boto:Params: {} 2024-04-13T22:52:57.534 DEBUG:boto:Token: None 2024-04-13T22:52:57.534 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/?versions 2024-04-13T22:52:57.534 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T22:52:57.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T22:52:57.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c553da96f31bcf8b-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.537 DEBUG:boto:b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.538 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.538 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.538 DEBUG:boto:path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.538 DEBUG:boto:auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.538 DEBUG:boto:Method: GET 2024-04-13T22:52:57.538 DEBUG:boto:Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T22:52:57.538 DEBUG:boto:Data: 2024-04-13T22:52:57.538 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.539 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.539 DEBUG:boto:Params: {} 2024-04-13T22:52:57.539 DEBUG:boto:Token: None 2024-04-13T22:52:57.539 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/?versions 2024-04-13T22:52:57.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T22:52:57.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T22:52:57.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1993a413f55062-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.542 DEBUG:boto:b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:52:57.542 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.543 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.543 DEBUG:boto:path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.543 DEBUG:boto:auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.543 DEBUG:boto:Method: GET 2024-04-13T22:52:57.543 DEBUG:boto:Path: /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.543 DEBUG:boto:Data: 2024-04-13T22:52:57.543 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.543 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.543 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.543 DEBUG:boto:Params: {} 2024-04-13T22:52:57.543 DEBUG:boto:Token: None 2024-04-13T22:52:57.544 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.544 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T22:52:57.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T22:52:57.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ad551ec4ab33c8-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.546 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.546 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.546 DEBUG:boto:path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.546 DEBUG:boto:auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.546 DEBUG:boto:Method: GET 2024-04-13T22:52:57.546 DEBUG:boto:Path: /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.546 DEBUG:boto:Data: 2024-04-13T22:52:57.546 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.547 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.547 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.547 DEBUG:boto:Params: {} 2024-04-13T22:52:57.547 DEBUG:boto:Token: None 2024-04-13T22:52:57.547 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.547 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T22:52:57.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T22:52:57.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2f97146c1185ef-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.549 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.549 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.549 DEBUG:boto:path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.549 DEBUG:boto:auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.549 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.549 DEBUG:boto:Path: /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.550 DEBUG:boto:Data: 2024-04-13T22:52:57.550 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.550 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.550 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.550 DEBUG:boto:Params: {} 2024-04-13T22:52:57.550 DEBUG:boto:Token: None 2024-04-13T22:52:57.550 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.550 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA= 2024-04-13T22:52:57.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA=', 'Content-Length': '0'} 2024-04-13T22:52:57.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb983673be05eb8-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.552 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.552 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.552 DEBUG:boto:path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.553 DEBUG:boto:auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.553 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.553 DEBUG:boto:Path: /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.553 DEBUG:boto:Data: 2024-04-13T22:52:57.553 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.553 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.553 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.553 DEBUG:boto:Params: {} 2024-04-13T22:52:57.553 DEBUG:boto:Token: None 2024-04-13T22:52:57.553 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.554 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA= 2024-04-13T22:52:57.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA=', 'Content-Length': '0'} 2024-04-13T22:52:57.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f624c4e7f76771ad-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:52:57.556 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.556 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.556 DEBUG:boto:path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.556 DEBUG:boto:auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.556 DEBUG:boto:Method: GET 2024-04-13T22:52:57.556 DEBUG:boto:Path: /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.556 DEBUG:boto:Data: 2024-04-13T22:52:57.556 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.556 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.556 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.557 DEBUG:boto:Params: {} 2024-04-13T22:52:57.557 DEBUG:boto:Token: None 2024-04-13T22:52:57.557 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T22:52:57.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T22:52:57.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6637a92007de2a8-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.559 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.559 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.559 DEBUG:boto:path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.559 DEBUG:boto:auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.559 DEBUG:boto:Method: GET 2024-04-13T22:52:57.560 DEBUG:boto:Path: /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.560 DEBUG:boto:Data: 2024-04-13T22:52:57.560 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.560 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.560 DEBUG:boto:Params: {} 2024-04-13T22:52:57.560 DEBUG:boto:Token: None 2024-04-13T22:52:57.560 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11?versionId=null 2024-04-13T22:52:57.560 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T22:52:57.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T22:52:57.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003301a549960f1b5c-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.562 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.563 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.563 DEBUG:boto:path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.563 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.563 DEBUG:boto:Method: GET 2024-04-13T22:52:57.563 DEBUG:boto:Path: /vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.563 DEBUG:boto:Data: 2024-04-13T22:52:57.563 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.563 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.563 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.563 DEBUG:boto:Params: {} 2024-04-13T22:52:57.564 DEBUG:boto:Token: None 2024-04-13T22:52:57.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b32bd03cb63c2d9b-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.566 DEBUG:boto:b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.566 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.567 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.567 DEBUG:boto:path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.567 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.567 DEBUG:boto:Method: GET 2024-04-13T22:52:57.567 DEBUG:boto:Path: /vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.567 DEBUG:boto:Data: 2024-04-13T22:52:57.567 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.567 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.567 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.567 DEBUG:boto:Params: {} 2024-04-13T22:52:57.567 DEBUG:boto:Token: None 2024-04-13T22:52:57.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.568 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d302c02f6749e3b-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.614 DEBUG:boto:b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.614 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.615 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.615 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.615 DEBUG:boto:Path: /vwnfns-66/obj11 2024-04-13T22:52:57.615 DEBUG:boto:Data: 2024-04-13T22:52:57.615 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.615 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.615 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.615 DEBUG:boto:Params: {} 2024-04-13T22:52:57.615 DEBUG:boto:Token: None 2024-04-13T22:52:57.615 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11 2024-04-13T22:52:57.616 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM= 2024-04-13T22:52:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM=', 'Content-Length': '0'} 2024-04-13T22:52:57.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0a232ca0f1b396-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.618 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.618 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.618 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.618 DEBUG:boto:Path: /vwnfns-66/obj11 2024-04-13T22:52:57.618 DEBUG:boto:Data: 2024-04-13T22:52:57.618 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.618 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.618 DEBUG:boto:Params: {} 2024-04-13T22:52:57.619 DEBUG:boto:Token: None 2024-04-13T22:52:57.619 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11 2024-04-13T22:52:57.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM= 2024-04-13T22:52:57.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM=', 'Content-Length': '0'} 2024-04-13T22:52:57.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b8d7602ba82acd-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-13T22:52:57.621 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.621 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.621 DEBUG:boto:Method: GET 2024-04-13T22:52:57.621 DEBUG:boto:Path: /vwnfns-66/obj11 2024-04-13T22:52:57.621 DEBUG:boto:Data: 2024-04-13T22:52:57.621 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.622 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.622 DEBUG:boto:Params: {} 2024-04-13T22:52:57.622 DEBUG:boto:Token: None 2024-04-13T22:52:57.622 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11 2024-04-13T22:52:57.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI= 2024-04-13T22:52:57.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI=', 'Content-Length': '0'} 2024-04-13T22:52:57.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038cb12ffd56d3b8d-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.624 DEBUG:boto:path=/vwnfns-66/obj11 2024-04-13T22:52:57.624 DEBUG:boto:auth_path=/vwnfns-66/obj11 2024-04-13T22:52:57.624 DEBUG:boto:Method: GET 2024-04-13T22:52:57.624 DEBUG:boto:Path: /vwnfns-66/obj11 2024-04-13T22:52:57.624 DEBUG:boto:Data: 2024-04-13T22:52:57.624 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.624 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.624 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.624 DEBUG:boto:Params: {} 2024-04-13T22:52:57.625 DEBUG:boto:Token: None 2024-04-13T22:52:57.625 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj11 2024-04-13T22:52:57.625 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI= 2024-04-13T22:52:57.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI=', 'Content-Length': '0'} 2024-04-13T22:52:57.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5fea8274bb5065-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.627 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.627 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.627 DEBUG:boto:path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.627 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.627 DEBUG:boto:Method: GET 2024-04-13T22:52:57.628 DEBUG:boto:Path: /vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.628 DEBUG:boto:Data: 2024-04-13T22:52:57.628 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.628 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.628 DEBUG:boto:Params: {} 2024-04-13T22:52:57.628 DEBUG:boto:Token: None 2024-04-13T22:52:57.628 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006709d942d30786a7-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.631 DEBUG:boto:b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.631 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.631 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.631 DEBUG:boto:path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.631 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.631 DEBUG:boto:Method: GET 2024-04-13T22:52:57.632 DEBUG:boto:Path: /vwnfns-66/?prefix=obj11 2024-04-13T22:52:57.632 DEBUG:boto:Data: 2024-04-13T22:52:57.632 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.632 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.632 DEBUG:boto:Params: {} 2024-04-13T22:52:57.632 DEBUG:boto:Token: None 2024-04-13T22:52:57.632 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002489f61da8d2e5cd-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.635 DEBUG:boto:b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:52:57.635 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.635 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.635 DEBUG:boto:path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.635 DEBUG:boto:auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.635 DEBUG:boto:Method: GET 2024-04-13T22:52:57.636 DEBUG:boto:Path: /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.636 DEBUG:boto:Data: 2024-04-13T22:52:57.636 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.636 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.636 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.636 DEBUG:boto:Params: {} 2024-04-13T22:52:57.636 DEBUG:boto:Token: None 2024-04-13T22:52:57.636 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.636 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T22:52:57.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T22:52:57.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0855810229e5677-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.678 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.678 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.678 DEBUG:boto:path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.679 DEBUG:boto:auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.679 DEBUG:boto:Method: GET 2024-04-13T22:52:57.679 DEBUG:boto:Path: /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.679 DEBUG:boto:Data: 2024-04-13T22:52:57.679 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.679 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.679 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.679 DEBUG:boto:Params: {} 2024-04-13T22:52:57.679 DEBUG:boto:Token: None 2024-04-13T22:52:57.679 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T22:52:57.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T22:52:57.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec0997af236afbf-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.682 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.682 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.682 DEBUG:boto:path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.682 DEBUG:boto:auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.682 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.682 DEBUG:boto:Path: /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.682 DEBUG:boto:Data: 2024-04-13T22:52:57.683 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.683 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.683 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.683 DEBUG:boto:Params: {} 2024-04-13T22:52:57.683 DEBUG:boto:Token: None 2024-04-13T22:52:57.683 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.684 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo= 2024-04-13T22:52:57.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo=', 'Content-Length': '0'} 2024-04-13T22:52:57.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ff78562c6f1ed2-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.686 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.686 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.686 DEBUG:boto:path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.686 DEBUG:boto:auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.687 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.687 DEBUG:boto:Path: /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.687 DEBUG:boto:Data: 2024-04-13T22:52:57.687 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.687 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.687 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.687 DEBUG:boto:Params: {} 2024-04-13T22:52:57.687 DEBUG:boto:Token: None 2024-04-13T22:52:57.687 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.687 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo= 2024-04-13T22:52:57.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo=', 'Content-Length': '0'} 2024-04-13T22:52:57.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccaaf88ca69f2089-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:52:57.689 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.690 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.690 DEBUG:boto:path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.690 DEBUG:boto:auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.690 DEBUG:boto:Method: GET 2024-04-13T22:52:57.690 DEBUG:boto:Path: /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.690 DEBUG:boto:Data: 2024-04-13T22:52:57.690 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.690 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.690 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.690 DEBUG:boto:Params: {} 2024-04-13T22:52:57.690 DEBUG:boto:Token: None 2024-04-13T22:52:57.691 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.691 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T22:52:57.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T22:52:57.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a814de98c3f31b2d-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.693 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.693 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.693 DEBUG:boto:path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.693 DEBUG:boto:auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.693 DEBUG:boto:Method: GET 2024-04-13T22:52:57.693 DEBUG:boto:Path: /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.693 DEBUG:boto:Data: 2024-04-13T22:52:57.693 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.693 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.694 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.694 DEBUG:boto:Params: {} 2024-04-13T22:52:57.694 DEBUG:boto:Token: None 2024-04-13T22:52:57.694 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22?versionId=null 2024-04-13T22:52:57.694 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T22:52:57.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T22:52:57.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006434dbb183b504d6-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.696 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.696 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.696 DEBUG:boto:path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.696 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.696 DEBUG:boto:Method: GET 2024-04-13T22:52:57.696 DEBUG:boto:Path: /vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.697 DEBUG:boto:Data: 2024-04-13T22:52:57.697 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.697 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.697 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.697 DEBUG:boto:Params: {} 2024-04-13T22:52:57.697 DEBUG:boto:Token: None 2024-04-13T22:52:57.697 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.697 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e7ee4001f95f89-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.700 DEBUG:boto:b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.700 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.700 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.700 DEBUG:boto:path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.700 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.700 DEBUG:boto:Method: GET 2024-04-13T22:52:57.700 DEBUG:boto:Path: /vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.701 DEBUG:boto:Data: 2024-04-13T22:52:57.701 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.701 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.701 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.701 DEBUG:boto:Params: {} 2024-04-13T22:52:57.701 DEBUG:boto:Token: None 2024-04-13T22:52:57.701 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.701 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2d41f66dd9f925-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.704 DEBUG:boto:b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.704 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.704 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.704 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.704 DEBUG:boto:Path: /vwnfns-66/obj22 2024-04-13T22:52:57.704 DEBUG:boto:Data: 2024-04-13T22:52:57.705 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.705 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.705 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.705 DEBUG:boto:Params: {} 2024-04-13T22:52:57.705 DEBUG:boto:Token: None 2024-04-13T22:52:57.705 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22 2024-04-13T22:52:57.705 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A= 2024-04-13T22:52:57.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A=', 'Content-Length': '0'} 2024-04-13T22:52:57.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ef9078502f2a78-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.707 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.707 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.707 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.707 DEBUG:boto:Path: /vwnfns-66/obj22 2024-04-13T22:52:57.708 DEBUG:boto:Data: 2024-04-13T22:52:57.708 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.708 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.708 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.708 DEBUG:boto:Params: {} 2024-04-13T22:52:57.708 DEBUG:boto:Token: None 2024-04-13T22:52:57.708 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22 2024-04-13T22:52:57.708 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A= 2024-04-13T22:52:57.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A=', 'Content-Length': '0'} 2024-04-13T22:52:57.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ee6bdfda21e58a-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-13T22:52:57.710 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.710 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.711 DEBUG:boto:Method: GET 2024-04-13T22:52:57.711 DEBUG:boto:Path: /vwnfns-66/obj22 2024-04-13T22:52:57.711 DEBUG:boto:Data: 2024-04-13T22:52:57.711 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.711 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.711 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.711 DEBUG:boto:Params: {} 2024-04-13T22:52:57.711 DEBUG:boto:Token: None 2024-04-13T22:52:57.711 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22 2024-04-13T22:52:57.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI= 2024-04-13T22:52:57.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI=', 'Content-Length': '0'} 2024-04-13T22:52:57.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42c0b7f1ea015b5-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.754 DEBUG:boto:path=/vwnfns-66/obj22 2024-04-13T22:52:57.754 DEBUG:boto:auth_path=/vwnfns-66/obj22 2024-04-13T22:52:57.754 DEBUG:boto:Method: GET 2024-04-13T22:52:57.755 DEBUG:boto:Path: /vwnfns-66/obj22 2024-04-13T22:52:57.755 DEBUG:boto:Data: 2024-04-13T22:52:57.755 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.755 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.755 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.755 DEBUG:boto:Params: {} 2024-04-13T22:52:57.755 DEBUG:boto:Token: None 2024-04-13T22:52:57.755 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj22 2024-04-13T22:52:57.756 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI= 2024-04-13T22:52:57.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI=', 'Content-Length': '0'} 2024-04-13T22:52:57.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153218e7433a00cf-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.758 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.758 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.758 DEBUG:boto:path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.758 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.758 DEBUG:boto:Method: GET 2024-04-13T22:52:57.758 DEBUG:boto:Path: /vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.758 DEBUG:boto:Data: 2024-04-13T22:52:57.758 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.758 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.759 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.759 DEBUG:boto:Params: {} 2024-04-13T22:52:57.759 DEBUG:boto:Token: None 2024-04-13T22:52:57.759 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.759 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f2b988bb1e33d4-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.761 DEBUG:boto:b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.762 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.762 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.762 DEBUG:boto:path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.762 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.762 DEBUG:boto:Method: GET 2024-04-13T22:52:57.762 DEBUG:boto:Path: /vwnfns-66/?prefix=obj22 2024-04-13T22:52:57.762 DEBUG:boto:Data: 2024-04-13T22:52:57.762 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.762 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.763 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.763 DEBUG:boto:Params: {} 2024-04-13T22:52:57.763 DEBUG:boto:Token: None 2024-04-13T22:52:57.763 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.763 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764d6e2451c045a4-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.765 DEBUG:boto:b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:52:57.766 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.766 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.766 DEBUG:boto:path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.766 DEBUG:boto:auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.766 DEBUG:boto:Method: GET 2024-04-13T22:52:57.766 DEBUG:boto:Path: /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.767 DEBUG:boto:Data: 2024-04-13T22:52:57.767 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.767 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.767 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.767 DEBUG:boto:Params: {} 2024-04-13T22:52:57.767 DEBUG:boto:Token: None 2024-04-13T22:52:57.767 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.767 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T22:52:57.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T22:52:57.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebb5853805d1377-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.769 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.769 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.769 DEBUG:boto:path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.770 DEBUG:boto:auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.770 DEBUG:boto:Method: GET 2024-04-13T22:52:57.770 DEBUG:boto:Path: /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.770 DEBUG:boto:Data: 2024-04-13T22:52:57.770 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.770 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.770 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.770 DEBUG:boto:Params: {} 2024-04-13T22:52:57.770 DEBUG:boto:Token: None 2024-04-13T22:52:57.770 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.771 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T22:52:57.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T22:52:57.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022706f577d7a9c6a-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.773 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.773 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.773 DEBUG:boto:path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.773 DEBUG:boto:auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.773 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.773 DEBUG:boto:Path: /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.773 DEBUG:boto:Data: 2024-04-13T22:52:57.773 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.773 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.773 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.774 DEBUG:boto:Params: {} 2024-04-13T22:52:57.774 DEBUG:boto:Token: None 2024-04-13T22:52:57.774 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.774 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8= 2024-04-13T22:52:57.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8=', 'Content-Length': '0'} 2024-04-13T22:52:57.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49c3dc408c7dcb1-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.776 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.776 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.776 DEBUG:boto:path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.776 DEBUG:boto:auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.776 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.776 DEBUG:boto:Path: /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.776 DEBUG:boto:Data: 2024-04-13T22:52:57.777 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.777 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.777 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.777 DEBUG:boto:Params: {} 2024-04-13T22:52:57.777 DEBUG:boto:Token: None 2024-04-13T22:52:57.778 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.778 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8= 2024-04-13T22:52:57.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8=', 'Content-Length': '0'} 2024-04-13T22:52:57.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090cf43b8eb939e8e-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:52:57.780 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.780 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.780 DEBUG:boto:path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.780 DEBUG:boto:auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.780 DEBUG:boto:Method: GET 2024-04-13T22:52:57.780 DEBUG:boto:Path: /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.780 DEBUG:boto:Data: 2024-04-13T22:52:57.780 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.780 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.780 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.781 DEBUG:boto:Params: {} 2024-04-13T22:52:57.781 DEBUG:boto:Token: None 2024-04-13T22:52:57.781 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.781 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T22:52:57.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T22:52:57.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b637b8b72fbcf38-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.783 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.783 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.784 DEBUG:boto:path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.784 DEBUG:boto:auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.784 DEBUG:boto:Method: GET 2024-04-13T22:52:57.784 DEBUG:boto:Path: /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.784 DEBUG:boto:Data: 2024-04-13T22:52:57.784 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.784 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.784 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.784 DEBUG:boto:Params: {} 2024-04-13T22:52:57.785 DEBUG:boto:Token: None 2024-04-13T22:52:57.785 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33?versionId=null 2024-04-13T22:52:57.785 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T22:52:57.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T22:52:57.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108375a68b348837-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.787 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.788 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.788 DEBUG:boto:path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.788 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.788 DEBUG:boto:Method: GET 2024-04-13T22:52:57.788 DEBUG:boto:Path: /vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.788 DEBUG:boto:Data: 2024-04-13T22:52:57.788 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.788 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.789 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.789 DEBUG:boto:Params: {} 2024-04-13T22:52:57.789 DEBUG:boto:Token: None 2024-04-13T22:52:57.789 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.789 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000047a749bf511cf56-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.792 DEBUG:boto:b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.792 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.792 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.792 DEBUG:boto:path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.792 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.792 DEBUG:boto:Method: GET 2024-04-13T22:52:57.792 DEBUG:boto:Path: /vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.793 DEBUG:boto:Data: 2024-04-13T22:52:57.793 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.793 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.793 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.793 DEBUG:boto:Params: {} 2024-04-13T22:52:57.793 DEBUG:boto:Token: None 2024-04-13T22:52:57.793 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.793 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e6acdac725dac6-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.796 DEBUG:boto:b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.796 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.796 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.796 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.796 DEBUG:boto:Path: /vwnfns-66/obj33 2024-04-13T22:52:57.796 DEBUG:boto:Data: 2024-04-13T22:52:57.796 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.797 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.797 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.797 DEBUG:boto:Params: {} 2024-04-13T22:52:57.797 DEBUG:boto:Token: None 2024-04-13T22:52:57.797 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33 2024-04-13T22:52:57.797 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc= 2024-04-13T22:52:57.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc=', 'Content-Length': '0'} 2024-04-13T22:52:57.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe66c3afa00ec1a-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.799 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.799 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.799 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.799 DEBUG:boto:Path: /vwnfns-66/obj33 2024-04-13T22:52:57.799 DEBUG:boto:Data: 2024-04-13T22:52:57.799 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.799 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.799 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.800 DEBUG:boto:Params: {} 2024-04-13T22:52:57.800 DEBUG:boto:Token: None 2024-04-13T22:52:57.800 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33 2024-04-13T22:52:57.800 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc= 2024-04-13T22:52:57.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc=', 'Content-Length': '0'} 2024-04-13T22:52:57.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce84b2ad19cd9154-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-13T22:52:57.802 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.802 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.803 DEBUG:boto:Method: GET 2024-04-13T22:52:57.803 DEBUG:boto:Path: /vwnfns-66/obj33 2024-04-13T22:52:57.803 DEBUG:boto:Data: 2024-04-13T22:52:57.803 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.803 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.803 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.803 DEBUG:boto:Params: {} 2024-04-13T22:52:57.803 DEBUG:boto:Token: None 2024-04-13T22:52:57.803 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33 2024-04-13T22:52:57.803 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE= 2024-04-13T22:52:57.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE=', 'Content-Length': '0'} 2024-04-13T22:52:57.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a9a5078a7c98aea-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.846 DEBUG:boto:path=/vwnfns-66/obj33 2024-04-13T22:52:57.846 DEBUG:boto:auth_path=/vwnfns-66/obj33 2024-04-13T22:52:57.847 DEBUG:boto:Method: GET 2024-04-13T22:52:57.847 DEBUG:boto:Path: /vwnfns-66/obj33 2024-04-13T22:52:57.847 DEBUG:boto:Data: 2024-04-13T22:52:57.847 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.847 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.847 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.847 DEBUG:boto:Params: {} 2024-04-13T22:52:57.847 DEBUG:boto:Token: None 2024-04-13T22:52:57.847 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj33 2024-04-13T22:52:57.847 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE= 2024-04-13T22:52:57.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE=', 'Content-Length': '0'} 2024-04-13T22:52:57.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001049bb88f047a2f5-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.849 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.850 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.850 DEBUG:boto:path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.850 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.850 DEBUG:boto:Method: GET 2024-04-13T22:52:57.850 DEBUG:boto:Path: /vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.850 DEBUG:boto:Data: 2024-04-13T22:52:57.850 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.850 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.850 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.850 DEBUG:boto:Params: {} 2024-04-13T22:52:57.851 DEBUG:boto:Token: None 2024-04-13T22:52:57.851 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.851 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2c41edf0880d49c-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.853 DEBUG:boto:b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.853 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.853 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.854 DEBUG:boto:path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.854 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.854 DEBUG:boto:Method: GET 2024-04-13T22:52:57.854 DEBUG:boto:Path: /vwnfns-66/?prefix=obj33 2024-04-13T22:52:57.854 DEBUG:boto:Data: 2024-04-13T22:52:57.854 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.854 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.854 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.855 DEBUG:boto:Params: {} 2024-04-13T22:52:57.855 DEBUG:boto:Token: None 2024-04-13T22:52:57.855 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.855 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4c0d5622d0e745-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.858 DEBUG:boto:b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:52:57.858 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.858 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.859 DEBUG:boto:path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.859 DEBUG:boto:auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.859 DEBUG:boto:Method: GET 2024-04-13T22:52:57.859 DEBUG:boto:Path: /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.859 DEBUG:boto:Data: 2024-04-13T22:52:57.859 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.859 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.859 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.859 DEBUG:boto:Params: {} 2024-04-13T22:52:57.859 DEBUG:boto:Token: None 2024-04-13T22:52:57.859 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T22:52:57.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T22:52:57.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115d6ce453babb03-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.862 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.862 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.862 DEBUG:boto:path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.862 DEBUG:boto:auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.862 DEBUG:boto:Method: GET 2024-04-13T22:52:57.862 DEBUG:boto:Path: /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.862 DEBUG:boto:Data: 2024-04-13T22:52:57.863 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.863 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.863 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.863 DEBUG:boto:Params: {} 2024-04-13T22:52:57.863 DEBUG:boto:Token: None 2024-04-13T22:52:57.863 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.863 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T22:52:57.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T22:52:57.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2615cbfc29e8832-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.865 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.865 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.865 DEBUG:boto:path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.866 DEBUG:boto:auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.866 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.866 DEBUG:boto:Path: /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.866 DEBUG:boto:Data: 2024-04-13T22:52:57.866 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.866 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.866 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.866 DEBUG:boto:Params: {} 2024-04-13T22:52:57.866 DEBUG:boto:Token: None 2024-04-13T22:52:57.866 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.867 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8= 2024-04-13T22:52:57.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8=', 'Content-Length': '0'} 2024-04-13T22:52:57.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006adc549fa2ffca75-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.868 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.869 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.869 DEBUG:boto:path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.869 DEBUG:boto:auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.869 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.869 DEBUG:boto:Path: /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.869 DEBUG:boto:Data: 2024-04-13T22:52:57.869 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.869 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.869 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.869 DEBUG:boto:Params: {} 2024-04-13T22:52:57.869 DEBUG:boto:Token: None 2024-04-13T22:52:57.870 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.870 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8= 2024-04-13T22:52:57.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8=', 'Content-Length': '0'} 2024-04-13T22:52:57.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc509b34aebbcaac-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:52:57.872 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.872 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.872 DEBUG:boto:path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.872 DEBUG:boto:auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.872 DEBUG:boto:Method: GET 2024-04-13T22:52:57.872 DEBUG:boto:Path: /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.872 DEBUG:boto:Data: 2024-04-13T22:52:57.873 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.873 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.873 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.873 DEBUG:boto:Params: {} 2024-04-13T22:52:57.873 DEBUG:boto:Token: None 2024-04-13T22:52:57.873 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.873 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T22:52:57.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T22:52:57.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162aebc5df0ed19f-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.875 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.875 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.875 DEBUG:boto:path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.875 DEBUG:boto:auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.875 DEBUG:boto:Method: GET 2024-04-13T22:52:57.876 DEBUG:boto:Path: /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.876 DEBUG:boto:Data: 2024-04-13T22:52:57.876 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.876 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.876 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.876 DEBUG:boto:Params: {} 2024-04-13T22:52:57.876 DEBUG:boto:Token: None 2024-04-13T22:52:57.876 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44?versionId=null 2024-04-13T22:52:57.876 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T22:52:57.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T22:52:57.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d461b633e47eb15e-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.879 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.879 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.879 DEBUG:boto:path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.879 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.879 DEBUG:boto:Method: GET 2024-04-13T22:52:57.879 DEBUG:boto:Path: /vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.879 DEBUG:boto:Data: 2024-04-13T22:52:57.879 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.879 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.879 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.880 DEBUG:boto:Params: {} 2024-04-13T22:52:57.880 DEBUG:boto:Token: None 2024-04-13T22:52:57.880 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.880 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000077ec7c00ae11f9-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.882 DEBUG:boto:b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.883 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.883 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.883 DEBUG:boto:path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.883 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.883 DEBUG:boto:Method: GET 2024-04-13T22:52:57.883 DEBUG:boto:Path: /vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.883 DEBUG:boto:Data: 2024-04-13T22:52:57.883 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.883 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.883 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.884 DEBUG:boto:Params: {} 2024-04-13T22:52:57.884 DEBUG:boto:Token: None 2024-04-13T22:52:57.884 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.884 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794439cf774fa91c-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.930 DEBUG:boto:b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.931 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.931 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.931 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.931 DEBUG:boto:Path: /vwnfns-66/obj44 2024-04-13T22:52:57.931 DEBUG:boto:Data: 2024-04-13T22:52:57.931 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.931 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.931 DEBUG:boto:Params: {} 2024-04-13T22:52:57.931 DEBUG:boto:Token: None 2024-04-13T22:52:57.932 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44 2024-04-13T22:52:57.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc= 2024-04-13T22:52:57.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc=', 'Content-Length': '0'} 2024-04-13T22:52:57.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d1fbdaf093eb74a-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.934 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.934 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.934 DEBUG:boto:Method: HEAD 2024-04-13T22:52:57.934 DEBUG:boto:Path: /vwnfns-66/obj44 2024-04-13T22:52:57.935 DEBUG:boto:Data: 2024-04-13T22:52:57.935 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.935 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.935 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.935 DEBUG:boto:Params: {} 2024-04-13T22:52:57.935 DEBUG:boto:Token: None 2024-04-13T22:52:57.935 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44 2024-04-13T22:52:57.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc= 2024-04-13T22:52:57.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc=', 'Content-Length': '0'} 2024-04-13T22:52:57.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359c20f1e3fe163d-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-13T22:52:57.937 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.937 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.938 DEBUG:boto:Method: GET 2024-04-13T22:52:57.938 DEBUG:boto:Path: /vwnfns-66/obj44 2024-04-13T22:52:57.938 DEBUG:boto:Data: 2024-04-13T22:52:57.938 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.938 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.938 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.938 DEBUG:boto:Params: {} 2024-04-13T22:52:57.938 DEBUG:boto:Token: None 2024-04-13T22:52:57.938 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44 2024-04-13T22:52:57.939 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A= 2024-04-13T22:52:57.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A=', 'Content-Length': '0'} 2024-04-13T22:52:57.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e927d6bf486b9180-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.940 DEBUG:boto:path=/vwnfns-66/obj44 2024-04-13T22:52:57.941 DEBUG:boto:auth_path=/vwnfns-66/obj44 2024-04-13T22:52:57.941 DEBUG:boto:Method: GET 2024-04-13T22:52:57.941 DEBUG:boto:Path: /vwnfns-66/obj44 2024-04-13T22:52:57.941 DEBUG:boto:Data: 2024-04-13T22:52:57.941 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.941 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.941 DEBUG:boto:Params: {} 2024-04-13T22:52:57.941 DEBUG:boto:Token: None 2024-04-13T22:52:57.941 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj44 2024-04-13T22:52:57.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A= 2024-04-13T22:52:57.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A=', 'Content-Length': '0'} 2024-04-13T22:52:57.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c04cdec2ec3faaa-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.943 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.944 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.944 DEBUG:boto:path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.944 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.944 DEBUG:boto:Method: GET 2024-04-13T22:52:57.944 DEBUG:boto:Path: /vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.944 DEBUG:boto:Data: 2024-04-13T22:52:57.944 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.944 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.944 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.944 DEBUG:boto:Params: {} 2024-04-13T22:52:57.944 DEBUG:boto:Token: None 2024-04-13T22:52:57.945 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.945 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f344c68e037fdab6-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.947 DEBUG:boto:b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.948 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:57.948 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:57.948 DEBUG:boto:path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.948 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.948 DEBUG:boto:Method: GET 2024-04-13T22:52:57.948 DEBUG:boto:Path: /vwnfns-66/?prefix=obj44 2024-04-13T22:52:57.948 DEBUG:boto:Data: 2024-04-13T22:52:57.948 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.949 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.949 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.949 DEBUG:boto:Params: {} 2024-04-13T22:52:57.949 DEBUG:boto:Token: None 2024-04-13T22:52:57.949 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/ 2024-04-13T22:52:57.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T22:52:57.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T22:52:57.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b384a82ac67f708-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.952 DEBUG:boto:b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:57.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:52:57.952 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:57.953 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:57.953 DEBUG:boto:path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.953 DEBUG:boto:auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.953 DEBUG:boto:Method: GET 2024-04-13T22:52:57.953 DEBUG:boto:Path: /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.954 DEBUG:boto:Data: 2024-04-13T22:52:57.954 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.954 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:57.954 DEBUG:boto:Port: 8001 2024-04-13T22:52:57.954 DEBUG:boto:Params: {} 2024-04-13T22:52:57.954 DEBUG:boto:Token: None 2024-04-13T22:52:57.955 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.955 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s= 2024-04-13T22:52:57.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s=', 'Content-Length': '0'} 2024-04-13T22:52:57.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003540c34588a5bc82-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:57.998 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:57.998 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:57.998 DEBUG:boto:path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.999 DEBUG:boto:auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.999 DEBUG:boto:Method: GET 2024-04-13T22:52:57.999 DEBUG:boto:Path: /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:57.999 DEBUG:boto:Data: 2024-04-13T22:52:57.999 DEBUG:boto:Headers: {} 2024-04-13T22:52:57.999 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:57.999 DEBUG:boto:Port: 8000 2024-04-13T22:52:57.999 DEBUG:boto:Params: {} 2024-04-13T22:52:57.999 DEBUG:boto:Token: None 2024-04-13T22:52:57.999 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:57 GMT /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.000 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s= 2024-04-13T22:52:58.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s=', 'Content-Length': '0'} 2024-04-13T22:52:58.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2cc81a8d5a2808-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.002 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.002 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.002 DEBUG:boto:path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.002 DEBUG:boto:auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.002 DEBUG:boto:Method: HEAD 2024-04-13T22:52:58.002 DEBUG:boto:Path: /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.002 DEBUG:boto:Data: 2024-04-13T22:52:58.002 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.002 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:58.003 DEBUG:boto:Port: 8001 2024-04-13T22:52:58.003 DEBUG:boto:Params: {} 2024-04-13T22:52:58.003 DEBUG:boto:Token: None 2024-04-13T22:52:58.003 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.003 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc= 2024-04-13T22:52:58.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc=', 'Content-Length': '0'} 2024-04-13T22:52:58.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c56149a856d39d82-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.005 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.005 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.005 DEBUG:boto:path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.005 DEBUG:boto:auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.005 DEBUG:boto:Method: HEAD 2024-04-13T22:52:58.006 DEBUG:boto:Path: /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.006 DEBUG:boto:Data: 2024-04-13T22:52:58.006 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.006 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:58.006 DEBUG:boto:Port: 8000 2024-04-13T22:52:58.006 DEBUG:boto:Params: {} 2024-04-13T22:52:58.006 DEBUG:boto:Token: None 2024-04-13T22:52:58.006 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.006 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc= 2024-04-13T22:52:58.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc=', 'Content-Length': '0'} 2024-04-13T22:52:58.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d8c743a0a4ef3e-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:52:58.009 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.009 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.009 DEBUG:boto:path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.009 DEBUG:boto:auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.009 DEBUG:boto:Method: GET 2024-04-13T22:52:58.009 DEBUG:boto:Path: /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.009 DEBUG:boto:Data: 2024-04-13T22:52:58.009 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.009 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:58.009 DEBUG:boto:Port: 8001 2024-04-13T22:52:58.009 DEBUG:boto:Params: {} 2024-04-13T22:52:58.010 DEBUG:boto:Token: None 2024-04-13T22:52:58.010 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.010 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8= 2024-04-13T22:52:58.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8=', 'Content-Length': '0'} 2024-04-13T22:52:58.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0cd36177e257d4-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.012 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.012 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.012 DEBUG:boto:path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.012 DEBUG:boto:auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.012 DEBUG:boto:Method: GET 2024-04-13T22:52:58.012 DEBUG:boto:Path: /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.012 DEBUG:boto:Data: 2024-04-13T22:52:58.012 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.013 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:58.013 DEBUG:boto:Port: 8000 2024-04-13T22:52:58.013 DEBUG:boto:Params: {} 2024-04-13T22:52:58.013 DEBUG:boto:Token: None 2024-04-13T22:52:58.013 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55?versionId=null 2024-04-13T22:52:58.013 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8= 2024-04-13T22:52:58.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8=', 'Content-Length': '0'} 2024-04-13T22:52:58.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3714b7918622559-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.015 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:58.015 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:58.015 DEBUG:boto:path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.016 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.016 DEBUG:boto:Method: GET 2024-04-13T22:52:58.016 DEBUG:boto:Path: /vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.016 DEBUG:boto:Data: 2024-04-13T22:52:58.016 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.016 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:58.016 DEBUG:boto:Port: 8001 2024-04-13T22:52:58.016 DEBUG:boto:Params: {} 2024-04-13T22:52:58.016 DEBUG:boto:Token: None 2024-04-13T22:52:58.016 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/ 2024-04-13T22:52:58.017 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T22:52:58.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T22:52:58.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b68c2eee809fa08-00661b0cca-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.019 DEBUG:boto:b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:58.019 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:58.020 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:58.020 DEBUG:boto:path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.020 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.020 DEBUG:boto:Method: GET 2024-04-13T22:52:58.020 DEBUG:boto:Path: /vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.020 DEBUG:boto:Data: 2024-04-13T22:52:58.020 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.020 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:58.020 DEBUG:boto:Port: 8000 2024-04-13T22:52:58.020 DEBUG:boto:Params: {} 2024-04-13T22:52:58.020 DEBUG:boto:Token: None 2024-04-13T22:52:58.021 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/ 2024-04-13T22:52:58.021 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T22:52:58.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T22:52:58.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180652a9ae213d6f-00661b0cca-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.066 DEBUG:boto:b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:58.066 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.067 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.067 DEBUG:boto:Method: HEAD 2024-04-13T22:52:58.067 DEBUG:boto:Path: /vwnfns-66/obj55 2024-04-13T22:52:58.067 DEBUG:boto:Data: 2024-04-13T22:52:58.067 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.067 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:58.067 DEBUG:boto:Port: 8001 2024-04-13T22:52:58.067 DEBUG:boto:Params: {} 2024-04-13T22:52:58.067 DEBUG:boto:Token: None 2024-04-13T22:52:58.067 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55 2024-04-13T22:52:58.068 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0= 2024-04-13T22:52:58.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0=', 'Content-Length': '0'} 2024-04-13T22:52:58.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5ede09950dc837-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.070 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.070 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.070 DEBUG:boto:Method: HEAD 2024-04-13T22:52:58.070 DEBUG:boto:Path: /vwnfns-66/obj55 2024-04-13T22:52:58.070 DEBUG:boto:Data: 2024-04-13T22:52:58.070 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.070 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:58.070 DEBUG:boto:Port: 8000 2024-04-13T22:52:58.071 DEBUG:boto:Params: {} 2024-04-13T22:52:58.071 DEBUG:boto:Token: None 2024-04-13T22:52:58.071 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55 2024-04-13T22:52:58.071 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0= 2024-04-13T22:52:58.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0=', 'Content-Length': '0'} 2024-04-13T22:52:58.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d315b6ef21003be-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-13T22:52:58.073 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.073 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.073 DEBUG:boto:Method: GET 2024-04-13T22:52:58.073 DEBUG:boto:Path: /vwnfns-66/obj55 2024-04-13T22:52:58.073 DEBUG:boto:Data: 2024-04-13T22:52:58.073 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:58.074 DEBUG:boto:Port: 8001 2024-04-13T22:52:58.074 DEBUG:boto:Params: {} 2024-04-13T22:52:58.074 DEBUG:boto:Token: None 2024-04-13T22:52:58.074 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55 2024-04-13T22:52:58.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ= 2024-04-13T22:52:58.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ=', 'Content-Length': '0'} 2024-04-13T22:52:58.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2b5bbe4a7ed456-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.076 DEBUG:boto:path=/vwnfns-66/obj55 2024-04-13T22:52:58.076 DEBUG:boto:auth_path=/vwnfns-66/obj55 2024-04-13T22:52:58.076 DEBUG:boto:Method: GET 2024-04-13T22:52:58.076 DEBUG:boto:Path: /vwnfns-66/obj55 2024-04-13T22:52:58.076 DEBUG:boto:Data: 2024-04-13T22:52:58.077 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.077 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:58.077 DEBUG:boto:Port: 8000 2024-04-13T22:52:58.077 DEBUG:boto:Params: {} 2024-04-13T22:52:58.077 DEBUG:boto:Token: None 2024-04-13T22:52:58.077 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/obj55 2024-04-13T22:52:58.077 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ= 2024-04-13T22:52:58.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ=', 'Content-Length': '0'} 2024-04-13T22:52:58.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197514d6c58266e9-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.079 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:58.079 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:58.079 DEBUG:boto:path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.079 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.079 DEBUG:boto:Method: GET 2024-04-13T22:52:58.079 DEBUG:boto:Path: /vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.079 DEBUG:boto:Data: 2024-04-13T22:52:58.080 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:52:58.080 DEBUG:boto:Port: 8001 2024-04-13T22:52:58.080 DEBUG:boto:Params: {} 2024-04-13T22:52:58.080 DEBUG:boto:Token: None 2024-04-13T22:52:58.080 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/ 2024-04-13T22:52:58.080 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T22:52:58.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T22:52:58.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004deb85208b35736a-00661b0cca-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.083 DEBUG:boto:b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:58.083 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:52:58.083 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:52:58.083 DEBUG:boto:path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.083 DEBUG:boto:auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.083 DEBUG:boto:Method: GET 2024-04-13T22:52:58.083 DEBUG:boto:Path: /vwnfns-66/?prefix=obj55 2024-04-13T22:52:58.084 DEBUG:boto:Data: 2024-04-13T22:52:58.084 DEBUG:boto:Headers: {} 2024-04-13T22:52:58.084 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:52:58.084 DEBUG:boto:Port: 8000 2024-04-13T22:52:58.084 DEBUG:boto:Params: {} 2024-04-13T22:52:58.084 DEBUG:boto:Token: None 2024-04-13T22:52:58.084 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 22:52:58 GMT /vwnfns-66/ 2024-04-13T22:52:58.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T22:52:58.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T22:52:58.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e41f6033ff502729-00661b0cca-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:52:58.087 DEBUG:boto:b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T22:52:58.087 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vwnfns-66 zones={a2, a1} 2024-04-13T22:52:58.087 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T22:52:58.087 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T22:52:58.088 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:52:58.085+0000 7fc70bec7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 116241) UID: 0 2024-04-13T22:52:58.088 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T22:52:58.085+0000 7fc71109c7c0 -1 shutting down 2024-04-13T22:53:03.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T22:53:04.187 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T22:53:04.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:04.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:04.188 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:04.247 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:04.247 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:04.294 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:53:04.294 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-66 2024-04-13T22:53:04.298 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 5 2024-04-13T22:53:04.337 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.337 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.337 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.338 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.338 DEBUG:boto:Data: 2024-04-13T22:53:04.338 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.338 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.338 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.338 DEBUG:boto:Params: {} 2024-04-13T22:53:04.338 DEBUG:boto:Token: None 2024-04-13T22:53:04.338 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.339 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de8c8d3a410a4bd-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.341 DEBUG:boto:path=/vwnfns-66/obj17 2024-04-13T22:53:04.342 DEBUG:boto:auth_path=/vwnfns-66/obj17 2024-04-13T22:53:04.342 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.342 DEBUG:boto:Path: /vwnfns-66/obj17 2024-04-13T22:53:04.342 DEBUG:boto:Data: 2024-04-13T22:53:04.342 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.342 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.342 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.342 DEBUG:boto:Params: {} 2024-04-13T22:53:04.342 DEBUG:boto:Token: None 2024-04-13T22:53:04.343 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj17 2024-04-13T22:53:04.343 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:32iosuymcFZrIP2KY7QZOb8UQIc= 2024-04-13T22:53:04.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:32iosuymcFZrIP2KY7QZOb8UQIc='} 2024-04-13T22:53:04.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007457c6f10c2e943e-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.348 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.348 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.348 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.349 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.349 DEBUG:boto:Data: 2024-04-13T22:53:04.349 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.349 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.349 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.349 DEBUG:boto:Params: {} 2024-04-13T22:53:04.349 DEBUG:boto:Token: None 2024-04-13T22:53:04.349 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.349 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000156b3cb79111cb29-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.352 DEBUG:boto:path=/vwnfns-66/obj34 2024-04-13T22:53:04.352 DEBUG:boto:auth_path=/vwnfns-66/obj34 2024-04-13T22:53:04.352 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.352 DEBUG:boto:Path: /vwnfns-66/obj34 2024-04-13T22:53:04.352 DEBUG:boto:Data: 2024-04-13T22:53:04.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.352 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.352 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.352 DEBUG:boto:Params: {} 2024-04-13T22:53:04.353 DEBUG:boto:Token: None 2024-04-13T22:53:04.353 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj34 2024-04-13T22:53:04.353 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lc6mUMfBi+zfpZwY6Pw9NlkK90k= 2024-04-13T22:53:04.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lc6mUMfBi+zfpZwY6Pw9NlkK90k='} 2024-04-13T22:53:04.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf35cb17ec7d61f8-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.399 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.399 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.399 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.399 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.399 DEBUG:boto:Data: 2024-04-13T22:53:04.399 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.399 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.400 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.400 DEBUG:boto:Params: {} 2024-04-13T22:53:04.400 DEBUG:boto:Token: None 2024-04-13T22:53:04.400 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.400 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df508f8846bfcc82-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.403 DEBUG:boto:path=/vwnfns-66/obj51 2024-04-13T22:53:04.403 DEBUG:boto:auth_path=/vwnfns-66/obj51 2024-04-13T22:53:04.403 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.403 DEBUG:boto:Path: /vwnfns-66/obj51 2024-04-13T22:53:04.403 DEBUG:boto:Data: 2024-04-13T22:53:04.404 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.404 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.404 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.404 DEBUG:boto:Params: {} 2024-04-13T22:53:04.404 DEBUG:boto:Token: None 2024-04-13T22:53:04.404 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj51 2024-04-13T22:53:04.404 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ju1B2O/CgxZ9Vr6C9E+43k6soSA= 2024-04-13T22:53:04.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ju1B2O/CgxZ9Vr6C9E+43k6soSA='} 2024-04-13T22:53:04.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a50e1eda6fe17127-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.409 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.410 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.410 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.410 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.410 DEBUG:boto:Data: 2024-04-13T22:53:04.410 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.410 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.410 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.410 DEBUG:boto:Params: {} 2024-04-13T22:53:04.410 DEBUG:boto:Token: None 2024-04-13T22:53:04.411 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.411 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbca12e170164b9e-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.413 DEBUG:boto:path=/vwnfns-66/obj68 2024-04-13T22:53:04.413 DEBUG:boto:auth_path=/vwnfns-66/obj68 2024-04-13T22:53:04.413 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.413 DEBUG:boto:Path: /vwnfns-66/obj68 2024-04-13T22:53:04.414 DEBUG:boto:Data: 2024-04-13T22:53:04.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.414 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.414 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.414 DEBUG:boto:Params: {} 2024-04-13T22:53:04.414 DEBUG:boto:Token: None 2024-04-13T22:53:04.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj68 2024-04-13T22:53:04.414 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YOXAFHwKCtLbz/kFxdXSW+WGXAs= 2024-04-13T22:53:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YOXAFHwKCtLbz/kFxdXSW+WGXAs='} 2024-04-13T22:53:04.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000039f9eebba76b7f0-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.459 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.459 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.459 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.459 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.459 DEBUG:boto:Data: 2024-04-13T22:53:04.459 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.459 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.459 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.459 DEBUG:boto:Params: {} 2024-04-13T22:53:04.460 DEBUG:boto:Token: None 2024-04-13T22:53:04.460 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.460 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baa00814d667e1b0-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.463 DEBUG:boto:path=/vwnfns-66/obj85 2024-04-13T22:53:04.463 DEBUG:boto:auth_path=/vwnfns-66/obj85 2024-04-13T22:53:04.463 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.463 DEBUG:boto:Path: /vwnfns-66/obj85 2024-04-13T22:53:04.463 DEBUG:boto:Data: 2024-04-13T22:53:04.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.463 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.463 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.463 DEBUG:boto:Params: {} 2024-04-13T22:53:04.463 DEBUG:boto:Token: None 2024-04-13T22:53:04.463 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj85 2024-04-13T22:53:04.464 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jfb8GMKxFrTRekHQ1q3y06otxRQ= 2024-04-13T22:53:04.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jfb8GMKxFrTRekHQ1q3y06otxRQ='} 2024-04-13T22:53:04.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba9860bd698fbbdd-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:04.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:04.470 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:04.521 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:04.521 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:04.565 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:53:04.565 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-66 2024-04-13T22:53:04.571 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 10 2024-04-13T22:53:04.616 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.617 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.617 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.617 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.617 DEBUG:boto:Data: 2024-04-13T22:53:04.617 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.617 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.617 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.617 DEBUG:boto:Params: {} 2024-04-13T22:53:04.617 DEBUG:boto:Token: None 2024-04-13T22:53:04.618 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.618 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ade0b18d3dd1394c-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.621 DEBUG:boto:path=/vwnfns-66/obj19 2024-04-13T22:53:04.621 DEBUG:boto:auth_path=/vwnfns-66/obj19 2024-04-13T22:53:04.621 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.621 DEBUG:boto:Path: /vwnfns-66/obj19 2024-04-13T22:53:04.621 DEBUG:boto:Data: 2024-04-13T22:53:04.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.621 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.621 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.622 DEBUG:boto:Params: {} 2024-04-13T22:53:04.622 DEBUG:boto:Token: None 2024-04-13T22:53:04.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj19 2024-04-13T22:53:04.622 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9JlSz+YNH9+a84xXHCuYwsl1ecE= 2024-04-13T22:53:04.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9JlSz+YNH9+a84xXHCuYwsl1ecE='} 2024-04-13T22:53:04.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47e9d33570e4640-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.627 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.627 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.627 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.628 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.628 DEBUG:boto:Data: 2024-04-13T22:53:04.628 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.628 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.628 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.628 DEBUG:boto:Params: {} 2024-04-13T22:53:04.628 DEBUG:boto:Token: None 2024-04-13T22:53:04.628 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000358a7222d26da155-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.631 DEBUG:boto:path=/vwnfns-66/obj38 2024-04-13T22:53:04.631 DEBUG:boto:auth_path=/vwnfns-66/obj38 2024-04-13T22:53:04.631 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.631 DEBUG:boto:Path: /vwnfns-66/obj38 2024-04-13T22:53:04.631 DEBUG:boto:Data: 2024-04-13T22:53:04.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.631 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.631 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.631 DEBUG:boto:Params: {} 2024-04-13T22:53:04.632 DEBUG:boto:Token: None 2024-04-13T22:53:04.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj38 2024-04-13T22:53:04.632 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:e8mwyVqgrlbkx+Y5JqCmpQa+SE4= 2024-04-13T22:53:04.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e8mwyVqgrlbkx+Y5JqCmpQa+SE4='} 2024-04-13T22:53:04.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000415aaa5849618f11-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.675 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.675 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.675 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.676 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.676 DEBUG:boto:Data: 2024-04-13T22:53:04.676 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.676 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.676 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.676 DEBUG:boto:Params: {} 2024-04-13T22:53:04.676 DEBUG:boto:Token: None 2024-04-13T22:53:04.676 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.676 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c6fcbfe0db5a8e4-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.679 DEBUG:boto:path=/vwnfns-66/obj57 2024-04-13T22:53:04.679 DEBUG:boto:auth_path=/vwnfns-66/obj57 2024-04-13T22:53:04.679 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.679 DEBUG:boto:Path: /vwnfns-66/obj57 2024-04-13T22:53:04.679 DEBUG:boto:Data: 2024-04-13T22:53:04.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.680 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.680 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.680 DEBUG:boto:Params: {} 2024-04-13T22:53:04.680 DEBUG:boto:Token: None 2024-04-13T22:53:04.680 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj57 2024-04-13T22:53:04.680 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WApo1ZL9XL0x2sxi2i0hiNYsgIY= 2024-04-13T22:53:04.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WApo1ZL9XL0x2sxi2i0hiNYsgIY='} 2024-04-13T22:53:04.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ecce265bf1404ba-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.684 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.684 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.684 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.684 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.684 DEBUG:boto:Data: 2024-04-13T22:53:04.684 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.685 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.685 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.685 DEBUG:boto:Params: {} 2024-04-13T22:53:04.685 DEBUG:boto:Token: None 2024-04-13T22:53:04.685 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.685 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d4159260582e8b3-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.687 DEBUG:boto:path=/vwnfns-66/obj76 2024-04-13T22:53:04.688 DEBUG:boto:auth_path=/vwnfns-66/obj76 2024-04-13T22:53:04.688 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.688 DEBUG:boto:Path: /vwnfns-66/obj76 2024-04-13T22:53:04.688 DEBUG:boto:Data: 2024-04-13T22:53:04.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.688 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.688 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.688 DEBUG:boto:Params: {} 2024-04-13T22:53:04.688 DEBUG:boto:Token: None 2024-04-13T22:53:04.688 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj76 2024-04-13T22:53:04.688 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lGdECkOAabY8CgwhEgRNUR1McSQ= 2024-04-13T22:53:04.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lGdECkOAabY8CgwhEgRNUR1McSQ='} 2024-04-13T22:53:04.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006027cdd029d76b2-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.731 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.731 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.731 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.731 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.731 DEBUG:boto:Data: 2024-04-13T22:53:04.731 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.731 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.731 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.731 DEBUG:boto:Params: {} 2024-04-13T22:53:04.731 DEBUG:boto:Token: None 2024-04-13T22:53:04.732 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.732 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059eafd043d67d3a1-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.735 DEBUG:boto:path=/vwnfns-66/obj95 2024-04-13T22:53:04.735 DEBUG:boto:auth_path=/vwnfns-66/obj95 2024-04-13T22:53:04.735 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.735 DEBUG:boto:Path: /vwnfns-66/obj95 2024-04-13T22:53:04.735 DEBUG:boto:Data: 2024-04-13T22:53:04.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.735 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.735 DEBUG:boto:Params: {} 2024-04-13T22:53:04.735 DEBUG:boto:Token: None 2024-04-13T22:53:04.736 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj95 2024-04-13T22:53:04.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cDKxU5W29xmaSltrkXFCBqoT0nQ= 2024-04-13T22:53:04.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cDKxU5W29xmaSltrkXFCBqoT0nQ='} 2024-04-13T22:53:04.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005acf70916692624d-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:04.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:04.740 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:04.793 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:04.794 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:04.841 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:53:04.841 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-66 2024-04-13T22:53:04.847 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 15 2024-04-13T22:53:04.896 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.896 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.897 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.897 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.897 DEBUG:boto:Data: 2024-04-13T22:53:04.897 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.897 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.897 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.897 DEBUG:boto:Params: {} 2024-04-13T22:53:04.898 DEBUG:boto:Token: None 2024-04-13T22:53:04.898 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.898 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a1472b7009d2a4c-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.901 DEBUG:boto:path=/vwnfns-66/obj23 2024-04-13T22:53:04.901 DEBUG:boto:auth_path=/vwnfns-66/obj23 2024-04-13T22:53:04.901 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.901 DEBUG:boto:Path: /vwnfns-66/obj23 2024-04-13T22:53:04.902 DEBUG:boto:Data: 2024-04-13T22:53:04.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.902 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.902 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.902 DEBUG:boto:Params: {} 2024-04-13T22:53:04.902 DEBUG:boto:Token: None 2024-04-13T22:53:04.902 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj23 2024-04-13T22:53:04.902 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AcVz9pUbHfR27k9CMVf/xCfG0Xk= 2024-04-13T22:53:04.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AcVz9pUbHfR27k9CMVf/xCfG0Xk='} 2024-04-13T22:53:04.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074f95932e8baa1f8-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.908 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.908 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.908 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.908 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.908 DEBUG:boto:Data: 2024-04-13T22:53:04.908 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.908 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.908 DEBUG:boto:Params: {} 2024-04-13T22:53:04.908 DEBUG:boto:Token: None 2024-04-13T22:53:04.909 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009183b5f9f633e911-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.912 DEBUG:boto:path=/vwnfns-66/obj46 2024-04-13T22:53:04.912 DEBUG:boto:auth_path=/vwnfns-66/obj46 2024-04-13T22:53:04.912 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.912 DEBUG:boto:Path: /vwnfns-66/obj46 2024-04-13T22:53:04.912 DEBUG:boto:Data: 2024-04-13T22:53:04.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.912 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.912 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.912 DEBUG:boto:Params: {} 2024-04-13T22:53:04.912 DEBUG:boto:Token: None 2024-04-13T22:53:04.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj46 2024-04-13T22:53:04.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7uwhu9f0G+SFYY2hnoGiDySC1cs= 2024-04-13T22:53:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7uwhu9f0G+SFYY2hnoGiDySC1cs='} 2024-04-13T22:53:04.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ca0ef2ea0ea8f9a-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.955 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.955 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.955 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.955 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.955 DEBUG:boto:Data: 2024-04-13T22:53:04.955 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.955 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.955 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.955 DEBUG:boto:Params: {} 2024-04-13T22:53:04.955 DEBUG:boto:Token: None 2024-04-13T22:53:04.956 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.956 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000503903b35d614545-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.959 DEBUG:boto:path=/vwnfns-66/obj69 2024-04-13T22:53:04.959 DEBUG:boto:auth_path=/vwnfns-66/obj69 2024-04-13T22:53:04.959 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.959 DEBUG:boto:Path: /vwnfns-66/obj69 2024-04-13T22:53:04.959 DEBUG:boto:Data: 2024-04-13T22:53:04.959 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.959 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.960 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.960 DEBUG:boto:Params: {} 2024-04-13T22:53:04.960 DEBUG:boto:Token: None 2024-04-13T22:53:04.960 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj69 2024-04-13T22:53:04.960 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5i42n6lpKmIcP1apQxfk/0D01qg= 2024-04-13T22:53:04.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5i42n6lpKmIcP1apQxfk/0D01qg='} 2024-04-13T22:53:04.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002322aa8ce54d21b-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.965 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:04.965 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:04.965 DEBUG:boto:Method: HEAD 2024-04-13T22:53:04.965 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:04.965 DEBUG:boto:Data: 2024-04-13T22:53:04.965 DEBUG:boto:Headers: {} 2024-04-13T22:53:04.966 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.966 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.966 DEBUG:boto:Params: {} 2024-04-13T22:53:04.966 DEBUG:boto:Token: None 2024-04-13T22:53:04.966 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/ 2024-04-13T22:53:04.966 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T22:53:04.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T22:53:04.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8b942f1b2d1fa9f-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:04.970 DEBUG:boto:path=/vwnfns-66/obj92 2024-04-13T22:53:04.970 DEBUG:boto:auth_path=/vwnfns-66/obj92 2024-04-13T22:53:04.970 DEBUG:boto:Method: PUT 2024-04-13T22:53:04.970 DEBUG:boto:Path: /vwnfns-66/obj92 2024-04-13T22:53:04.970 DEBUG:boto:Data: 2024-04-13T22:53:04.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:04.970 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:04.971 DEBUG:boto:Port: 8000 2024-04-13T22:53:04.971 DEBUG:boto:Params: {} 2024-04-13T22:53:04.971 DEBUG:boto:Token: None 2024-04-13T22:53:04.971 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:04 GMT /vwnfns-66/obj92 2024-04-13T22:53:04.971 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DpyvH5dUE231pH0/kRw549SM37g= 2024-04-13T22:53:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpyvH5dUE231pH0/kRw549SM37g='} 2024-04-13T22:53:05.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027abc30edc7779ad-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.015 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.015 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.015 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.015 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.015 DEBUG:boto:Data: 2024-04-13T22:53:05.015 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.015 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.015 DEBUG:boto:Params: {} 2024-04-13T22:53:05.015 DEBUG:boto:Token: None 2024-04-13T22:53:05.016 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000298d1319338dfe24-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.019 DEBUG:boto:path=/vwnfns-66/obj115 2024-04-13T22:53:05.019 DEBUG:boto:auth_path=/vwnfns-66/obj115 2024-04-13T22:53:05.019 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.019 DEBUG:boto:Path: /vwnfns-66/obj115 2024-04-13T22:53:05.019 DEBUG:boto:Data: 2024-04-13T22:53:05.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.019 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.019 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.019 DEBUG:boto:Params: {} 2024-04-13T22:53:05.020 DEBUG:boto:Token: None 2024-04-13T22:53:05.020 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj115 2024-04-13T22:53:05.020 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:FzvS69U1MASxo4SchS5TV6D37HM= 2024-04-13T22:53:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FzvS69U1MASxo4SchS5TV6D37HM='} 2024-04-13T22:53:05.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003845caa07add70e9-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.025 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:05.076 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:05.076 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:05.125 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:53:05.125 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-66 2024-04-13T22:53:05.132 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 20 2024-04-13T22:53:05.184 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.184 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.184 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.184 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.184 DEBUG:boto:Data: 2024-04-13T22:53:05.184 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.184 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.184 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.184 DEBUG:boto:Params: {} 2024-04-13T22:53:05.185 DEBUG:boto:Token: None 2024-04-13T22:53:05.185 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.185 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a07d9a3aaa548c3-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.189 DEBUG:boto:path=/vwnfns-66/obj29 2024-04-13T22:53:05.189 DEBUG:boto:auth_path=/vwnfns-66/obj29 2024-04-13T22:53:05.189 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.189 DEBUG:boto:Path: /vwnfns-66/obj29 2024-04-13T22:53:05.189 DEBUG:boto:Data: 2024-04-13T22:53:05.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.189 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.189 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.190 DEBUG:boto:Params: {} 2024-04-13T22:53:05.190 DEBUG:boto:Token: None 2024-04-13T22:53:05.190 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj29 2024-04-13T22:53:05.190 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NEnT98IxW89zvnG9CEvZEmeGfqg= 2024-04-13T22:53:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NEnT98IxW89zvnG9CEvZEmeGfqg='} 2024-04-13T22:53:05.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f00af851005d79b-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.239 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.239 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.239 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.239 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.239 DEBUG:boto:Data: 2024-04-13T22:53:05.239 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.239 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.239 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.239 DEBUG:boto:Params: {} 2024-04-13T22:53:05.240 DEBUG:boto:Token: None 2024-04-13T22:53:05.240 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.240 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000152f16c938460a20-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.243 DEBUG:boto:path=/vwnfns-66/obj58 2024-04-13T22:53:05.243 DEBUG:boto:auth_path=/vwnfns-66/obj58 2024-04-13T22:53:05.243 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.243 DEBUG:boto:Path: /vwnfns-66/obj58 2024-04-13T22:53:05.243 DEBUG:boto:Data: 2024-04-13T22:53:05.243 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.243 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.243 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.243 DEBUG:boto:Params: {} 2024-04-13T22:53:05.244 DEBUG:boto:Token: None 2024-04-13T22:53:05.244 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj58 2024-04-13T22:53:05.244 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:TCuFDiOiXDMG4GGvZisyLtEk0Is= 2024-04-13T22:53:05.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TCuFDiOiXDMG4GGvZisyLtEk0Is='} 2024-04-13T22:53:05.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e148bab5d694922e-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.249 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.249 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.249 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.250 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.250 DEBUG:boto:Data: 2024-04-13T22:53:05.250 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.250 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.250 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.250 DEBUG:boto:Params: {} 2024-04-13T22:53:05.250 DEBUG:boto:Token: None 2024-04-13T22:53:05.250 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.250 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be58b75eeb7d6ee5-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.253 DEBUG:boto:path=/vwnfns-66/obj87 2024-04-13T22:53:05.254 DEBUG:boto:auth_path=/vwnfns-66/obj87 2024-04-13T22:53:05.254 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.254 DEBUG:boto:Path: /vwnfns-66/obj87 2024-04-13T22:53:05.254 DEBUG:boto:Data: 2024-04-13T22:53:05.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.254 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.254 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.254 DEBUG:boto:Params: {} 2024-04-13T22:53:05.254 DEBUG:boto:Token: None 2024-04-13T22:53:05.254 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj87 2024-04-13T22:53:05.255 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9dJscfjC7aDipTQBz2L15omQaUk= 2024-04-13T22:53:05.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9dJscfjC7aDipTQBz2L15omQaUk='} 2024-04-13T22:53:05.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a5393c1e0c765eb-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.298 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.299 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.299 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.299 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.299 DEBUG:boto:Data: 2024-04-13T22:53:05.299 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.299 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.299 DEBUG:boto:Params: {} 2024-04-13T22:53:05.299 DEBUG:boto:Token: None 2024-04-13T22:53:05.299 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000223fca108b102d-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.303 DEBUG:boto:path=/vwnfns-66/obj116 2024-04-13T22:53:05.303 DEBUG:boto:auth_path=/vwnfns-66/obj116 2024-04-13T22:53:05.303 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.303 DEBUG:boto:Path: /vwnfns-66/obj116 2024-04-13T22:53:05.303 DEBUG:boto:Data: 2024-04-13T22:53:05.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.304 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.304 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.304 DEBUG:boto:Params: {} 2024-04-13T22:53:05.304 DEBUG:boto:Token: None 2024-04-13T22:53:05.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj116 2024-04-13T22:53:05.304 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:vQRvluzu98ff1GFe5YoABoUe2/c= 2024-04-13T22:53:05.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vQRvluzu98ff1GFe5YoABoUe2/c='} 2024-04-13T22:53:05.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075400ce5ab1f2cff-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.309 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.309 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.309 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.310 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.310 DEBUG:boto:Data: 2024-04-13T22:53:05.310 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.310 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.310 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.310 DEBUG:boto:Params: {} 2024-04-13T22:53:05.310 DEBUG:boto:Token: None 2024-04-13T22:53:05.310 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.311 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a842f5d2e728c1c6-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.314 DEBUG:boto:path=/vwnfns-66/obj145 2024-04-13T22:53:05.314 DEBUG:boto:auth_path=/vwnfns-66/obj145 2024-04-13T22:53:05.314 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.314 DEBUG:boto:Path: /vwnfns-66/obj145 2024-04-13T22:53:05.314 DEBUG:boto:Data: 2024-04-13T22:53:05.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.314 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.314 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.314 DEBUG:boto:Params: {} 2024-04-13T22:53:05.315 DEBUG:boto:Token: None 2024-04-13T22:53:05.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj145 2024-04-13T22:53:05.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eCiQ5B6hcoN3ee+SWkbKBChgX5A= 2024-04-13T22:53:05.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eCiQ5B6hcoN3ee+SWkbKBChgX5A='} 2024-04-13T22:53:05.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a67b50a139fc5ce-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.359 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:05.410 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:05.410 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:05.462 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:53:05.462 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-66 2024-04-13T22:53:05.474 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 25 2024-04-13T22:53:05.534 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.534 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.534 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.534 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.534 DEBUG:boto:Data: 2024-04-13T22:53:05.534 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.534 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.535 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.535 DEBUG:boto:Params: {} 2024-04-13T22:53:05.535 DEBUG:boto:Token: None 2024-04-13T22:53:05.535 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.535 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85a127b41820905-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.539 DEBUG:boto:path=/vwnfns-66/obj31 2024-04-13T22:53:05.539 DEBUG:boto:auth_path=/vwnfns-66/obj31 2024-04-13T22:53:05.539 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.539 DEBUG:boto:Path: /vwnfns-66/obj31 2024-04-13T22:53:05.539 DEBUG:boto:Data: 2024-04-13T22:53:05.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.539 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.539 DEBUG:boto:Params: {} 2024-04-13T22:53:05.540 DEBUG:boto:Token: None 2024-04-13T22:53:05.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj31 2024-04-13T22:53:05.540 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:736ACA/fLgQIUgTq5AHENfrPsQw= 2024-04-13T22:53:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:736ACA/fLgQIUgTq5AHENfrPsQw='} 2024-04-13T22:53:05.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1ab8b1183c3e12c-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.545 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.545 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.545 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.546 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.546 DEBUG:boto:Data: 2024-04-13T22:53:05.546 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.546 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.546 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.546 DEBUG:boto:Params: {} 2024-04-13T22:53:05.546 DEBUG:boto:Token: None 2024-04-13T22:53:05.546 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.546 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8d562451efbc9f7-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.550 DEBUG:boto:path=/vwnfns-66/obj62 2024-04-13T22:53:05.550 DEBUG:boto:auth_path=/vwnfns-66/obj62 2024-04-13T22:53:05.550 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.550 DEBUG:boto:Path: /vwnfns-66/obj62 2024-04-13T22:53:05.550 DEBUG:boto:Data: 2024-04-13T22:53:05.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.551 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.551 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.551 DEBUG:boto:Params: {} 2024-04-13T22:53:05.551 DEBUG:boto:Token: None 2024-04-13T22:53:05.551 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj62 2024-04-13T22:53:05.551 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iT5OyyNbM1Sw8S8cViTT95PoKwQ= 2024-04-13T22:53:05.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iT5OyyNbM1Sw8S8cViTT95PoKwQ='} 2024-04-13T22:53:05.556 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dcd27eeb8bd48b7-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.556 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.556 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.557 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.557 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.557 DEBUG:boto:Data: 2024-04-13T22:53:05.557 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.557 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.557 DEBUG:boto:Params: {} 2024-04-13T22:53:05.557 DEBUG:boto:Token: None 2024-04-13T22:53:05.557 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.558 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092a5a6c9a83db4b5-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.561 DEBUG:boto:path=/vwnfns-66/obj93 2024-04-13T22:53:05.561 DEBUG:boto:auth_path=/vwnfns-66/obj93 2024-04-13T22:53:05.561 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.561 DEBUG:boto:Path: /vwnfns-66/obj93 2024-04-13T22:53:05.561 DEBUG:boto:Data: 2024-04-13T22:53:05.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.561 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.562 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.562 DEBUG:boto:Params: {} 2024-04-13T22:53:05.562 DEBUG:boto:Token: None 2024-04-13T22:53:05.562 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj93 2024-04-13T22:53:05.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:A5KsK04X+7RZUEyZyECtWNRDftE= 2024-04-13T22:53:05.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A5KsK04X+7RZUEyZyECtWNRDftE='} 2024-04-13T22:53:05.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ceabf06bb9c8ece-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.606 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.607 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.607 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.607 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.607 DEBUG:boto:Data: 2024-04-13T22:53:05.607 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.607 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.607 DEBUG:boto:Params: {} 2024-04-13T22:53:05.607 DEBUG:boto:Token: None 2024-04-13T22:53:05.607 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.608 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020d8ebe6ff6c0ac3-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.611 DEBUG:boto:path=/vwnfns-66/obj124 2024-04-13T22:53:05.611 DEBUG:boto:auth_path=/vwnfns-66/obj124 2024-04-13T22:53:05.611 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.611 DEBUG:boto:Path: /vwnfns-66/obj124 2024-04-13T22:53:05.611 DEBUG:boto:Data: 2024-04-13T22:53:05.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.612 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.612 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.612 DEBUG:boto:Params: {} 2024-04-13T22:53:05.612 DEBUG:boto:Token: None 2024-04-13T22:53:05.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj124 2024-04-13T22:53:05.613 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2VF0bWtuvlTOCJWBx7ywbzF4ZZA= 2024-04-13T22:53:05.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2VF0bWtuvlTOCJWBx7ywbzF4ZZA='} 2024-04-13T22:53:05.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd59dfc839821f77-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.618 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.618 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.618 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.618 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.618 DEBUG:boto:Data: 2024-04-13T22:53:05.618 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.618 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.618 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.618 DEBUG:boto:Params: {} 2024-04-13T22:53:05.619 DEBUG:boto:Token: None 2024-04-13T22:53:05.619 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.619 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ada50197a1a9e40-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.622 DEBUG:boto:path=/vwnfns-66/obj155 2024-04-13T22:53:05.622 DEBUG:boto:auth_path=/vwnfns-66/obj155 2024-04-13T22:53:05.622 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.622 DEBUG:boto:Path: /vwnfns-66/obj155 2024-04-13T22:53:05.623 DEBUG:boto:Data: 2024-04-13T22:53:05.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.623 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.623 DEBUG:boto:Params: {} 2024-04-13T22:53:05.623 DEBUG:boto:Token: None 2024-04-13T22:53:05.623 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj155 2024-04-13T22:53:05.623 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hyYc/OXyvDUoPhzndLItZZbhtIw= 2024-04-13T22:53:05.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hyYc/OXyvDUoPhzndLItZZbhtIw='} 2024-04-13T22:53:05.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee84af3458064978-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.667 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:05.718 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:05.718 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:05.770 INFO:teuthology.orchestra.run.smithi132.stdout:tenant: 2024-04-13T22:53:05.770 INFO:teuthology.orchestra.run.smithi132.stdout:bucket name: vwnfns-66 2024-04-13T22:53:05.781 INFO:teuthology.orchestra.run.smithi132.stdout:total entries: 30 2024-04-13T22:53:05.843 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.843 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.843 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.843 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.843 DEBUG:boto:Data: 2024-04-13T22:53:05.843 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.844 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.844 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.844 DEBUG:boto:Params: {} 2024-04-13T22:53:05.844 DEBUG:boto:Token: None 2024-04-13T22:53:05.844 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.844 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8043e81bd3a160e-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.848 DEBUG:boto:path=/vwnfns-66/obj37 2024-04-13T22:53:05.848 DEBUG:boto:auth_path=/vwnfns-66/obj37 2024-04-13T22:53:05.848 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.848 DEBUG:boto:Path: /vwnfns-66/obj37 2024-04-13T22:53:05.849 DEBUG:boto:Data: 2024-04-13T22:53:05.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.849 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.849 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.849 DEBUG:boto:Params: {} 2024-04-13T22:53:05.849 DEBUG:boto:Token: None 2024-04-13T22:53:05.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj37 2024-04-13T22:53:05.849 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:HdsBJTG++tgjNFORUPEl3xpIBK4= 2024-04-13T22:53:05.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HdsBJTG++tgjNFORUPEl3xpIBK4='} 2024-04-13T22:53:05.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9df37f5942c2d05-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.855 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.855 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.855 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.855 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.855 DEBUG:boto:Data: 2024-04-13T22:53:05.855 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.855 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.855 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.855 DEBUG:boto:Params: {} 2024-04-13T22:53:05.855 DEBUG:boto:Token: None 2024-04-13T22:53:05.856 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.856 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000860261f5f4d3881c-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.859 DEBUG:boto:path=/vwnfns-66/obj74 2024-04-13T22:53:05.859 DEBUG:boto:auth_path=/vwnfns-66/obj74 2024-04-13T22:53:05.859 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.859 DEBUG:boto:Path: /vwnfns-66/obj74 2024-04-13T22:53:05.860 DEBUG:boto:Data: 2024-04-13T22:53:05.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.860 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.860 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.860 DEBUG:boto:Params: {} 2024-04-13T22:53:05.860 DEBUG:boto:Token: None 2024-04-13T22:53:05.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj74 2024-04-13T22:53:05.860 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:wYVPsi5lI43dd8r/ILMAJ2/Z6go= 2024-04-13T22:53:05.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wYVPsi5lI43dd8r/ILMAJ2/Z6go='} 2024-04-13T22:53:05.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005551ccdd057bacfe-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.903 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.903 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.903 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.903 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.903 DEBUG:boto:Data: 2024-04-13T22:53:05.903 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.903 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.903 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.903 DEBUG:boto:Params: {} 2024-04-13T22:53:05.904 DEBUG:boto:Token: None 2024-04-13T22:53:05.904 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.904 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d845acc62ba9d31-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.908 DEBUG:boto:path=/vwnfns-66/obj111 2024-04-13T22:53:05.908 DEBUG:boto:auth_path=/vwnfns-66/obj111 2024-04-13T22:53:05.908 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.908 DEBUG:boto:Path: /vwnfns-66/obj111 2024-04-13T22:53:05.908 DEBUG:boto:Data: 2024-04-13T22:53:05.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.908 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.908 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.908 DEBUG:boto:Params: {} 2024-04-13T22:53:05.908 DEBUG:boto:Token: None 2024-04-13T22:53:05.909 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj111 2024-04-13T22:53:05.909 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DYvDZnoBX+pNCHDC2pKvhZgVQjg= 2024-04-13T22:53:05.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DYvDZnoBX+pNCHDC2pKvhZgVQjg='} 2024-04-13T22:53:05.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001636d8347c7db51c-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.915 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.915 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.915 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.915 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.915 DEBUG:boto:Data: 2024-04-13T22:53:05.915 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.915 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.915 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.916 DEBUG:boto:Params: {} 2024-04-13T22:53:05.916 DEBUG:boto:Token: None 2024-04-13T22:53:05.916 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.916 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050704e233873d838-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.919 DEBUG:boto:path=/vwnfns-66/obj148 2024-04-13T22:53:05.919 DEBUG:boto:auth_path=/vwnfns-66/obj148 2024-04-13T22:53:05.920 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.920 DEBUG:boto:Path: /vwnfns-66/obj148 2024-04-13T22:53:05.920 DEBUG:boto:Data: 2024-04-13T22:53:05.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.920 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.920 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.920 DEBUG:boto:Params: {} 2024-04-13T22:53:05.920 DEBUG:boto:Token: None 2024-04-13T22:53:05.920 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj148 2024-04-13T22:53:05.920 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hZbbagdDsrjrAbBcgDeu3EI3ze8= 2024-04-13T22:53:05.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hZbbagdDsrjrAbBcgDeu3EI3ze8='} 2024-04-13T22:53:05.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5b51908322b5d88-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.962 DEBUG:boto:path=/vwnfns-66/ 2024-04-13T22:53:05.963 DEBUG:boto:auth_path=/vwnfns-66/ 2024-04-13T22:53:05.963 DEBUG:boto:Method: HEAD 2024-04-13T22:53:05.963 DEBUG:boto:Path: /vwnfns-66/ 2024-04-13T22:53:05.963 DEBUG:boto:Data: 2024-04-13T22:53:05.963 DEBUG:boto:Headers: {} 2024-04-13T22:53:05.963 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.963 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.963 DEBUG:boto:Params: {} 2024-04-13T22:53:05.963 DEBUG:boto:Token: None 2024-04-13T22:53:05.963 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/ 2024-04-13T22:53:05.963 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T22:53:05.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T22:53:05.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c6066b2486cceeb-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.967 DEBUG:boto:path=/vwnfns-66/obj185 2024-04-13T22:53:05.967 DEBUG:boto:auth_path=/vwnfns-66/obj185 2024-04-13T22:53:05.968 DEBUG:boto:Method: PUT 2024-04-13T22:53:05.968 DEBUG:boto:Path: /vwnfns-66/obj185 2024-04-13T22:53:05.968 DEBUG:boto:Data: 2024-04-13T22:53:05.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T22:53:05.968 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:53:05.968 DEBUG:boto:Port: 8000 2024-04-13T22:53:05.968 DEBUG:boto:Params: {} 2024-04-13T22:53:05.968 DEBUG:boto:Token: None 2024-04-13T22:53:05.968 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 13 Apr 2024 22:53:05 GMT /vwnfns-66/obj185 2024-04-13T22:53:05.968 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fnSuFQcYrketdfcOmYz8/pOYXnI= 2024-04-13T22:53:05.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fnSuFQcYrketdfcOmYz8/pOYXnI='} 2024-04-13T22:53:05.973 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6ba08cc444a9d5b-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:53:05.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:53:05.974 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:53:06.022 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:06.022 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:06.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:06.050+0000 7fef185c3780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-13T22:53:06.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:06.053+0000 7fef185c3780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-13T22:53:06.077 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-13T22:53:06.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T22:53:06.080 INFO:tasks.rgw.c1.client.1:Started 2024-04-13T22:53:06.080 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:53:06.099 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:53:06.100 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:53:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-13T22:53:06.101 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T22:53:07.102 DEBUG:teuthology.orchestra.run.smithi132:> curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T22:53:07.123 INFO:teuthology.orchestra.run.smithi132.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T22:53:07.123 INFO:teuthology.orchestra.run.smithi132.stderr: Dload Upload Total Spent Left Speed 2024-04-13T22:53:07.124 INFO:teuthology.orchestra.run.smithi132.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T22:53:07.125 INFO:teuthology.orchestra.run.smithi132.stdout:anonymous 2024-04-13T22:53:07.125 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=vwnfns-66 2024-04-13T22:53:07.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:53:07.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T22:53:07.125 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T22:53:07.219 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:53:07.219 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:53:07.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:07.258+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:08.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T22:53:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:12.259+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:17.260+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:22.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:22.261+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:27.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:27.262+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:32.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:32.263+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:37.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:37.262+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:42.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:42.263+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:47.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:47.264+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:52.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:52.264+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:53:57.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:53:57.266+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:02.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:02.266+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:07.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:07.267+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:12.268+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:17.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:17.268+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:22.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:22.268+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:27.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:27.269+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:32.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:32.270+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:37.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:37.271+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:42.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:42.272+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:47.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:47.273+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:52.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:52.273+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:54:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:54:57.273+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:55:02.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:02.274+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:55:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:07.274+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T22:55:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:12.275+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:17.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:17.277+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:22.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:22.277+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:27.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:27.278+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:32.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:32.278+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:37.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:37.279+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:42.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:42.279+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:47.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:47.279+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:52.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:52.280+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:55:57.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:55:57.281+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:02.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:02.281+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:07.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:07.282+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:12.282+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:17.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:17.282+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:22.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:22.283+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:27.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:27.284+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:32.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:32.285+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:37.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:37.286+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:42.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:42.287+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:47.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:47.287+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:52.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:52.287+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:56:57.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:56:57.288+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:02.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:02.289+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:07.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:07.289+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:12.291+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:17.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:17.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:22.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:22.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:27.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:27.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:32.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:32.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:37.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:37.293+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:42.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:42.294+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:47.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:47.295+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:52.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:52.296+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:57:57.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:57:57.296+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T22:58:02.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:02.297+0000 7fbf1ea8f780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-13T22:58:02.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:02.297+0000 7fbf1ea8f780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T22:58:02.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:02.297+0000 7fbf1ea8f780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T22:58:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-13T22:58:02.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-04-13T22:58:02.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:02.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:02.325 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:02.379 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.379 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:02.413 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:58:02.427 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:58:02.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.427 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:53:05.788346Z" 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.429 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:02.429 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:58:02.429 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:02.429 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:58:02.430 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:02.430 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:58:02.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:02.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:02.431 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:02.479 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.479 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:02.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:02.509+0000 7fe4b9dfb780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:02.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:02.510+0000 7fe4b9dfb780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:02.527 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:02.527 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:02.527 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:02.527 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:02.528 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:02.529 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:02.530 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:02.531 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:02.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.533 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:02.533 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:02.533 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:02.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:02.533 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:02Z" 2024-04-13T22:58:02.533 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:02.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:02Z" } 2024-04-13T22:58:02.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:02.534 DEBUG:rgw_multi.tests:log_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:02.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:02.534 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:58:02.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.535 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:02.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:02.636 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T22:58:02.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.637 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T22:58:02.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.735 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:02.832 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:02.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.833 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:02.879 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.879 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:02.926 INFO:teuthology.orchestra.run.smithi132.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-13T22:58:02.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:02.927 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:02.971 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:02.971 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:03.020 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:03.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.020 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:03.072 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:03.072 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:03.124 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:03.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.125 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:03.178 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:03.178 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:03.226 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:03.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.227 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:03.277 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:03.277 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:03.326 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T22:58:03.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.327 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:03.372 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:03.372 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:03.425 INFO:teuthology.orchestra.run.smithi132.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-13T22:58:03.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.426 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:03.472 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:03.472 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:03.527 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 13, 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.528 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:58:03.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:58:03.530 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:58:03.531 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:58:03.532 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T22:58:03.533 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T22:58:03.534 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T22:58:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: "directional": [ 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: "source_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: "dest_zone": "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T22:58:03.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T22:58:03.537 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T22:58:03.538 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T22:58:03.539 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "enabled" 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.540 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T22:58:03.541 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:58:03.541 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.541 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T22:58:03.542 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T22:58:03.542 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T22:58:03.542 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T22:58:03.542 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:58:03.542 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T22:58:03.543 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.543 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T22:58:03.543 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T22:58:03.543 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:58:03.543 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:58:03.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.544 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T22:58:03.544 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T22:58:03.544 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:58:03.544 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T22:58:03.545 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.545 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:58:03.545 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:58:03.545 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:58:03.545 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:58:03.546 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:58:03.546 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:58:03.546 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:58:03.546 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:58:03.546 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:58:03.547 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:58:03.547 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:58:03.547 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.548 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:58:03.548 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T22:58:03.548 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:58:03.548 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T22:58:03.548 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.549 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:58:03.549 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:58:03.549 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:58:03.549 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:58:03.549 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:58:03.550 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:58:03.550 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:58:03.550 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:58:03.550 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:58:03.550 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:58:03.550 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:58:03.551 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.551 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.551 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T22:58:03.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.551 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T22:58:03.552 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T22:58:03.552 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T22:58:03.552 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T22:58:03.552 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.553 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.553 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T22:58:03.553 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:58:03.553 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T22:58:03.554 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T22:58:03.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.554 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T22:58:03.556 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:58:03.556 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.556 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.557 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:58:03.557 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T22:58:03.557 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.557 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T22:58:03.558 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T22:58:03.558 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.558 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:58:03.558 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T22:58:03.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.559 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:58:03.559 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T22:58:03.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:03.560 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:58:03.560 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T22:58:03.560 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.561 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:03.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.561 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T22:58:03.561 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:58:03.561 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T22:58:03.562 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T22:58:03.562 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T22:58:03.562 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T22:58:03.562 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T22:58:03.563 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T22:58:03.563 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T22:58:03.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.563 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T22:58:03.563 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T22:58:03.564 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T22:58:03.564 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T22:58:03.564 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T22:58:03.564 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T22:58:03.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.565 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T22:58:03.565 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:58:03.565 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:58:03.565 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:58:03.565 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:58:03.566 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:58:03.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.566 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T22:58:03.566 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:58:03.567 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:58:03.567 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:58:03.567 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:58:03.567 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:58:03.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.568 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T22:58:03.568 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:58:03.568 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:58:03.568 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:58:03.569 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:58:03.569 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:58:03.569 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:03.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:03.569 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:58:03.570 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T22:58:03.570 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:03.570 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:03.571 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-67 2024-04-13T22:58:03.571 DEBUG:boto:path=/vwnfns-67/ 2024-04-13T22:58:03.571 DEBUG:boto:auth_path=/vwnfns-67/ 2024-04-13T22:58:03.572 DEBUG:boto:Method: PUT 2024-04-13T22:58:03.572 DEBUG:boto:Path: /vwnfns-67/ 2024-04-13T22:58:03.572 DEBUG:boto:Data: 2024-04-13T22:58:03.572 DEBUG:boto:Headers: {} 2024-04-13T22:58:03.572 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:58:03.573 DEBUG:boto:Port: 8000 2024-04-13T22:58:03.573 DEBUG:boto:Params: {} 2024-04-13T22:58:03.574 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:58:03.574 DEBUG:boto:Token: None 2024-04-13T22:58:03.574 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:58:03 GMT /vwnfns-67/ 2024-04-13T22:58:03.575 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uJyxblWbz/jhphh/UznDoPJ6mU0= 2024-04-13T22:58:03.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:58:03 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uJyxblWbz/jhphh/UznDoPJ6mU0=', 'Content-Length': '0'} 2024-04-13T22:58:03.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6e165727414577f-00661b0dfb-24c8-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:58:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:58:03.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:03.897 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:03.973 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:03.973 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.098 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T22:58:04.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.099 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.168 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.168 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.270 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:04.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.271 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.332 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.332 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.420 INFO:teuthology.orchestra.run.smithi132.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-13T22:58:04.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.421 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.468 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.468 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.521 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:04.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.522 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.575 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.576 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.626 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:04.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.627 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.742 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:58:04.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.743 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.855 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T22:58:04.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:58:04.856 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:58:04.903 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:04.903 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:04.957 INFO:teuthology.orchestra.run.smithi132.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-04-13T22:58:04.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:04.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:04.958 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:05.004 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:05.004 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:05.044 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:58:05.057 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:58:05.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.057 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.837717_1041.1", 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:58:04.837717Z" 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.059 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:05.059 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:58:05.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:05.059 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:58:05.060 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:05.060 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:58:05.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:05.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:05.060 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:05.107 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:05.108 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:05.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:05.134+0000 7f2f4cf22780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:05.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:05.135+0000 7f2f4cf22780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:05.150 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:05.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:05.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:05.153 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:05.154 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:05.155 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:05.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:05.157 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:05Z" 2024-04-13T22:58:05.157 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:05.159 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:05Z" } 2024-04-13T22:58:05.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:05.159 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:05.159 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:05.159 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:10.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:10.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:10.161 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:10.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:10.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:10.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:10.247+0000 7f1c7e750780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:10.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:10.248+0000 7f1c7e750780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:10.265 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:10.265 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:10.265 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:10.265 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:10.265 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:10.265 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:10.266 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:10.267 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:10.268 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:10.269 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:10Z" 2024-04-13T22:58:10.271 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:10.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:10Z" } 2024-04-13T22:58:10.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:10.272 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:10.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:10.272 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:15.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:15.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:15.274 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:15.330 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:15.370 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:15.367+0000 7f67766a7780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:15.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:15.368+0000 7f67766a7780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:15.388 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:15.389 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:15.391 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:15.392 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:15.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:15.394 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:15Z" 2024-04-13T22:58:15.394 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:15.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:15Z" } 2024-04-13T22:58:15.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:15.395 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:15.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:15.395 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:20.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:20.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:20.397 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:20.451 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:20.451 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:20.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:20.488+0000 7fc4d5b45780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:20.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:20.489+0000 7fc4d5b45780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:20.508 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:20.509 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:20.510 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:20.511 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:20.512 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:20Z" 2024-04-13T22:58:20.513 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:20.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:20Z" } 2024-04-13T22:58:20.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:20.515 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:20.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:20.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:25.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:25.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:25.517 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:25.570 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:25.570 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:25.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:25.601+0000 7f0e1cb69780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:25.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:25.602+0000 7f0e1cb69780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:25.618 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:25.619 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:25.620 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:25.621 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:25.622 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:25.623 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:25.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:25.624 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:25Z" 2024-04-13T22:58:25.624 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:25.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:25Z" } 2024-04-13T22:58:25.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:25.625 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:25.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:25.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:30.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:30.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:30.627 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:30.683 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:30.684 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:30.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:30.725+0000 7fefebbc7780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:30.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:30.727+0000 7fefebbc7780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:30.749 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:30.750 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:30.751 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:30.752 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:30.753 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:30.754 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:30.755 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:30Z" 2024-04-13T22:58:30.755 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:30.756 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:30Z" } 2024-04-13T22:58:30.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:30.756 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:30.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:30.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:35.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:35.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:35.758 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:35.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:35.814 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:35.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:35.848+0000 7f5f9d8ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:35.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:35.849+0000 7f5f9d8ca780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:35.870 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:35.870 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:35.870 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:35.870 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:35.871 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:35.872 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:35.873 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:35.874 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:35.875 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:35Z" 2024-04-13T22:58:35.876 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:35.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:35Z" } 2024-04-13T22:58:35.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:35.878 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:35.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:35.878 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:40.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:40.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:40.879 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:40.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:40.936 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:40.973+0000 7f3cfeee5780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:40.974+0000 7f3cfeee5780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:40.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:40.998 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:40.999 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:41.000 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:40Z" 2024-04-13T22:58:41.001 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:41.001 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:40Z" } 2024-04-13T22:58:41.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:41.002 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:41.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:41.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:46.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:46.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:46.004 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:46.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:46.091+0000 7faa55ca6780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:46.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:46.092+0000 7faa55ca6780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:46.109 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:46.109 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:46.109 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:46.109 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:46.110 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:46.111 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:46.112 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:46.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:46Z" 2024-04-13T22:58:46.115 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:46.116 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:46Z" } 2024-04-13T22:58:46.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:46.117 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:46.117 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:46.117 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:51.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:51.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:51.118 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:51.173 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:51.173 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:51.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:51.204+0000 7fde4aa4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:51.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:51.205+0000 7fde4aa4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:51.222 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:51.222 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:51.222 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:51.222 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:51.222 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:51.222 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:51.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:51.224 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:51.225 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:51.226 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:51.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:51.228 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:51.228 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:51.228 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:51.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:51.228 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:51Z" 2024-04-13T22:58:51.228 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:51.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:51Z" } 2024-04-13T22:58:51.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:51.230 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:51.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:51.230 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:58:56.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:56.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:58:56.231 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:58:56.287 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:58:56.287 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:58:56.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:56.324+0000 7f91079d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:56.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:58:56.325+0000 7f91079d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:58:56.342 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:56.343 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:56.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:56.345 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:56.346 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:58:56.347 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:58:56.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:58:56.349 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:58:56Z" 2024-04-13T22:58:56.349 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:58:56.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:58:56Z" } 2024-04-13T22:58:56.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:58:56.350 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:56.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:58:56.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:59:01.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:01.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:01.352 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:01.408 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:01.408 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:01.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:01.444+0000 7f8dbef35780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:01.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:01.445+0000 7f8dbef35780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:01.462 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:01.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:01.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:01.465 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:01.466 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:01.467 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:01Z" 2024-04-13T22:59:01.468 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:01.469 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:01Z" } 2024-04-13T22:59:01.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:01.470 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:01.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:01.470 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:59:06.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:06.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:06.472 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:06.525 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:06.526 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:06.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:06.564+0000 7f95e063f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:06.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:06.565+0000 7f95e063f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:06.583 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:06.584 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:06.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:06.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:06.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:06Z" 2024-04-13T22:59:06.588 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:06.589 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:06Z" } 2024-04-13T22:59:06.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:06.589 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:06.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:06.590 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:59:11.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:11.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:11.592 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:11.646 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:11.647 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:11.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:11.678+0000 7f4aca54e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:11.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:11.679+0000 7f4aca54e780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:11.695 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:11.695 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:11.695 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:11.695 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:11.695 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:11.695 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713048785.788346_1005.1", 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:11.696 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.788346Z", 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:11.697 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:11.698 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:11.699 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:11.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:11.701 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:11.701 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:11.701 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:11.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:11.701 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:11Z" 2024-04-13T22:59:11.701 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:11.702 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713048785.788346_1005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:53:05.788346Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:11Z" } 2024-04-13T22:59:11.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048785.788346_1005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.788346Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:11.702 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:11.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1713048785.788346_1005.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:11.703 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713048785.788346_1005.1 2024-04-13T22:59:16.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:16.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:16.704 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:16.758 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:16.759 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:16.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:16.801+0000 7fa37fdfd780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:16.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:16.802+0000 7fa37fdfd780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:16.822 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:16.822 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:16.822 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:16.822 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:16.822 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.607182_1036.1", 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:16.823 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.607182Z", 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:16.824 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:16.825 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:16.826 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:16.827 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:16.828 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:16.828 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:16.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:16.828 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:16Z" 2024-04-13T22:59:16.828 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:16.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.607182_1036.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.607182Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:16Z" } 2024-04-13T22:59:16.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.607182_1036.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.607182Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:16.829 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:16.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.607182_1036.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:16.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713049084.607182_1036.1 2024-04-13T22:59:21.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:21.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:21.831 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:21.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:21.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:21.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:21.921+0000 7efdd8f5f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:21.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:21.922+0000 7efdd8f5f780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:21.944 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:21.944 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:21.944 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:21.944 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:21.944 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.607182_1036.1", 2024-04-13T22:59:21.945 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.607182Z", 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:21.948 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:21.949 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:21Z" 2024-04-13T22:59:21.950 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:21.951 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.607182_1036.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.607182Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:21Z" } 2024-04-13T22:59:21.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.607182_1036.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.607182Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:21.951 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:21.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.607182_1036.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:21.952 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049084.837717_1041.1 target=1_1713049084.607182_1036.1 2024-04-13T22:59:26.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:26.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:26.953 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:27.008 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.008 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:27.042+0000 7f4049d8b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:27.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:27.043+0000 7f4049d8b780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:27.062 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.837717_1041.1", 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.063 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.837717Z", 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.064 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:27.065 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.066 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:27.066 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.066 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:27.066 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.067 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.067 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.068 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:27Z" 2024-04-13T22:59:27.069 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:27.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.837717_1041.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.837717Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:27Z" } 2024-04-13T22:59:27.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.837717_1041.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.837717Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:27.071 DEBUG:rgw_multi.tests:log_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:27.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:27.071 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:59:27.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.072 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:27.121 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.122 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.176 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:59:27.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.177 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:27.231 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.231 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.282 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:59:27.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.283 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:27.336 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.337 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.389 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T22:59:27.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.390 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:27.437 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.437 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.497 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T22:59:27.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'vwnfns-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:27.498 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket vwnfns-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:27.544 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.545 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.598 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T22:59:27.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:27.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:27.598 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:27.645 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.645 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.682 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:59:27.480760Z" 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.698 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:59:27.699 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:27.699 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:59:27.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:27.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:27.700 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:27.747 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:27.747 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:27.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:27.779+0000 7f9635b12780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:27.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:27.780+0000 7f9635b12780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:27.797 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:27.797 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:27.797 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.798 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.837717_1041.1", 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.837717Z", 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.799 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:27.800 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:27.801 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:27.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.803 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:27.803 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:27.803 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:27.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:27.803 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:27Z" 2024-04-13T22:59:27.803 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:27.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.837717_1041.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.837717Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:27Z" } 2024-04-13T22:59:27.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.837717_1041.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.837717Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:27.804 DEBUG:rgw_multi.tests:log_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:27.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:27.805 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049167.480760_1049.1 target=1_1713049084.837717_1041.1 2024-04-13T22:59:32.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:32.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:32.807 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:32.862 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:32.862 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:32.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:32.893+0000 7fd5e1e79780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:32.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:32.894+0000 7fd5e1e79780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:32.911 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:32.911 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:32.911 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:32.911 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:32.911 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:32.911 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.837717_1041.1", 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:32.912 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.837717Z", 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:32.913 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:32.914 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:32.915 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:32.916 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:32.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:32.917 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:32Z" 2024-04-13T22:59:32.917 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:32.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.837717_1041.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.837717Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:32Z" } 2024-04-13T22:59:32.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.837717_1041.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.837717Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:32.918 DEBUG:rgw_multi.tests:log_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:32.918 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:32.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049167.480760_1049.1 target=1_1713049084.837717_1041.1 2024-04-13T22:59:37.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:37.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:37.920 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:37.975 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:37.975 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:38.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:38.016+0000 7fc92c90a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:38.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:38.017+0000 7fc92c90a780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:38.034 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.837717_1041.1", 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:38.035 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.837717Z", 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:38.036 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:38.037 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:38.038 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:38.039 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:38.040 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:38.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:38.040 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:38Z" 2024-04-13T22:59:38.040 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:38.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.837717_1041.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.837717Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:38Z" } 2024-04-13T22:59:38.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.837717_1041.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.837717Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:38.041 DEBUG:rgw_multi.tests:log_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:38.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:38.041 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049167.480760_1049.1 target=1_1713049084.837717_1041.1 2024-04-13T22:59:43.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:43.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:43.043 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:43.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:43.098 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:43.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:43.132+0000 7f7597141780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:43.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:43.133+0000 7f7597141780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:43.149 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049084.837717_1041.1", 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:58:04.837717Z", 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:43.150 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:43.151 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:43.152 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:43.153 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:43Z" 2024-04-13T22:59:43.154 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:43.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049084.837717_1041.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:58:04.837717Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:43Z" } 2024-04-13T22:59:43.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049084.837717_1041.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:58:04.837717Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:43.156 DEBUG:rgw_multi.tests:log_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:43.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049084.837717_1041.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:43.156 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049167.480760_1049.1 target=1_1713049084.837717_1041.1 2024-04-13T22:59:48.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:48.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:48.158 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:48.212 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.212 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:48.248+0000 7fdaf3886780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:48.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:48.249+0000 7fdaf3886780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.265 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T22:59:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:48.267 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:48.268 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:48.269 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:48.270 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:48.271 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:48.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.271 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:48Z" 2024-04-13T22:59:48.271 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:48.272 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:48Z" } 2024-04-13T22:59:48.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:48.273 DEBUG:rgw_multi.tests:log_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:48.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:48.273 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:59:48.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.273 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:48.323 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.323 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.369 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:59:48.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.370 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:48.423 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.423 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.470 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-13T22:59:48.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.471 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:48.522 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.522 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.574 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T22:59:48.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.575 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:48.620 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.620 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.669 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T22:59:48.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.670 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:48.715 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.715 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.761 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T22:59:48.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:48.762 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:48.809 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.809 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.870 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 14, 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.871 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:48.873 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:48.874 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:48.875 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T22:59:48.876 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T22:59:48.877 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.878 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:48.879 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:48.880 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T22:59:48.886 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:48.887 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T22:59:48.888 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T22:59:48.889 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.890 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T22:59:48.891 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T22:59:48.892 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:59:48.893 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:59:48.894 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T22:59:48.895 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:48.896 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:48.897 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-04-13T22:59:48.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:48.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:48.898 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:48.945 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:48.945 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:48.982 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:59:48.998 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:59:48.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.998 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T22:59:48.998 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:59:27.480760Z" 2024-04-13T22:59:48.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:59:48.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:49.000 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:59:49.000 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:49.001 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:59:49.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:49.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:49.001 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:49.058 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.058 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:49.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:49.104+0000 7f87a5f80780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:49.105+0000 7f87a5f80780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:49.138 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:49Z" 2024-04-13T22:59:49.144 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:49.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:49Z" } 2024-04-13T22:59:49.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:49.144 DEBUG:rgw_multi.tests:log_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:49.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:49.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T22:59:49.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.145 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:49.205 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.206 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:49.359 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T22:59:49.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.360 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:49.435 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.435 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:49.531 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T22:59:49.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.531 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:49.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:49.675 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T22:59:49.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.676 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:49.738 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.738 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:49.805 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T22:59:49.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.806 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:49.854 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.854 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:49.901 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T22:59:49.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:49.901 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:49.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:49.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:50.004 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 15, 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T22:59:50.005 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:50.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:50.007 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T22:59:50.008 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:50.009 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T22:59:50.010 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T22:59:50.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.012 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:50.013 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T22:59:50.014 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "enabled" 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:50.015 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T22:59:50.016 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:50.017 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T22:59:50.019 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.021 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.022 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T22:59:50.024 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T22:59:50.025 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:59:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T22:59:50.027 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:50.028 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:50.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:50.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T22:59:50.030 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T22:59:50.073 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:50.074 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:50.125 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T22:59:50.125 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-68 2024-04-13T22:59:50.125 DEBUG:boto:path=/vwnfns-68/ 2024-04-13T22:59:50.125 DEBUG:boto:auth_path=/vwnfns-68/ 2024-04-13T22:59:50.126 DEBUG:boto:Method: PUT 2024-04-13T22:59:50.126 DEBUG:boto:Path: /vwnfns-68/ 2024-04-13T22:59:50.126 DEBUG:boto:Data: 2024-04-13T22:59:50.126 DEBUG:boto:Headers: {} 2024-04-13T22:59:50.126 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:59:50.126 DEBUG:boto:Port: 8000 2024-04-13T22:59:50.126 DEBUG:boto:Params: {} 2024-04-13T22:59:50.127 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T22:59:50.127 DEBUG:boto:Token: None 2024-04-13T22:59:50.127 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:59:50 GMT /vwnfns-68/ 2024-04-13T22:59:50.127 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:d89plTwXN/1CxgHcsiPrVSVxMCk= 2024-04-13T22:59:50.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:59:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:d89plTwXN/1CxgHcsiPrVSVxMCk=', 'Content-Length': '0'} 2024-04-13T22:59:50.379 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d31d02e4dee4820-00661b0e66-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:59:50.379 DEBUG:boto:path=/vwnfns-68/ 2024-04-13T22:59:50.379 DEBUG:boto:auth_path=/vwnfns-68/ 2024-04-13T22:59:50.380 DEBUG:boto:Method: HEAD 2024-04-13T22:59:50.380 DEBUG:boto:Path: /vwnfns-68/ 2024-04-13T22:59:50.380 DEBUG:boto:Data: 2024-04-13T22:59:50.380 DEBUG:boto:Headers: {} 2024-04-13T22:59:50.380 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:59:50.380 DEBUG:boto:Port: 8000 2024-04-13T22:59:50.380 DEBUG:boto:Params: {} 2024-04-13T22:59:50.380 DEBUG:boto:Token: None 2024-04-13T22:59:50.380 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:59:50 GMT /vwnfns-68/ 2024-04-13T22:59:50.381 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1ub8zpr7DOPET1vvQI6vI4CawrM= 2024-04-13T22:59:50.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:59:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1ub8zpr7DOPET1vvQI6vI4CawrM=', 'Content-Length': '0'} 2024-04-13T22:59:50.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba972555c7ce9971-00661b0e66-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:59:50.386 DEBUG:boto:path=/vwnfns-68/obj1 2024-04-13T22:59:50.386 DEBUG:boto:auth_path=/vwnfns-68/obj1 2024-04-13T22:59:50.387 DEBUG:boto:Method: PUT 2024-04-13T22:59:50.387 DEBUG:boto:Path: /vwnfns-68/obj1 2024-04-13T22:59:50.387 DEBUG:boto:Data: 2024-04-13T22:59:50.387 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:59:50.387 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T22:59:50.387 DEBUG:boto:Port: 8000 2024-04-13T22:59:50.387 DEBUG:boto:Params: {} 2024-04-13T22:59:50.387 DEBUG:boto:Token: None 2024-04-13T22:59:50.387 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:59:50 GMT /vwnfns-68/obj1 2024-04-13T22:59:50.388 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:F4GmGQ+NNBhvMjnRt0/0VlBKms4= 2024-04-13T22:59:50.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:59:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F4GmGQ+NNBhvMjnRt0/0VlBKms4='} 2024-04-13T22:59:50.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d75ca69baa0d6a7d-00661b0e66-2678-a1'), ('Date', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:59:50.406 INFO:rgw_multi.tests:create bucket zone=a2 name=vwnfns-69 2024-04-13T22:59:50.406 DEBUG:boto:path=/vwnfns-69/ 2024-04-13T22:59:50.407 DEBUG:boto:auth_path=/vwnfns-69/ 2024-04-13T22:59:50.407 DEBUG:boto:Method: PUT 2024-04-13T22:59:50.407 DEBUG:boto:Path: /vwnfns-69/ 2024-04-13T22:59:50.407 DEBUG:boto:Data: 2024-04-13T22:59:50.407 DEBUG:boto:Headers: {} 2024-04-13T22:59:50.407 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:59:50.407 DEBUG:boto:Port: 8001 2024-04-13T22:59:50.407 DEBUG:boto:Params: {} 2024-04-13T22:59:50.408 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T22:59:50.408 DEBUG:boto:Token: None 2024-04-13T22:59:50.408 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 22:59:50 GMT /vwnfns-69/ 2024-04-13T22:59:50.408 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:UWw53vXXkMOiAQO3vNYq7tSEn8Q= 2024-04-13T22:59:50.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:59:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UWw53vXXkMOiAQO3vNYq7tSEn8Q=', 'Content-Length': '0'} 2024-04-13T22:59:50.453 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a9541a6efbdbbc7-00661b0e66-2674-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:59:50.453 DEBUG:boto:path=/vwnfns-69/ 2024-04-13T22:59:50.453 DEBUG:boto:auth_path=/vwnfns-69/ 2024-04-13T22:59:50.453 DEBUG:boto:Method: HEAD 2024-04-13T22:59:50.453 DEBUG:boto:Path: /vwnfns-69/ 2024-04-13T22:59:50.453 DEBUG:boto:Data: 2024-04-13T22:59:50.453 DEBUG:boto:Headers: {} 2024-04-13T22:59:50.453 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:59:50.454 DEBUG:boto:Port: 8001 2024-04-13T22:59:50.454 DEBUG:boto:Params: {} 2024-04-13T22:59:50.454 DEBUG:boto:Token: None 2024-04-13T22:59:50.454 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 22:59:50 GMT /vwnfns-69/ 2024-04-13T22:59:50.454 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sPVdJLLvQpI6WnvZ4AqbezF/RB4= 2024-04-13T22:59:50.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:59:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sPVdJLLvQpI6WnvZ4AqbezF/RB4=', 'Content-Length': '0'} 2024-04-13T22:59:50.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae848b2e458b857e-00661b0e66-2674-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:59:50.458 DEBUG:boto:path=/vwnfns-69/obj2 2024-04-13T22:59:50.458 DEBUG:boto:auth_path=/vwnfns-69/obj2 2024-04-13T22:59:50.458 DEBUG:boto:Method: PUT 2024-04-13T22:59:50.458 DEBUG:boto:Path: /vwnfns-69/obj2 2024-04-13T22:59:50.458 DEBUG:boto:Data: 2024-04-13T22:59:50.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T22:59:50.458 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T22:59:50.458 DEBUG:boto:Port: 8001 2024-04-13T22:59:50.458 DEBUG:boto:Params: {} 2024-04-13T22:59:50.459 DEBUG:boto:Token: None 2024-04-13T22:59:50.459 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 22:59:50 GMT /vwnfns-69/obj2 2024-04-13T22:59:50.459 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:57r/tfFwivjizuKWrlM2kHcfdc0= 2024-04-13T22:59:50.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:59:50 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:57r/tfFwivjizuKWrlM2kHcfdc0='} 2024-04-13T22:59:50.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c05afc0429463af3-00661b0e66-2674-a2'), ('Date', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T22:59:50.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:50.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:50.474 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:50.537 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:50.538 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:50.590 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T22:59:50.613 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T22:59:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049190.433608_1085.1", 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:59:50.433608Z" 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.615 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T22:59:50.616 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:50.616 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T22:59:50.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:50.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:50.617 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:50.674 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:50.675 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:50.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:50.718+0000 7fb984ca0780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:50.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:50.719+0000 7fb984ca0780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:50.741 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.742 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:50.744 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:50.745 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:50.746 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:50.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:50.748 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:50Z" 2024-04-13T22:59:50.748 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:50.749 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:50Z" } 2024-04-13T22:59:50.749 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:50.749 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:50.749 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:50.750 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T22:59:55.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:55.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T22:59:55.751 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T22:59:55.807 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T22:59:55.807 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T22:59:55.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:55.837+0000 7f0fb9c82780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:55.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T22:59:55.839+0000 7f0fb9c82780 20 RGW-SYNC:meta: read sync status 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T22:59:55.855 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:55.856 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:55.857 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:55.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:55.859 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T22:59:55.860 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T22:59:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T22:59:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T22:59:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T22:59:55.862 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T22:59:55Z" 2024-04-13T22:59:55.862 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T22:59:55.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T22:59:55Z" } 2024-04-13T22:59:55.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T22:59:55.863 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:55.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T22:59:55.864 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:00.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:00.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:00.865 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:00.921 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:00.921 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:00.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:00.952+0000 7f64ef95f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:00.953+0000 7f64ef95f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:00.974 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:00.976 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:00.977 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:00.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:00.979 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:00Z" 2024-04-13T23:00:00.979 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:00.980 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:00Z" } 2024-04-13T23:00:00.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:00.980 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:00.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:00.980 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:05.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:05.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:05.982 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:06.037 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:06.038 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:06.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:06.077+0000 7f085a3e4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:06.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:06.078+0000 7f085a3e4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:06.099 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:06.100 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:06.101 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:06.102 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:06.103 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:06.104 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:06.105 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:06.105 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:06.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:06.105 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:06Z" 2024-04-13T23:00:06.105 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:06.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:06Z" } 2024-04-13T23:00:06.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:06.109 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:06.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:06.110 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:11.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:11.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:11.111 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:11.168 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:11.168 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:11.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:11.198+0000 7f294e2cc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:11.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:11.200+0000 7f294e2cc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:11.217 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:11.218 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:11.219 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:11.220 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:11.221 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:11.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:11.223 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:11.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:11.223 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:11.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:11.223 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:11Z" 2024-04-13T23:00:11.223 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:11.224 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:11Z" } 2024-04-13T23:00:11.224 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:11.224 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:11.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:11.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:16.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:16.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:16.227 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:16.284 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:16.284 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:16.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:16.321+0000 7fc1bf6e3780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:16.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:16.322+0000 7fc1bf6e3780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:16.347 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:16.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:16.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:16.351 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:16.352 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:16.353 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:16.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:16.353 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:16Z" 2024-04-13T23:00:16.353 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:16.354 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:16Z" } 2024-04-13T23:00:16.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:16.354 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:16.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:16.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:21.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:21.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:21.356 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:21.412 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:21.413 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:21.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:21.443+0000 7f5e7c0a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:21.444+0000 7f5e7c0a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:21.460 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:21.460 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:21.460 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:21.460 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:21.460 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:21.460 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:21.461 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:21.462 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:21.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:21.464 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:21.465 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:21.466 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:21.467 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:21.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:21.467 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:21Z" 2024-04-13T23:00:21.467 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:21.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:21Z" } 2024-04-13T23:00:21.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:21.468 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:21.469 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:21.469 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:26.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:26.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:26.471 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:26.526 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:26.526 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:26.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:26.558+0000 7f98cbdfc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:26.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:26.559+0000 7f98cbdfc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:26.575 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:26.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:26.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:26.579 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:26Z" 2024-04-13T23:00:26.580 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:26.581 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:26Z" } 2024-04-13T23:00:26.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:26.581 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:26.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:26.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:31.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:31.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:31.583 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:31.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:31.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:31.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:31.676+0000 7faa9b39a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:31.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:31.677+0000 7faa9b39a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:31.694 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:31.694 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:31.694 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:31.694 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:31.694 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:31.695 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:31.696 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:31.697 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:31.698 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:31.699 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:31.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:31.701 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:31Z" 2024-04-13T23:00:31.701 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:31.702 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:31Z" } 2024-04-13T23:00:31.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:31.703 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:31.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:31.703 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:36.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:36.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:36.704 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:36.758 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:36.758 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:36.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:36.788+0000 7f74fc2b8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:36.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:36.789+0000 7f74fc2b8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:36.805 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:36.805 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:36.806 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:36.807 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:36.808 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:36.809 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:36.810 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:36.811 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:36Z" 2024-04-13T23:00:36.811 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:36.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:36Z" } 2024-04-13T23:00:36.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:36.812 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:36.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:36.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:41.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:41.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:41.814 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:41.870 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:41.870 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:41.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:41.911+0000 7f0e03dd2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:41.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:41.912+0000 7f0e03dd2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:41.932 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:41.932 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:41.932 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:41.933 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:41.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:41.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:41.936 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:41Z" 2024-04-13T23:00:41.939 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:41.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:41Z" } 2024-04-13T23:00:41.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:41.941 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:41.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:41.941 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:46.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:46.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:46.942 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:46.997 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:46.998 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:47.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:47.031+0000 7f7a4cbfc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:47.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:47.032+0000 7f7a4cbfc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:47.049 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:47.049 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:47.049 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:47.049 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:47.049 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:47.050 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:47.051 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:47.052 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:47.053 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:47.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:47.057 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:47.057 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:47.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:47.057 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:47.057 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:47.057 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:47.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:47.058 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:47Z" 2024-04-13T23:00:47.058 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:47.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:47Z" } 2024-04-13T23:00:47.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:47.059 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:47.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:47.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:52.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:52.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:52.061 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:52.116 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:52.116 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:52.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:52.152+0000 7f22f07e4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:52.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:52.153+0000 7f22f07e4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:52.173 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:52.173 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:52.173 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:52.173 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:52.174 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:52.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:52.177 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:52Z" 2024-04-13T23:00:52.179 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:52.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:52Z" } 2024-04-13T23:00:52.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:52.181 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:52.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:52.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:00:57.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:57.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:00:57.183 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:00:57.237 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:00:57.237 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:00:57.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:57.277+0000 7fcc47a44780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:57.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:00:57.278+0000 7fcc47a44780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:00:57.296 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:00:57.296 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:00:57.296 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:00:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:00:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:00:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:00:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:00:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:00:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:00:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:00:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:00:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:00:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:00:57Z" 2024-04-13T23:00:57.304 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:00:57.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:00:57Z" } 2024-04-13T23:00:57.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:00:57.305 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:57.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:00:57.306 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:02.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:02.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:02.307 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:02.364 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:02.364 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:02.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:02.395+0000 7f9fe2aa8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:02.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:02.396+0000 7f9fe2aa8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:02.418 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:02.418 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:02.418 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:02.418 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:02.418 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:01:02.419 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:02.420 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:02.421 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:02.422 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:02.423 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:02.424 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:02.424 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:02.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:02.424 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:02Z" 2024-04-13T23:01:02.424 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:02.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:02Z" } 2024-04-13T23:01:02.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:02.425 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:02.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:02.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:07.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:07.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:07.426 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:07.482 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:07.482 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:07.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:07.525+0000 7f1d903fa780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:07.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:07.526+0000 7f1d903fa780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:07.549 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:07.549 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:07.549 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:07.549 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:07.550 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:07.551 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:07.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:07.553 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:07.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:07.555 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:07Z" 2024-04-13T23:01:07.556 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:07.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:07Z" } 2024-04-13T23:01:07.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:07.557 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:07.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:07.557 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:12.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:12.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:12.558 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:12.613 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:12.613 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:12.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:12.653+0000 7fa2a9521780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:12.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:12.654+0000 7fa2a9521780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:12.670 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:12.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:12.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:12.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:12.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:12Z" 2024-04-13T23:01:12.675 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:12.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:12Z" } 2024-04-13T23:01:12.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:12.676 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:12.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:12.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:17.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:17.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:17.678 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:17.730 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:17.731 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:17.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:17.781+0000 7ff6d077c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:17.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:17.782+0000 7ff6d077c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:17.806 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:17.807 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:17.808 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:17.809 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:17.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:17.811 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:17Z" 2024-04-13T23:01:17.812 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:17.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:17Z" } 2024-04-13T23:01:17.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:17.813 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:17.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:17.814 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:22.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:22.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:22.816 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:22.871 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:22.871 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:22.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:22.912+0000 7f28c99d7780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:22.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:22.913+0000 7f28c99d7780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:22.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:22.931 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:22.933 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:22.934 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:22.935 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:22.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:22.936 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:22Z" 2024-04-13T23:01:22.936 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:22.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:22Z" } 2024-04-13T23:01:22.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:22.937 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:22.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:22.937 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:27.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:27.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:27.939 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:27.994 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:27.994 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:28.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:28.029+0000 7f859490b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:28.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:28.030+0000 7f859490b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:28.054 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:28.055 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049167.480760_1049.1", 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:27.480760Z", 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:28.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:28.057 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:28.058 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:28.059 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:28.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:28.061 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:28Z" 2024-04-13T23:01:28.061 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:28.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049167.480760_1049.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:27.480760Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:28Z" } 2024-04-13T23:01:28.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049167.480760_1049.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:27.480760Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:28.062 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:28.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049167.480760_1049.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:28.062 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049190.433608_1085.1 target=1_1713049167.480760_1049.1 2024-04-13T23:01:33.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:33.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:33.064 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:33.119 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:33.119 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:33.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:33.151+0000 7f84ffc2b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:33.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:33.152+0000 7f84ffc2b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:33.169 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049190.433608_1085.1", 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.433608Z", 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:33.173 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.174 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:33Z" 2024-04-13T23:01:33.175 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:33.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049190.433608_1085.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:50.433608Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:33Z" } 2024-04-13T23:01:33.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049190.433608_1085.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.433608Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:33.177 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:33.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:33.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:01:33.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:33.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:33.177 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:33.224 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:33.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:33.292 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:01:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.294 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:01:33.294 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:01:33.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.294 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:01:33.295 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:33.295 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:01:33.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:33.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:33.296 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:33.345 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:33.345 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:33.411 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:33.411 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:33.411 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:33.412 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046930", 2024-04-13T23:01:33.413 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.912411Z" 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:33.414 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053083", 2024-04-13T23:01:33.415 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.971913Z" 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:33.416 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:33Z" 2024-04-13T23:01:33.417 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:33.417 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.912411Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053083", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:53:05.971913Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:01:33Z" } 2024-04-13T23:01:33.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.912411Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053083', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.971913Z'}}] 2024-04-13T23:01:33.418 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:33.418 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000046930', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053083'} 2024-04-13T23:01:33.418 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047071 target=00000000000000000000:00000000000000046930, shard=3 master=00000000000000000000:00000000000000053225 target=00000000000000000000:00000000000000053083 2024-04-13T23:01:38.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:38.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:38.420 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:38.477 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:38.477 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:38.529 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:38.529 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:38.530 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046930", 2024-04-13T23:01:38.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.912411Z" 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:38.532 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053083", 2024-04-13T23:01:38.533 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.971913Z" 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:01:38.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:38.535 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:38Z" 2024-04-13T23:01:38.535 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:38.536 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.912411Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053083", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:53:05.971913Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:01:38Z" } 2024-04-13T23:01:38.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.912411Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053083', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.971913Z'}}] 2024-04-13T23:01:38.536 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:38.536 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000046930', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053083'} 2024-04-13T23:01:38.536 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047071 target=00000000000000000000:00000000000000046930, shard=3 master=00000000000000000000:00000000000000053225 target=00000000000000000000:00000000000000053083 2024-04-13T23:01:43.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:43.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:43.541 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:43.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:43.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:43.661 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:43.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046930", 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.912411Z" 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:43.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:43.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053083", 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.971913Z" 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:43.665 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:43Z" 2024-04-13T23:01:43.666 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:43.667 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.912411Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053083", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:53:05.971913Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:01:43Z" } 2024-04-13T23:01:43.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.912411Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053083', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.971913Z'}}] 2024-04-13T23:01:43.668 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:43.668 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000046930', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053083'} 2024-04-13T23:01:43.668 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047071 target=00000000000000000000:00000000000000046930, shard=3 master=00000000000000000000:00000000000000053225 target=00000000000000000000:00000000000000053083 2024-04-13T23:01:48.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:48.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:48.670 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:48.728 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:48.728 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:48.779 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:48.780 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:48.781 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046930", 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.912411Z" 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:48.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:48.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053083", 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.971913Z" 2024-04-13T23:01:48.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:48Z" 2024-04-13T23:01:48.785 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:48.786 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.912411Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053083", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:53:05.971913Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:01:48Z" } 2024-04-13T23:01:48.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.912411Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053083', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.971913Z'}}] 2024-04-13T23:01:48.787 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:48.787 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000046930', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053083'} 2024-04-13T23:01:48.787 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047071 target=00000000000000000000:00000000000000046930, shard=3 master=00000000000000000000:00000000000000053225 target=00000000000000000000:00000000000000053083 2024-04-13T23:01:53.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:53.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:53.789 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:53.844 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:53.844 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:53.903 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:01:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:53Z" 2024-04-13T23:01:53.909 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:53.910 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:01:53Z" } 2024-04-13T23:01:53.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:01:53.910 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:53.910 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:53.910 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:01:53.911 DEBUG:boto:path=/vwnfns-68/ 2024-04-13T23:01:53.911 DEBUG:boto:auth_path=/vwnfns-68/ 2024-04-13T23:01:53.911 DEBUG:boto:Method: HEAD 2024-04-13T23:01:53.911 DEBUG:boto:Path: /vwnfns-68/ 2024-04-13T23:01:53.911 DEBUG:boto:Data: 2024-04-13T23:01:53.911 DEBUG:boto:Headers: {} 2024-04-13T23:01:53.911 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:01:53.911 DEBUG:boto:Port: 8001 2024-04-13T23:01:53.911 DEBUG:boto:Params: {} 2024-04-13T23:01:53.912 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:01:53.912 DEBUG:boto:Token: None 2024-04-13T23:01:53.912 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:53 GMT /vwnfns-68/ 2024-04-13T23:01:53.913 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1GYUwIP2/xZgwwFs7tnGMQTYY1w= 2024-04-13T23:01:53.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1GYUwIP2/xZgwwFs7tnGMQTYY1w=', 'Content-Length': '0'} 2024-04-13T23:01:53.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000769393924b2e61b8-00661b0ee1-2674-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:53.918 DEBUG:boto:path=/vwnfns-68/obj1 2024-04-13T23:01:53.918 DEBUG:boto:auth_path=/vwnfns-68/obj1 2024-04-13T23:01:53.918 DEBUG:boto:Method: HEAD 2024-04-13T23:01:53.918 DEBUG:boto:Path: /vwnfns-68/obj1 2024-04-13T23:01:53.918 DEBUG:boto:Data: 2024-04-13T23:01:53.918 DEBUG:boto:Headers: {} 2024-04-13T23:01:53.918 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:01:53.918 DEBUG:boto:Port: 8001 2024-04-13T23:01:53.919 DEBUG:boto:Params: {} 2024-04-13T23:01:53.919 DEBUG:boto:Token: None 2024-04-13T23:01:53.919 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:53 GMT /vwnfns-68/obj1 2024-04-13T23:01:53.919 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PeOj5PbshZ34WDFfxXoADxzD+OM= 2024-04-13T23:01:53.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PeOj5PbshZ34WDFfxXoADxzD+OM=', 'Content-Length': '0'} 2024-04-13T23:01:53.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-68:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cacdbfdc68aaabc0-00661b0ee1-2674-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:01:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:53.922 DEBUG:boto:path=/vwnfns-68/obj1 2024-04-13T23:01:53.922 DEBUG:boto:auth_path=/vwnfns-68/obj1 2024-04-13T23:01:53.922 DEBUG:boto:Method: GET 2024-04-13T23:01:53.922 DEBUG:boto:Path: /vwnfns-68/obj1 2024-04-13T23:01:53.922 DEBUG:boto:Data: 2024-04-13T23:01:53.922 DEBUG:boto:Headers: {} 2024-04-13T23:01:53.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:01:53.922 DEBUG:boto:Port: 8001 2024-04-13T23:01:53.922 DEBUG:boto:Params: {} 2024-04-13T23:01:53.923 DEBUG:boto:Token: None 2024-04-13T23:01:53.923 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:01:53 GMT /vwnfns-68/obj1 2024-04-13T23:01:53.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tjy5i/92c3toxWdksDVYnh1zheU= 2024-04-13T23:01:53.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tjy5i/92c3toxWdksDVYnh1zheU=', 'Content-Length': '0'} 2024-04-13T23:01:53.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-68:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000913a9f38fa6e91fc-00661b0ee1-2674-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:01:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:53.925 DEBUG:boto:path=/vwnfns-69/ 2024-04-13T23:01:53.925 DEBUG:boto:auth_path=/vwnfns-69/ 2024-04-13T23:01:53.925 DEBUG:boto:Method: HEAD 2024-04-13T23:01:53.925 DEBUG:boto:Path: /vwnfns-69/ 2024-04-13T23:01:53.925 DEBUG:boto:Data: 2024-04-13T23:01:53.925 DEBUG:boto:Headers: {} 2024-04-13T23:01:53.926 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:53.926 DEBUG:boto:Port: 8000 2024-04-13T23:01:53.926 DEBUG:boto:Params: {} 2024-04-13T23:01:53.926 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:01:53.926 DEBUG:boto:Token: None 2024-04-13T23:01:53.926 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:53 GMT /vwnfns-69/ 2024-04-13T23:01:53.926 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:2Xf7n2lGZ6MjksYo+eTmY0aNURI= 2024-04-13T23:01:53.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2Xf7n2lGZ6MjksYo+eTmY0aNURI=', 'Content-Length': '0'} 2024-04-13T23:01:53.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcfb776f8ed24989-00661b0ee1-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:53.930 DEBUG:boto:path=/vwnfns-69/obj2 2024-04-13T23:01:53.930 DEBUG:boto:auth_path=/vwnfns-69/obj2 2024-04-13T23:01:53.931 DEBUG:boto:Method: HEAD 2024-04-13T23:01:53.931 DEBUG:boto:Path: /vwnfns-69/obj2 2024-04-13T23:01:53.931 DEBUG:boto:Data: 2024-04-13T23:01:53.931 DEBUG:boto:Headers: {} 2024-04-13T23:01:53.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:53.931 DEBUG:boto:Port: 8000 2024-04-13T23:01:53.931 DEBUG:boto:Params: {} 2024-04-13T23:01:53.931 DEBUG:boto:Token: None 2024-04-13T23:01:53.931 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:53 GMT /vwnfns-69/obj2 2024-04-13T23:01:53.931 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ViVmD+BVOp78WRwxyfk9gw91ar8= 2024-04-13T23:01:53.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ViVmD+BVOp78WRwxyfk9gw91ar8=', 'Content-Length': '0'} 2024-04-13T23:01:53.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-69:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000232fec1231a81a97-00661b0ee1-2678-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:01:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:53.933 DEBUG:boto:path=/vwnfns-69/obj2 2024-04-13T23:01:53.933 DEBUG:boto:auth_path=/vwnfns-69/obj2 2024-04-13T23:01:53.933 DEBUG:boto:Method: GET 2024-04-13T23:01:53.933 DEBUG:boto:Path: /vwnfns-69/obj2 2024-04-13T23:01:53.933 DEBUG:boto:Data: 2024-04-13T23:01:53.933 DEBUG:boto:Headers: {} 2024-04-13T23:01:53.933 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:53.934 DEBUG:boto:Port: 8000 2024-04-13T23:01:53.934 DEBUG:boto:Params: {} 2024-04-13T23:01:53.934 DEBUG:boto:Token: None 2024-04-13T23:01:53.934 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:01:53 GMT /vwnfns-69/obj2 2024-04-13T23:01:53.934 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mGJY78/2BL8x2MiKOSl3606p3EU= 2024-04-13T23:01:53.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mGJY78/2BL8x2MiKOSl3606p3EU=', 'Content-Length': '0'} 2024-04-13T23:01:53.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:59:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-69:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c29f3a15bccd231-00661b0ee1-2678-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:01:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:53.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:53.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:53.936 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:53.982 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:53.983 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.035 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:01:54.036 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2024-04-13T23:01:54.037 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-13T23:01:54.038 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-13T23:01:54.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.039 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:54.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.130 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:01:54.146 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:01:54.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.146 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049190.433608_1085.1", 2024-04-13T23:01:54.146 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:59:50.433608Z" 2024-04-13T23:01:54.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.147 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:54.148 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:01:54.148 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.148 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:01:54.149 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:54.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:01:54.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.150 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:54.197 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.197 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:54.239+0000 7f7d27c1c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:54.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:54.240+0000 7f7d27c1c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:54.259 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:54.259 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:54.259 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.260 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049190.433608_1085.1", 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.433608Z", 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.261 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.263 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.264 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.265 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:54Z" 2024-04-13T23:01:54.266 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:54.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049190.433608_1085.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T22:59:50.433608Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:54Z" } 2024-04-13T23:01:54.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049190.433608_1085.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.433608Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:54.267 DEBUG:rgw_multi.tests:log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:54.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:54.267 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:01:54.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.268 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:54.311 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.311 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.362 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T23:01:54.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.363 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:54.446 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.447 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.511 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T23:01:54.511 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-70 2024-04-13T23:01:54.512 DEBUG:boto:path=/vwnfns-70/ 2024-04-13T23:01:54.512 DEBUG:boto:auth_path=/vwnfns-70/ 2024-04-13T23:01:54.512 DEBUG:boto:Method: PUT 2024-04-13T23:01:54.512 DEBUG:boto:Path: /vwnfns-70/ 2024-04-13T23:01:54.512 DEBUG:boto:Data: 2024-04-13T23:01:54.512 DEBUG:boto:Headers: {} 2024-04-13T23:01:54.512 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:54.512 DEBUG:boto:Port: 8000 2024-04-13T23:01:54.512 DEBUG:boto:Params: {} 2024-04-13T23:01:54.512 DEBUG:boto:Token: None 2024-04-13T23:01:54.513 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:01:54 GMT /vwnfns-70/ 2024-04-13T23:01:54.513 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:R9vqkRMcyn5kKPrD2+9+ougi9JI= 2024-04-13T23:01:54.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R9vqkRMcyn5kKPrD2+9+ougi9JI=', 'Content-Length': '0'} 2024-04-13T23:01:54.537 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f09d5ea014ed0331-00661b0ee2-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:54.538 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-71 2024-04-13T23:01:54.538 DEBUG:boto:path=/vwnfns-71/ 2024-04-13T23:01:54.538 DEBUG:boto:auth_path=/vwnfns-71/ 2024-04-13T23:01:54.538 DEBUG:boto:Method: PUT 2024-04-13T23:01:54.538 DEBUG:boto:Path: /vwnfns-71/ 2024-04-13T23:01:54.538 DEBUG:boto:Data: 2024-04-13T23:01:54.538 DEBUG:boto:Headers: {} 2024-04-13T23:01:54.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:54.539 DEBUG:boto:Port: 8000 2024-04-13T23:01:54.539 DEBUG:boto:Params: {} 2024-04-13T23:01:54.539 DEBUG:boto:Token: None 2024-04-13T23:01:54.539 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:01:54 GMT /vwnfns-71/ 2024-04-13T23:01:54.539 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fumts7JMLHEfidZHCR1oIK6BBBI= 2024-04-13T23:01:54.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fumts7JMLHEfidZHCR1oIK6BBBI=', 'Content-Length': '0'} 2024-04-13T23:01:54.561 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000032e4b253bd5d3fc8-00661b0ee2-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:54.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.562 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:54.614 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.614 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.673 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:01:54.686 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:01:54.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.686 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:54.558113Z" 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.688 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:54.688 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:01:54.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.688 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:01:54.689 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:54.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:01:54.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:54.689 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:54.737 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.737 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:54.766+0000 7f05b6661780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:54.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:54.767+0000 7f05b6661780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:54.782 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.783 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.784 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.785 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:54.786 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:54.787 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:54Z" 2024-04-13T23:01:54.788 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:54.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049314.558113_1103.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:01:54.558113Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:54Z" } 2024-04-13T23:01:54.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049314.558113_1103.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:01:54.558113Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:54.790 DEBUG:rgw_multi.tests:log_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:54.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:54.790 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:01:54.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.790 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:54.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:54.914 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:01:54.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:54.915 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:54.967 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:54.967 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:55.043 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:01:55.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=vwnfns-70', '--dest-bucket=vwnfns-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=vwnfns-70', '--dest-bucket=vwnfns-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.044 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=vwnfns-70 --dest-bucket=vwnfns-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:55.159 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:01:55.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.160 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:55.207 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:55.208 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:55.257 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:01:55.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.258 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:55.301 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:55.301 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:55.353 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:01:55.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.354 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:55.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:55.440 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:55.513 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:55.513 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:55.513 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 16, 2024-04-13T23:01:55.513 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:01:55.513 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:01:55.513 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.514 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:01:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:01:55.517 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:01:55.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:01:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.520 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-70", 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-70", 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:01:55.522 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "enabled" 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.523 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:01:55.524 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:01:55.525 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:01:55.526 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:01:55.527 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:01:55.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:01:55.529 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:01:55.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:01:55.541 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:01:55.542 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:01:55.543 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.544 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:55.545 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:55.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:01:55.547 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:01:55.590 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:55.591 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:55.658 INFO:teuthology.orchestra.run.smithi132.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"vwnfns-70"},"dest":{"zone":"a1","bucket":"vwnfns-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"vwnfns-70"},"dest":{"zone":"a2","bucket":"vwnfns-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-13T23:01:55.658 DEBUG:boto:path=/vwnfns-70/ 2024-04-13T23:01:55.658 DEBUG:boto:auth_path=/vwnfns-70/ 2024-04-13T23:01:55.658 DEBUG:boto:Method: HEAD 2024-04-13T23:01:55.659 DEBUG:boto:Path: /vwnfns-70/ 2024-04-13T23:01:55.659 DEBUG:boto:Data: 2024-04-13T23:01:55.659 DEBUG:boto:Headers: {} 2024-04-13T23:01:55.659 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.659 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.659 DEBUG:boto:Params: {} 2024-04-13T23:01:55.659 DEBUG:boto:Token: None 2024-04-13T23:01:55.659 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-70/ 2024-04-13T23:01:55.659 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo= 2024-04-13T23:01:55.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo=', 'Content-Length': '0'} 2024-04-13T23:01:55.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068cfbeae1aa1dc0c-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.740 DEBUG:boto:path=/vwnfns-70/obj1 2024-04-13T23:01:55.740 DEBUG:boto:auth_path=/vwnfns-70/obj1 2024-04-13T23:01:55.740 DEBUG:boto:Method: PUT 2024-04-13T23:01:55.740 DEBUG:boto:Path: /vwnfns-70/obj1 2024-04-13T23:01:55.740 DEBUG:boto:Data: 2024-04-13T23:01:55.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:01:55.740 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.740 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.740 DEBUG:boto:Params: {} 2024-04-13T23:01:55.740 DEBUG:boto:Token: None 2024-04-13T23:01:55.741 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-70/obj1 2024-04-13T23:01:55.741 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:rRKU++zLXh9ZRCaWcMKq4Ufq57U= 2024-04-13T23:01:55.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rRKU++zLXh9ZRCaWcMKq4Ufq57U='} 2024-04-13T23:01:55.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e328cd2f8dbebd-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.771 DEBUG:boto:path=/vwnfns-70/ 2024-04-13T23:01:55.771 DEBUG:boto:auth_path=/vwnfns-70/ 2024-04-13T23:01:55.771 DEBUG:boto:Method: HEAD 2024-04-13T23:01:55.771 DEBUG:boto:Path: /vwnfns-70/ 2024-04-13T23:01:55.771 DEBUG:boto:Data: 2024-04-13T23:01:55.772 DEBUG:boto:Headers: {} 2024-04-13T23:01:55.772 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.772 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.772 DEBUG:boto:Params: {} 2024-04-13T23:01:55.772 DEBUG:boto:Token: None 2024-04-13T23:01:55.772 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-70/ 2024-04-13T23:01:55.772 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo= 2024-04-13T23:01:55.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo=', 'Content-Length': '0'} 2024-04-13T23:01:55.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcaf3ca3b6b348bb-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.775 DEBUG:boto:path=/vwnfns-70/obj2 2024-04-13T23:01:55.775 DEBUG:boto:auth_path=/vwnfns-70/obj2 2024-04-13T23:01:55.776 DEBUG:boto:Method: PUT 2024-04-13T23:01:55.776 DEBUG:boto:Path: /vwnfns-70/obj2 2024-04-13T23:01:55.776 DEBUG:boto:Data: 2024-04-13T23:01:55.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:01:55.776 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.776 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.776 DEBUG:boto:Params: {} 2024-04-13T23:01:55.776 DEBUG:boto:Token: None 2024-04-13T23:01:55.776 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-70/obj2 2024-04-13T23:01:55.776 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EPEGU/UiwhyiHVQfb69dxFgs+bE= 2024-04-13T23:01:55.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EPEGU/UiwhyiHVQfb69dxFgs+bE='} 2024-04-13T23:01:55.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b987d62623efa1c-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.823 DEBUG:boto:path=/vwnfns-70/ 2024-04-13T23:01:55.823 DEBUG:boto:auth_path=/vwnfns-70/ 2024-04-13T23:01:55.823 DEBUG:boto:Method: HEAD 2024-04-13T23:01:55.823 DEBUG:boto:Path: /vwnfns-70/ 2024-04-13T23:01:55.823 DEBUG:boto:Data: 2024-04-13T23:01:55.823 DEBUG:boto:Headers: {} 2024-04-13T23:01:55.823 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.823 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.824 DEBUG:boto:Params: {} 2024-04-13T23:01:55.824 DEBUG:boto:Token: None 2024-04-13T23:01:55.824 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-70/ 2024-04-13T23:01:55.824 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo= 2024-04-13T23:01:55.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo=', 'Content-Length': '0'} 2024-04-13T23:01:55.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004889fc88941e29f5-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.828 DEBUG:boto:path=/vwnfns-70/obj3 2024-04-13T23:01:55.828 DEBUG:boto:auth_path=/vwnfns-70/obj3 2024-04-13T23:01:55.828 DEBUG:boto:Method: PUT 2024-04-13T23:01:55.828 DEBUG:boto:Path: /vwnfns-70/obj3 2024-04-13T23:01:55.828 DEBUG:boto:Data: 2024-04-13T23:01:55.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:01:55.828 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.828 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.829 DEBUG:boto:Params: {} 2024-04-13T23:01:55.829 DEBUG:boto:Token: None 2024-04-13T23:01:55.829 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-70/obj3 2024-04-13T23:01:55.829 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:bi3SQM/vlu5RcSAgo/9Ub41gPMQ= 2024-04-13T23:01:55.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bi3SQM/vlu5RcSAgo/9Ub41gPMQ='} 2024-04-13T23:01:55.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001289099cfb88730b-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.839 DEBUG:boto:path=/vwnfns-71/ 2024-04-13T23:01:55.839 DEBUG:boto:auth_path=/vwnfns-71/ 2024-04-13T23:01:55.840 DEBUG:boto:Method: HEAD 2024-04-13T23:01:55.840 DEBUG:boto:Path: /vwnfns-71/ 2024-04-13T23:01:55.840 DEBUG:boto:Data: 2024-04-13T23:01:55.840 DEBUG:boto:Headers: {} 2024-04-13T23:01:55.840 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.840 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.840 DEBUG:boto:Params: {} 2024-04-13T23:01:55.840 DEBUG:boto:Token: None 2024-04-13T23:01:55.840 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-71/ 2024-04-13T23:01:55.840 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O+CJTCAvIcGw8yKFrd7Mc2Tt+gg= 2024-04-13T23:01:55.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O+CJTCAvIcGw8yKFrd7Mc2Tt+gg=', 'Content-Length': '0'} 2024-04-13T23:01:55.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e5a11adbd153505-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.845 DEBUG:boto:path=/vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:01:55.846 DEBUG:boto:auth_path=/vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:01:55.846 DEBUG:boto:Method: PUT 2024-04-13T23:01:55.846 DEBUG:boto:Path: /vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:01:55.846 DEBUG:boto:Data: 2024-04-13T23:01:55.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:01:55.846 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:01:55.846 DEBUG:boto:Port: 8000 2024-04-13T23:01:55.846 DEBUG:boto:Params: {} 2024-04-13T23:01:55.846 DEBUG:boto:Token: None 2024-04-13T23:01:55.847 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:01:55 GMT /vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:01:55.847 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tsCuTQfXAHrED8h42W84xYVkrQY= 2024-04-13T23:01:55.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tsCuTQfXAHrED8h42W84xYVkrQY='} 2024-04-13T23:01:55.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d58eb918c3fcb7c5-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:01:55.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:55.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:55.891 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:55.958 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:55.958 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:56.057 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:01:56.104 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:01:56.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.104 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:01:56.104 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:54.558113Z" 2024-04-13T23:01:56.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.105 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:56.106 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:01:56.106 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.106 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:01:56.107 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:56.107 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:01:56.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:56.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:56.107 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:56.178 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:56.178 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:56.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:56.233+0000 7f6fd2709780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:56.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:01:56.237+0000 7f6fd2709780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.285 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.286 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:56.287 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:01:56.288 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.289 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:56.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.291 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:56.291 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:01:56.291 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:01:56.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.291 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:56Z" 2024-04-13T23:01:56.291 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:56.292 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049314.558113_1103.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:01:54.558113Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:01:56Z" } 2024-04-13T23:01:56.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049314.558113_1103.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:01:54.558113Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:01:56.292 DEBUG:rgw_multi.tests:log_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:56.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:01:56.293 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:01:56.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:56.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:56.293 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:56.350 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:56.350 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:01:56.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.468 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:01:56.470 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:01:56.470 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:01:56.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:56.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:01:56.470 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:01:56.529 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:01:56.529 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:01:56.615 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:01:56.615 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:01:56.615 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:01:56.615 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:01:56.615 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:01:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.618 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:01:56.619 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:01:56Z" 2024-04-13T23:01:56.620 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:01:56.621 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:01:56Z" } 2024-04-13T23:01:56.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:01:56.622 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:01:56.622 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:01:56.622 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:01.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:01.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:01.624 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:01.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:01.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:01.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:01.747 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:01.748 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:01.751 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:01.752 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:01.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:01.752 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:01Z" 2024-04-13T23:02:01.752 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:01.753 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:01Z" } 2024-04-13T23:02:01.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:01.753 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:01.753 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:01.753 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:06.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:06.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:06.755 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:06.812 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:06.812 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:06.880 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:06.880 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:06.881 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:06.882 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:06.883 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:06.885 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:06.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:06.886 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:06Z" 2024-04-13T23:02:06.886 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:06.887 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:06Z" } 2024-04-13T23:02:06.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:06.887 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:06.887 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:06.887 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:11.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:11.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:11.889 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:11.946 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:11.947 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:12.006 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:12.006 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:12.006 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:12.007 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:12.008 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:12.009 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:12.010 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:12.011 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:12.012 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:12.012 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:12.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:12.012 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:11Z" 2024-04-13T23:02:12.012 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:12.013 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:11Z" } 2024-04-13T23:02:12.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:12.013 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:12.013 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:12.014 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:17.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:17.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:17.015 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:17.071 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:17.071 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:17.131 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:17.131 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:17.131 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:17.132 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:17.133 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:17.134 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:17.135 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:17.136 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:17.137 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:17Z" 2024-04-13T23:02:17.138 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:17.139 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:17Z" } 2024-04-13T23:02:17.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:17.139 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:17.139 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:17.139 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:22.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:22.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:22.141 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:22.198 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:22.198 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:22.249 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:22.249 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:22.249 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:22.249 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:22.250 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:22.251 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:22.252 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:22.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:22.254 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:22Z" 2024-04-13T23:02:22.255 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:22.256 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:22Z" } 2024-04-13T23:02:22.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:22.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:22.257 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:22.257 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:27.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:27.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:27.258 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:27.316 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:27.316 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:27.388 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:27.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:27.390 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:27.391 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:27.392 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:27.393 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:27.394 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:27Z" 2024-04-13T23:02:27.395 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:27.396 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:27Z" } 2024-04-13T23:02:27.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:27.396 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:27.396 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:27.397 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:32.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:32.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:32.398 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:32.455 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:32.455 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:32.512 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:32.512 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:32.512 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:32.512 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:32.512 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:32.513 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:32.514 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:32.515 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:32.516 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:32.517 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:32Z" 2024-04-13T23:02:32.518 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:32.519 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:32Z" } 2024-04-13T23:02:32.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:32.519 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:32.519 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:32.519 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:37.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:37.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:37.521 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:37.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:37.579 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:37.641 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:37.641 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:37.642 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:37.643 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:37.644 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:37.645 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:37.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:37.647 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:37Z" 2024-04-13T23:02:37.647 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:37.648 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:37Z" } 2024-04-13T23:02:37.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:37.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:37.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:37.649 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:42.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:42.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:42.651 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:42.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:42.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:42.770 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:42.771 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:42.772 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:42.773 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:42.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:42.775 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:42Z" 2024-04-13T23:02:42.776 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:42.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:42Z" } 2024-04-13T23:02:42.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:42.777 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:42.777 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:42.778 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:47.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:47.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:47.779 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:47.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:47.838 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:47.900 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:47Z" 2024-04-13T23:02:47.906 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:47.907 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:47Z" } 2024-04-13T23:02:47.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:47.908 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:47.908 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:47.908 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:52.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:52.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:52.909 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:53.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:53.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:53.022 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:53.023 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:53Z" 2024-04-13T23:02:53.024 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:53.025 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:53Z" } 2024-04-13T23:02:53.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:53.026 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:53.026 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:53.026 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:02:58.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:58.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:02:58.028 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:02:58.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:02:58.086 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:02:58.144 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:58.145 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:02:58.146 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:02:58.147 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:02:58.148 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:58.149 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:02:58.150 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:02:58.150 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:02:58.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:02:58.150 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:02:58Z" 2024-04-13T23:02:58.150 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:02:58.151 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:02:58Z" } 2024-04-13T23:02:58.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:02:58.152 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:02:58.152 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:02:58.152 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:03.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:03.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:03.153 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:03.211 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:03.211 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:03.261 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:03.261 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:03.261 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:03.261 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:03.261 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:03.261 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:03.262 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:03.263 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:03.264 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:03.265 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:03Z" 2024-04-13T23:03:03.266 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:03.267 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:03Z" } 2024-04-13T23:03:03.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:03.268 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:03.268 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:03.268 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:08.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:08.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:08.269 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:08.326 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:08.327 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:08.390 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:08.390 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:08.391 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:08.392 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:08.393 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:08.394 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:08.395 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:08.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:08.396 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:08Z" 2024-04-13T23:03:08.396 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:08.397 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:08Z" } 2024-04-13T23:03:08.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:08.397 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:08.397 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:08.397 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:13.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:13.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:13.399 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:13.456 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:13.456 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:13.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:13.519 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:13.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:13.521 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:13.522 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:13Z" 2024-04-13T23:03:13.523 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:13.524 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:13Z" } 2024-04-13T23:03:13.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:13.525 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:13.525 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:13.525 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:18.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:18.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:18.526 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:18.583 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:18.584 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:18.635 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:18.636 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:18.637 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:18.638 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:18.639 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:18.640 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:18.641 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:18.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:18.641 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:18Z" 2024-04-13T23:03:18.641 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:18.642 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:18Z" } 2024-04-13T23:03:18.642 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:18.642 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:18.642 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:18.642 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:23.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:23.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:23.644 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:23.701 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:23.702 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:23.762 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:23.762 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:23.763 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:23.764 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:23.765 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:23.766 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:23.767 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:23.768 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:23.768 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:23.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:23.768 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:23Z" 2024-04-13T23:03:23.768 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:23.769 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:23Z" } 2024-04-13T23:03:23.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:23.769 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:23.769 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:23.770 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:28.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:28.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:28.771 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:28.828 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:28.829 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:28.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:28.883 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:28.884 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:28.885 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:28.886 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:28Z" 2024-04-13T23:03:28.887 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:28.887 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:28Z" } 2024-04-13T23:03:28.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:28.888 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:28.888 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:28.888 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:33.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:33.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:33.890 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:33.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:33.948 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:34.004 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:34.005 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:34.006 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:34.007 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:34.008 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:34.009 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:33Z" 2024-04-13T23:03:34.010 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:34.011 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:33Z" } 2024-04-13T23:03:34.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:34.011 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:34.011 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:34.012 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:39.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:39.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:39.013 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:39.070 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:39.071 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:39.133 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:39.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:39.136 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:39.137 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:39.138 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:39.139 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:39Z" 2024-04-13T23:03:39.140 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:39.141 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:39Z" } 2024-04-13T23:03:39.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:39.142 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:39.142 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:39.142 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:44.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:44.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:44.143 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:44.201 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:44.201 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:44.250 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:44.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:44.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:44.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:44.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:44.255 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:44.256 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:44.257 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:44.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:44.257 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:44Z" 2024-04-13T23:03:44.257 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:44.258 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:44Z" } 2024-04-13T23:03:44.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:44.259 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:44.259 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:44.260 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:49.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:49.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:49.261 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:49.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:49.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:49.372 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:49Z" 2024-04-13T23:03:49.378 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:49.379 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:49Z" } 2024-04-13T23:03:49.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:49.379 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:49.380 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:49.380 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:54.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:54.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:54.381 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:54.438 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:54.438 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:54.501 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:54.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:54.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:54.504 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:03:54.505 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:54Z" 2024-04-13T23:03:54.506 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:54.507 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046153", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:53:05.852657Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047071", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:59:57.484330Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053225", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T22:59:50.402677Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:54Z" } 2024-04-13T23:03:54.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:03:54.508 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:54.508 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:03:54.508 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:03:59.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:59.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:59.509 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:59.569 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:59.569 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:59.624 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:59.624 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:59.624 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:59.624 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:03:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:03:59.626 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:59.627 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.628 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:59.629 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:03:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:03:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.630 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:59Z" 2024-04-13T23:03:59.630 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:59.631 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:03:59Z" } 2024-04-13T23:03:59.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:03:59.631 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:59.631 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:03:59.631 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:03:59.632 DEBUG:boto:path=/vwnfns-70/ 2024-04-13T23:03:59.632 DEBUG:boto:auth_path=/vwnfns-70/ 2024-04-13T23:03:59.632 DEBUG:boto:Method: HEAD 2024-04-13T23:03:59.632 DEBUG:boto:Path: /vwnfns-70/ 2024-04-13T23:03:59.632 DEBUG:boto:Data: 2024-04-13T23:03:59.632 DEBUG:boto:Headers: {} 2024-04-13T23:03:59.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:03:59.632 DEBUG:boto:Port: 8001 2024-04-13T23:03:59.632 DEBUG:boto:Params: {} 2024-04-13T23:03:59.633 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:03:59.633 DEBUG:boto:Token: None 2024-04-13T23:03:59.633 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:03:59 GMT /vwnfns-70/ 2024-04-13T23:03:59.634 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tFOjMIDMNatUSv+2bBi2gslb7bM= 2024-04-13T23:03:59.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tFOjMIDMNatUSv+2bBi2gslb7bM=', 'Content-Length': '0'} 2024-04-13T23:03:59.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003825a8407ffc6108-00661b0f5f-27d0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:03:59.640 DEBUG:boto:path=/vwnfns-70/obj1 2024-04-13T23:03:59.640 DEBUG:boto:auth_path=/vwnfns-70/obj1 2024-04-13T23:03:59.640 DEBUG:boto:Method: HEAD 2024-04-13T23:03:59.640 DEBUG:boto:Path: /vwnfns-70/obj1 2024-04-13T23:03:59.640 DEBUG:boto:Data: 2024-04-13T23:03:59.640 DEBUG:boto:Headers: {} 2024-04-13T23:03:59.640 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:03:59.640 DEBUG:boto:Port: 8001 2024-04-13T23:03:59.641 DEBUG:boto:Params: {} 2024-04-13T23:03:59.641 DEBUG:boto:Token: None 2024-04-13T23:03:59.641 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:03:59 GMT /vwnfns-70/obj1 2024-04-13T23:03:59.641 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oosvsvkrx9cpjs1o2Oqd59RRyJQ= 2024-04-13T23:03:59.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oosvsvkrx9cpjs1o2Oqd59RRyJQ=', 'Content-Length': '0'} 2024-04-13T23:03:59.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-70:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aa9e0c8c323491b-00661b0f5f-27d0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:03:59.643 DEBUG:boto:path=/vwnfns-70/obj1 2024-04-13T23:03:59.643 DEBUG:boto:auth_path=/vwnfns-70/obj1 2024-04-13T23:03:59.643 DEBUG:boto:Method: GET 2024-04-13T23:03:59.643 DEBUG:boto:Path: /vwnfns-70/obj1 2024-04-13T23:03:59.644 DEBUG:boto:Data: 2024-04-13T23:03:59.644 DEBUG:boto:Headers: {} 2024-04-13T23:03:59.644 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:03:59.644 DEBUG:boto:Port: 8001 2024-04-13T23:03:59.644 DEBUG:boto:Params: {} 2024-04-13T23:03:59.644 DEBUG:boto:Token: None 2024-04-13T23:03:59.644 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:03:59 GMT /vwnfns-70/obj1 2024-04-13T23:03:59.644 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5EzvppZWRdIQFcw8GwtAElEbyx0= 2024-04-13T23:03:59.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5EzvppZWRdIQFcw8GwtAElEbyx0=', 'Content-Length': '0'} 2024-04-13T23:03:59.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-70:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a62f44203992043-00661b0f5f-27d0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:03:59.646 DEBUG:boto:path=/vwnfns-70/obj2 2024-04-13T23:03:59.646 DEBUG:boto:auth_path=/vwnfns-70/obj2 2024-04-13T23:03:59.647 DEBUG:boto:Method: HEAD 2024-04-13T23:03:59.647 DEBUG:boto:Path: /vwnfns-70/obj2 2024-04-13T23:03:59.647 DEBUG:boto:Data: 2024-04-13T23:03:59.647 DEBUG:boto:Headers: {} 2024-04-13T23:03:59.647 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:03:59.647 DEBUG:boto:Port: 8001 2024-04-13T23:03:59.647 DEBUG:boto:Params: {} 2024-04-13T23:03:59.647 DEBUG:boto:Token: None 2024-04-13T23:03:59.647 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:03:59 GMT /vwnfns-70/obj2 2024-04-13T23:03:59.647 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eVTB+XAz03UfRDPTfFtH6+dOMGQ= 2024-04-13T23:03:59.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eVTB+XAz03UfRDPTfFtH6+dOMGQ=', 'Content-Length': '0'} 2024-04-13T23:03:59.649 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000022766e6910bc4bff-00661b0f5f-27d0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:03:59.652 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2024-04-13T23:03:59.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:59.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:59.653 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:59.701 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:59.702 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:59.755 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:03:59.775 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:54.558113Z" 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.776 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.777 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:03:59.778 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:03:59.778 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:03:59.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:59.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:03:59.779 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:03:59.827 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:59.827 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:59.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:03:59.861+0000 7f50b473a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:03:59.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:03:59.862+0000 7f50b473a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:03:59.880 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:03:59.880 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.881 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.882 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:03:59.885 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:03:59.885 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:03:59.885 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.886 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:03:59.887 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:03:59.887 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:03:59.887 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:03:59.887 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:03:59Z" 2024-04-13T23:03:59.887 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:03:59.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049314.558113_1103.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:01:54.558113Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:03:59Z" } 2024-04-13T23:03:59.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049314.558113_1103.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:01:54.558113Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:03:59.888 DEBUG:rgw_multi.tests:log_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:03:59.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:03:59.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:03:59.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:03:59.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:03:59.888 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:03:59.933 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:03:59.933 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:03:59.987 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:03:59.987 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-72 2024-04-13T23:03:59.987 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:03:59.987 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:03:59.987 DEBUG:boto:Method: PUT 2024-04-13T23:03:59.988 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:03:59.988 DEBUG:boto:Data: 2024-04-13T23:03:59.988 DEBUG:boto:Headers: {} 2024-04-13T23:03:59.988 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:03:59.988 DEBUG:boto:Port: 8000 2024-04-13T23:03:59.988 DEBUG:boto:Params: {} 2024-04-13T23:03:59.988 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:03:59.988 DEBUG:boto:Token: None 2024-04-13T23:03:59.989 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:03:59 GMT /vwnfns-72/ 2024-04-13T23:03:59.989 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JOPxjVuYCNx1F0sBBfNYWt7QVCE= 2024-04-13T23:03:59.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JOPxjVuYCNx1F0sBBfNYWt7QVCE=', 'Content-Length': '0'} 2024-04-13T23:04:00.014 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034a6fbe62b2c3bc9-00661b0f5f-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:04:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:04:00.014 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-73 2024-04-13T23:04:00.015 DEBUG:boto:path=/vwnfns-73/ 2024-04-13T23:04:00.015 DEBUG:boto:auth_path=/vwnfns-73/ 2024-04-13T23:04:00.015 DEBUG:boto:Method: PUT 2024-04-13T23:04:00.015 DEBUG:boto:Path: /vwnfns-73/ 2024-04-13T23:04:00.015 DEBUG:boto:Data: 2024-04-13T23:04:00.015 DEBUG:boto:Headers: {} 2024-04-13T23:04:00.015 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:04:00.015 DEBUG:boto:Port: 8000 2024-04-13T23:04:00.015 DEBUG:boto:Params: {} 2024-04-13T23:04:00.016 DEBUG:boto:Token: None 2024-04-13T23:04:00.016 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:04:00 GMT /vwnfns-73/ 2024-04-13T23:04:00.016 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:RXvAsvJhLU8PrEkQRoompo5STtk= 2024-04-13T23:04:00.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:04:00 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RXvAsvJhLU8PrEkQRoompo5STtk=', 'Content-Length': '0'} 2024-04-13T23:04:00.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000063aaec6ac66484c2-00661b0f60-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:04:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:04:00.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:00.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:00.042 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:00.089 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.089 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.124 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:04:00.138 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:04:00.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.138 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:04:00.140 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:04:00.035533Z" 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.142 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:04:00.142 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:04:00.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.142 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:04:00.143 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:04:00.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:04:00.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:00.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:00.144 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:00.194 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.195 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:04:00.246+0000 7fd9d39d9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:04:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:04:00.247+0000 7fd9d39d9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:04:00.264 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:00.264 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:00.264 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:00.264 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:00.264 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:00.264 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:04:00.265 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:00.267 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:00.268 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.269 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:00.270 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:04:00.270 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:04:00.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.270 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:00Z" 2024-04-13T23:04:00.270 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:00.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:04:00Z" } 2024-04-13T23:04:00.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:04:00.271 DEBUG:rgw_multi.tests:log_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:04:00.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:04:00.271 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:04:00.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.272 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:04:00.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.369 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:04:00.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.370 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:04:00.422 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.422 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.479 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:04:00.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-72', '--dest-bucket=vwnfns-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-72', '--dest-bucket=vwnfns-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.480 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=vwnfns-72 --dest-bucket=vwnfns-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:04:00.533 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.603 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-72","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:04:00.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.604 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:04:00.654 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.654 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.705 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-72","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:04:00.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.706 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:04:00.753 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.753 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.810 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:00.810 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:04:00.810 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 17, 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:04:00.811 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:04:00.812 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:04:00.813 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:04:00.814 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:04:00.815 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:04:00.816 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T23:04:00.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:04:00.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-72", 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-73", 2024-04-13T23:04:00.819 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "enabled" 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:04:00.822 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:04:00.823 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:04:00.824 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.825 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:04:00.826 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.827 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:04:00.828 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:04:00.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.829 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:04:00.829 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:04:00.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.829 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:04:00.830 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:04:00.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.830 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:04:00.831 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:04:00.832 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:04:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:04:00.834 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:04:00.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:00.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:00.835 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:04:00.835 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:04:00.835 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:00.835 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:00.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:04:00.836 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:04:00.880 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:00.880 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:00.929 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-72","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:04:00.930 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:04:00.930 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:04:00.931 DEBUG:boto:Method: HEAD 2024-04-13T23:04:00.931 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:04:00.931 DEBUG:boto:Data: 2024-04-13T23:04:00.931 DEBUG:boto:Headers: {} 2024-04-13T23:04:00.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:04:00.932 DEBUG:boto:Port: 8000 2024-04-13T23:04:00.932 DEBUG:boto:Params: {} 2024-04-13T23:04:00.932 DEBUG:boto:Token: None 2024-04-13T23:04:00.932 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:04:00 GMT /vwnfns-72/ 2024-04-13T23:04:00.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:eWlof0csPn+285SWsIXyYw9haxQ= 2024-04-13T23:04:00.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:04:00 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eWlof0csPn+285SWsIXyYw9haxQ=', 'Content-Length': '0'} 2024-04-13T23:04:01.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000221eaa260def4747-00661b0f61-2910-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:04:01.185 DEBUG:boto:path=/vwnfns-72/obj1 2024-04-13T23:04:01.185 DEBUG:boto:auth_path=/vwnfns-72/obj1 2024-04-13T23:04:01.185 DEBUG:boto:Method: PUT 2024-04-13T23:04:01.185 DEBUG:boto:Path: /vwnfns-72/obj1 2024-04-13T23:04:01.185 DEBUG:boto:Data: 2024-04-13T23:04:01.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:04:01.185 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:04:01.185 DEBUG:boto:Port: 8000 2024-04-13T23:04:01.185 DEBUG:boto:Params: {} 2024-04-13T23:04:01.185 DEBUG:boto:Token: None 2024-04-13T23:04:01.186 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:04:01 GMT /vwnfns-72/obj1 2024-04-13T23:04:01.186 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uXxyuEJD7Y7yCY/PTb+0erjAojs= 2024-04-13T23:04:01.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:04:01 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uXxyuEJD7Y7yCY/PTb+0erjAojs='} 2024-04-13T23:04:01.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fad8e91981ff414f-00661b0f61-2910-a1'), ('Date', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:04:01.210 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:04:01.210 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:04:01.210 DEBUG:boto:Method: HEAD 2024-04-13T23:04:01.210 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:04:01.210 DEBUG:boto:Data: 2024-04-13T23:04:01.210 DEBUG:boto:Headers: {} 2024-04-13T23:04:01.210 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:04:01.210 DEBUG:boto:Port: 8000 2024-04-13T23:04:01.210 DEBUG:boto:Params: {} 2024-04-13T23:04:01.211 DEBUG:boto:Token: None 2024-04-13T23:04:01.211 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:04:01 GMT /vwnfns-72/ 2024-04-13T23:04:01.211 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:MR4qCpl59iaKKbQX13mVm0DE2vc= 2024-04-13T23:04:01.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:04:01 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MR4qCpl59iaKKbQX13mVm0DE2vc=', 'Content-Length': '0'} 2024-04-13T23:04:01.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a8c690ce65957d2-00661b0f61-2910-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:04:01.216 DEBUG:boto:path=/vwnfns-72/obj2 2024-04-13T23:04:01.216 DEBUG:boto:auth_path=/vwnfns-72/obj2 2024-04-13T23:04:01.216 DEBUG:boto:Method: PUT 2024-04-13T23:04:01.216 DEBUG:boto:Path: /vwnfns-72/obj2 2024-04-13T23:04:01.216 DEBUG:boto:Data: 2024-04-13T23:04:01.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:04:01.217 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:04:01.217 DEBUG:boto:Port: 8000 2024-04-13T23:04:01.217 DEBUG:boto:Params: {} 2024-04-13T23:04:01.217 DEBUG:boto:Token: None 2024-04-13T23:04:01.217 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:04:01 GMT /vwnfns-72/obj2 2024-04-13T23:04:01.217 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:sDsbF0MJm5u9fKBI16M1ht7pNoo= 2024-04-13T23:04:01.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:04:01 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sDsbF0MJm5u9fKBI16M1ht7pNoo='} 2024-04-13T23:04:01.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4f3c3c38ba76e67-00661b0f61-2910-a1'), ('Date', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:04:01.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.263 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:01.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:01.331 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:01.406 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:04:01.440 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:04:00.035533Z" 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:04:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.442 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:04:01.443 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:04:01.444 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:04:01.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.444 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:01.505 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:01.505 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:01.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:04:01.568+0000 7f8b62298780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:04:01.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:04:01.571+0000 7f8b62298780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:04:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:01Z" 2024-04-13T23:04:01.617 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:01.617 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:04:01Z" } 2024-04-13T23:04:01.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:04:01.618 DEBUG:rgw_multi.tests:log_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:04:01.618 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:04:01.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:04:01.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.619 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:01.676 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:01.677 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:01.777 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:04:01.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.777 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:01.777 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:01.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047351", 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:04:01.232088Z" 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:04:01.205428Z" 2024-04-13T23:04:01.778 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.779 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:04:01.780 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:01.780 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:04:01.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:01.780 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:01.836 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:01.836 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:01.897 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.898 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:01.901 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:01.902 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:01Z" 2024-04-13T23:04:01.903 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:01.904 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:01Z" } 2024-04-13T23:04:01.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:01.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:01.904 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:01.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:06.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:06.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:06.906 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:06.964 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:06.965 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:07.016 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:07.017 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:07.018 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:07.019 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:07.020 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:07.021 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:07.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:07.022 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:07.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:07.022 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:07Z" 2024-04-13T23:04:07.022 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:07.023 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:07Z" } 2024-04-13T23:04:07.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:07.023 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:07.024 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:07.024 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:12.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:12.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:12.025 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:12.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:12.085 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:12.150 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:12Z" 2024-04-13T23:04:12.156 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:12.157 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:12Z" } 2024-04-13T23:04:12.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:12.158 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:12.158 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:12.158 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:17.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:17.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:17.159 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:17.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:17.218 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:17.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:17.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:17.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:17.280 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:17Z" 2024-04-13T23:04:17.281 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:17.282 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:17Z" } 2024-04-13T23:04:17.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:17.282 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:17.282 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:17.283 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:22.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:22.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:22.284 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:22.342 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:22.342 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:22.396 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:22.397 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:22.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:22.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:22.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:22.401 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:22Z" 2024-04-13T23:04:22.402 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:22.403 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:22Z" } 2024-04-13T23:04:22.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:22.404 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:22.404 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:22.404 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:27.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:27.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:27.405 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:27.463 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:27.463 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:27.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:27.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:27.527 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:27.528 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:27.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:27.530 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:27Z" 2024-04-13T23:04:27.530 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:27.530 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:27Z" } 2024-04-13T23:04:27.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:27.531 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:27.531 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:27.531 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:32.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:32.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:32.533 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:32.587 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:32.588 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:32.642 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:32.642 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:32.644 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:32.645 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:32.647 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:32.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:32.648 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:32Z" 2024-04-13T23:04:32.648 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:32.649 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:32Z" } 2024-04-13T23:04:32.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:32.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:32.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:32.649 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:37.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:37.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:37.651 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:37.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:37.708 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:37.772 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:37Z" 2024-04-13T23:04:37.779 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:37.780 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:37Z" } 2024-04-13T23:04:37.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:37.781 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:37.781 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:37.781 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:42.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:42.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:42.782 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:42.839 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:42.840 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:42.899 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:42.899 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:42.899 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:42.900 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:42.901 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:42.902 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:42.903 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:42.904 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:42.905 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:42.905 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:42.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:42.905 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:42Z" 2024-04-13T23:04:42.905 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:42.906 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:42Z" } 2024-04-13T23:04:42.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:42.906 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:42.906 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:42.907 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:47.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:47.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:47.908 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:47.966 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:47.967 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:48.020 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:48.021 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:48.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:48.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:48.024 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:48.025 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:48.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:48.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:48.025 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:48.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:48.026 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:48.026 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:48.026 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:48.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:48.027 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:48Z" 2024-04-13T23:04:48.027 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:48.028 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:48Z" } 2024-04-13T23:04:48.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:48.028 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:48.028 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:48.028 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:53.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:53.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:53.029 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:53.086 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:53.086 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:53.151 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:53.151 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:53.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:53.153 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:53.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:53.155 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:53.156 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:53.157 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:53.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:53.157 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:53Z" 2024-04-13T23:04:53.157 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:53.158 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:53Z" } 2024-04-13T23:04:53.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:53.158 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:53.158 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:53.158 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:04:58.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:58.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:04:58.160 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:04:58.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:04:58.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:58.280 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:58.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:58.282 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:04:58.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:04:58.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:04:58Z" 2024-04-13T23:04:58.285 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:04:58.286 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:04:58Z" } 2024-04-13T23:04:58.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:04:58.287 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:04:58.287 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:04:58.287 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:03.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:03.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:03.288 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:03.346 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:03.346 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:03.406 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:03.407 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:03.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:03.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:03.411 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:03Z" 2024-04-13T23:05:03.412 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:03.413 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:03Z" } 2024-04-13T23:05:03.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:03.413 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:03.414 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:03.414 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:08.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:08.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:08.415 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:08.472 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:08.472 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:08.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:08.537 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:08.538 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:08.539 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:08.540 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:08Z" 2024-04-13T23:05:08.541 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:08.542 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:08Z" } 2024-04-13T23:05:08.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:08.543 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:08.543 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:08.543 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:13.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:13.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:13.544 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:13.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:13.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:13.652 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:13.652 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:13.652 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:13.652 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:13.653 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:13.654 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:13.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:13.656 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:13.657 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:13.658 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:13.659 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:13.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:13.659 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:13Z" 2024-04-13T23:05:13.659 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:13.660 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:13Z" } 2024-04-13T23:05:13.660 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:13.660 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:13.660 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:13.661 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:18.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:18.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:18.663 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:18.721 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:18.721 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:18.789 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:18.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:18.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:18.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:18.794 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:18Z" 2024-04-13T23:05:18.795 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:18.796 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:18Z" } 2024-04-13T23:05:18.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:18.797 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:18.797 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:18.797 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:23.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:23.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:23.798 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:23.857 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:23.857 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:23.909 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:23.909 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:23.909 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:23.910 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:23.911 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:23.912 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:23.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:23.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:23Z" 2024-04-13T23:05:23.915 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:23.916 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:23Z" } 2024-04-13T23:05:23.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:23.917 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:23.917 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:23.917 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:28.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:28.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:28.919 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:28.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:28.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:29.041 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:29.041 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:29.042 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:29.043 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:29.045 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:29.046 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:29.047 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:29.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:29.047 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:29Z" 2024-04-13T23:05:29.047 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:29.048 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:29Z" } 2024-04-13T23:05:29.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:29.048 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:29.048 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:29.048 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:32.199 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T23:05:32.197+0000 7f5b022dd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T23:05:34.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:34.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:34.050 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:34.110 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:34.110 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:34.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:34.168 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:34.169 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:34.170 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:34.171 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:34.172 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:34.172 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:34.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:34.172 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:34Z" 2024-04-13T23:05:34.172 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:34.173 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:34Z" } 2024-04-13T23:05:34.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:34.173 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:34.173 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:34.173 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:39.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:39.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:39.175 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:39.232 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:39.232 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:39.292 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:39.292 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:39.292 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:39.292 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:39.292 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:39.292 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:39.293 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:39.294 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:39.295 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:39.296 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:39Z" 2024-04-13T23:05:39.297 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:39.298 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:39Z" } 2024-04-13T23:05:39.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:39.299 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:39.299 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:39.299 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:44.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:44.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:44.301 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:44.359 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:44.359 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:44.415 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:44Z" 2024-04-13T23:05:44.421 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:44.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:44Z" } 2024-04-13T23:05:44.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:44.422 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:44.422 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:44.422 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:49.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:49.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:49.423 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:49.483 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:49.483 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:49.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:49.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:49.549 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:49.549 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:49.549 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:49.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:49.549 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:49Z" 2024-04-13T23:05:49.549 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:49.550 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:49Z" } 2024-04-13T23:05:49.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:49.550 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:49.550 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:49.550 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:54.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:54.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:54.552 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:54.609 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:54.610 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:54.671 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:54.672 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:54.673 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:54.674 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:54.677 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:54.677 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:54.677 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:54.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:54.677 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:54Z" 2024-04-13T23:05:54.677 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:54.678 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:54Z" } 2024-04-13T23:05:54.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:54.679 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:54.679 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:54.679 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:05:59.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:59.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:05:59.680 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:05:59.737 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:05:59.738 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:05:59.791 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:05:59.791 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:05:59.791 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:05:59.791 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:05:59.791 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:59.792 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:59.793 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:05:59.794 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:05:59.795 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:05:59Z" 2024-04-13T23:05:59.796 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:05:59.797 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.789755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053505", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:01:55.836207Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:05:59Z" } 2024-04-13T23:05:59.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:05:59.798 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:05:59.798 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:05:59.798 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047351 target=00000000000000000000:00000000000000047211, shard=3 master=00000000000000000000:00000000000000053645 target=00000000000000000000:00000000000000053505 2024-04-13T23:06:04.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:04.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:04.799 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:04.856 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:04.856 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:05.914 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:05.915 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047351", 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:05.916 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.232088Z" 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:05.917 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:05.918 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.205428Z" 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:05Z" 2024-04-13T23:06:05.919 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:05.920 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047351", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:04:01.232088Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:04:01.205428Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:06:05Z" } 2024-04-13T23:06:05.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047351', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.232088Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.205428Z'}}] 2024-04-13T23:06:05.921 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:06:05.921 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:06:05.921 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:06:05.921 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:06:05.921 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:06:05.922 DEBUG:boto:Method: HEAD 2024-04-13T23:06:05.922 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:06:05.922 DEBUG:boto:Data: 2024-04-13T23:06:05.922 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.922 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.922 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.922 DEBUG:boto:Params: {} 2024-04-13T23:06:05.923 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:06:05.923 DEBUG:boto:Token: None 2024-04-13T23:06:05.923 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-72/ 2024-04-13T23:06:05.923 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lKaHd3yO/WQi6AeAJP4Db5OzNzY= 2024-04-13T23:06:05.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lKaHd3yO/WQi6AeAJP4Db5OzNzY=', 'Content-Length': '0'} 2024-04-13T23:06:05.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff5edc165c0efa15-00661b0fdd-290c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.931 DEBUG:boto:path=/vwnfns-72/obj1 2024-04-13T23:06:05.931 DEBUG:boto:auth_path=/vwnfns-72/obj1 2024-04-13T23:06:05.931 DEBUG:boto:Method: HEAD 2024-04-13T23:06:05.931 DEBUG:boto:Path: /vwnfns-72/obj1 2024-04-13T23:06:05.931 DEBUG:boto:Data: 2024-04-13T23:06:05.931 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.931 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.931 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.931 DEBUG:boto:Params: {} 2024-04-13T23:06:05.932 DEBUG:boto:Token: None 2024-04-13T23:06:05.932 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-72/obj1 2024-04-13T23:06:05.932 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:42ouYefhI2bjLPYowk3SPQ2vQFw= 2024-04-13T23:06:05.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:42ouYefhI2bjLPYowk3SPQ2vQFw=', 'Content-Length': '0'} 2024-04-13T23:06:05.933 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002c550d62566ff6be-00661b0fdd-290c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.934 DEBUG:boto:path=/vwnfns-72/obj2 2024-04-13T23:06:05.934 DEBUG:boto:auth_path=/vwnfns-72/obj2 2024-04-13T23:06:05.934 DEBUG:boto:Method: HEAD 2024-04-13T23:06:05.934 DEBUG:boto:Path: /vwnfns-72/obj2 2024-04-13T23:06:05.934 DEBUG:boto:Data: 2024-04-13T23:06:05.934 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.934 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.934 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.934 DEBUG:boto:Params: {} 2024-04-13T23:06:05.934 DEBUG:boto:Token: None 2024-04-13T23:06:05.935 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-72/obj2 2024-04-13T23:06:05.935 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jOm3D6RJtxhSM54+NVHlQPzk3Zg= 2024-04-13T23:06:05.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jOm3D6RJtxhSM54+NVHlQPzk3Zg=', 'Content-Length': '0'} 2024-04-13T23:06:05.936 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c58cef79b94bc0be-00661b0fdd-290c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.936 DEBUG:boto:path=/vwnfns-73/ 2024-04-13T23:06:05.936 DEBUG:boto:auth_path=/vwnfns-73/ 2024-04-13T23:06:05.937 DEBUG:boto:Method: HEAD 2024-04-13T23:06:05.937 DEBUG:boto:Path: /vwnfns-73/ 2024-04-13T23:06:05.937 DEBUG:boto:Data: 2024-04-13T23:06:05.937 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.937 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.937 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.937 DEBUG:boto:Params: {} 2024-04-13T23:06:05.937 DEBUG:boto:Token: None 2024-04-13T23:06:05.937 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-73/ 2024-04-13T23:06:05.938 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:32w2XD70KzpbIuTBJv3fnet4UaQ= 2024-04-13T23:06:05.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:32w2XD70KzpbIuTBJv3fnet4UaQ=', 'Content-Length': '0'} 2024-04-13T23:06:05.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3cddd57d1c9cfa7-00661b0fdd-290c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.941 DEBUG:boto:path=/vwnfns-73/obj1 2024-04-13T23:06:05.941 DEBUG:boto:auth_path=/vwnfns-73/obj1 2024-04-13T23:06:05.941 DEBUG:boto:Method: HEAD 2024-04-13T23:06:05.941 DEBUG:boto:Path: /vwnfns-73/obj1 2024-04-13T23:06:05.941 DEBUG:boto:Data: 2024-04-13T23:06:05.941 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.941 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.941 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.941 DEBUG:boto:Params: {} 2024-04-13T23:06:05.942 DEBUG:boto:Token: None 2024-04-13T23:06:05.942 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-73/obj1 2024-04-13T23:06:05.942 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xwtOEx8Hs31mIQ4y5mR63X2ARMQ= 2024-04-13T23:06:05.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xwtOEx8Hs31mIQ4y5mR63X2ARMQ=', 'Content-Length': '0'} 2024-04-13T23:06:05.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdd2e79deb803148-00661b0fdd-290c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.944 DEBUG:boto:path=/vwnfns-73/obj1 2024-04-13T23:06:05.944 DEBUG:boto:auth_path=/vwnfns-73/obj1 2024-04-13T23:06:05.944 DEBUG:boto:Method: GET 2024-04-13T23:06:05.945 DEBUG:boto:Path: /vwnfns-73/obj1 2024-04-13T23:06:05.945 DEBUG:boto:Data: 2024-04-13T23:06:05.945 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.945 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.945 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.945 DEBUG:boto:Params: {} 2024-04-13T23:06:05.945 DEBUG:boto:Token: None 2024-04-13T23:06:05.945 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-73/obj1 2024-04-13T23:06:05.945 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:NeLtbkqxbKkWA8wYYF7CUsA5Ers= 2024-04-13T23:06:05.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NeLtbkqxbKkWA8wYYF7CUsA5Ers=', 'Content-Length': '0'} 2024-04-13T23:06:05.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d3529780597ba7a-00661b0fdd-290c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.948 DEBUG:boto:path=/vwnfns-73/obj2 2024-04-13T23:06:05.948 DEBUG:boto:auth_path=/vwnfns-73/obj2 2024-04-13T23:06:05.948 DEBUG:boto:Method: HEAD 2024-04-13T23:06:05.948 DEBUG:boto:Path: /vwnfns-73/obj2 2024-04-13T23:06:05.948 DEBUG:boto:Data: 2024-04-13T23:06:05.948 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.948 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.948 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.948 DEBUG:boto:Params: {} 2024-04-13T23:06:05.949 DEBUG:boto:Token: None 2024-04-13T23:06:05.949 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-73/obj2 2024-04-13T23:06:05.949 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:xONNCRBC9PGhMF3VnYtEYnMwmm4= 2024-04-13T23:06:05.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:xONNCRBC9PGhMF3VnYtEYnMwmm4=', 'Content-Length': '0'} 2024-04-13T23:06:05.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aeb0e09302158eb-00661b0fdd-290c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.951 DEBUG:boto:path=/vwnfns-73/obj2 2024-04-13T23:06:05.951 DEBUG:boto:auth_path=/vwnfns-73/obj2 2024-04-13T23:06:05.951 DEBUG:boto:Method: GET 2024-04-13T23:06:05.951 DEBUG:boto:Path: /vwnfns-73/obj2 2024-04-13T23:06:05.951 DEBUG:boto:Data: 2024-04-13T23:06:05.951 DEBUG:boto:Headers: {} 2024-04-13T23:06:05.951 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:06:05.951 DEBUG:boto:Port: 8001 2024-04-13T23:06:05.951 DEBUG:boto:Params: {} 2024-04-13T23:06:05.951 DEBUG:boto:Token: None 2024-04-13T23:06:05.952 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:06:05 GMT /vwnfns-73/obj2 2024-04-13T23:06:05.952 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:x71ZcMj+i+HHX5gOJ0C9gAuvx3U= 2024-04-13T23:06:05.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:06:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x71ZcMj+i+HHX5gOJ0C9gAuvx3U=', 'Content-Length': '0'} 2024-04-13T23:06:05.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:04:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e278480e03a7b87b-00661b0fdd-290c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:06:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:06:05.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:05.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:05.995 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.049 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.049 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.109 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-13T23:06:06.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.110 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.161 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.162 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.220 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:06:06.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.221 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.268 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.268 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.316 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:06:06.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.316 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.363 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.363 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 18, 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:06:06.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:06:06.439 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:06:06.440 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:06:06.441 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:06:06.442 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:06:06.443 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.444 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.446 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:06:06.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "allowed" 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.448 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:06:06.449 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:06:06.450 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:06:06.451 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:06:06.452 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.453 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:06:06.454 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:06:06.455 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:06:06.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.456 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:06:06.456 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:06:06.457 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:06:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:06:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:06:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:06:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:06:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:06.462 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:06.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.465 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.596 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:06:06.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.597 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.777 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:06:06.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.778 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:06.847 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:06.847 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:06.956 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:06:06.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=vwnfns-72', '--source-bucket=*', '--dest-bucket=vwnfns-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=vwnfns-72', '--source-bucket=*', '--dest-bucket=vwnfns-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:06.956 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=vwnfns-72 '--source-bucket=*' --dest-bucket=vwnfns-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:07.020 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:07.021 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:07.120 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-13T23:06:07.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:07.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:07.121 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:07.180 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:07.180 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:07.266 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T23:06:07.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:07.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:06:07.267 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:06:07.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:07.320 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:07.399 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T23:06:07.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:07.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:07.400 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:07.452 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:07.452 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:07.495 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:06:07.508 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:06:07.241183Z" 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:06:07.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:07.510 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:06:07.511 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:07.512 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:06:07.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:07.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:07.512 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:07.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:07.561 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:07.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:07.592+0000 7f5d9861a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:07.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:07.593+0000 7f5d9861a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:07.611 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.612 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:07.613 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:07.615 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:07.616 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:07.617 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:07Z" 2024-04-13T23:06:07.618 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:07.619 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:07Z" } 2024-04-13T23:06:07.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:07.619 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:07.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:07.619 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:12.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:12.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:12.620 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:12.675 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:12.676 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:12.713+0000 7fac14b61780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:12.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:12.714+0000 7fac14b61780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:12.730 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:12.730 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:12.730 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:12.730 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:12.730 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:12.730 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:12.731 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:12.732 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:12.733 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:12.734 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:12.735 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:12.736 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:12.736 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:12.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:12.736 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:12Z" 2024-04-13T23:06:12.736 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:12.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:12Z" } 2024-04-13T23:06:12.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:12.737 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:12.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:12.737 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:17.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:17.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:17.739 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:17.795 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:17.795 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:17.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:17.835+0000 7f0cc3d68780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:17.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:17.836+0000 7f0cc3d68780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:17.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:17.853 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:17.854 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:17.855 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:17.856 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:17.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:17.858 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:17Z" 2024-04-13T23:06:17.858 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:17.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:17Z" } 2024-04-13T23:06:17.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:17.859 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:17.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:17.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:22.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:22.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:22.861 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:22.916 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:22.916 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:22.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:22.957+0000 7f779ebb3780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:22.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:22.958+0000 7f779ebb3780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:22.975 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:22.975 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:22.975 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:22.976 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:22.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:22.979 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:22.980 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:22.981 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:22Z" 2024-04-13T23:06:22.982 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:22.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:22Z" } 2024-04-13T23:06:22.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:22.983 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:22.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:22.983 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:27.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:27.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:27.985 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:28.041 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:28.041 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:28.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:28.072+0000 7f70d1159780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:28.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:28.073+0000 7f70d1159780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:28.093 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:28Z" 2024-04-13T23:06:28.099 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:28.100 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:28Z" } 2024-04-13T23:06:28.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:28.100 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:28.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:28.101 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:33.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:33.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:33.102 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:33.158 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:33.158 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:33.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:33.202+0000 7fe57defd780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:33.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:33.203+0000 7fe57defd780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:33.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:33.226 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:33.227 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:33.228 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:33.229 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:33.230 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:33.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:33.231 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:33.231 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:33.231 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:33.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:33.232 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:33Z" 2024-04-13T23:06:33.232 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:33.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:33Z" } 2024-04-13T23:06:33.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:33.234 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:33.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:33.234 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:38.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:38.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:38.236 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:38.291 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:38.291 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:38.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:38.327+0000 7f55d499b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:38.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:38.328+0000 7f55d499b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:38.349 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:38.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:38.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:38.352 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:38.353 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:38.354 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:38.355 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:38Z" 2024-04-13T23:06:38.356 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:38.356 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:38Z" } 2024-04-13T23:06:38.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:38.357 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:38.357 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:38.357 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:43.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:43.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:43.358 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:43.414 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:43.414 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:43.447+0000 7f84e7fd5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:43.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:43.448+0000 7f84e7fd5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:43.465 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:43.465 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:43.465 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:43.465 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:43.465 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:43.465 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:43.466 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:43.467 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:43.468 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:43.469 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:43.471 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:43.471 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:43.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:43.471 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:43Z" 2024-04-13T23:06:43.471 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:43.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:43Z" } 2024-04-13T23:06:43.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:43.472 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:43.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:43.472 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:48.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:48.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:48.474 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:48.531 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:48.531 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:48.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:48.564+0000 7f7ffb454780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:48.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:48.566+0000 7f7ffb454780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:48.584 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:48Z" 2024-04-13T23:06:48.590 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:48.591 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:48Z" } 2024-04-13T23:06:48.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:48.591 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:48.592 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:48.592 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:53.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:53.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:53.594 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:53.650 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:53.650 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:53.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:53.685+0000 7fd30238f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:53.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:53.686+0000 7fd30238f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:53.705 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:53Z" 2024-04-13T23:06:53.712 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:53.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:53Z" } 2024-04-13T23:06:53.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:53.713 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:53.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:53.713 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:06:58.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:58.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:06:58.714 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:06:58.770 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:06:58.770 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:06:58.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:58.811+0000 7faf4c667780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:58.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:06:58.812+0000 7faf4c667780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:58.832 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:58.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:58.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:06:58.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:06:58Z" 2024-04-13T23:06:58.837 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:06:58.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:06:58Z" } 2024-04-13T23:06:58.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:06:58.838 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:58.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:06:58.839 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:03.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:03.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:03.840 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:03.896 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:03.896 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:03.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:03.927+0000 7f2b7c933780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:03.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:03.928+0000 7f2b7c933780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:03.945 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:03.946 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:03.947 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:03.948 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:03.948 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:03.948 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:03.948 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:03.948 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:03.948 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:03.949 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:03.950 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:03.950 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:03.950 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:03.950 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:03.950 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:03.950 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:03.951 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:03.952 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:03.953 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:03.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:03.953 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:03Z" 2024-04-13T23:07:03.953 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:03.954 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:03Z" } 2024-04-13T23:07:03.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:03.954 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:03.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:03.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:08.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:08.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:08.957 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:09.013 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:09.013 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:09.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:09.050+0000 7f7591373780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:09.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:09.051+0000 7f7591373780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:09.070 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:09.071 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:09.072 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:09.073 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:09.074 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:09.075 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:09.076 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:09.076 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:09.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:09.076 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:09Z" 2024-04-13T23:07:09.076 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:09.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:09Z" } 2024-04-13T23:07:09.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:09.077 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:09.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:09.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:14.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:14.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:14.079 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:14.134 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:14.135 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:14.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:14.167+0000 7f63eeebb780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:14.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:14.168+0000 7f63eeebb780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:14.186 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:14.187 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:14.188 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:14.189 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:14.190 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:14.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:14.192 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:14Z" 2024-04-13T23:07:14.192 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:14.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:14Z" } 2024-04-13T23:07:14.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:14.193 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:14.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:14.193 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:19.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:19.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:19.195 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:19.250 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:19.251 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:19.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:19.298+0000 7fe4339ef780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:19.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:19.299+0000 7fe4339ef780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:19.321 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:19.322 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:19.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:19.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:19.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:19.326 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:19Z" 2024-04-13T23:07:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:19.328 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:19Z" } 2024-04-13T23:07:19.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:19.328 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:19.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:19.329 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:24.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:24.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:24.330 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:24.385 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:24.386 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:24.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:24.419+0000 7f5a49599780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:24.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:24.420+0000 7f5a49599780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:24.436 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:24.436 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:24.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:24.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:24.440 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:24.441 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:24.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:24.443 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:24Z" 2024-04-13T23:07:24.443 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:24.444 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:24Z" } 2024-04-13T23:07:24.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:24.444 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:24.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:24.445 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:29.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:29.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:29.447 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:29.504 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:29.504 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:29.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:29.550+0000 7f7870684780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:29.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:29.551+0000 7f7870684780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:29.572 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:29.573 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:29.575 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:29.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:29.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:29.578 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:29Z" 2024-04-13T23:07:29.578 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:29.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:29Z" } 2024-04-13T23:07:29.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:29.579 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:29.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:29.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:34.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:34.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:34.581 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:34.637 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:34.637 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:34.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:34.670+0000 7f6c8b20a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:34.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:34.671+0000 7f6c8b20a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:34.687 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:34.687 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:34.687 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:34.687 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:34.687 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:34.688 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:34.689 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:34.690 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:34.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:34.692 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:34Z" 2024-04-13T23:07:34.693 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:34.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:34Z" } 2024-04-13T23:07:34.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:34.695 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:34.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:34.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:39.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:39.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:39.697 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:39.754 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:39.754 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:39.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:39.797+0000 7f4ac0c26780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:39.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:39.798+0000 7f4ac0c26780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:39.815 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:39.815 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:39.815 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:39.816 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:39.817 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:39.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:39.819 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:39.820 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:39.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:39.822 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:39Z" 2024-04-13T23:07:39.822 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:39.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:39Z" } 2024-04-13T23:07:39.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:39.824 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:39.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:39.824 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:44.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:44.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:44.825 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:44.882 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:44.882 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:44.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:44.915+0000 7fe2cb55d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:44.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:44.916+0000 7fe2cb55d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:44.933 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:44.934 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:44.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:44.936 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:44.937 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:44.938 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:44.939 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:44.939 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:44.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:44.939 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:44Z" 2024-04-13T23:07:44.939 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:44.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:44Z" } 2024-04-13T23:07:44.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:44.940 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:44.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:44.941 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:49.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:49.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:49.943 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:49.998 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:49.998 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:50.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:50.031+0000 7f7a7c6da780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:50.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:50.032+0000 7f7a7c6da780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:50.056 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:50.057 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:50.058 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:50.060 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:50.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:50.062 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:50Z" 2024-04-13T23:07:50.062 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:50.063 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:50Z" } 2024-04-13T23:07:50.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:50.063 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:50.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:50.063 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:07:55.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:55.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:07:55.065 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:07:55.120 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:07:55.120 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:07:55.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:55.154+0000 7f0f8b038780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:55.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:07:55.155+0000 7f0f8b038780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:07:55.172 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:07:55.172 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:07:55.172 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:07:55.172 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:07:55.172 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:07:55.172 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:07:55.173 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:07:55.174 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:55.175 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:55.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:07:55.177 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:07:55Z" 2024-04-13T23:07:55.178 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:07:55.179 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:07:55Z" } 2024-04-13T23:07:55.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:07:55.180 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:55.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:07:55.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:08:00.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:00.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:00.181 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:00.238 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:00.238 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:00.269+0000 7f9fbf42b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:00.270+0000 7f9fbf42b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:00.291 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:00.294 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:00.295 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:00.296 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:00.297 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:00.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:00.298 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:00Z" 2024-04-13T23:08:00.298 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:00.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:00Z" } 2024-04-13T23:08:00.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:00.299 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:00.299 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:00.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:08:05.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:05.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:05.301 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:05.357 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:05.357 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:05.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:05.388+0000 7f6dc4ab3780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:05.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:05.389+0000 7f6dc4ab3780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:05.404 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049440.035533_1130.1", 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:05.405 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:00.035533Z", 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:05.406 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:05.407 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:05.408 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:05.410 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:05.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:05.410 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:05Z" 2024-04-13T23:08:05.410 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:05.411 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049440.035533_1130.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:04:00.035533Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:05Z" } 2024-04-13T23:08:05.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049440.035533_1130.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:04:00.035533Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:05.411 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:05.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049440.035533_1130.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:05.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049567.241183_1172.1 target=1_1713049440.035533_1130.1 2024-04-13T23:08:10.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.413 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:10.468 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:10.468 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:10.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:10.514+0000 7fb953e09780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:10.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:10.515+0000 7fb953e09780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:10.530 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:10.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:06:07.241183Z", 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.534 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.535 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:10.536 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:10.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.537 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:10Z" 2024-04-13T23:08:10.537 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:10.538 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049567.241183_1172.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:06:07.241183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:10Z" } 2024-04-13T23:08:10.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049567.241183_1172.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:06:07.241183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:10.538 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:10.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:10.538 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:08:10.539 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:08:10.539 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:08:10.539 DEBUG:boto:Method: HEAD 2024-04-13T23:08:10.539 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:08:10.540 DEBUG:boto:Data: 2024-04-13T23:08:10.540 DEBUG:boto:Headers: {} 2024-04-13T23:08:10.540 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:08:10.540 DEBUG:boto:Port: 8000 2024-04-13T23:08:10.540 DEBUG:boto:Params: {} 2024-04-13T23:08:10.540 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:08:10.541 DEBUG:boto:Token: None 2024-04-13T23:08:10.541 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:10 GMT /vwnfns-72/ 2024-04-13T23:08:10.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VXDGrsCmlQ9zXMMbB9hIXMygQqs= 2024-04-13T23:08:10.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VXDGrsCmlQ9zXMMbB9hIXMygQqs=', 'Content-Length': '0'} 2024-04-13T23:08:10.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9891406c6e22a86-00661b105a-2a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:10.548 DEBUG:boto:path=/vwnfns-72/obj5 2024-04-13T23:08:10.549 DEBUG:boto:auth_path=/vwnfns-72/obj5 2024-04-13T23:08:10.549 DEBUG:boto:Method: PUT 2024-04-13T23:08:10.549 DEBUG:boto:Path: /vwnfns-72/obj5 2024-04-13T23:08:10.549 DEBUG:boto:Data: 2024-04-13T23:08:10.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:08:10.549 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:08:10.549 DEBUG:boto:Port: 8000 2024-04-13T23:08:10.549 DEBUG:boto:Params: {} 2024-04-13T23:08:10.549 DEBUG:boto:Token: None 2024-04-13T23:08:10.550 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:08:10 GMT /vwnfns-72/obj5 2024-04-13T23:08:10.550 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ckrnBmMIjQ1J35J1ErC+iRQjABE= 2024-04-13T23:08:10.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:08:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ckrnBmMIjQ1J35J1ErC+iRQjABE='} 2024-04-13T23:08:10.556 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5e15bd04523b80e-00661b105a-2a22-a1'), ('Date', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:10.556 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:08:10.556 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:08:10.557 DEBUG:boto:Method: HEAD 2024-04-13T23:08:10.557 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:08:10.557 DEBUG:boto:Data: 2024-04-13T23:08:10.557 DEBUG:boto:Headers: {} 2024-04-13T23:08:10.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:08:10.557 DEBUG:boto:Port: 8000 2024-04-13T23:08:10.557 DEBUG:boto:Params: {} 2024-04-13T23:08:10.557 DEBUG:boto:Token: None 2024-04-13T23:08:10.557 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:10 GMT /vwnfns-72/ 2024-04-13T23:08:10.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VXDGrsCmlQ9zXMMbB9hIXMygQqs= 2024-04-13T23:08:10.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VXDGrsCmlQ9zXMMbB9hIXMygQqs=', 'Content-Length': '0'} 2024-04-13T23:08:10.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021cd9898fa850d32-00661b105a-2a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:10.560 DEBUG:boto:path=/vwnfns-72/obj6 2024-04-13T23:08:10.560 DEBUG:boto:auth_path=/vwnfns-72/obj6 2024-04-13T23:08:10.560 DEBUG:boto:Method: PUT 2024-04-13T23:08:10.560 DEBUG:boto:Path: /vwnfns-72/obj6 2024-04-13T23:08:10.560 DEBUG:boto:Data: 2024-04-13T23:08:10.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:08:10.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:08:10.561 DEBUG:boto:Port: 8000 2024-04-13T23:08:10.561 DEBUG:boto:Params: {} 2024-04-13T23:08:10.561 DEBUG:boto:Token: None 2024-04-13T23:08:10.561 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:08:10 GMT /vwnfns-72/obj6 2024-04-13T23:08:10.561 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l7g8JfjW/QReMMorP6e1DiEvK64= 2024-04-13T23:08:10.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:08:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l7g8JfjW/QReMMorP6e1DiEvK64='} 2024-04-13T23:08:10.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000166cef5c39f9dee4-00661b105a-2a22-a1'), ('Date', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:10.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.603 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:10.650 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:10.650 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:10.685 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:08:10.706 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:08:10.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.706 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:10.706 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:06:07.241183Z" 2024-04-13T23:08:10.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:08:10.707 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.708 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:08:10.709 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:10.709 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:08:10.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.709 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:10.757 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:10.757 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:10.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:10.797+0000 7f0713dba780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:10.798+0000 7f0713dba780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:10.816 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:10.816 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.817 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:06:07.241183Z", 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.818 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:10.821 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:10.822 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:10.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.823 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:10Z" 2024-04-13T23:08:10.823 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:10.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049567.241183_1172.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:06:07.241183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:10Z" } 2024-04-13T23:08:10.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049567.241183_1172.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:06:07.241183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:10.824 DEBUG:rgw_multi.tests:log_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:10.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:10.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:08:10.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.825 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:10.871 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:10.872 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:10.927 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:08:10.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:08:10.928 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:08:10.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:10.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:10.929 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:08:10.929 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:08:10.929 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:10.929 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:08:10.930 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:10.930 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:08:10.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:10.931 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:11.030 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:11.030 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:11.030 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:11.030 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:11.030 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:11.030 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:11.031 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047351", 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.232088Z" 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.205428Z" 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:11Z" 2024-04-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:11.036 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047351", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:04:01.232088Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:04:01.205428Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:08:11Z" } 2024-04-13T23:08:11.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047351', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.232088Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.205428Z'}}] 2024-04-13T23:08:11.037 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:11.037 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:08:11.037 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047492 target=00000000000000000000:00000000000000047351, shard=3 master=00000000000000000000:00000000000000053786 target=00000000000000000000:00000000000000053645 2024-04-13T23:08:16.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:16.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:16.038 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:16.096 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:16.096 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:16.151 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:16.151 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:16.151 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:16.151 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:16.151 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:16.151 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:16.152 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047351", 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:16.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.232088Z" 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:08:16.154 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:16.155 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.205428Z" 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:16.156 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:16Z" 2024-04-13T23:08:16.157 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:16.158 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047351", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:04:01.232088Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:04:01.205428Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:08:16Z" } 2024-04-13T23:08:16.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047351', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.232088Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.205428Z'}}] 2024-04-13T23:08:16.158 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:16.158 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:08:16.158 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047492 target=00000000000000000000:00000000000000047351, shard=3 master=00000000000000000000:00000000000000053786 target=00000000000000000000:00000000000000053645 2024-04-13T23:08:21.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:21.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:21.160 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:21.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:21.217 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:21.289 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:21.290 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:21.291 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047351", 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.232088Z" 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:21.292 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:21.293 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.205428Z" 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:08:21.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:21.295 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:21Z" 2024-04-13T23:08:21.295 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:21.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047351", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:04:01.232088Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:04:01.205428Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:08:21Z" } 2024-04-13T23:08:21.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047351', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.232088Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.205428Z'}}] 2024-04-13T23:08:21.296 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:21.296 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:08:21.296 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047492 target=00000000000000000000:00000000000000047351, shard=3 master=00000000000000000000:00000000000000053786 target=00000000000000000000:00000000000000053645 2024-04-13T23:08:26.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:26.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:26.297 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:26.353 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:26.353 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:26.404 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:26.405 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:26.406 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047351", 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.232088Z" 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:26.407 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053645", 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:26.408 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:04:01.205428Z" 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:26Z" 2024-04-13T23:08:26.409 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:26.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047351", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:04:01.232088Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053645", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:04:01.205428Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:08:26Z" } 2024-04-13T23:08:26.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047351', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.232088Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053645', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:04:01.205428Z'}}] 2024-04-13T23:08:26.411 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:26.411 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047351', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053645'} 2024-04-13T23:08:26.411 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047492 target=00000000000000000000:00000000000000047351, shard=3 master=00000000000000000000:00000000000000053786 target=00000000000000000000:00000000000000053645 2024-04-13T23:08:31.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:31.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:31.412 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:31.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:31.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:31.537 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:31.537 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:31.537 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:31.537 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:31.537 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:31.538 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:08:31.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.541 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:31Z" 2024-04-13T23:08:31.542 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:31.543 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046295", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:01:55.861427Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033468", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T22:53:11.276425Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:08:31Z" } 2024-04-13T23:08:31.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:08:31.543 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:31.543 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:08:31.544 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:08:31.544 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:08:31.544 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:08:31.544 DEBUG:boto:Method: HEAD 2024-04-13T23:08:31.544 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:08:31.544 DEBUG:boto:Data: 2024-04-13T23:08:31.544 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.544 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.545 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.545 DEBUG:boto:Params: {} 2024-04-13T23:08:31.545 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:08:31.545 DEBUG:boto:Token: None 2024-04-13T23:08:31.545 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-72/ 2024-04-13T23:08:31.545 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9T3iOcJHzyCCc/W+lqVt67+g97U= 2024-04-13T23:08:31.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9T3iOcJHzyCCc/W+lqVt67+g97U=', 'Content-Length': '0'} 2024-04-13T23:08:31.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000649b4c090cce450a-00661b106f-2a13-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.551 DEBUG:boto:path=/vwnfns-72/obj5 2024-04-13T23:08:31.551 DEBUG:boto:auth_path=/vwnfns-72/obj5 2024-04-13T23:08:31.552 DEBUG:boto:Method: HEAD 2024-04-13T23:08:31.552 DEBUG:boto:Path: /vwnfns-72/obj5 2024-04-13T23:08:31.552 DEBUG:boto:Data: 2024-04-13T23:08:31.552 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.552 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.552 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.552 DEBUG:boto:Params: {} 2024-04-13T23:08:31.552 DEBUG:boto:Token: None 2024-04-13T23:08:31.552 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-72/obj5 2024-04-13T23:08:31.552 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:JuF9rE6ORonEvokavVM0izUS21U= 2024-04-13T23:08:31.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JuF9rE6ORonEvokavVM0izUS21U=', 'Content-Length': '0'} 2024-04-13T23:08:31.554 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002b86ece22038cf45-00661b106f-2a13-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.554 DEBUG:boto:path=/vwnfns-72/obj6 2024-04-13T23:08:31.554 DEBUG:boto:auth_path=/vwnfns-72/obj6 2024-04-13T23:08:31.554 DEBUG:boto:Method: HEAD 2024-04-13T23:08:31.554 DEBUG:boto:Path: /vwnfns-72/obj6 2024-04-13T23:08:31.554 DEBUG:boto:Data: 2024-04-13T23:08:31.554 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.555 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.555 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.555 DEBUG:boto:Params: {} 2024-04-13T23:08:31.555 DEBUG:boto:Token: None 2024-04-13T23:08:31.555 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-72/obj6 2024-04-13T23:08:31.555 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ucJGgMVPYDPsTsOrT2wgZdpRkNo= 2024-04-13T23:08:31.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ucJGgMVPYDPsTsOrT2wgZdpRkNo=', 'Content-Length': '0'} 2024-04-13T23:08:31.556 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003d431a698beaa2c7-00661b106f-2a13-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.556 DEBUG:boto:path=/vwnfns-73/ 2024-04-13T23:08:31.557 DEBUG:boto:auth_path=/vwnfns-73/ 2024-04-13T23:08:31.557 DEBUG:boto:Method: HEAD 2024-04-13T23:08:31.557 DEBUG:boto:Path: /vwnfns-73/ 2024-04-13T23:08:31.557 DEBUG:boto:Data: 2024-04-13T23:08:31.557 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.557 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.557 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.557 DEBUG:boto:Params: {} 2024-04-13T23:08:31.557 DEBUG:boto:Token: None 2024-04-13T23:08:31.557 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-73/ 2024-04-13T23:08:31.557 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:grr/xlKqEAHVfsEakWddOaPS1sg= 2024-04-13T23:08:31.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:grr/xlKqEAHVfsEakWddOaPS1sg=', 'Content-Length': '0'} 2024-04-13T23:08:31.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005af7b8658ef8a1be-00661b106f-2a13-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.560 DEBUG:boto:path=/vwnfns-73/obj5 2024-04-13T23:08:31.560 DEBUG:boto:auth_path=/vwnfns-73/obj5 2024-04-13T23:08:31.560 DEBUG:boto:Method: HEAD 2024-04-13T23:08:31.560 DEBUG:boto:Path: /vwnfns-73/obj5 2024-04-13T23:08:31.560 DEBUG:boto:Data: 2024-04-13T23:08:31.560 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.560 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.560 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.561 DEBUG:boto:Params: {} 2024-04-13T23:08:31.561 DEBUG:boto:Token: None 2024-04-13T23:08:31.561 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-73/obj5 2024-04-13T23:08:31.561 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:q6PnDtxmerWtfAXNT9EZ57gtFBo= 2024-04-13T23:08:31.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q6PnDtxmerWtfAXNT9EZ57gtFBo=', 'Content-Length': '0'} 2024-04-13T23:08:31.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f3898191c3175f5-00661b106f-2a13-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.563 DEBUG:boto:path=/vwnfns-73/obj5 2024-04-13T23:08:31.563 DEBUG:boto:auth_path=/vwnfns-73/obj5 2024-04-13T23:08:31.563 DEBUG:boto:Method: GET 2024-04-13T23:08:31.563 DEBUG:boto:Path: /vwnfns-73/obj5 2024-04-13T23:08:31.563 DEBUG:boto:Data: 2024-04-13T23:08:31.564 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.564 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.564 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.564 DEBUG:boto:Params: {} 2024-04-13T23:08:31.564 DEBUG:boto:Token: None 2024-04-13T23:08:31.564 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-73/obj5 2024-04-13T23:08:31.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9uE3t76lmTNveQljJ6nnkqcE++o= 2024-04-13T23:08:31.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9uE3t76lmTNveQljJ6nnkqcE++o=', 'Content-Length': '0'} 2024-04-13T23:08:31.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fa569f44b0a26e1-00661b106f-2a13-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.606 DEBUG:boto:path=/vwnfns-73/obj6 2024-04-13T23:08:31.606 DEBUG:boto:auth_path=/vwnfns-73/obj6 2024-04-13T23:08:31.607 DEBUG:boto:Method: HEAD 2024-04-13T23:08:31.607 DEBUG:boto:Path: /vwnfns-73/obj6 2024-04-13T23:08:31.607 DEBUG:boto:Data: 2024-04-13T23:08:31.607 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.607 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.607 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.607 DEBUG:boto:Params: {} 2024-04-13T23:08:31.607 DEBUG:boto:Token: None 2024-04-13T23:08:31.607 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-73/obj6 2024-04-13T23:08:31.607 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qSAjdmjIZsMoI4M1sN9dYlhUR9s= 2024-04-13T23:08:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qSAjdmjIZsMoI4M1sN9dYlhUR9s=', 'Content-Length': '0'} 2024-04-13T23:08:31.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064609a9f730dffcf-00661b106f-2a13-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.609 DEBUG:boto:path=/vwnfns-73/obj6 2024-04-13T23:08:31.610 DEBUG:boto:auth_path=/vwnfns-73/obj6 2024-04-13T23:08:31.610 DEBUG:boto:Method: GET 2024-04-13T23:08:31.610 DEBUG:boto:Path: /vwnfns-73/obj6 2024-04-13T23:08:31.610 DEBUG:boto:Data: 2024-04-13T23:08:31.610 DEBUG:boto:Headers: {} 2024-04-13T23:08:31.610 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:31.610 DEBUG:boto:Port: 8001 2024-04-13T23:08:31.610 DEBUG:boto:Params: {} 2024-04-13T23:08:31.610 DEBUG:boto:Token: None 2024-04-13T23:08:31.610 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:08:31 GMT /vwnfns-73/obj6 2024-04-13T23:08:31.610 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:m/G4XlGE3F5TFyFLwpBW9V2dmBk= 2024-04-13T23:08:31.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m/G4XlGE3F5TFyFLwpBW9V2dmBk=', 'Content-Length': '0'} 2024-04-13T23:08:31.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-72:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087462c1082d78491-00661b106f-2a13-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:08:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:31.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:31.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:31.613 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:31.661 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:31.661 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:31.724 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:08:31.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:31.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:31.725 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:31.772 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:31.772 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:31.804 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:08:31.821 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:08:31.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.821 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049711.704344_1178.1", 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:31.704344Z" 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:08:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:08:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.823 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:08:31.824 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:31.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:08:31.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:31.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:31.825 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:31.873 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:31.873 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:31.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:31.911+0000 7f883d3f1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:31.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:31.912+0000 7f883d3f1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:31.932 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:31.932 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:31.932 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:31.932 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:31.933 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:06:07.241183Z", 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:31.934 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:31.935 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:31.936 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.937 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:31Z" 2024-04-13T23:08:31.938 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:31.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049567.241183_1172.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:06:07.241183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:31Z" } 2024-04-13T23:08:31.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049567.241183_1172.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:06:07.241183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:31.939 DEBUG:rgw_multi.tests:log_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:31.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:31.940 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049711.704344_1178.1 target=1_1713049567.241183_1172.1 2024-04-13T23:08:36.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:36.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:36.941 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:36.996 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:36.996 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:37.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:37.027+0000 7fae06743780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:37.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:37.028+0000 7fae06743780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:37.051 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:06:07.241183Z", 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:37.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:37.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:37.054 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:37.055 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:37Z" 2024-04-13T23:08:37.056 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:37.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049567.241183_1172.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:06:07.241183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:37Z" } 2024-04-13T23:08:37.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049567.241183_1172.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:06:07.241183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:37.057 DEBUG:rgw_multi.tests:log_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:37.058 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:37.058 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049711.704344_1178.1 target=1_1713049567.241183_1172.1 2024-04-13T23:08:42.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:42.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:42.059 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:42.114 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:42.114 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:42.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:42.157+0000 7f32b8ace780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:42.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:42.158+0000 7f32b8ace780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:42.177 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:06:07.241183Z", 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:42.178 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:42.179 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:42.180 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:42.181 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:42.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:42.183 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:42.183 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:42.183 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:42.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:42.183 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:42Z" 2024-04-13T23:08:42.183 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:42.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049567.241183_1172.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:06:07.241183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:42Z" } 2024-04-13T23:08:42.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049567.241183_1172.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:06:07.241183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:42.184 DEBUG:rgw_multi.tests:log_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:42.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:42.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049711.704344_1178.1 target=1_1713049567.241183_1172.1 2024-04-13T23:08:47.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:47.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:47.186 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:47.242 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:47.242 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:47.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:47.274+0000 7fe82666d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:47.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:47.275+0000 7fe82666d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:47.291 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:47.291 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:47.291 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:47.291 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:47.291 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049567.241183_1172.1", 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:47.292 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:06:07.241183Z", 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:47.293 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:47.294 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:47.295 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:47.296 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:47Z" 2024-04-13T23:08:47.297 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:47.298 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049567.241183_1172.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:06:07.241183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:47Z" } 2024-04-13T23:08:47.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049567.241183_1172.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:06:07.241183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:47.298 DEBUG:rgw_multi.tests:log_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:47.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049567.241183_1172.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:47.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049711.704344_1178.1 target=1_1713049567.241183_1172.1 2024-04-13T23:08:52.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:52.300 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:52.354 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:52.355 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:52.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:52.386+0000 7fd4ab355780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:52.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:52.387+0000 7fd4ab355780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:52.403 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:52.403 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:52.404 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049711.704344_1178.1", 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:31.704344Z", 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:52.405 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:52.406 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:52.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:52.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:52Z" 2024-04-13T23:08:52.409 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:52.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049711.704344_1178.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:31.704344Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:52Z" } 2024-04-13T23:08:52.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049711.704344_1178.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:31.704344Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:52.411 DEBUG:rgw_multi.tests:log_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:52.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:52.411 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:08:52.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.411 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:52.458 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:52.459 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:52.508 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T23:08:52.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.509 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:52.559 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:52.559 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:52.623 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:08:52.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.624 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:52.677 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:52.677 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:52.750 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:08:52.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=vwnfns-72', '--source-bucket=vwnfns-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=vwnfns-72', '--source-bucket=vwnfns-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.751 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=vwnfns-72 --source-bucket=vwnfns-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:52.806 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:52.806 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:52.867 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-73","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-13T23:08:52.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.868 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:52.918 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:52.918 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:52.973 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-73","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T23:08:52.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:08:52.974 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:08:53.019 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:53.020 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:53.068 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-73","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T23:08:53.068 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:08:53.068 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:08:53.068 DEBUG:boto:Method: HEAD 2024-04-13T23:08:53.068 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:08:53.069 DEBUG:boto:Data: 2024-04-13T23:08:53.069 DEBUG:boto:Headers: {} 2024-04-13T23:08:53.069 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.069 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.069 DEBUG:boto:Params: {} 2024-04-13T23:08:53.069 DEBUG:boto:Token: None 2024-04-13T23:08:53.069 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-72/ 2024-04-13T23:08:53.070 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mrhsiqUg0cfSJ+lEw8QXQeJ5W8Y= 2024-04-13T23:08:53.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mrhsiqUg0cfSJ+lEw8QXQeJ5W8Y=', 'Content-Length': '0'} 2024-04-13T23:08:53.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004728547d34e5b2b7-00661b1085-2a13-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.073 DEBUG:boto:path=/vwnfns-72/obj7 2024-04-13T23:08:53.073 DEBUG:boto:auth_path=/vwnfns-72/obj7 2024-04-13T23:08:53.073 DEBUG:boto:Method: PUT 2024-04-13T23:08:53.073 DEBUG:boto:Path: /vwnfns-72/obj7 2024-04-13T23:08:53.073 DEBUG:boto:Data: 2024-04-13T23:08:53.073 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:08:53.073 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.073 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.073 DEBUG:boto:Params: {} 2024-04-13T23:08:53.073 DEBUG:boto:Token: None 2024-04-13T23:08:53.074 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-72/obj7 2024-04-13T23:08:53.074 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:puJ/cT+MY25nFoQDo7k56/BHArs= 2024-04-13T23:08:53.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:puJ/cT+MY25nFoQDo7k56/BHArs='} 2024-04-13T23:08:53.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000170f917e21f3e1b9-00661b1085-2a13-a2'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.080 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:08:53.080 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:08:53.080 DEBUG:boto:Method: HEAD 2024-04-13T23:08:53.080 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:08:53.080 DEBUG:boto:Data: 2024-04-13T23:08:53.080 DEBUG:boto:Headers: {} 2024-04-13T23:08:53.080 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.080 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.080 DEBUG:boto:Params: {} 2024-04-13T23:08:53.080 DEBUG:boto:Token: None 2024-04-13T23:08:53.081 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-72/ 2024-04-13T23:08:53.081 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:mrhsiqUg0cfSJ+lEw8QXQeJ5W8Y= 2024-04-13T23:08:53.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mrhsiqUg0cfSJ+lEw8QXQeJ5W8Y=', 'Content-Length': '0'} 2024-04-13T23:08:53.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000501166590c687973-00661b1085-2a13-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.083 DEBUG:boto:path=/vwnfns-72/obj8 2024-04-13T23:08:53.083 DEBUG:boto:auth_path=/vwnfns-72/obj8 2024-04-13T23:08:53.083 DEBUG:boto:Method: PUT 2024-04-13T23:08:53.083 DEBUG:boto:Path: /vwnfns-72/obj8 2024-04-13T23:08:53.083 DEBUG:boto:Data: 2024-04-13T23:08:53.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:08:53.083 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.084 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.084 DEBUG:boto:Params: {} 2024-04-13T23:08:53.084 DEBUG:boto:Token: None 2024-04-13T23:08:53.084 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-72/obj8 2024-04-13T23:08:53.084 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tv+09PvDSuVCpzBXtDqLImALCG8= 2024-04-13T23:08:53.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tv+09PvDSuVCpzBXtDqLImALCG8='} 2024-04-13T23:08:53.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000798aae0e9929809a-00661b1085-2a13-a2'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.127 DEBUG:boto:path=/vwnfns-73/ 2024-04-13T23:08:53.127 DEBUG:boto:auth_path=/vwnfns-73/ 2024-04-13T23:08:53.127 DEBUG:boto:Method: HEAD 2024-04-13T23:08:53.127 DEBUG:boto:Path: /vwnfns-73/ 2024-04-13T23:08:53.127 DEBUG:boto:Data: 2024-04-13T23:08:53.127 DEBUG:boto:Headers: {} 2024-04-13T23:08:53.127 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.128 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.128 DEBUG:boto:Params: {} 2024-04-13T23:08:53.128 DEBUG:boto:Token: None 2024-04-13T23:08:53.128 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-73/ 2024-04-13T23:08:53.128 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M2XzAjhbWVRJHeeoIrIyEsb0mKI= 2024-04-13T23:08:53.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M2XzAjhbWVRJHeeoIrIyEsb0mKI=', 'Content-Length': '0'} 2024-04-13T23:08:53.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ab3117b79b5ef4b-00661b1085-2a13-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.130 DEBUG:boto:path=/vwnfns-73/obj9 2024-04-13T23:08:53.131 DEBUG:boto:auth_path=/vwnfns-73/obj9 2024-04-13T23:08:53.131 DEBUG:boto:Method: PUT 2024-04-13T23:08:53.131 DEBUG:boto:Path: /vwnfns-73/obj9 2024-04-13T23:08:53.131 DEBUG:boto:Data: 2024-04-13T23:08:53.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:08:53.131 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.131 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.131 DEBUG:boto:Params: {} 2024-04-13T23:08:53.131 DEBUG:boto:Token: None 2024-04-13T23:08:53.132 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-73/obj9 2024-04-13T23:08:53.132 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kPwLEK34bHDx5fd+BMpV1ueJK+A= 2024-04-13T23:08:53.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kPwLEK34bHDx5fd+BMpV1ueJK+A='} 2024-04-13T23:08:53.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc4037eb47307adc-00661b1085-2a13-a2'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.137 DEBUG:boto:path=/vwnfns-73/ 2024-04-13T23:08:53.137 DEBUG:boto:auth_path=/vwnfns-73/ 2024-04-13T23:08:53.138 DEBUG:boto:Method: HEAD 2024-04-13T23:08:53.138 DEBUG:boto:Path: /vwnfns-73/ 2024-04-13T23:08:53.138 DEBUG:boto:Data: 2024-04-13T23:08:53.138 DEBUG:boto:Headers: {} 2024-04-13T23:08:53.138 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.138 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.138 DEBUG:boto:Params: {} 2024-04-13T23:08:53.138 DEBUG:boto:Token: None 2024-04-13T23:08:53.138 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-73/ 2024-04-13T23:08:53.139 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:M2XzAjhbWVRJHeeoIrIyEsb0mKI= 2024-04-13T23:08:53.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:M2XzAjhbWVRJHeeoIrIyEsb0mKI=', 'Content-Length': '0'} 2024-04-13T23:08:53.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c962cd77c5b6ea19-00661b1085-2a13-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.141 DEBUG:boto:path=/vwnfns-73/obj10 2024-04-13T23:08:53.141 DEBUG:boto:auth_path=/vwnfns-73/obj10 2024-04-13T23:08:53.141 DEBUG:boto:Method: PUT 2024-04-13T23:08:53.141 DEBUG:boto:Path: /vwnfns-73/obj10 2024-04-13T23:08:53.142 DEBUG:boto:Data: 2024-04-13T23:08:53.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:08:53.142 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:08:53.142 DEBUG:boto:Port: 8001 2024-04-13T23:08:53.142 DEBUG:boto:Params: {} 2024-04-13T23:08:53.142 DEBUG:boto:Token: None 2024-04-13T23:08:53.142 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:08:53 GMT /vwnfns-73/obj10 2024-04-13T23:08:53.142 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:daQUsEJ9B0kuWCa8k28FVDswXF8= 2024-04-13T23:08:53.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:08:53 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:daQUsEJ9B0kuWCa8k28FVDswXF8='} 2024-04-13T23:08:53.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8a5832511f7a8e0-00661b1085-2a13-a2'), ('Date', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:08:53.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:53.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:53.191 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:53.239 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:53.239 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:53.280 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049732.956694_1186.1", 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:52.956694Z" 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:08:53.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:53.296 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:08:53.297 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:53.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:08:53.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:53.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:53.298 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:53.345 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:53.345 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:53.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:53.376+0000 7f6f5fc6c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:53.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:53.377+0000 7f6f5fc6c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:53.391 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:53.391 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:53.391 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:53.391 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:53.392 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049711.704344_1178.1", 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:31.704344Z", 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.393 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:53.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:53.395 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:53.396 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:53Z" 2024-04-13T23:08:53.397 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:53.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049711.704344_1178.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:31.704344Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:53Z" } 2024-04-13T23:08:53.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049711.704344_1178.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:31.704344Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:53.399 DEBUG:rgw_multi.tests:log_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:53.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:53.399 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049732.956694_1186.1 target=1_1713049711.704344_1178.1 2024-04-13T23:08:58.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:58.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:08:58.401 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:08:58.455 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:08:58.455 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:08:58.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:58.487+0000 7f7c96687780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:58.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:08:58.488+0000 7f7c96687780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:08:58.506 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:08:58.506 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:08:58.507 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049711.704344_1178.1", 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:31.704344Z", 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:08:58.508 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:58.509 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:58.510 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:08:58.511 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:08:58.512 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:08:58Z" 2024-04-13T23:08:58.513 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:08:58.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049711.704344_1178.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:31.704344Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:08:58Z" } 2024-04-13T23:08:58.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049711.704344_1178.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:31.704344Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:08:58.514 DEBUG:rgw_multi.tests:log_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:58.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:08:58.514 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049732.956694_1186.1 target=1_1713049711.704344_1178.1 2024-04-13T23:09:03.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:03.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:03.516 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:03.575 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:03.575 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:03.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:03.610+0000 7f1bc964d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:03.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:03.611+0000 7f1bc964d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:03.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049711.704344_1178.1", 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:03.629 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:31.704344Z", 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:03.630 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:03.631 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:03.632 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:03Z" 2024-04-13T23:09:03.633 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:03.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049711.704344_1178.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:31.704344Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:03Z" } 2024-04-13T23:09:03.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049711.704344_1178.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:31.704344Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:03.635 DEBUG:rgw_multi.tests:log_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:03.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:03.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049732.956694_1186.1 target=1_1713049711.704344_1178.1 2024-04-13T23:09:08.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:08.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:08.637 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:08.693 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:08.693 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:08.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:08.731+0000 7fe217261780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:08.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:08.732+0000 7fe217261780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:08.754 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:08.754 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049711.704344_1178.1", 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:08.755 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:31.704344Z", 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:08.756 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:08.757 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:08.759 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:08.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:08.760 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:08Z" 2024-04-13T23:09:08.760 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:08.761 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049711.704344_1178.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:31.704344Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:08Z" } 2024-04-13T23:09:08.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049711.704344_1178.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:31.704344Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:08.761 DEBUG:rgw_multi.tests:log_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:08.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049711.704344_1178.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:08.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049732.956694_1186.1 target=1_1713049711.704344_1178.1 2024-04-13T23:09:13.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:13.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:13.763 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:13.819 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:13.819 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:13.855+0000 7fa26599c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:13.856+0000 7fa26599c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:13.872 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:13.872 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:13.872 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:13.872 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:13.872 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049732.956694_1186.1", 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:13.873 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:52.956694Z", 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:13.874 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:13.875 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:13.876 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:13Z" 2024-04-13T23:09:13.878 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:13.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049732.956694_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:52.956694Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:13Z" } 2024-04-13T23:09:13.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049732.956694_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:52.956694Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:13.879 DEBUG:rgw_multi.tests:log_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:13.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:13.879 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:09:13.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:13.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:13.880 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:13.928 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:13.928 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000018838", 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:53.135996Z" 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000015353", 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:27.934702Z" 2024-04-13T23:09:13.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000008261", 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:53.146478Z" 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000019956", 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:53.078327Z" 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:13.981 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:09:13.982 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000018838', 1: '00000000000000000000:00000000000000015353', 2: '00000000000000000000:00000000000000008261', 3: '00000000000000000000:00000000000000019956'} 2024-04-13T23:09:13.982 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T23:09:13.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:13.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:13.983 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:14.032 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:14.032 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:14.082 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 6585290900071302380 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.083 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000018838", 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:53.135996Z" 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.084 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000015353", 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:27.934702Z" 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.085 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000008261", 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:53.146478Z" 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.086 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000019956", 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:53.078327Z" 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:14.087 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 0, 2024-04-13T23:09:14.088 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 0 2024-04-13T23:09:14.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.088 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:14Z" 2024-04-13T23:09:14.088 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:14.089 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585290900071302380 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018838", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:53.135996Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015353", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:27.934702Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008261", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:53.146478Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019956", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:53.078327Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-13T23:09:14Z" } 2024-04-13T23:09:14.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018838', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:53.135996Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015353', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:27.934702Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008261', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:53.146478Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019956', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:53.078327Z'}}] 2024-04-13T23:09:14.089 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000018838', 1: '00000000000000000000:00000000000000015353', 2: '00000000000000000000:00000000000000008261', 3: '00000000000000000000:00000000000000019956'} 2024-04-13T23:09:14.089 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000018838', 1: '00000000000000000000:00000000000000015353', 2: '00000000000000000000:00000000000000008261', 3: '00000000000000000000:00000000000000019956'} 2024-04-13T23:09:14.089 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-13T23:09:14.090 DEBUG:boto:path=/vwnfns-72/ 2024-04-13T23:09:14.090 DEBUG:boto:auth_path=/vwnfns-72/ 2024-04-13T23:09:14.090 DEBUG:boto:Method: HEAD 2024-04-13T23:09:14.090 DEBUG:boto:Path: /vwnfns-72/ 2024-04-13T23:09:14.090 DEBUG:boto:Data: 2024-04-13T23:09:14.090 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.090 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.090 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.091 DEBUG:boto:Params: {} 2024-04-13T23:09:14.091 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:09:14.091 DEBUG:boto:Token: None 2024-04-13T23:09:14.091 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/ 2024-04-13T23:09:14.091 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:fS7gO5B8QsZu5sIxS03kV2nwLO0= 2024-04-13T23:09:14.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fS7gO5B8QsZu5sIxS03kV2nwLO0=', 'Content-Length': '0'} 2024-04-13T23:09:14.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000892faa490becec5c-00661b109a-2a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.096 DEBUG:boto:path=/vwnfns-72/obj7 2024-04-13T23:09:14.096 DEBUG:boto:auth_path=/vwnfns-72/obj7 2024-04-13T23:09:14.096 DEBUG:boto:Method: HEAD 2024-04-13T23:09:14.096 DEBUG:boto:Path: /vwnfns-72/obj7 2024-04-13T23:09:14.096 DEBUG:boto:Data: 2024-04-13T23:09:14.096 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.097 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.097 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.097 DEBUG:boto:Params: {} 2024-04-13T23:09:14.097 DEBUG:boto:Token: None 2024-04-13T23:09:14.097 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/obj7 2024-04-13T23:09:14.097 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+7V0Og9tVB0cnbopkwvEDvL+Vbc= 2024-04-13T23:09:14.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+7V0Og9tVB0cnbopkwvEDvL+Vbc=', 'Content-Length': '0'} 2024-04-13T23:09:14.099 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008f9f51278a970ac7-00661b109a-2a22-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.099 DEBUG:boto:path=/vwnfns-72/obj8 2024-04-13T23:09:14.099 DEBUG:boto:auth_path=/vwnfns-72/obj8 2024-04-13T23:09:14.099 DEBUG:boto:Method: HEAD 2024-04-13T23:09:14.099 DEBUG:boto:Path: /vwnfns-72/obj8 2024-04-13T23:09:14.099 DEBUG:boto:Data: 2024-04-13T23:09:14.099 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.099 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.099 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.099 DEBUG:boto:Params: {} 2024-04-13T23:09:14.100 DEBUG:boto:Token: None 2024-04-13T23:09:14.100 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/obj8 2024-04-13T23:09:14.100 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0Y0NY77V8WryfAT2GninJT0k+Ws= 2024-04-13T23:09:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0Y0NY77V8WryfAT2GninJT0k+Ws=', 'Content-Length': '0'} 2024-04-13T23:09:14.101 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c75d18dc44a52093-00661b109a-2a22-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.102 DEBUG:boto:path=/vwnfns-72/obj9 2024-04-13T23:09:14.102 DEBUG:boto:auth_path=/vwnfns-72/obj9 2024-04-13T23:09:14.102 DEBUG:boto:Method: HEAD 2024-04-13T23:09:14.102 DEBUG:boto:Path: /vwnfns-72/obj9 2024-04-13T23:09:14.102 DEBUG:boto:Data: 2024-04-13T23:09:14.102 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.102 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.102 DEBUG:boto:Params: {} 2024-04-13T23:09:14.102 DEBUG:boto:Token: None 2024-04-13T23:09:14.103 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/obj9 2024-04-13T23:09:14.103 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ciLA0OR8ZDt5AxNkDlkN2Wndt0Y= 2024-04-13T23:09:14.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ciLA0OR8ZDt5AxNkDlkN2Wndt0Y=', 'Content-Length': '0'} 2024-04-13T23:09:14.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-73:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022044c34083d4938-00661b109a-2a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.105 DEBUG:boto:path=/vwnfns-72/obj9 2024-04-13T23:09:14.105 DEBUG:boto:auth_path=/vwnfns-72/obj9 2024-04-13T23:09:14.105 DEBUG:boto:Method: GET 2024-04-13T23:09:14.105 DEBUG:boto:Path: /vwnfns-72/obj9 2024-04-13T23:09:14.105 DEBUG:boto:Data: 2024-04-13T23:09:14.105 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.105 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.105 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.105 DEBUG:boto:Params: {} 2024-04-13T23:09:14.106 DEBUG:boto:Token: None 2024-04-13T23:09:14.106 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/obj9 2024-04-13T23:09:14.106 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:op05Vtj3ngh+zn7K+ZjTp61Yz00= 2024-04-13T23:09:14.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:op05Vtj3ngh+zn7K+ZjTp61Yz00=', 'Content-Length': '0'} 2024-04-13T23:09:14.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-73:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000053a5b945fbb7bca-00661b109a-2a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.108 DEBUG:boto:path=/vwnfns-72/obj10 2024-04-13T23:09:14.108 DEBUG:boto:auth_path=/vwnfns-72/obj10 2024-04-13T23:09:14.108 DEBUG:boto:Method: HEAD 2024-04-13T23:09:14.108 DEBUG:boto:Path: /vwnfns-72/obj10 2024-04-13T23:09:14.108 DEBUG:boto:Data: 2024-04-13T23:09:14.108 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.109 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.109 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.109 DEBUG:boto:Params: {} 2024-04-13T23:09:14.109 DEBUG:boto:Token: None 2024-04-13T23:09:14.109 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/obj10 2024-04-13T23:09:14.109 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:GKgFWNMtL4tp5RV/5mJAgiWVBZg= 2024-04-13T23:09:14.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:GKgFWNMtL4tp5RV/5mJAgiWVBZg=', 'Content-Length': '0'} 2024-04-13T23:09:14.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-73:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000114e756926c2fec0-00661b109a-2a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.111 DEBUG:boto:path=/vwnfns-72/obj10 2024-04-13T23:09:14.111 DEBUG:boto:auth_path=/vwnfns-72/obj10 2024-04-13T23:09:14.111 DEBUG:boto:Method: GET 2024-04-13T23:09:14.111 DEBUG:boto:Path: /vwnfns-72/obj10 2024-04-13T23:09:14.112 DEBUG:boto:Data: 2024-04-13T23:09:14.112 DEBUG:boto:Headers: {} 2024-04-13T23:09:14.112 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:14.112 DEBUG:boto:Port: 8000 2024-04-13T23:09:14.112 DEBUG:boto:Params: {} 2024-04-13T23:09:14.112 DEBUG:boto:Token: None 2024-04-13T23:09:14.112 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:09:14 GMT /vwnfns-72/obj10 2024-04-13T23:09:14.112 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:H6eRFsfhNDByGRxnk7C234hAUMk= 2024-04-13T23:09:14.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:14 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H6eRFsfhNDByGRxnk7C234hAUMk=', 'Content-Length': '0'} 2024-04-13T23:09:14.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:08:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '5da80fbc-5edc-4ba7-826d-471359a9fc92:vwnfns-73:f6f208d6-f791-479b-bc3d-112512eec2d4.10190.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f829cfcc216f874d-00661b109a-2a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:09:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:14.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:14.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:14.115 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket vwnfns-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:14.163 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:14.163 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:14.218 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:09:14.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:14.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:14.219 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:14.264 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:14.264 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:14.328 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:09:14.329 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2024-04-13T23:09:14.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:14.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:14.331 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:14.377 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:14.377 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:14.429 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049754.198915_1188.1", 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:09:14.198915Z" 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.448 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.449 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:09:14.450 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049754.198915_1188.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:14.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:09:14.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:14.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:14.451 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:14.498 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:14.498 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:14.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:14.533+0000 7f8f612b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:14.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:14.533+0000 7f8f612b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:14.549 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:14.549 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:14.549 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:14.549 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:14.550 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049732.956694_1186.1", 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:52.956694Z", 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.551 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:14.552 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:14.553 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:14.554 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:14Z" 2024-04-13T23:09:14.555 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:14.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049732.956694_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:52.956694Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:14Z" } 2024-04-13T23:09:14.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049732.956694_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:52.956694Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:14.556 DEBUG:rgw_multi.tests:log_status={0: '1_1713049754.198915_1188.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:14.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:14.557 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049754.198915_1188.1 target=1_1713049732.956694_1186.1 2024-04-13T23:09:19.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:19.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:19.558 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:19.616 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:19.616 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:19.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:19.658+0000 7f9de2695780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:19.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:19.659+0000 7f9de2695780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:19.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049732.956694_1186.1", 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:52.956694Z", 2024-04-13T23:09:19.684 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:19.685 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:19.686 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:19.687 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:19.688 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:19Z" 2024-04-13T23:09:19.689 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:19.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049732.956694_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:52.956694Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:19Z" } 2024-04-13T23:09:19.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049732.956694_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:52.956694Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:19.690 DEBUG:rgw_multi.tests:log_status={0: '1_1713049754.198915_1188.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:19.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:19.690 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049754.198915_1188.1 target=1_1713049732.956694_1186.1 2024-04-13T23:09:24.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:24.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:24.691 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:24.748 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:24.748 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:24.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:24.800+0000 7fe356427780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:24.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:24.801+0000 7fe356427780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:24.821 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:24.821 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:24.821 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:24.822 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049732.956694_1186.1", 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:52.956694Z", 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:24.823 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:24.824 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:24.825 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:24.826 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:24Z" 2024-04-13T23:09:24.827 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:24.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049732.956694_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:08:52.956694Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:24Z" } 2024-04-13T23:09:24.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049732.956694_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:08:52.956694Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:24.828 DEBUG:rgw_multi.tests:log_status={0: '1_1713049754.198915_1188.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:24.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049732.956694_1186.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:24.828 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049754.198915_1188.1 target=1_1713049732.956694_1186.1 2024-04-13T23:09:29.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:29.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:29.830 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:29.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:29.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:29.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:29.925+0000 7eff7c228780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:29.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:29.926+0000 7eff7c228780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:29.942 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049754.198915_1188.1", 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:14.198915Z", 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:29Z" 2024-04-13T23:09:29.948 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:29.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049754.198915_1188.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:14.198915Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:29Z" } 2024-04-13T23:09:29.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049754.198915_1188.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:14.198915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:29.949 DEBUG:rgw_multi.tests:log_status={0: '1_1713049754.198915_1188.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:29.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049754.198915_1188.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:29.950 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:09:29.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:29.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:29.950 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:29.996 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:29.996 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.047 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T23:09:30.047 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-74 2024-04-13T23:09:30.048 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:09:30.048 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:09:30.048 DEBUG:boto:Method: PUT 2024-04-13T23:09:30.048 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:09:30.048 DEBUG:boto:Data: 2024-04-13T23:09:30.048 DEBUG:boto:Headers: {} 2024-04-13T23:09:30.048 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:30.048 DEBUG:boto:Port: 8000 2024-04-13T23:09:30.048 DEBUG:boto:Params: {} 2024-04-13T23:09:30.049 DEBUG:boto:Token: None 2024-04-13T23:09:30.049 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:09:30 GMT /vwnfns-74/ 2024-04-13T23:09:30.049 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:B4B3i/SnO+cUB12Wu37JxzsTFXo= 2024-04-13T23:09:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:B4B3i/SnO+cUB12Wu37JxzsTFXo=', 'Content-Length': '0'} 2024-04-13T23:09:30.080 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d582dc2dae8c596-00661b10aa-2a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:30.080 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-75 2024-04-13T23:09:30.080 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:09:30.080 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:09:30.080 DEBUG:boto:Method: PUT 2024-04-13T23:09:30.080 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:09:30.081 DEBUG:boto:Data: 2024-04-13T23:09:30.081 DEBUG:boto:Headers: {} 2024-04-13T23:09:30.081 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:30.081 DEBUG:boto:Port: 8000 2024-04-13T23:09:30.081 DEBUG:boto:Params: {} 2024-04-13T23:09:30.081 DEBUG:boto:Token: None 2024-04-13T23:09:30.081 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:09:30 GMT /vwnfns-75/ 2024-04-13T23:09:30.081 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0Dvfmo2Ta7v22HoLMF8VmFiQzws= 2024-04-13T23:09:30.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0Dvfmo2Ta7v22HoLMF8VmFiQzws=', 'Content-Length': '0'} 2024-04-13T23:09:30.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3c0ad840e808051-00661b10aa-2a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:30.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:30.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:30.110 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:30.160 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.160 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.203 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:09:30.216 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:09:30.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:09:30.107263Z" 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:09:30.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.218 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:30.218 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:09:30.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:30.218 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:09:30.219 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:30.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:09:30.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:30.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:30.220 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:30.268 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.269 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:30.315+0000 7f2216fb5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:30.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:30.316+0000 7f2216fb5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.340 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.341 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:30.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:30.343 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:30.344 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:30Z" 2024-04-13T23:09:30.345 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:30.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:30Z" } 2024-04-13T23:09:30.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:30.347 DEBUG:rgw_multi.tests:log_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:30.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:30.347 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:09:30.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.347 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:30.395 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.395 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.472 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:09:30.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.473 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:30.525 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.525 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.586 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:09:30.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-74', '--dest-bucket=vwnfns-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-74', '--dest-bucket=vwnfns-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.587 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-vwnfns-74 --source-zones=a1 --dest-zones=a2 --source-bucket=vwnfns-74 --dest-bucket=vwnfns-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:30.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.703 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:09:30.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-75', '--dest-bucket=vwnfns-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-75', '--dest-bucket=vwnfns-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.704 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-vwnfns-75 --source-zones=a1 --dest-zones=a2 --source-bucket=vwnfns-75 --dest-bucket=vwnfns-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:30.757 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.757 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.804 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-75","source":{"bucket":"vwnfns-75","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:09:30.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.805 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:30.853 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.853 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:30.904 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-75","source":{"bucket":"vwnfns-75","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:09:30.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:30.905 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:30.951 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:30.951 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 19, 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:09:31.004 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:09:31.005 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.006 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:09:31.007 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:09:31.008 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.009 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.010 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "directional": [ 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "source_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: "dest_zone": "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:09:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe-vwnfns-74", 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-74", 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-75", 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe-vwnfns-75", 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:09:31.014 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-75", 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-75", 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.015 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "enabled" 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:09:31.018 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:09:31.019 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:09:31.020 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:09:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:09:31.022 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:09:31.022 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:09:31.022 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:09:31.022 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:09:31.022 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:09:31.023 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:09:31.023 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:09:31.023 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:09:31.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:09:31.027 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:09:31.028 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:09:31.029 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:09:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:09:31.031 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.031 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:09:31.031 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:09:31.031 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:31.031 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:31.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:31.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:09:31.032 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:09:31.075 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:31.075 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:31.139 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-75","source":{"bucket":"vwnfns-75","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:09:31.139 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:09:31.140 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:09:31.140 DEBUG:boto:Method: HEAD 2024-04-13T23:09:31.140 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:09:31.140 DEBUG:boto:Data: 2024-04-13T23:09:31.140 DEBUG:boto:Headers: {} 2024-04-13T23:09:31.140 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.140 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.140 DEBUG:boto:Params: {} 2024-04-13T23:09:31.140 DEBUG:boto:Token: None 2024-04-13T23:09:31.141 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-74/ 2024-04-13T23:09:31.141 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IQGihAxcSihl/7K6sOcpNRC79ck= 2024-04-13T23:09:31.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IQGihAxcSihl/7K6sOcpNRC79ck=', 'Content-Length': '0'} 2024-04-13T23:09:31.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f11712d47f75df2-00661b10ab-2c4e-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.334 DEBUG:boto:path=/vwnfns-74/obj1 2024-04-13T23:09:31.334 DEBUG:boto:auth_path=/vwnfns-74/obj1 2024-04-13T23:09:31.334 DEBUG:boto:Method: PUT 2024-04-13T23:09:31.334 DEBUG:boto:Path: /vwnfns-74/obj1 2024-04-13T23:09:31.335 DEBUG:boto:Data: 2024-04-13T23:09:31.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:09:31.335 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.335 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.335 DEBUG:boto:Params: {} 2024-04-13T23:09:31.335 DEBUG:boto:Token: None 2024-04-13T23:09:31.335 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-74/obj1 2024-04-13T23:09:31.335 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:dM+Mjx42cu+vfUa6Ui/w1A+mWtk= 2024-04-13T23:09:31.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dM+Mjx42cu+vfUa6Ui/w1A+mWtk='} 2024-04-13T23:09:31.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca29d4b6d7f481c1-00661b10ab-2c4e-a1'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.366 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:09:31.366 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:09:31.366 DEBUG:boto:Method: HEAD 2024-04-13T23:09:31.366 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:09:31.366 DEBUG:boto:Data: 2024-04-13T23:09:31.366 DEBUG:boto:Headers: {} 2024-04-13T23:09:31.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.367 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.367 DEBUG:boto:Params: {} 2024-04-13T23:09:31.367 DEBUG:boto:Token: None 2024-04-13T23:09:31.367 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-74/ 2024-04-13T23:09:31.367 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:IQGihAxcSihl/7K6sOcpNRC79ck= 2024-04-13T23:09:31.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IQGihAxcSihl/7K6sOcpNRC79ck=', 'Content-Length': '0'} 2024-04-13T23:09:31.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9380473d47bc2c5-00661b10ab-2c4e-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.370 DEBUG:boto:path=/vwnfns-74/obj2 2024-04-13T23:09:31.371 DEBUG:boto:auth_path=/vwnfns-74/obj2 2024-04-13T23:09:31.371 DEBUG:boto:Method: PUT 2024-04-13T23:09:31.371 DEBUG:boto:Path: /vwnfns-74/obj2 2024-04-13T23:09:31.371 DEBUG:boto:Data: 2024-04-13T23:09:31.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:09:31.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.371 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.371 DEBUG:boto:Params: {} 2024-04-13T23:09:31.371 DEBUG:boto:Token: None 2024-04-13T23:09:31.372 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-74/obj2 2024-04-13T23:09:31.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uDw0LXFNwg0B3SMxJjC/CoetMmE= 2024-04-13T23:09:31.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uDw0LXFNwg0B3SMxJjC/CoetMmE='} 2024-04-13T23:09:31.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075a89df8cb95d5e8-00661b10ab-2c4e-a1'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.415 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:09:31.415 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:09:31.415 DEBUG:boto:Method: HEAD 2024-04-13T23:09:31.415 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:09:31.415 DEBUG:boto:Data: 2024-04-13T23:09:31.416 DEBUG:boto:Headers: {} 2024-04-13T23:09:31.416 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.416 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.416 DEBUG:boto:Params: {} 2024-04-13T23:09:31.416 DEBUG:boto:Token: None 2024-04-13T23:09:31.416 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-75/ 2024-04-13T23:09:31.416 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gep2OemVKnUt8PJmub7rpl1M1nQ= 2024-04-13T23:09:31.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gep2OemVKnUt8PJmub7rpl1M1nQ=', 'Content-Length': '0'} 2024-04-13T23:09:31.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bd7f5f48f4a9afa-00661b10ab-2c4e-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.422 DEBUG:boto:path=/vwnfns-75/obj3 2024-04-13T23:09:31.422 DEBUG:boto:auth_path=/vwnfns-75/obj3 2024-04-13T23:09:31.422 DEBUG:boto:Method: PUT 2024-04-13T23:09:31.422 DEBUG:boto:Path: /vwnfns-75/obj3 2024-04-13T23:09:31.422 DEBUG:boto:Data: 2024-04-13T23:09:31.423 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:09:31.423 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.423 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.423 DEBUG:boto:Params: {} 2024-04-13T23:09:31.423 DEBUG:boto:Token: None 2024-04-13T23:09:31.423 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-75/obj3 2024-04-13T23:09:31.423 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:nlL9LzL8iuIhWzUCw/g4RDPxgvE= 2024-04-13T23:09:31.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nlL9LzL8iuIhWzUCw/g4RDPxgvE='} 2024-04-13T23:09:31.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012bd8db2ea4de39d-00661b10ab-2c4e-a1'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.443 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:09:31.443 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:09:31.443 DEBUG:boto:Method: HEAD 2024-04-13T23:09:31.443 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:09:31.443 DEBUG:boto:Data: 2024-04-13T23:09:31.443 DEBUG:boto:Headers: {} 2024-04-13T23:09:31.443 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.443 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.443 DEBUG:boto:Params: {} 2024-04-13T23:09:31.443 DEBUG:boto:Token: None 2024-04-13T23:09:31.444 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-75/ 2024-04-13T23:09:31.444 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Gep2OemVKnUt8PJmub7rpl1M1nQ= 2024-04-13T23:09:31.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Gep2OemVKnUt8PJmub7rpl1M1nQ=', 'Content-Length': '0'} 2024-04-13T23:09:31.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f88994683eba30c7-00661b10ab-2c4e-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.450 DEBUG:boto:path=/vwnfns-75/obj4 2024-04-13T23:09:31.450 DEBUG:boto:auth_path=/vwnfns-75/obj4 2024-04-13T23:09:31.450 DEBUG:boto:Method: PUT 2024-04-13T23:09:31.451 DEBUG:boto:Path: /vwnfns-75/obj4 2024-04-13T23:09:31.451 DEBUG:boto:Data: 2024-04-13T23:09:31.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:09:31.451 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:09:31.451 DEBUG:boto:Port: 8000 2024-04-13T23:09:31.451 DEBUG:boto:Params: {} 2024-04-13T23:09:31.451 DEBUG:boto:Token: None 2024-04-13T23:09:31.451 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:09:31 GMT /vwnfns-75/obj4 2024-04-13T23:09:31.451 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:8l3VDaWgRlE4NuWVk9LyEifbENw= 2024-04-13T23:09:31.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:09:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8l3VDaWgRlE4NuWVk9LyEifbENw='} 2024-04-13T23:09:31.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000999f990b937588f1-00661b10ab-2c4e-a1'), ('Date', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:09:31.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:31.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:31.495 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:31.565 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:31.565 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:31.661 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:09:31.722 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:09:31.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:09:30.107263Z" 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:09:31.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.724 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:31.724 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:09:31.724 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.724 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:09:31.725 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:31.725 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:09:31.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:31.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:31.725 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:31.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:31.796 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:31.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:31.872+0000 7fd872d8d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:31.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:09:31.874+0000 7fd872d8d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:09:31.906 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:31.906 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:31.906 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:31.907 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.908 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:31.910 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:31.911 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:31Z" 2024-04-13T23:09:31.912 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:31.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:09:31Z" } 2024-04-13T23:09:31.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:09:31.914 DEBUG:rgw_multi.tests:log_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:31.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:09:31.914 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:09:31.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:31.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:31.915 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:31.969 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:32.071 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:09:32.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047633", 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:09:31.392463Z" 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033750", 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:09:31.464865Z" 2024-04-13T23:09:32.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054068", 2024-04-13T23:09:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:09:31.437863Z" 2024-04-13T23:09:32.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:32.073 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:09:32.074 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:32.074 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:09:32.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:32.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:32.075 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:32.136 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:32.136 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:32.189 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:32.189 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:32.190 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:09:32.192 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:32.193 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:32.194 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:09:32.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:32.196 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:32Z" 2024-04-13T23:09:32.196 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:32.197 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:09:32Z" } 2024-04-13T23:09:32.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:09:32.197 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:32.197 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:09:32.197 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:09:37.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:37.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:37.200 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:37.258 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:37.258 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:37.319 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:37.319 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:37.319 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:37.320 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:37.320 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:37.320 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:09:37.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:37.321 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:37.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:37.321 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:37.321 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:37.321 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:37.322 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:37.322 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:37.322 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:37.322 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:37.323 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:37.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:37.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:37.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:37.324 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:37.324 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:37.324 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:37.324 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:09:37.324 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:37.325 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:37.325 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:37.325 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:09:37.325 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:37.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:37.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:37.326 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:37.326 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:37.326 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:37.326 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:37.327 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:37.328 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:37.328 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:09:37.328 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:37.328 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:09:37.328 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:37.328 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:09:37.329 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:09:37.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:37.330 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:37Z" 2024-04-13T23:09:37.330 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:37.331 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:09:37Z" } 2024-04-13T23:09:37.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:09:37.331 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:37.331 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:09:37.331 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:09:42.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:42.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:42.335 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:42.392 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:42.392 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:42.454 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:42.454 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:42.455 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:42.456 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:09:42.457 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:42.458 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:42Z" 2024-04-13T23:09:42.459 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:42.460 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:09:42Z" } 2024-04-13T23:09:42.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:09:42.461 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:42.461 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:09:42.461 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:09:47.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:47.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:47.462 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:47.523 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:47.523 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:47.592 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:47.593 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:47.593 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:47.593 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:47.593 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:47.593 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:09:47.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:47.594 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:09:47.595 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:47.596 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:47.597 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:47.598 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:47Z" 2024-04-13T23:09:47.599 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:47.600 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:09:47Z" } 2024-04-13T23:09:47.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:09:47.601 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:47.601 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:09:47.601 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:09:52.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:52.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:52.603 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:52.660 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:52.660 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:52.728 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:52.728 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:52.728 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:52.728 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:52.728 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:52.729 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:09:52.730 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:09:52.731 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:09:52.732 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:09:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:52.734 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:52Z" 2024-04-13T23:09:52.734 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:52.735 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:09:52Z" } 2024-04-13T23:09:52.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:09:52.735 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:52.735 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:09:52.735 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:09:57.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:57.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:09:57.736 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:09:57.793 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:09:57.793 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:09:57.869 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:09:57.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:57.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:09:57.872 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:57.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:57.874 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:09:57Z" 2024-04-13T23:09:57.875 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:09:57.876 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:09:57Z" } 2024-04-13T23:09:57.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:09:57.877 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:09:57.877 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:09:57.877 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:02.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:02.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:02.878 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:02.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:02.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:02.989 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:02.990 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:02.991 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:02.992 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:02.993 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:02.994 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:02.995 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:02.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:02.995 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:02Z" 2024-04-13T23:10:02.995 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:02.996 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:02Z" } 2024-04-13T23:10:02.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:02.996 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:02.996 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:02.996 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:07.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:07.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:07.998 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:08.056 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:08.056 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:08.130 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:08.131 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:08.132 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:08.133 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:08.134 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:08.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:08.136 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:08.136 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:08.136 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:08.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:08.136 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:08Z" 2024-04-13T23:10:08.136 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:08.137 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:08Z" } 2024-04-13T23:10:08.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:08.137 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:08.137 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:08.137 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:13.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:13.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:13.139 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:13.196 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:13.196 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:13.249 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:13.249 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:13.249 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:13.249 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:13.249 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:13.249 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:13.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:13.252 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:13.253 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:13Z" 2024-04-13T23:10:13.254 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:13.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:13Z" } 2024-04-13T23:10:13.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:13.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:13.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:13.256 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:18.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:18.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:18.258 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:18.316 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:18.316 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:18.397 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:18Z" 2024-04-13T23:10:18.402 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:18.403 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:18Z" } 2024-04-13T23:10:18.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:18.404 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:18.404 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:18.404 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:23.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:23.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:23.405 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:23.462 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:23.463 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:24Z" 2024-04-13T23:10:24.532 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:24.533 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:24Z" } 2024-04-13T23:10:24.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:24.533 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:24.533 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:24.533 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:29.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:29.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:29.535 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:29.593 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:29.594 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:29.651 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:29.651 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:29.651 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:29.651 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:29.652 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:29.653 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:29.654 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:29.655 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:29.656 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:29.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:29.657 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:29Z" 2024-04-13T23:10:29.657 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:29.658 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:29Z" } 2024-04-13T23:10:29.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:29.658 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:29.658 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:29.658 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:34.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:34.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:34.660 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:34.719 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:34.719 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:34.771 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:34.772 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:34.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:34.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:34.775 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:34Z" 2024-04-13T23:10:34.776 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:34.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:34Z" } 2024-04-13T23:10:34.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:34.777 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:34.777 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:34.778 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:39.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:39.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:39.779 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:39.834 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:39.834 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:39.902 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:39.902 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:39.903 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:39.903 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:39.903 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:39.903 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:39.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:39.903 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:39.904 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:39.905 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:39.906 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:39.906 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:39.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:39.906 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:39.906 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:39Z" 2024-04-13T23:10:39.910 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:39.910 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:39Z" } 2024-04-13T23:10:39.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:39.911 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:39.911 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:39.911 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:44.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:44.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:44.913 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:44.970 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:44.970 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:45.029 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:45Z" 2024-04-13T23:10:45.030 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:45.031 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:45Z" } 2024-04-13T23:10:45.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:45.032 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:45.032 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:45.032 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:50.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:50.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:50.033 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:50.090 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:50.090 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:50.140 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:50.140 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:50.140 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:50.140 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:50.140 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:50.140 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:50.141 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:50.142 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:50.143 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:50.144 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:50.145 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:50Z" 2024-04-13T23:10:50.146 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:50.146 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:50Z" } 2024-04-13T23:10:50.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:50.147 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:50.147 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:50.147 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:10:55.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:55.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:10:55.148 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:10:55.206 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:10:55.206 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:10:55.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:55.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:55.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:10:55.265 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:55.266 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:10:55.267 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:10:55.267 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:10:55.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:10:55.267 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:10:55Z" 2024-04-13T23:10:55.267 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:10:55.268 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:10:55Z" } 2024-04-13T23:10:55.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:10:55.268 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:10:55.268 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:10:55.268 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:00.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:00.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:00.270 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:00.326 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:00.327 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:00.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:00.378 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:00.379 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:00.380 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:00.381 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:00.382 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:00.383 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:00Z" 2024-04-13T23:11:00.383 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:00.384 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:00Z" } 2024-04-13T23:11:00.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:00.384 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:00.384 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:00.384 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:05.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:05.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:05.386 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:05.443 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:05.444 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:05.523 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:05.523 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:05.523 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:05.523 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:05.523 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:05.523 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:05.524 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:05.525 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:05.526 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:05.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:05.529 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:05Z" 2024-04-13T23:11:05.529 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:05.529 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:05Z" } 2024-04-13T23:11:05.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:05.530 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:05.530 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:05.530 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:10.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:10.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:10.532 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:10.590 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:10.590 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:10.655 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:10.657 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:10.658 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:10.659 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:10Z" 2024-04-13T23:11:10.660 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:10.661 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:10Z" } 2024-04-13T23:11:10.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:10.662 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:10.662 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:10.662 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:15.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:15.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:15.664 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:15.721 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:15.721 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:15.774 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:15Z" 2024-04-13T23:11:15.780 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:15.781 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:15Z" } 2024-04-13T23:11:15.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:15.782 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:15.782 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:15.782 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:20.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:20.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:20.787 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:20.842 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:20.842 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:20.916 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:20.916 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:20.916 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:20.917 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:20.918 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:20.919 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:20.920 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:20.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:20.922 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:20.922 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:20.922 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:20.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:20.922 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:20Z" 2024-04-13T23:11:20.923 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:20.924 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:20Z" } 2024-04-13T23:11:20.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:20.925 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:20.925 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:20.925 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:25.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:25.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:25.927 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:25.984 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:25.984 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:26.045 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:26.045 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:26.045 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:26.045 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:26.045 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:26.046 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:26.047 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:26.048 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:26.049 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:26.050 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:26Z" 2024-04-13T23:11:26.051 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:26.051 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:26Z" } 2024-04-13T23:11:26.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:26.052 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:26.052 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:26.052 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:31.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:31.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:31.054 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:31.111 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:31.111 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:31.164 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:31.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:31.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047492", 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.564829Z" 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:31.167 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:31.168 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:31.168 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:31.168 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:31.168 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033609", 2024-04-13T23:11:31.168 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:31.168 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013679Z" 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000053786", 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:10.554830Z" 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:31Z" 2024-04-13T23:11:31.171 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:31.172 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047492", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:10.564829Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033609", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:08:58.013679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053786", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:08:10.554830Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:31Z" } 2024-04-13T23:11:31.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047492', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.564829Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033609', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053786', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:08:10.554830Z'}}] 2024-04-13T23:11:31.172 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:31.173 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047492', 2: '00000000000000000000:00000000000000033609', 3: '00000000000000000000:00000000000000053786'} 2024-04-13T23:11:31.173 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047633 target=00000000000000000000:00000000000000047492, shard=2 master=00000000000000000000:00000000000000033750 target=00000000000000000000:00000000000000033609, shard=3 master=00000000000000000000:00000000000000054068 target=00000000000000000000:00000000000000053786 2024-04-13T23:11:36.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:36.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:36.175 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:36.230 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.230 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:36.279 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:36.279 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:36.281 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047633", 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.392463Z" 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:36.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033750", 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.464865Z" 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:36.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054068", 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.437863Z" 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:36Z" 2024-04-13T23:11:36.285 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:36.286 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047633", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:09:31.392463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033750", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:09:31.464865Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054068", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:09:31.437863Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:11:36Z" } 2024-04-13T23:11:36.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047633', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.392463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033750', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.464865Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054068', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.437863Z'}}] 2024-04-13T23:11:36.286 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:36.286 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:11:36.287 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:11:36.287 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:11:36.287 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:11:36.287 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.287 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:11:36.287 DEBUG:boto:Data: 2024-04-13T23:11:36.287 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.287 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.288 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.288 DEBUG:boto:Params: {} 2024-04-13T23:11:36.289 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:11:36.289 DEBUG:boto:Token: None 2024-04-13T23:11:36.289 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-74/ 2024-04-13T23:11:36.289 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:z85JIJ5NYK0yT90jB3BdZEW0lt8= 2024-04-13T23:11:36.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z85JIJ5NYK0yT90jB3BdZEW0lt8=', 'Content-Length': '0'} 2024-04-13T23:11:36.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb1fbd8175296cc6-00661b1128-2c4c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.295 DEBUG:boto:path=/vwnfns-74/obj1 2024-04-13T23:11:36.296 DEBUG:boto:auth_path=/vwnfns-74/obj1 2024-04-13T23:11:36.296 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.296 DEBUG:boto:Path: /vwnfns-74/obj1 2024-04-13T23:11:36.296 DEBUG:boto:Data: 2024-04-13T23:11:36.296 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.296 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.296 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.296 DEBUG:boto:Params: {} 2024-04-13T23:11:36.296 DEBUG:boto:Token: None 2024-04-13T23:11:36.296 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-74/obj1 2024-04-13T23:11:36.296 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:lzeld9Ww1huhOzLZoTWiy1LFCfY= 2024-04-13T23:11:36.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lzeld9Ww1huhOzLZoTWiy1LFCfY=', 'Content-Length': '0'} 2024-04-13T23:11:36.298 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008ed00d783f0aaab7-00661b1128-2c4c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.298 DEBUG:boto:path=/vwnfns-74/obj2 2024-04-13T23:11:36.298 DEBUG:boto:auth_path=/vwnfns-74/obj2 2024-04-13T23:11:36.299 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.299 DEBUG:boto:Path: /vwnfns-74/obj2 2024-04-13T23:11:36.299 DEBUG:boto:Data: 2024-04-13T23:11:36.299 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.299 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.299 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.299 DEBUG:boto:Params: {} 2024-04-13T23:11:36.299 DEBUG:boto:Token: None 2024-04-13T23:11:36.299 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-74/obj2 2024-04-13T23:11:36.300 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Fi3NvLKS+Owz/uQ8g4qmMtg0AH0= 2024-04-13T23:11:36.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Fi3NvLKS+Owz/uQ8g4qmMtg0AH0=', 'Content-Length': '0'} 2024-04-13T23:11:36.301 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000204ca25e8d3def3e-00661b1128-2c4c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.301 DEBUG:boto:path=/vwnfns-74/obj3 2024-04-13T23:11:36.301 DEBUG:boto:auth_path=/vwnfns-74/obj3 2024-04-13T23:11:36.301 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.302 DEBUG:boto:Path: /vwnfns-74/obj3 2024-04-13T23:11:36.302 DEBUG:boto:Data: 2024-04-13T23:11:36.302 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.302 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.302 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.302 DEBUG:boto:Params: {} 2024-04-13T23:11:36.302 DEBUG:boto:Token: None 2024-04-13T23:11:36.302 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-74/obj3 2024-04-13T23:11:36.302 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cGFot8JWJko4SY/NWCvT7+TfuEs= 2024-04-13T23:11:36.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cGFot8JWJko4SY/NWCvT7+TfuEs=', 'Content-Length': '0'} 2024-04-13T23:11:36.304 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007e975f928daba396-00661b1128-2c4c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.304 DEBUG:boto:path=/vwnfns-74/obj4 2024-04-13T23:11:36.304 DEBUG:boto:auth_path=/vwnfns-74/obj4 2024-04-13T23:11:36.305 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.305 DEBUG:boto:Path: /vwnfns-74/obj4 2024-04-13T23:11:36.305 DEBUG:boto:Data: 2024-04-13T23:11:36.305 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.305 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.305 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.305 DEBUG:boto:Params: {} 2024-04-13T23:11:36.305 DEBUG:boto:Token: None 2024-04-13T23:11:36.305 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-74/obj4 2024-04-13T23:11:36.305 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:9cNmzU6x+/QcLy2IXs4REkjOhE4= 2024-04-13T23:11:36.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9cNmzU6x+/QcLy2IXs4REkjOhE4=', 'Content-Length': '0'} 2024-04-13T23:11:36.307 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e0f31cfb6e1e8a6d-00661b1128-2c4c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.307 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:11:36.307 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:11:36.307 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.307 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:11:36.308 DEBUG:boto:Data: 2024-04-13T23:11:36.308 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.308 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.308 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.308 DEBUG:boto:Params: {} 2024-04-13T23:11:36.308 DEBUG:boto:Token: None 2024-04-13T23:11:36.308 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/ 2024-04-13T23:11:36.308 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:4rTDNAraKIPK6JIaXiba9MzgvI0= 2024-04-13T23:11:36.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4rTDNAraKIPK6JIaXiba9MzgvI0=', 'Content-Length': '0'} 2024-04-13T23:11:36.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e19c33ea25c7035-00661b1128-2c4c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.311 DEBUG:boto:path=/vwnfns-75/obj1 2024-04-13T23:11:36.311 DEBUG:boto:auth_path=/vwnfns-75/obj1 2024-04-13T23:11:36.311 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.311 DEBUG:boto:Path: /vwnfns-75/obj1 2024-04-13T23:11:36.311 DEBUG:boto:Data: 2024-04-13T23:11:36.311 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.311 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.311 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.311 DEBUG:boto:Params: {} 2024-04-13T23:11:36.312 DEBUG:boto:Token: None 2024-04-13T23:11:36.312 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj1 2024-04-13T23:11:36.312 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5ThexP2FTMTZz3CvJhm9w+crXnc= 2024-04-13T23:11:36.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5ThexP2FTMTZz3CvJhm9w+crXnc=', 'Content-Length': '0'} 2024-04-13T23:11:36.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-74:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aafd8e21ad08f6c3-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.314 DEBUG:boto:path=/vwnfns-75/obj1 2024-04-13T23:11:36.314 DEBUG:boto:auth_path=/vwnfns-75/obj1 2024-04-13T23:11:36.314 DEBUG:boto:Method: GET 2024-04-13T23:11:36.314 DEBUG:boto:Path: /vwnfns-75/obj1 2024-04-13T23:11:36.314 DEBUG:boto:Data: 2024-04-13T23:11:36.314 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.315 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.315 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.315 DEBUG:boto:Params: {} 2024-04-13T23:11:36.315 DEBUG:boto:Token: None 2024-04-13T23:11:36.315 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj1 2024-04-13T23:11:36.315 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:KyYwruRZerxqOCt05BxYlkod3UQ= 2024-04-13T23:11:36.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KyYwruRZerxqOCt05BxYlkod3UQ=', 'Content-Length': '0'} 2024-04-13T23:11:36.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-74:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc8d65779bcf2f68-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.358 DEBUG:boto:path=/vwnfns-75/obj2 2024-04-13T23:11:36.359 DEBUG:boto:auth_path=/vwnfns-75/obj2 2024-04-13T23:11:36.359 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.359 DEBUG:boto:Path: /vwnfns-75/obj2 2024-04-13T23:11:36.359 DEBUG:boto:Data: 2024-04-13T23:11:36.359 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.359 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.359 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.359 DEBUG:boto:Params: {} 2024-04-13T23:11:36.359 DEBUG:boto:Token: None 2024-04-13T23:11:36.360 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj2 2024-04-13T23:11:36.360 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+w2s9M3qy13tVpShBshjSS6UKEo= 2024-04-13T23:11:36.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+w2s9M3qy13tVpShBshjSS6UKEo=', 'Content-Length': '0'} 2024-04-13T23:11:36.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-74:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001778710f6aabf6b4-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.362 DEBUG:boto:path=/vwnfns-75/obj2 2024-04-13T23:11:36.362 DEBUG:boto:auth_path=/vwnfns-75/obj2 2024-04-13T23:11:36.362 DEBUG:boto:Method: GET 2024-04-13T23:11:36.362 DEBUG:boto:Path: /vwnfns-75/obj2 2024-04-13T23:11:36.362 DEBUG:boto:Data: 2024-04-13T23:11:36.362 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.362 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.362 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.363 DEBUG:boto:Params: {} 2024-04-13T23:11:36.363 DEBUG:boto:Token: None 2024-04-13T23:11:36.363 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj2 2024-04-13T23:11:36.363 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ie5u078rnVpaH5An4HYjqcgJWEg= 2024-04-13T23:11:36.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ie5u078rnVpaH5An4HYjqcgJWEg=', 'Content-Length': '0'} 2024-04-13T23:11:36.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-74:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fa47c1a710f29eb-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.365 DEBUG:boto:path=/vwnfns-75/obj3 2024-04-13T23:11:36.365 DEBUG:boto:auth_path=/vwnfns-75/obj3 2024-04-13T23:11:36.365 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.365 DEBUG:boto:Path: /vwnfns-75/obj3 2024-04-13T23:11:36.366 DEBUG:boto:Data: 2024-04-13T23:11:36.366 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.366 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.366 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.366 DEBUG:boto:Params: {} 2024-04-13T23:11:36.366 DEBUG:boto:Token: None 2024-04-13T23:11:36.366 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj3 2024-04-13T23:11:36.366 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kkj8yejAofOe8jqIbuoPtviW0LY= 2024-04-13T23:11:36.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kkj8yejAofOe8jqIbuoPtviW0LY=', 'Content-Length': '0'} 2024-04-13T23:11:36.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c279e1629593e5f1-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.368 DEBUG:boto:path=/vwnfns-75/obj3 2024-04-13T23:11:36.368 DEBUG:boto:auth_path=/vwnfns-75/obj3 2024-04-13T23:11:36.368 DEBUG:boto:Method: GET 2024-04-13T23:11:36.368 DEBUG:boto:Path: /vwnfns-75/obj3 2024-04-13T23:11:36.368 DEBUG:boto:Data: 2024-04-13T23:11:36.368 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.369 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.369 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.369 DEBUG:boto:Params: {} 2024-04-13T23:11:36.369 DEBUG:boto:Token: None 2024-04-13T23:11:36.369 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj3 2024-04-13T23:11:36.369 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:tzO/Lzz7OIiERrdtWVNnqOrPquQ= 2024-04-13T23:11:36.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tzO/Lzz7OIiERrdtWVNnqOrPquQ=', 'Content-Length': '0'} 2024-04-13T23:11:36.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3c6f9e529672060-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.371 DEBUG:boto:path=/vwnfns-75/obj4 2024-04-13T23:11:36.371 DEBUG:boto:auth_path=/vwnfns-75/obj4 2024-04-13T23:11:36.371 DEBUG:boto:Method: HEAD 2024-04-13T23:11:36.371 DEBUG:boto:Path: /vwnfns-75/obj4 2024-04-13T23:11:36.371 DEBUG:boto:Data: 2024-04-13T23:11:36.371 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.371 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.371 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.372 DEBUG:boto:Params: {} 2024-04-13T23:11:36.372 DEBUG:boto:Token: None 2024-04-13T23:11:36.372 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj4 2024-04-13T23:11:36.372 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:10yjTmELu8lacPELBGFLgvCwoZg= 2024-04-13T23:11:36.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:10yjTmELu8lacPELBGFLgvCwoZg=', 'Content-Length': '0'} 2024-04-13T23:11:36.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2ac7732a6dcb4ae-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.374 DEBUG:boto:path=/vwnfns-75/obj4 2024-04-13T23:11:36.374 DEBUG:boto:auth_path=/vwnfns-75/obj4 2024-04-13T23:11:36.374 DEBUG:boto:Method: GET 2024-04-13T23:11:36.374 DEBUG:boto:Path: /vwnfns-75/obj4 2024-04-13T23:11:36.374 DEBUG:boto:Data: 2024-04-13T23:11:36.374 DEBUG:boto:Headers: {} 2024-04-13T23:11:36.374 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:11:36.374 DEBUG:boto:Port: 8001 2024-04-13T23:11:36.374 DEBUG:boto:Params: {} 2024-04-13T23:11:36.375 DEBUG:boto:Token: None 2024-04-13T23:11:36.375 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:11:36 GMT /vwnfns-75/obj4 2024-04-13T23:11:36.375 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:cTOJPJRj2hG1ThaQChRV5wg3NPM= 2024-04-13T23:11:36.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:11:36 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cTOJPJRj2hG1ThaQChRV5wg3NPM=', 'Content-Length': '0'} 2024-04-13T23:11:36.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:09:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e636f158296589b6-00661b1128-2c4c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:11:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:11:36.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.377 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-vwnfns-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:36.426 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.427 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:36.494 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-vwnfns-75","source":{"bucket":"vwnfns-75","zones":["a1"]},"dest":{"bucket":"vwnfns-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:11:36.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.495 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-vwnfns-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:36.545 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.545 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:36.628 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-13T23:11:36.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.629 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:36.681 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:36.734 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-13T23:11:36.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.735 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:36.786 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.787 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:36.836 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-13T23:11:36.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.836 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:36.887 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.887 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:36.934 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:11:36.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:36.934 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:37.031 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:11:37.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.032 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:37.078 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:37.078 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 20, 2024-04-13T23:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:11:37.141 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:11:37.142 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.146 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:11:37.147 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:11:37.149 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:11:37.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:11:37.151 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:11:37.152 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:11:37.153 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:11:37.154 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "allowed" 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:11:37.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:11:37.158 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.159 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:11:37.160 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:11:37.161 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:11:37.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:11:37.164 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:11:37.165 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:11:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:11:37.167 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:11:37.168 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:11:37.169 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:37.170 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:37.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.170 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:37.226 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:37.227 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:37.331 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:11:37.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.332 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket vwnfns-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:37.542 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:11:37.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.543 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket vwnfns-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:37.612 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:37.612 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:37.730 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:11:37.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-74', '--source-bucket=vwnfns-74', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-74', '--source-bucket=vwnfns-74', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.730 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-vwnfns-74 --source-zones=a1 --dest-zones=a2 --bucket=vwnfns-74 --source-bucket=vwnfns-74 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:37.795 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:37.795 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:37.879 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-13T23:11:37.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-74', '--source-bucket=vwnfns-75', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-74', '--source-bucket=vwnfns-75', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:37.880 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-vwnfns-75 --source-zones=a1 --dest-zones=a2 --bucket=vwnfns-74 --source-bucket=vwnfns-75 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:37.944 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:37.945 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:38.078 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-vwnfns-75","source":{"bucket":"vwnfns-75","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-13T23:11:38.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:38.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:38.079 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket vwnfns-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:38.136 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:38.137 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:38.219 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-74","source":{"bucket":"vwnfns-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-vwnfns-75","source":{"bucket":"vwnfns-75","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T23:11:38.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:38.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:11:38.219 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:11:38.263 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:38.263 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:38.337 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:11:38.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:38.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:38.338 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:38.384 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:38.385 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:38.432 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:11:38.447 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:11:38.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.447 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:11:38.447 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:11:38.190756Z" 2024-04-13T23:11:38.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:11:38.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:38.449 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:11:38.450 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:38.450 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:11:38.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:38.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:38.450 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:38.497 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:38.497 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:38.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:38.528+0000 7fd32d603780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:38.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:38.529+0000 7fd32d603780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:38.550 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:38.550 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:38.550 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:11:38.551 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:38.552 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:38.553 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:38.554 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.555 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:38.556 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:11:38.556 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:11:38.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:38.556 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:38Z" 2024-04-13T23:11:38.556 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:38.557 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:11:38Z" } 2024-04-13T23:11:38.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:11:38.558 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:38.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:38.558 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:11:43.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:43.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:43.559 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:43.616 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:43.616 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:43.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:43.657+0000 7fc5d25e8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:43.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:43.658+0000 7fc5d25e8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:43.675 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:43.675 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:43.675 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:43.675 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:43.675 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:11:43.676 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:43.677 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:11:43.678 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:43.679 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:43.680 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:11:43.681 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:11:43.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:43.681 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:43Z" 2024-04-13T23:11:43.681 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:43.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:11:43Z" } 2024-04-13T23:11:43.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:11:43.682 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:43.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:43.682 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:11:48.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:48.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:48.683 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:48.740 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:48.740 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:48.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:48.778+0000 7fb39204f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:48.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:48.779+0000 7fb39204f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:48.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:48.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:48.798 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:48.799 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:48.800 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:48.801 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:11:48.802 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:11:48.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:48.802 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:48Z" 2024-04-13T23:11:48.802 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:48.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:11:48Z" } 2024-04-13T23:11:48.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:11:48.803 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:48.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:48.803 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:11:53.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:53.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:53.805 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:53.860 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:53.860 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:53.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:53.894+0000 7f60ef135780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:53.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:53.895+0000 7f60ef135780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:53.913 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:53.913 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:53.913 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:53.914 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:53.915 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:53.916 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:53.917 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:53.918 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:53Z" 2024-04-13T23:11:53.919 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:53.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:11:53Z" } 2024-04-13T23:11:53.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:11:53.920 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:53.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:53.921 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:11:58.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:58.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:11:58.922 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:11:58.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:11:58.977 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:11:59.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:59.020+0000 7f7271b4c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:59.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:11:59.021+0000 7f7271b4c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:11:59.038 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:11:59.038 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:11:59.038 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:11:59.038 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:11:59.038 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:11:59.038 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:11:59.039 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:59.040 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:59.041 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:11:59.042 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:11:59.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:11:59.044 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:11:59Z" 2024-04-13T23:11:59.044 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:11:59.044 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:11:59Z" } 2024-04-13T23:11:59.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:11:59.045 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:59.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:11:59.045 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:04.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:04.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:04.047 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:04.102 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:04.102 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:04.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:04.134+0000 7fd42d2a2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:04.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:04.135+0000 7fd42d2a2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:04.150 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:04.150 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:04.150 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:04.150 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:04.150 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:04.150 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:04.151 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:04.152 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:04.153 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:04.154 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:04.155 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:04Z" 2024-04-13T23:12:04.156 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:04.157 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:04Z" } 2024-04-13T23:12:04.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:04.158 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:04.158 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:04.158 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:09.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:09.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:09.160 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:09.216 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:09.216 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:09.259+0000 7f3265962780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:09.260+0000 7f3265962780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:09.281 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:09.282 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:09.283 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:09.284 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:09.285 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:09Z" 2024-04-13T23:12:09.286 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:09.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:09Z" } 2024-04-13T23:12:09.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:09.288 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:09.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:09.288 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:14.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:14.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:14.290 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:14.346 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:14.347 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:14.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:14.396+0000 7ff0197d2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:14.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:14.398+0000 7ff0197d2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:14.419 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:14.419 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:14.419 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:14.419 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:14.419 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:14.419 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:14.420 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:14.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:14.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:14.424 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:14Z" 2024-04-13T23:12:14.425 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:14.426 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:14Z" } 2024-04-13T23:12:14.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:14.427 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:14.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:14.427 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:19.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:19.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:19.429 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:19.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:19.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:19.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:19.514+0000 7f4978c7a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:19.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:19.516+0000 7f4978c7a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:19.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:19.533 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:19.534 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:19.535 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:19.537 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:19.538 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:19Z" 2024-04-13T23:12:19.538 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:19.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:19Z" } 2024-04-13T23:12:19.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:19.539 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:19.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:19.539 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:24.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:24.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:24.541 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:24.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:24.596 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:24.637+0000 7fc39dccd780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:24.638+0000 7fc39dccd780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:24.662 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:24.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:24.666 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:24.667 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:24.668 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:24.669 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:24Z" 2024-04-13T23:12:24.669 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:24.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:24Z" } 2024-04-13T23:12:24.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:24.671 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:24.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:24.671 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:29.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:29.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:29.673 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:29.729 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:29.729 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:29.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:29.779+0000 7fc28e14a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:29.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:29.780+0000 7fc28e14a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:29.796 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:29.797 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:29.798 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:29.799 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:29.800 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:29.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:29.802 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:29.802 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:29.802 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:29.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:29.802 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:29Z" 2024-04-13T23:12:29.802 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:29.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:29Z" } 2024-04-13T23:12:29.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:29.803 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:29.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:29.804 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:34.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:34.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:34.805 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:34.858 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:34.858 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:34.892+0000 7fe0fccaa780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:34.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:34.893+0000 7fe0fccaa780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:34.911 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:34.911 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:34.911 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:34.911 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:34.912 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:34.913 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:34.914 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:34.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:34.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:34Z" 2024-04-13T23:12:34.917 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:34.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:34Z" } 2024-04-13T23:12:34.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:34.918 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:34.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:34.919 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:39.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:39.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:39.920 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:39.975 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:39.975 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:40.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:40.019+0000 7f054af99780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:40.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:40.020+0000 7f054af99780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:40.038 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:40.039 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:40Z" 2024-04-13T23:12:40.040 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:40.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:40Z" } 2024-04-13T23:12:40.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:40.041 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:40.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:40.042 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:45.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:45.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:45.043 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:45.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:45.097 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:45.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:45.137+0000 7f488a3cb780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:45.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:45.138+0000 7f488a3cb780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:45.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:45.162 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:45.163 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:45.164 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:45.165 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:45Z" 2024-04-13T23:12:45.166 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:45.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:45Z" } 2024-04-13T23:12:45.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:45.168 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:45.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:45.168 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:50.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:50.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:50.170 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:50.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:50.225 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:50.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:50.273+0000 7fd405c43780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:50.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:50.274+0000 7fd405c43780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:50.290 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:50.290 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:50.290 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:50.290 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:50.291 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:50.292 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:50.293 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:50.294 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:50.295 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:50Z" 2024-04-13T23:12:50.296 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:50.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:50Z" } 2024-04-13T23:12:50.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:50.297 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:50.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:50.297 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:12:55.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:55.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:12:55.299 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:12:55.355 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:12:55.355 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:12:55.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:55.388+0000 7f9a26721780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:55.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:12:55.389+0000 7f9a26721780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:12:55.405 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:55.406 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:55.407 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:55.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:55.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:12:55.410 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:12:55Z" 2024-04-13T23:12:55.411 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:12:55.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:12:55Z" } 2024-04-13T23:12:55.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:12:55.412 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:55.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:12:55.413 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:00.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:00.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:00.414 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:00.470 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:00.470 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:00.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:00.522+0000 7f0870fcf780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:00.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:00.523+0000 7f0870fcf780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:00.546 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:00.546 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:00.546 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:00.547 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:00.548 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:00.549 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:00.550 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:00.551 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:00.552 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:00Z" 2024-04-13T23:13:00.552 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:00.553 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:00Z" } 2024-04-13T23:13:00.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:00.553 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:00.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:00.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:05.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:05.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:05.555 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:05.610 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:05.610 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:05.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:05.657+0000 7f7eefbb1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:05.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:05.658+0000 7f7eefbb1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:05.674 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:05Z" 2024-04-13T23:13:05.680 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:05.681 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:05Z" } 2024-04-13T23:13:05.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:05.681 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:05.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:05.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:10.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:10.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:10.683 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:10.741 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:10.741 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:10.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:10.775+0000 7fca399c0780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:10.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:10.776+0000 7fca399c0780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:10.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:10.795 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:10.796 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:10.797 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:10.798 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:10.799 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:10.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:10.800 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:10Z" 2024-04-13T23:13:10.800 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:10.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:10Z" } 2024-04-13T23:13:10.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:10.801 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:10.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:10.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:15.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:15.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:15.803 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:15.857 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:15.857 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:15.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:15.903+0000 7ff2bb676780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:15.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:15.903+0000 7ff2bb676780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:15.922 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:15.922 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:15.922 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:15.923 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:15.924 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:15.925 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:15.926 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:15.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:15.928 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:15Z" 2024-04-13T23:13:15.928 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:15.928 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:15Z" } 2024-04-13T23:13:15.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:15.929 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:15.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:15.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:20.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:20.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:20.931 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:20.986 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:20.987 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:21.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:21.024+0000 7f7887187780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:21.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:21.026+0000 7f7887187780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:21.041 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:21.041 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:21.041 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:21.041 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:21.042 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:21.043 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:21.044 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:21.045 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:21.046 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:21.047 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:21.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:21.047 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:21Z" 2024-04-13T23:13:21.047 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:21.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:21Z" } 2024-04-13T23:13:21.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:21.048 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:21.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:21.048 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:26.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:26.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:26.050 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:26.105 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:26.105 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:26.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:26.138+0000 7f3274cb6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:26.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:26.139+0000 7f3274cb6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:26.157 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:26.157 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:26.158 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:26.159 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:26.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:26.162 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:26Z" 2024-04-13T23:13:26.163 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:26.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:26Z" } 2024-04-13T23:13:26.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:26.165 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:26.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:26.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:31.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:31.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:31.167 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:31.222 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:31.222 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:31.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:31.259+0000 7f7b85749780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:31.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:31.260+0000 7f7b85749780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:31.278 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:31.278 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:31.278 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:31.278 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:31.279 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:31.280 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:31.281 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:31.282 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:31.283 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:31.284 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:31.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:31.284 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:31Z" 2024-04-13T23:13:31.284 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:31.285 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:31Z" } 2024-04-13T23:13:31.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:31.285 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:31.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:31.285 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:36.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:36.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:36.287 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:36.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:36.374+0000 7f3e7991b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:36.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:36.375+0000 7f3e7991b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:36.391 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:36.391 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:36.391 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:36.391 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:36.392 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049770.107263_1196.1", 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:30.107263Z", 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:36.394 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:36.395 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:36.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:36.397 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:36.397 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:36.397 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:36.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:36.397 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:36Z" 2024-04-13T23:13:36.397 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:36.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049770.107263_1196.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:09:30.107263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:36Z" } 2024-04-13T23:13:36.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049770.107263_1196.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:09:30.107263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:36.398 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:36.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049770.107263_1196.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:36.399 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713049898.190756_1244.1 target=1_1713049770.107263_1196.1 2024-04-13T23:13:41.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.400 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:41.455 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:41.456 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:41.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:41.506+0000 7f371be3f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:41.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:41.507+0000 7f371be3f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:41.531 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:41.531 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:41.531 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:41.531 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:41.531 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:41.531 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.532 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:11:38.190756Z", 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:41.533 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.534 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:41.535 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.536 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:41.537 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:41.537 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:41.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.537 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:41Z" 2024-04-13T23:13:41.537 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:41.538 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049898.190756_1244.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:11:38.190756Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:41Z" } 2024-04-13T23:13:41.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049898.190756_1244.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:11:38.190756Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:41.538 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:41.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:41.538 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:13:41.539 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:13:41.539 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:13:41.539 DEBUG:boto:Method: HEAD 2024-04-13T23:13:41.539 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:13:41.539 DEBUG:boto:Data: 2024-04-13T23:13:41.539 DEBUG:boto:Headers: {} 2024-04-13T23:13:41.539 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.540 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.540 DEBUG:boto:Params: {} 2024-04-13T23:13:41.540 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:13:41.540 DEBUG:boto:Token: None 2024-04-13T23:13:41.541 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-74/ 2024-04-13T23:13:41.541 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ak/60as4dt7+tnTEmZdVJVEa3ZE= 2024-04-13T23:13:41.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ak/60as4dt7+tnTEmZdVJVEa3ZE=', 'Content-Length': '0'} 2024-04-13T23:13:41.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba9ed67739e0839-00661b11a5-2d6b-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.548 DEBUG:boto:path=/vwnfns-74/obj5 2024-04-13T23:13:41.548 DEBUG:boto:auth_path=/vwnfns-74/obj5 2024-04-13T23:13:41.548 DEBUG:boto:Method: PUT 2024-04-13T23:13:41.548 DEBUG:boto:Path: /vwnfns-74/obj5 2024-04-13T23:13:41.548 DEBUG:boto:Data: 2024-04-13T23:13:41.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:13:41.548 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.549 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.549 DEBUG:boto:Params: {} 2024-04-13T23:13:41.549 DEBUG:boto:Token: None 2024-04-13T23:13:41.549 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-74/obj5 2024-04-13T23:13:41.549 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:3bjGj4NeWJcnotdubp/r8xuLgvg= 2024-04-13T23:13:41.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3bjGj4NeWJcnotdubp/r8xuLgvg='} 2024-04-13T23:13:41.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9af7e8b9a7f5630-00661b11a5-2d6b-a1'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.558 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:13:41.558 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:13:41.558 DEBUG:boto:Method: HEAD 2024-04-13T23:13:41.558 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:13:41.558 DEBUG:boto:Data: 2024-04-13T23:13:41.558 DEBUG:boto:Headers: {} 2024-04-13T23:13:41.558 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.558 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.558 DEBUG:boto:Params: {} 2024-04-13T23:13:41.562 DEBUG:boto:Token: None 2024-04-13T23:13:41.562 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-74/ 2024-04-13T23:13:41.562 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Ak/60as4dt7+tnTEmZdVJVEa3ZE= 2024-04-13T23:13:41.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ak/60as4dt7+tnTEmZdVJVEa3ZE=', 'Content-Length': '0'} 2024-04-13T23:13:41.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de8a9af9c1622f06-00661b11a5-2d6b-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.565 DEBUG:boto:path=/vwnfns-74/obj6 2024-04-13T23:13:41.565 DEBUG:boto:auth_path=/vwnfns-74/obj6 2024-04-13T23:13:41.565 DEBUG:boto:Method: PUT 2024-04-13T23:13:41.565 DEBUG:boto:Path: /vwnfns-74/obj6 2024-04-13T23:13:41.565 DEBUG:boto:Data: 2024-04-13T23:13:41.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:13:41.565 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.566 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.566 DEBUG:boto:Params: {} 2024-04-13T23:13:41.566 DEBUG:boto:Token: None 2024-04-13T23:13:41.566 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-74/obj6 2024-04-13T23:13:41.566 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5j8HZli/R32W5+rlX318jlQivhI= 2024-04-13T23:13:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5j8HZli/R32W5+rlX318jlQivhI='} 2024-04-13T23:13:41.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5abc1e387e0d182-00661b11a5-2d6b-a1'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.572 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:13:41.572 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:13:41.573 DEBUG:boto:Method: HEAD 2024-04-13T23:13:41.573 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:13:41.573 DEBUG:boto:Data: 2024-04-13T23:13:41.573 DEBUG:boto:Headers: {} 2024-04-13T23:13:41.573 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.573 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.573 DEBUG:boto:Params: {} 2024-04-13T23:13:41.573 DEBUG:boto:Token: None 2024-04-13T23:13:41.573 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-75/ 2024-04-13T23:13:41.574 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1KdrcT6/wBkE2Hy0YagEVehzAzI= 2024-04-13T23:13:41.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1KdrcT6/wBkE2Hy0YagEVehzAzI=', 'Content-Length': '0'} 2024-04-13T23:13:41.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f1aa1cf9e82a4de-00661b11a5-2d6b-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.577 DEBUG:boto:path=/vwnfns-75/obj7 2024-04-13T23:13:41.577 DEBUG:boto:auth_path=/vwnfns-75/obj7 2024-04-13T23:13:41.577 DEBUG:boto:Method: PUT 2024-04-13T23:13:41.577 DEBUG:boto:Path: /vwnfns-75/obj7 2024-04-13T23:13:41.577 DEBUG:boto:Data: 2024-04-13T23:13:41.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:13:41.577 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.577 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.577 DEBUG:boto:Params: {} 2024-04-13T23:13:41.577 DEBUG:boto:Token: None 2024-04-13T23:13:41.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-75/obj7 2024-04-13T23:13:41.578 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EgWG+UVgkJrjTv1DOa+7vrrZMmc= 2024-04-13T23:13:41.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EgWG+UVgkJrjTv1DOa+7vrrZMmc='} 2024-04-13T23:13:41.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b507a4373d535dc-00661b11a5-2d6b-a1'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.623 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:13:41.623 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:13:41.623 DEBUG:boto:Method: HEAD 2024-04-13T23:13:41.623 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:13:41.623 DEBUG:boto:Data: 2024-04-13T23:13:41.623 DEBUG:boto:Headers: {} 2024-04-13T23:13:41.623 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.623 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.623 DEBUG:boto:Params: {} 2024-04-13T23:13:41.624 DEBUG:boto:Token: None 2024-04-13T23:13:41.624 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-75/ 2024-04-13T23:13:41.624 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1KdrcT6/wBkE2Hy0YagEVehzAzI= 2024-04-13T23:13:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1KdrcT6/wBkE2Hy0YagEVehzAzI=', 'Content-Length': '0'} 2024-04-13T23:13:41.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3492c9b04fd9f04-00661b11a5-2d6b-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.626 DEBUG:boto:path=/vwnfns-75/obj8 2024-04-13T23:13:41.627 DEBUG:boto:auth_path=/vwnfns-75/obj8 2024-04-13T23:13:41.627 DEBUG:boto:Method: PUT 2024-04-13T23:13:41.627 DEBUG:boto:Path: /vwnfns-75/obj8 2024-04-13T23:13:41.627 DEBUG:boto:Data: 2024-04-13T23:13:41.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:13:41.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:13:41.627 DEBUG:boto:Port: 8000 2024-04-13T23:13:41.627 DEBUG:boto:Params: {} 2024-04-13T23:13:41.627 DEBUG:boto:Token: None 2024-04-13T23:13:41.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:13:41 GMT /vwnfns-75/obj8 2024-04-13T23:13:41.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:24YQf+X/ja3Idf22cQqL3JepVH8= 2024-04-13T23:13:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:13:41 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:24YQf+X/ja3Idf22cQqL3JepVH8='} 2024-04-13T23:13:41.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b24dcd455aa4912-00661b11a5-2d6b-a1'), ('Date', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:13:41.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.635 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:41.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:41.682 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:41.715 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:13:41.737 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:13:41.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.737 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:13:41.737 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:11:38.190756Z" 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.739 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:41.739 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:13:41.739 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.739 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:13:41.740 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:41.740 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:13:41.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.740 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:41.787 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:41.788 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:41.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:41.836+0000 7f4cafb7d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:41.839 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:13:41.837+0000 7f4cafb7d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:13:41.857 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:41.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:11:38.190756Z", 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:13:41.860 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:41.861 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:13:41.862 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:13:41.863 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:13:41.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.864 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:41Z" 2024-04-13T23:13:41.864 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:41.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049898.190756_1244.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:11:38.190756Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:13:41Z" } 2024-04-13T23:13:41.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049898.190756_1244.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:11:38.190756Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:13:41.865 DEBUG:rgw_multi.tests:log_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:41.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:13:41.865 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:13:41.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.866 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:41.912 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:41.912 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:41.968 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:13:41.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:13:41.969 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:13:41.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:41.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:41.970 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:13:41.970 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:13:41.970 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:41.970 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:13:41.971 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:13:41.971 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:13:41.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:41.972 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:42.021 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:42.021 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:42.089 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:42.089 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:42.090 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:42.091 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047633", 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.392463Z" 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:42.092 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033750", 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.464865Z" 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:42.093 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054068", 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.437863Z" 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:42.094 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:13:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:13:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:42.095 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:42Z" 2024-04-13T23:13:42.095 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:42.096 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047633", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:09:31.392463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033750", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:09:31.464865Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054068", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:09:31.437863Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:13:42Z" } 2024-04-13T23:13:42.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047633', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.392463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033750', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.464865Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054068', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.437863Z'}}] 2024-04-13T23:13:42.096 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:13:42.096 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:13:42.096 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047774 target=00000000000000000000:00000000000000047633, shard=2 master=00000000000000000000:00000000000000033891 target=00000000000000000000:00000000000000033750, shard=3 master=00000000000000000000:00000000000000054350 target=00000000000000000000:00000000000000054068 2024-04-13T23:13:47.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:47.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:47.098 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:47.155 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:47.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:47.214 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:47.214 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:47.214 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:47.215 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:47.215 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:47.215 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:13:47.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:47.216 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:47.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:47.216 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:47.216 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:47.216 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:47.217 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:13:47.217 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:47.217 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:47.217 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:47.218 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:13:47.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:47.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:47.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:47.218 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047633", 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.392463Z" 2024-04-13T23:13:47.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033750", 2024-04-13T23:13:47.220 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.464865Z" 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:47.221 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054068", 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.437863Z" 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:47.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:47.223 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:47.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:13:47.223 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:13:47.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:47.223 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:47Z" 2024-04-13T23:13:47.223 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:47.224 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047633", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:09:31.392463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033750", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:09:31.464865Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054068", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:09:31.437863Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:13:47Z" } 2024-04-13T23:13:47.224 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047633', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.392463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033750', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.464865Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054068', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.437863Z'}}] 2024-04-13T23:13:47.224 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:13:47.224 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:13:47.225 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047774 target=00000000000000000000:00000000000000047633, shard=2 master=00000000000000000000:00000000000000033891 target=00000000000000000000:00000000000000033750, shard=3 master=00000000000000000000:00000000000000054350 target=00000000000000000000:00000000000000054068 2024-04-13T23:13:52.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:52.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:52.226 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:52.283 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:52.283 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:52.359 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047633", 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.392463Z" 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033750", 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.464865Z" 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054068", 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.437863Z" 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:52.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:52.365 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:52.365 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:13:52.365 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:13:52.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:52.365 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:52Z" 2024-04-13T23:13:52.365 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:52.366 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047633", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:09:31.392463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033750", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:09:31.464865Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054068", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:09:31.437863Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:13:52Z" } 2024-04-13T23:13:52.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047633', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.392463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033750', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.464865Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054068', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.437863Z'}}] 2024-04-13T23:13:52.366 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:13:52.366 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:13:52.366 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047774 target=00000000000000000000:00000000000000047633, shard=2 master=00000000000000000000:00000000000000033891 target=00000000000000000000:00000000000000033750, shard=3 master=00000000000000000000:00000000000000054350 target=00000000000000000000:00000000000000054068 2024-04-13T23:13:57.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:57.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:13:57.368 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:13:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:13:57.427 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:13:57.486 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:13:57.486 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:13:57.486 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:13:57.487 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047633", 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:57.488 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.392463Z" 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033750", 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:57.489 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.464865Z" 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054068", 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:13:57.490 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:09:31.437863Z" 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:13:57.491 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:13:57Z" 2024-04-13T23:13:57.492 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:13:57.492 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047633", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:09:31.392463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033750", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:09:31.464865Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054068", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:09:31.437863Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:13:57Z" } 2024-04-13T23:13:57.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047633', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.392463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033750', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.464865Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054068', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:09:31.437863Z'}}] 2024-04-13T23:13:57.493 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:13:57.493 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047633', 2: '00000000000000000000:00000000000000033750', 3: '00000000000000000000:00000000000000054068'} 2024-04-13T23:13:57.493 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047774 target=00000000000000000000:00000000000000047633, shard=2 master=00000000000000000000:00000000000000033891 target=00000000000000000000:00000000000000033750, shard=3 master=00000000000000000000:00000000000000054350 target=00000000000000000000:00000000000000054068 2024-04-13T23:14:02.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:02.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:02.494 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:02.551 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:02.551 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:02.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:02.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:02.616 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:02Z" 2024-04-13T23:14:02.617 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:02.618 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:02Z" } 2024-04-13T23:14:02.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:02.618 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:02.618 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:02.618 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:14:02.619 DEBUG:boto:path=/vwnfns-75/ 2024-04-13T23:14:02.619 DEBUG:boto:auth_path=/vwnfns-75/ 2024-04-13T23:14:02.619 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.619 DEBUG:boto:Path: /vwnfns-75/ 2024-04-13T23:14:02.619 DEBUG:boto:Data: 2024-04-13T23:14:02.619 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.619 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.619 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.620 DEBUG:boto:Params: {} 2024-04-13T23:14:02.620 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:14:02.620 DEBUG:boto:Token: None 2024-04-13T23:14:02.620 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-75/ 2024-04-13T23:14:02.621 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:5x13okFhK9hRnVcln+KU5XHB7t4= 2024-04-13T23:14:02.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5x13okFhK9hRnVcln+KU5XHB7t4=', 'Content-Length': '0'} 2024-04-13T23:14:02.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045fef78c346180dc-00661b11ba-2d6d-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.626 DEBUG:boto:path=/vwnfns-75/obj5 2024-04-13T23:14:02.626 DEBUG:boto:auth_path=/vwnfns-75/obj5 2024-04-13T23:14:02.627 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.627 DEBUG:boto:Path: /vwnfns-75/obj5 2024-04-13T23:14:02.627 DEBUG:boto:Data: 2024-04-13T23:14:02.627 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.627 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.627 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.627 DEBUG:boto:Params: {} 2024-04-13T23:14:02.627 DEBUG:boto:Token: None 2024-04-13T23:14:02.627 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-75/obj5 2024-04-13T23:14:02.628 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:V/BdTGxNNKth8Mi+9fkCQ3G94a8= 2024-04-13T23:14:02.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:V/BdTGxNNKth8Mi+9fkCQ3G94a8=', 'Content-Length': '0'} 2024-04-13T23:14:02.629 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c238cacf3435e48b-00661b11ba-2d6d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.629 DEBUG:boto:path=/vwnfns-75/obj6 2024-04-13T23:14:02.629 DEBUG:boto:auth_path=/vwnfns-75/obj6 2024-04-13T23:14:02.629 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.629 DEBUG:boto:Path: /vwnfns-75/obj6 2024-04-13T23:14:02.629 DEBUG:boto:Data: 2024-04-13T23:14:02.629 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.630 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.630 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.630 DEBUG:boto:Params: {} 2024-04-13T23:14:02.630 DEBUG:boto:Token: None 2024-04-13T23:14:02.630 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-75/obj6 2024-04-13T23:14:02.630 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:PV22toySfTOzH4gyjPGEcI/ia9I= 2024-04-13T23:14:02.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PV22toySfTOzH4gyjPGEcI/ia9I=', 'Content-Length': '0'} 2024-04-13T23:14:02.631 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001a4c1d920d229ed3-00661b11ba-2d6d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.632 DEBUG:boto:path=/vwnfns-75/obj7 2024-04-13T23:14:02.632 DEBUG:boto:auth_path=/vwnfns-75/obj7 2024-04-13T23:14:02.632 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.632 DEBUG:boto:Path: /vwnfns-75/obj7 2024-04-13T23:14:02.632 DEBUG:boto:Data: 2024-04-13T23:14:02.632 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.632 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.632 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.632 DEBUG:boto:Params: {} 2024-04-13T23:14:02.632 DEBUG:boto:Token: None 2024-04-13T23:14:02.632 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-75/obj7 2024-04-13T23:14:02.633 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:CoQyHHb7ol1YpgoaB6OFg+HKyX8= 2024-04-13T23:14:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CoQyHHb7ol1YpgoaB6OFg+HKyX8=', 'Content-Length': '0'} 2024-04-13T23:14:02.634 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000068f9e9d63ae78c2-00661b11ba-2d6d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.634 DEBUG:boto:path=/vwnfns-75/obj8 2024-04-13T23:14:02.634 DEBUG:boto:auth_path=/vwnfns-75/obj8 2024-04-13T23:14:02.634 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.634 DEBUG:boto:Path: /vwnfns-75/obj8 2024-04-13T23:14:02.635 DEBUG:boto:Data: 2024-04-13T23:14:02.635 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.635 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.635 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.635 DEBUG:boto:Params: {} 2024-04-13T23:14:02.635 DEBUG:boto:Token: None 2024-04-13T23:14:02.635 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-75/obj8 2024-04-13T23:14:02.635 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:hF7oNGey8xPKXMGXSBLRly0Wqvs= 2024-04-13T23:14:02.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hF7oNGey8xPKXMGXSBLRly0Wqvs=', 'Content-Length': '0'} 2024-04-13T23:14:02.637 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000097fd15e6f15ff137-00661b11ba-2d6d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.637 DEBUG:boto:path=/vwnfns-74/ 2024-04-13T23:14:02.637 DEBUG:boto:auth_path=/vwnfns-74/ 2024-04-13T23:14:02.637 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.637 DEBUG:boto:Path: /vwnfns-74/ 2024-04-13T23:14:02.637 DEBUG:boto:Data: 2024-04-13T23:14:02.638 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.638 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.638 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.638 DEBUG:boto:Params: {} 2024-04-13T23:14:02.638 DEBUG:boto:Token: None 2024-04-13T23:14:02.638 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/ 2024-04-13T23:14:02.638 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ATT7uwak6Sb0EAh0ZyjzTvQs4rc= 2024-04-13T23:14:02.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ATT7uwak6Sb0EAh0ZyjzTvQs4rc=', 'Content-Length': '0'} 2024-04-13T23:14:02.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac8fd804fd21bea3-00661b11ba-2d6d-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.640 DEBUG:boto:path=/vwnfns-74/obj7 2024-04-13T23:14:02.641 DEBUG:boto:auth_path=/vwnfns-74/obj7 2024-04-13T23:14:02.641 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.641 DEBUG:boto:Path: /vwnfns-74/obj7 2024-04-13T23:14:02.641 DEBUG:boto:Data: 2024-04-13T23:14:02.641 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.641 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.641 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.641 DEBUG:boto:Params: {} 2024-04-13T23:14:02.641 DEBUG:boto:Token: None 2024-04-13T23:14:02.641 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj7 2024-04-13T23:14:02.642 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oUV1nk9d7P8nY2qUEBm1unAL9Kw= 2024-04-13T23:14:02.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oUV1nk9d7P8nY2qUEBm1unAL9Kw=', 'Content-Length': '0'} 2024-04-13T23:14:02.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c05ea69c0ab760a-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.644 DEBUG:boto:path=/vwnfns-74/obj7 2024-04-13T23:14:02.644 DEBUG:boto:auth_path=/vwnfns-74/obj7 2024-04-13T23:14:02.644 DEBUG:boto:Method: GET 2024-04-13T23:14:02.644 DEBUG:boto:Path: /vwnfns-74/obj7 2024-04-13T23:14:02.644 DEBUG:boto:Data: 2024-04-13T23:14:02.644 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.644 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.644 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.644 DEBUG:boto:Params: {} 2024-04-13T23:14:02.645 DEBUG:boto:Token: None 2024-04-13T23:14:02.645 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj7 2024-04-13T23:14:02.645 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0Ga+1sbpdUkjNHhk12L3dSC5O9c= 2024-04-13T23:14:02.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0Ga+1sbpdUkjNHhk12L3dSC5O9c=', 'Content-Length': '0'} 2024-04-13T23:14:02.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa54e527cdd18879-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.647 DEBUG:boto:path=/vwnfns-74/obj8 2024-04-13T23:14:02.647 DEBUG:boto:auth_path=/vwnfns-74/obj8 2024-04-13T23:14:02.647 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.647 DEBUG:boto:Path: /vwnfns-74/obj8 2024-04-13T23:14:02.647 DEBUG:boto:Data: 2024-04-13T23:14:02.648 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.648 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.648 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.648 DEBUG:boto:Params: {} 2024-04-13T23:14:02.648 DEBUG:boto:Token: None 2024-04-13T23:14:02.648 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj8 2024-04-13T23:14:02.648 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1HbBVfsQUs+21ERRNKxa4J7YbYY= 2024-04-13T23:14:02.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1HbBVfsQUs+21ERRNKxa4J7YbYY=', 'Content-Length': '0'} 2024-04-13T23:14:02.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024875de95430091d-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.650 DEBUG:boto:path=/vwnfns-74/obj8 2024-04-13T23:14:02.650 DEBUG:boto:auth_path=/vwnfns-74/obj8 2024-04-13T23:14:02.650 DEBUG:boto:Method: GET 2024-04-13T23:14:02.650 DEBUG:boto:Path: /vwnfns-74/obj8 2024-04-13T23:14:02.651 DEBUG:boto:Data: 2024-04-13T23:14:02.651 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.651 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.651 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.651 DEBUG:boto:Params: {} 2024-04-13T23:14:02.651 DEBUG:boto:Token: None 2024-04-13T23:14:02.651 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj8 2024-04-13T23:14:02.651 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qsN8M2woeGeapLjqe+Iz6EqOkf0= 2024-04-13T23:14:02.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qsN8M2woeGeapLjqe+Iz6EqOkf0=', 'Content-Length': '0'} 2024-04-13T23:14:02.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007f638a4cc28d6dd-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.653 DEBUG:boto:path=/vwnfns-74/obj7 2024-04-13T23:14:02.654 DEBUG:boto:auth_path=/vwnfns-74/obj7 2024-04-13T23:14:02.654 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.654 DEBUG:boto:Path: /vwnfns-74/obj7 2024-04-13T23:14:02.654 DEBUG:boto:Data: 2024-04-13T23:14:02.654 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.654 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.654 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.654 DEBUG:boto:Params: {} 2024-04-13T23:14:02.654 DEBUG:boto:Token: None 2024-04-13T23:14:02.654 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj7 2024-04-13T23:14:02.655 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:oUV1nk9d7P8nY2qUEBm1unAL9Kw= 2024-04-13T23:14:02.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oUV1nk9d7P8nY2qUEBm1unAL9Kw=', 'Content-Length': '0'} 2024-04-13T23:14:02.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d44ee719bf35dc1-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.656 DEBUG:boto:path=/vwnfns-74/obj7 2024-04-13T23:14:02.656 DEBUG:boto:auth_path=/vwnfns-74/obj7 2024-04-13T23:14:02.657 DEBUG:boto:Method: GET 2024-04-13T23:14:02.657 DEBUG:boto:Path: /vwnfns-74/obj7 2024-04-13T23:14:02.657 DEBUG:boto:Data: 2024-04-13T23:14:02.657 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.657 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.657 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.657 DEBUG:boto:Params: {} 2024-04-13T23:14:02.657 DEBUG:boto:Token: None 2024-04-13T23:14:02.657 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj7 2024-04-13T23:14:02.657 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:0Ga+1sbpdUkjNHhk12L3dSC5O9c= 2024-04-13T23:14:02.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0Ga+1sbpdUkjNHhk12L3dSC5O9c=', 'Content-Length': '0'} 2024-04-13T23:14:02.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000541e7f8eec186937-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.659 DEBUG:boto:path=/vwnfns-74/obj8 2024-04-13T23:14:02.659 DEBUG:boto:auth_path=/vwnfns-74/obj8 2024-04-13T23:14:02.659 DEBUG:boto:Method: HEAD 2024-04-13T23:14:02.660 DEBUG:boto:Path: /vwnfns-74/obj8 2024-04-13T23:14:02.660 DEBUG:boto:Data: 2024-04-13T23:14:02.660 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.660 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.660 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.660 DEBUG:boto:Params: {} 2024-04-13T23:14:02.660 DEBUG:boto:Token: None 2024-04-13T23:14:02.660 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj8 2024-04-13T23:14:02.660 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:1HbBVfsQUs+21ERRNKxa4J7YbYY= 2024-04-13T23:14:02.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1HbBVfsQUs+21ERRNKxa4J7YbYY=', 'Content-Length': '0'} 2024-04-13T23:14:02.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000708f2518c95e3fed-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.662 DEBUG:boto:path=/vwnfns-74/obj8 2024-04-13T23:14:02.662 DEBUG:boto:auth_path=/vwnfns-74/obj8 2024-04-13T23:14:02.663 DEBUG:boto:Method: GET 2024-04-13T23:14:02.663 DEBUG:boto:Path: /vwnfns-74/obj8 2024-04-13T23:14:02.663 DEBUG:boto:Data: 2024-04-13T23:14:02.663 DEBUG:boto:Headers: {} 2024-04-13T23:14:02.663 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:14:02.663 DEBUG:boto:Port: 8001 2024-04-13T23:14:02.663 DEBUG:boto:Params: {} 2024-04-13T23:14:02.663 DEBUG:boto:Token: None 2024-04-13T23:14:02.663 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:14:02 GMT /vwnfns-74/obj8 2024-04-13T23:14:02.664 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:qsN8M2woeGeapLjqe+Iz6EqOkf0= 2024-04-13T23:14:02.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:02 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qsN8M2woeGeapLjqe+Iz6EqOkf0=', 'Content-Length': '0'} 2024-04-13T23:14:02.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:13:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-75:f6f208d6-f791-479b-bc3d-112512eec2d4.10791.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d68778af55ee1dc5-00661b11ba-2d6d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:14:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:02.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:02.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:02.666 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket vwnfns-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:02.713 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:02.714 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:02.777 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:14:02.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:02.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:02.778 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:02.824 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:02.824 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:02.880 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:14:02.881 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2024-04-13T23:14:02.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:02.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:02.883 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:02.972 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:14:02.987 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050042.750550_1246.1", 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:14:02.750550Z" 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:02.988 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:14:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:14:02.989 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:02.989 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:14:02.990 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:02.990 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:14:02.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:02.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:02.991 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:03.039 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:03.040 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:03.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:03.087+0000 7f6b937cf780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:03.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:03.088+0000 7f6b937cf780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:03.105 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:03.105 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:11:38.190756Z", 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:03.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:03.108 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:03.109 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:03.110 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:03Z" 2024-04-13T23:14:03.111 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:03.112 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049898.190756_1244.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:11:38.190756Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:03Z" } 2024-04-13T23:14:03.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049898.190756_1244.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:11:38.190756Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:03.113 DEBUG:rgw_multi.tests:log_status={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:03.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:03.113 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050042.750550_1246.1 target=1_1713049898.190756_1244.1 2024-04-13T23:14:08.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:08.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:08.114 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:08.169 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:08.170 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:08.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:08.200+0000 7fe9c4b7a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:08.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:08.201+0000 7fe9c4b7a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:08.218 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:11:38.190756Z", 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:08.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:08.220 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:08.221 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:08.222 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:08.224 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:08.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:08.224 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:08Z" 2024-04-13T23:14:08.224 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:08.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049898.190756_1244.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:11:38.190756Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:08Z" } 2024-04-13T23:14:08.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049898.190756_1244.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:11:38.190756Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:08.225 DEBUG:rgw_multi.tests:log_status={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:08.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:08.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050042.750550_1246.1 target=1_1713049898.190756_1244.1 2024-04-13T23:14:13.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:13.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:13.227 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:13.282 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:13.282 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:13.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:13.323+0000 7f93c0d16780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:13.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:13.324+0000 7f93c0d16780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:13.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:13.344 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:11:38.190756Z", 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:13.345 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:13.346 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:13.347 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:13.348 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:13.349 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:13.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:13.349 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:13Z" 2024-04-13T23:14:13.349 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:13.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049898.190756_1244.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:11:38.190756Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:13Z" } 2024-04-13T23:14:13.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049898.190756_1244.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:11:38.190756Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:13.350 DEBUG:rgw_multi.tests:log_status={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:13.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:13.351 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050042.750550_1246.1 target=1_1713049898.190756_1244.1 2024-04-13T23:14:18.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:18.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:18.352 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:18.407 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:18.407 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:18.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:18.453+0000 7f35b0085780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:18.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:18.454+0000 7f35b0085780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713049898.190756_1244.1", 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:11:38.190756Z", 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:18.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:18.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:18.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:18.474 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:18Z" 2024-04-13T23:14:18.475 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:18.476 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713049898.190756_1244.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:11:38.190756Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:18Z" } 2024-04-13T23:14:18.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049898.190756_1244.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:11:38.190756Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:18.476 DEBUG:rgw_multi.tests:log_status={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:18.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1713049898.190756_1244.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:18.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050042.750550_1246.1 target=1_1713049898.190756_1244.1 2024-04-13T23:14:23.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:23.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:23.478 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:23.534 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:23.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:23.582+0000 7f89ad684780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:23.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:23.583+0000 7f89ad684780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:23.605 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050042.750550_1246.1", 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:23.606 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:02.750550Z", 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.607 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:23.608 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.609 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:23.610 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:23.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.611 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:23Z" 2024-04-13T23:14:23.611 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:23.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050042.750550_1246.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:02.750550Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:23Z" } 2024-04-13T23:14:23.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050042.750550_1246.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:02.750550Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:23.612 DEBUG:rgw_multi.tests:log_status={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:23.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050042.750550_1246.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:23.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:14:23.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:23.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:23.613 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:23.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:23.658 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:23.706 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[]} 2024-04-13T23:14:23.709 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-76 2024-04-13T23:14:23.709 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:14:23.710 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:14:23.710 DEBUG:boto:Method: PUT 2024-04-13T23:14:23.710 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:14:23.710 DEBUG:boto:Data: 2024-04-13T23:14:23.710 DEBUG:boto:Headers: {} 2024-04-13T23:14:23.710 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:14:23.710 DEBUG:boto:Port: 8000 2024-04-13T23:14:23.710 DEBUG:boto:Params: {} 2024-04-13T23:14:23.711 DEBUG:boto:Token: None 2024-04-13T23:14:23.711 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:14:23 GMT /vwnfns-76/ 2024-04-13T23:14:23.711 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:uUWbzsHtnBY2if5xEPCmYeoy020= 2024-04-13T23:14:23.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uUWbzsHtnBY2if5xEPCmYeoy020=', 'Content-Length': '0'} 2024-04-13T23:14:23.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000003bb2e37dcc20804-00661b11cf-2d6b-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:14:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:23.734 INFO:rgw_multi.tests:create bucket zone=a1 name=vwnfns-77 2024-04-13T23:14:23.734 DEBUG:boto:path=/vwnfns-77/ 2024-04-13T23:14:23.734 DEBUG:boto:auth_path=/vwnfns-77/ 2024-04-13T23:14:23.735 DEBUG:boto:Method: PUT 2024-04-13T23:14:23.735 DEBUG:boto:Path: /vwnfns-77/ 2024-04-13T23:14:23.735 DEBUG:boto:Data: 2024-04-13T23:14:23.735 DEBUG:boto:Headers: {} 2024-04-13T23:14:23.735 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:14:23.735 DEBUG:boto:Port: 8000 2024-04-13T23:14:23.735 DEBUG:boto:Params: {} 2024-04-13T23:14:23.735 DEBUG:boto:Token: None 2024-04-13T23:14:23.735 DEBUG:boto:StringToSign: PUT Sat, 13 Apr 2024 23:14:23 GMT /vwnfns-77/ 2024-04-13T23:14:23.736 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ZjrQzgCw6m4cD6+2Cw9vOrYPorw= 2024-04-13T23:14:23.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZjrQzgCw6m4cD6+2Cw9vOrYPorw=', 'Content-Length': '0'} 2024-04-13T23:14:23.754 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014eb954f99812e96-00661b11cf-2d6b-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:14:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:23.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:23.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:23.755 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:23.802 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:23.802 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:23.851 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:14:23.881 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:14:23.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:14:23.751827Z" 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.882 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.883 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:14:23.884 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:23.884 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:14:23.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:23.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:23.885 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:23.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:23.935 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:23.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:23.973+0000 7f45e434c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:23.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:23.973+0000 7f45e434c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:23.995 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:23.995 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:23.995 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:23.995 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:23.995 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:23.995 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.722919_1247.1", 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:23.996 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.722919Z", 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.997 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.998 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:23.999 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:24.000 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:24.001 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:24.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:24.001 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:23Z" 2024-04-13T23:14:24.001 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:24.002 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.722919_1247.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.722919Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:23Z" } 2024-04-13T23:14:24.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.722919_1247.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.722919Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:24.002 DEBUG:rgw_multi.tests:log_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:24.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.722919_1247.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:24.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050063.751827_1254.1 target=1_1713050063.722919_1247.1 2024-04-13T23:14:29.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:29.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:29.004 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:29.060 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.061 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:29.100+0000 7ff0a4d34780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:29.101+0000 7ff0a4d34780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:29.123 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:29.123 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:29.123 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:29.123 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:29.123 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:29.124 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:29.125 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:29.126 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:29.127 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.128 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:29Z" 2024-04-13T23:14:29.129 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:29.130 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:29Z" } 2024-04-13T23:14:29.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:29.131 DEBUG:rgw_multi.tests:log_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:29.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:29.131 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:14:29.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.131 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.250 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:14:29.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.251 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.301 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.302 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.353 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:14:29.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-76', '--dest-bucket=vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-76', '--dest-bucket=vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.354 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-vwnfns-76 --source-zones=a1 --dest-zones=a2 --source-bucket=vwnfns-76 --dest-bucket=vwnfns-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.407 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.407 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.468 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:14:29.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-77', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-76', '--dest-bucket=vwnfns-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-vwnfns-77', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-76', '--dest-bucket=vwnfns-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.469 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-vwnfns-77 --source-zones=a1 --dest-zones=a2 --source-bucket=vwnfns-76 --dest-bucket=vwnfns-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.522 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.522 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.600 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:14:29.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-76', '--dest-bucket=vwnfns-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=vwnfns-76', '--dest-bucket=vwnfns-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.601 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=vwnfns-76 --dest-bucket=vwnfns-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.653 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.654 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.720 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:14:29.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.721 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.768 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.769 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.837 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:14:29.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.838 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:29.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:29.885 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:29.940 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 21, 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:14:29.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:14:29.942 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:14:29.943 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:14:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:14:29.945 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:14:29.946 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:14:29.947 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe-vwnfns-76", 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:14:29.950 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.951 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe-vwnfns-77", 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:14:29.952 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-77", 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:14:29.953 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-77", 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.955 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:14:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "enabled" 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:14:29.957 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:14:29.958 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:14:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:14:29.959 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:14:29.960 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:14:29.962 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:14:29.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:14:29.964 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:14:29.966 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:14:29.967 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:14:29.968 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.969 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:29.970 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:29.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:14:29.971 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:14:30.016 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:30.016 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:30.077 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:14:30.078 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:14:30.078 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:14:30.078 DEBUG:boto:Method: HEAD 2024-04-13T23:14:30.078 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:14:30.078 DEBUG:boto:Data: 2024-04-13T23:14:30.079 DEBUG:boto:Headers: {} 2024-04-13T23:14:30.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:14:30.079 DEBUG:boto:Port: 8000 2024-04-13T23:14:30.079 DEBUG:boto:Params: {} 2024-04-13T23:14:30.079 DEBUG:boto:Token: None 2024-04-13T23:14:30.079 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:30 GMT /vwnfns-76/ 2024-04-13T23:14:30.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iBOTwvtsIjpA9N+TVk1E3YCrdGY= 2024-04-13T23:14:30.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iBOTwvtsIjpA9N+TVk1E3YCrdGY=', 'Content-Length': '0'} 2024-04-13T23:14:30.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073c453936d866220-00661b11d6-2f5a-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:30.309 DEBUG:boto:path=/vwnfns-76/obj1 2024-04-13T23:14:30.309 DEBUG:boto:auth_path=/vwnfns-76/obj1 2024-04-13T23:14:30.309 DEBUG:boto:Method: PUT 2024-04-13T23:14:30.309 DEBUG:boto:Path: /vwnfns-76/obj1 2024-04-13T23:14:30.309 DEBUG:boto:Data: 2024-04-13T23:14:30.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:14:30.309 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:14:30.309 DEBUG:boto:Port: 8000 2024-04-13T23:14:30.309 DEBUG:boto:Params: {} 2024-04-13T23:14:30.310 DEBUG:boto:Token: None 2024-04-13T23:14:30.310 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:14:30 GMT /vwnfns-76/obj1 2024-04-13T23:14:30.310 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ch4lsYIPVW//2nnPvVTeOffrpQE= 2024-04-13T23:14:30.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ch4lsYIPVW//2nnPvVTeOffrpQE='} 2024-04-13T23:14:30.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a279813aa442fe41-00661b11d6-2f5a-a1'), ('Date', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:30.333 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:14:30.333 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:14:30.333 DEBUG:boto:Method: HEAD 2024-04-13T23:14:30.333 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:14:30.333 DEBUG:boto:Data: 2024-04-13T23:14:30.333 DEBUG:boto:Headers: {} 2024-04-13T23:14:30.334 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:14:30.334 DEBUG:boto:Port: 8000 2024-04-13T23:14:30.334 DEBUG:boto:Params: {} 2024-04-13T23:14:30.334 DEBUG:boto:Token: None 2024-04-13T23:14:30.334 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:14:30 GMT /vwnfns-76/ 2024-04-13T23:14:30.334 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:iBOTwvtsIjpA9N+TVk1E3YCrdGY= 2024-04-13T23:14:30.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iBOTwvtsIjpA9N+TVk1E3YCrdGY=', 'Content-Length': '0'} 2024-04-13T23:14:30.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae3b3118a73e7a7c-00661b11d6-2f5a-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:30.341 DEBUG:boto:path=/vwnfns-76/obj2 2024-04-13T23:14:30.341 DEBUG:boto:auth_path=/vwnfns-76/obj2 2024-04-13T23:14:30.341 DEBUG:boto:Method: PUT 2024-04-13T23:14:30.341 DEBUG:boto:Path: /vwnfns-76/obj2 2024-04-13T23:14:30.341 DEBUG:boto:Data: 2024-04-13T23:14:30.341 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:14:30.341 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:14:30.341 DEBUG:boto:Port: 8000 2024-04-13T23:14:30.342 DEBUG:boto:Params: {} 2024-04-13T23:14:30.342 DEBUG:boto:Token: None 2024-04-13T23:14:30.342 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:14:30 GMT /vwnfns-76/obj2 2024-04-13T23:14:30.342 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:Iibc7RAZnJlXb2Af+7pgUi2E6No= 2024-04-13T23:14:30.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Iibc7RAZnJlXb2Af+7pgUi2E6No='} 2024-04-13T23:14:30.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091f7f4a713e08e8a-00661b11d6-2f5a-a1'), ('Date', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:14:30.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.367 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:30.434 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:30.435 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:30.537 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:14:30.581 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:14:30.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.581 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:14:30.581 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:14:23.751827Z" 2024-04-13T23:14:30.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.582 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:30.583 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:14:30.583 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.583 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:14:30.584 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:30.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:14:30.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.584 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:30.653 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:30.653 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:30.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:30.743+0000 7fb02a2d5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:30.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:14:30.744+0000 7fb02a2d5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:14:30.791 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:30.792 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:30.793 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.794 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:30.795 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:30.796 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:14:30.797 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:14:30.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.798 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:30Z" 2024-04-13T23:14:30.798 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:30.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:14:30Z" } 2024-04-13T23:14:30.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:14:30.800 DEBUG:rgw_multi.tests:log_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:30.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:14:30.800 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:14:30.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.801 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:30.855 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:30.855 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:30.977 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:14:30.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.977 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:30.977 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:30.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:30.978 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:14:30.979 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:14:30.979 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:30.979 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:14:30.980 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:30.980 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:14:30.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:30.980 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:31.038 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:31.039 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:31.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:31.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:31.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:31.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:31.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:31Z" 2024-04-13T23:14:31.128 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:31.129 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:31Z" } 2024-04-13T23:14:31.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:31.130 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:31.130 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:31.130 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:14:36.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:36.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:36.131 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:36.190 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:36.190 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:36.252 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:36.253 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:36.254 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:36.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:36.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:36.257 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:36Z" 2024-04-13T23:14:36.258 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:36.259 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:36Z" } 2024-04-13T23:14:36.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:36.259 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:36.259 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:36.260 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:14:41.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:41.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:41.261 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:41.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:41.319 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:41.384 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:41.384 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:41.384 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:41.385 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:41.386 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:41.387 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:41.388 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:41.389 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:41Z" 2024-04-13T23:14:41.390 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:41.391 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:41Z" } 2024-04-13T23:14:41.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:41.391 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:41.391 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:41.391 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:14:46.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:46.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:46.392 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:46.450 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:46.450 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:46.535 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:46.535 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:46.535 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:46.536 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:46.537 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:46.538 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:46.539 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:46.540 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:46.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:46.541 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:46Z" 2024-04-13T23:14:46.541 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:46.542 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:46Z" } 2024-04-13T23:14:46.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:46.542 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:46.542 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:46.542 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:14:51.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:51.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:51.544 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:51.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:51.602 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:51.664 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:51.665 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:51.666 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:51.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:51.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:51.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:51.670 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:51Z" 2024-04-13T23:14:51.670 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:51.671 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:51Z" } 2024-04-13T23:14:51.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:51.671 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:51.672 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:51.672 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:14:56.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:56.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:14:56.673 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:14:56.731 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:14:56.731 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:14:56.814 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:14:56.814 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:14:56.814 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:14:56.814 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:14:56.814 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:56.815 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:56.816 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:14:56.817 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:14:56.818 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:14:56.819 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:14:56.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:14:56.820 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:14:56Z" 2024-04-13T23:14:56.820 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:14:56.821 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:14:56Z" } 2024-04-13T23:14:56.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:14:56.821 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:14:56.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:14:56.821 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:01.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:01.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:01.823 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:01.881 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:01.939 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:01.939 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:01.939 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:01.940 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:01.941 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:01.942 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:01.943 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:01.944 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:01.945 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:01.945 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:01.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:01.945 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:01Z" 2024-04-13T23:15:01.945 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:01.946 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:01Z" } 2024-04-13T23:15:01.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:01.946 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:01.946 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:01.946 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:06.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:06.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:06.948 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:07.005 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:07.005 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:07.079 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:07.079 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:07.079 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:07.079 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:07.079 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:07.079 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:07.080 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:07.081 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:07.082 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:07.083 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:07Z" 2024-04-13T23:15:07.084 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:07.085 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:07Z" } 2024-04-13T23:15:07.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:07.086 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:07.086 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:07.086 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:12.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:12.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:12.087 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:12.144 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:12.144 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:12.207 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:12.207 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:12.207 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:12.207 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:12.207 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:12.207 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:12.208 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:12.209 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:12.210 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:12.211 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:12.212 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:12.213 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:12.213 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:12.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:12.213 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:12Z" 2024-04-13T23:15:12.213 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:12.214 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:12Z" } 2024-04-13T23:15:12.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:12.214 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:12.214 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:12.215 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:17.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:17.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:17.216 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:17.275 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:17.275 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:17.332 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:17.332 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:17.332 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:17.332 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:17.332 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:17.333 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:17.334 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:17.335 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:17.335 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:17.335 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:17.335 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:17.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:17.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:17.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:17.337 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:17Z" 2024-04-13T23:15:17.338 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:17.339 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:17Z" } 2024-04-13T23:15:17.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:17.340 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:17.340 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:17.340 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:22.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:22.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:22.342 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:22.399 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:22.400 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:22.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:22.477 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:22.478 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:22.479 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:22.480 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:22Z" 2024-04-13T23:15:22.481 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:22.482 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:22Z" } 2024-04-13T23:15:22.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:22.483 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:22.483 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:22.483 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:27.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:27.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:27.485 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:27.543 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:27.543 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:27.609 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:27.609 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:27.609 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:27.610 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:27.611 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:27.612 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:27.613 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:27.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:27.615 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:27.615 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:27.615 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:27.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:27.615 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:27Z" 2024-04-13T23:15:27.615 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:27.616 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:27Z" } 2024-04-13T23:15:27.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:27.617 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:27.617 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:27.617 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:32.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:32.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:32.619 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:32.676 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:32.676 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:33.738 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:33.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:33.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:33.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:33.742 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:33Z" 2024-04-13T23:15:33.743 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:33.744 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:33Z" } 2024-04-13T23:15:33.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:33.745 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:33.745 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:33.745 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:38.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:38.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:38.747 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:38.804 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:38.805 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:38.862 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:38.862 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:38.863 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:38.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:38.865 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:38.866 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:38.867 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:38.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:38.869 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:38Z" 2024-04-13T23:15:38.869 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:38.870 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:38Z" } 2024-04-13T23:15:38.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:38.870 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:38.870 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:38.870 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:43.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:43.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:43.871 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:43.984 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:43.984 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:43.984 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:43.984 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:43.986 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:43.987 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:43.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:43.989 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:43Z" 2024-04-13T23:15:43.990 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:43.991 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:43Z" } 2024-04-13T23:15:43.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:43.992 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:43.992 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:43.992 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:48.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:48.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:48.993 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:49.050 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:49.050 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:49.111 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:49.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:49.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:49.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:49Z" 2024-04-13T23:15:49.117 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:49.118 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:49Z" } 2024-04-13T23:15:49.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:49.118 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:49.118 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:49.118 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:54.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:54.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:54.120 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:54.177 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:54.177 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:54.233 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:54.234 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:54.235 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:54.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:54.237 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:54Z" 2024-04-13T23:15:54.238 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:54.242 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:54Z" } 2024-04-13T23:15:54.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:54.243 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:54.243 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:54.243 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:15:59.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:59.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:15:59.244 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:15:59.301 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:15:59.301 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:15:59.355 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047774", 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.571210Z" 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.582662Z" 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:15:59Z" 2024-04-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:15:59.362 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:13:41.571210Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:13:41.582662Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:15:59Z" } 2024-04-13T23:15:59.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.571210Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.582662Z'}}] 2024-04-13T23:15:59.363 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:15:59.363 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047774', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054350'} 2024-04-13T23:15:59.363 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000047915 target=00000000000000000000:00000000000000047774, shard=3 master=00000000000000000000:00000000000000054491 target=00000000000000000000:00000000000000054350 2024-04-13T23:16:04.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:04.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:04.364 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:04.421 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:04.421 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:04.476 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:04.476 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:04.477 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:04.478 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:04.479 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:04.480 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:16:04.481 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:16:04.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:04.482 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:04Z" 2024-04-13T23:16:04.482 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:04.483 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:16:04Z" } 2024-04-13T23:16:04.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:16:04.483 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:04.483 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:04.483 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:16:04.484 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:16:04.484 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:16:04.484 DEBUG:boto:Method: HEAD 2024-04-13T23:16:04.484 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:16:04.484 DEBUG:boto:Data: 2024-04-13T23:16:04.484 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.484 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.484 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.484 DEBUG:boto:Params: {} 2024-04-13T23:16:04.485 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:16:04.485 DEBUG:boto:Token: None 2024-04-13T23:16:04.485 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-76/ 2024-04-13T23:16:04.486 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:O2IgMzbQKfgRL5GAu4XDS7EmQmQ= 2024-04-13T23:16:04.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O2IgMzbQKfgRL5GAu4XDS7EmQmQ=', 'Content-Length': '0'} 2024-04-13T23:16:04.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008671d3a41c366eda-00661b1234-2f5c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.492 DEBUG:boto:path=/vwnfns-76/obj1 2024-04-13T23:16:04.492 DEBUG:boto:auth_path=/vwnfns-76/obj1 2024-04-13T23:16:04.493 DEBUG:boto:Method: HEAD 2024-04-13T23:16:04.493 DEBUG:boto:Path: /vwnfns-76/obj1 2024-04-13T23:16:04.493 DEBUG:boto:Data: 2024-04-13T23:16:04.493 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.493 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.493 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.493 DEBUG:boto:Params: {} 2024-04-13T23:16:04.493 DEBUG:boto:Token: None 2024-04-13T23:16:04.493 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-76/obj1 2024-04-13T23:16:04.493 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:zpG3dmISnRww1UfCsHg4NaWgH7M= 2024-04-13T23:16:04.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zpG3dmISnRww1UfCsHg4NaWgH7M=', 'Content-Length': '0'} 2024-04-13T23:16:04.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a776e867414e9bbc-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.498 DEBUG:boto:path=/vwnfns-76/obj1 2024-04-13T23:16:04.498 DEBUG:boto:auth_path=/vwnfns-76/obj1 2024-04-13T23:16:04.498 DEBUG:boto:Method: GET 2024-04-13T23:16:04.498 DEBUG:boto:Path: /vwnfns-76/obj1 2024-04-13T23:16:04.498 DEBUG:boto:Data: 2024-04-13T23:16:04.498 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.498 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.499 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.499 DEBUG:boto:Params: {} 2024-04-13T23:16:04.499 DEBUG:boto:Token: None 2024-04-13T23:16:04.499 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-76/obj1 2024-04-13T23:16:04.499 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:7fvPAK1ODVrmB7o6j0e4xWAR6JI= 2024-04-13T23:16:04.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7fvPAK1ODVrmB7o6j0e4xWAR6JI=', 'Content-Length': '0'} 2024-04-13T23:16:04.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f44ed164f3f302ed-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.501 DEBUG:boto:path=/vwnfns-76/obj2 2024-04-13T23:16:04.501 DEBUG:boto:auth_path=/vwnfns-76/obj2 2024-04-13T23:16:04.502 DEBUG:boto:Method: HEAD 2024-04-13T23:16:04.502 DEBUG:boto:Path: /vwnfns-76/obj2 2024-04-13T23:16:04.502 DEBUG:boto:Data: 2024-04-13T23:16:04.502 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.502 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.502 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.502 DEBUG:boto:Params: {} 2024-04-13T23:16:04.502 DEBUG:boto:Token: None 2024-04-13T23:16:04.502 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-76/obj2 2024-04-13T23:16:04.503 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:jFuLMbq5jJzV9DZnnxRoaovl/Tw= 2024-04-13T23:16:04.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jFuLMbq5jJzV9DZnnxRoaovl/Tw=', 'Content-Length': '0'} 2024-04-13T23:16:04.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a50737c665a327b2-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.504 DEBUG:boto:path=/vwnfns-76/obj2 2024-04-13T23:16:04.504 DEBUG:boto:auth_path=/vwnfns-76/obj2 2024-04-13T23:16:04.504 DEBUG:boto:Method: GET 2024-04-13T23:16:04.504 DEBUG:boto:Path: /vwnfns-76/obj2 2024-04-13T23:16:04.505 DEBUG:boto:Data: 2024-04-13T23:16:04.505 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.505 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.505 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.505 DEBUG:boto:Params: {} 2024-04-13T23:16:04.505 DEBUG:boto:Token: None 2024-04-13T23:16:04.505 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-76/obj2 2024-04-13T23:16:04.505 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:OCTuhVeEMpEIQ2RxIvmP9v+vbJ0= 2024-04-13T23:16:04.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OCTuhVeEMpEIQ2RxIvmP9v+vbJ0=', 'Content-Length': '0'} 2024-04-13T23:16:04.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e07499408c2b34a-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.507 DEBUG:boto:path=/vwnfns-77/ 2024-04-13T23:16:04.507 DEBUG:boto:auth_path=/vwnfns-77/ 2024-04-13T23:16:04.507 DEBUG:boto:Method: HEAD 2024-04-13T23:16:04.508 DEBUG:boto:Path: /vwnfns-77/ 2024-04-13T23:16:04.508 DEBUG:boto:Data: 2024-04-13T23:16:04.508 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.508 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.508 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.508 DEBUG:boto:Params: {} 2024-04-13T23:16:04.508 DEBUG:boto:Token: None 2024-04-13T23:16:04.508 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-77/ 2024-04-13T23:16:04.508 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:l7Zm4D9b5NqqkTmZqSS6zgDzc1o= 2024-04-13T23:16:04.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:l7Zm4D9b5NqqkTmZqSS6zgDzc1o=', 'Content-Length': '0'} 2024-04-13T23:16:04.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003836e818d8a49f2f-00661b1234-2f5c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.511 DEBUG:boto:path=/vwnfns-77/obj1 2024-04-13T23:16:04.511 DEBUG:boto:auth_path=/vwnfns-77/obj1 2024-04-13T23:16:04.511 DEBUG:boto:Method: HEAD 2024-04-13T23:16:04.511 DEBUG:boto:Path: /vwnfns-77/obj1 2024-04-13T23:16:04.511 DEBUG:boto:Data: 2024-04-13T23:16:04.511 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.511 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.512 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.512 DEBUG:boto:Params: {} 2024-04-13T23:16:04.512 DEBUG:boto:Token: None 2024-04-13T23:16:04.512 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-77/obj1 2024-04-13T23:16:04.512 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:grZRrjGjjjtNaga87WVZC7Q5kJE= 2024-04-13T23:16:04.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:grZRrjGjjjtNaga87WVZC7Q5kJE=', 'Content-Length': '0'} 2024-04-13T23:16:04.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095c549f0bfe017d5-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.514 DEBUG:boto:path=/vwnfns-77/obj1 2024-04-13T23:16:04.514 DEBUG:boto:auth_path=/vwnfns-77/obj1 2024-04-13T23:16:04.514 DEBUG:boto:Method: GET 2024-04-13T23:16:04.514 DEBUG:boto:Path: /vwnfns-77/obj1 2024-04-13T23:16:04.514 DEBUG:boto:Data: 2024-04-13T23:16:04.514 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.514 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.515 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.515 DEBUG:boto:Params: {} 2024-04-13T23:16:04.515 DEBUG:boto:Token: None 2024-04-13T23:16:04.515 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-77/obj1 2024-04-13T23:16:04.515 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:YdRtdwv0IkjURRjoO5JGIMeS6ko= 2024-04-13T23:16:04.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YdRtdwv0IkjURRjoO5JGIMeS6ko=', 'Content-Length': '0'} 2024-04-13T23:16:04.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069cd5f272d102b0a-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.562 DEBUG:boto:path=/vwnfns-77/obj2 2024-04-13T23:16:04.563 DEBUG:boto:auth_path=/vwnfns-77/obj2 2024-04-13T23:16:04.563 DEBUG:boto:Method: HEAD 2024-04-13T23:16:04.563 DEBUG:boto:Path: /vwnfns-77/obj2 2024-04-13T23:16:04.563 DEBUG:boto:Data: 2024-04-13T23:16:04.563 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.563 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.563 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.563 DEBUG:boto:Params: {} 2024-04-13T23:16:04.563 DEBUG:boto:Token: None 2024-04-13T23:16:04.564 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-77/obj2 2024-04-13T23:16:04.564 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:SdgvIO8XOVqJKUN7BLyRJ4LlHtM= 2024-04-13T23:16:04.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SdgvIO8XOVqJKUN7BLyRJ4LlHtM=', 'Content-Length': '0'} 2024-04-13T23:16:04.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d0604cd38334dcb-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.566 DEBUG:boto:path=/vwnfns-77/obj2 2024-04-13T23:16:04.566 DEBUG:boto:auth_path=/vwnfns-77/obj2 2024-04-13T23:16:04.566 DEBUG:boto:Method: GET 2024-04-13T23:16:04.566 DEBUG:boto:Path: /vwnfns-77/obj2 2024-04-13T23:16:04.566 DEBUG:boto:Data: 2024-04-13T23:16:04.566 DEBUG:boto:Headers: {} 2024-04-13T23:16:04.566 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:16:04.567 DEBUG:boto:Port: 8001 2024-04-13T23:16:04.567 DEBUG:boto:Params: {} 2024-04-13T23:16:04.567 DEBUG:boto:Token: None 2024-04-13T23:16:04.567 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:16:04 GMT /vwnfns-77/obj2 2024-04-13T23:16:04.567 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:AGy3AWW/FkgErEnqxy1Kjxykf80= 2024-04-13T23:16:04.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AGy3AWW/FkgErEnqxy1Kjxykf80=', 'Content-Length': '0'} 2024-04-13T23:16:04.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:14:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032570a06974af04f-00661b1234-2f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:16:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:04.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.569 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:04.619 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:04.619 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:04.692 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:16:04.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.693 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:04.744 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:04.745 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:04.776 INFO:teuthology.orchestra.run.smithi132.stderr:WARNING: cannot find source zone id for name=* 2024-04-13T23:16:04.776 INFO:teuthology.orchestra.run.smithi132.stderr:WARNING: cannot find source zone id for name=* 2024-04-13T23:16:04.792 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:16:04.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.793 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:04.841 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:04.841 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:04.896 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:16:04.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:04.897 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:04.944 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:04.944 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout: "epoch": 22, 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": [], 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout: "period_map": { 2024-04-13T23:16:05.005 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "zonegroups": [ 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "a", 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a", 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": true, 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:16:05.006 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a2", 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:16:05.007 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "a1", 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:16:05.009 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:16:05.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:16:05.011 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [ 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-group", 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "data_flow": { 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "symmetrical": [ 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-flow1", 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: "pipes": [ 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe-vwnfns-76", 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:16:05.013 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.014 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:16:05.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe-vwnfns-77", 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-76", 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "vwnfns-77", 2024-04-13T23:16:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "5da80fbc-5edc-4ba7-826d-471359a9fc92" 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:16:05.017 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "sync-pipe", 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: "*" 2024-04-13T23:16:05.018 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": { 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket": "*", 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "*" 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "params": { 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "source": { 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "filter": { 2024-04-13T23:16:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [] 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: "dest": {}, 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: "priority": 0, 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: "mode": "system", 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: "user": "" 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "allowed" 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "b", 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b", 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "api_name": "", 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "is_master": false, 2024-04-13T23:16:05.021 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames": [], 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: "hostnames_s3website": [], 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: "zones": [ 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.022 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b2", 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:16:05.023 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "b1", 2024-04-13T23:16:05.024 INFO:teuthology.orchestra.run.smithi132.stdout: "endpoints": [ 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "log_meta": false, 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "log_data": true, 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_index_max_shards": 11, 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "read_only": false, 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "tier_type": "", 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from_all": true, 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_from": [], 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "redirect_zone": "", 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "supported_features": [ 2024-04-13T23:16:05.025 INFO:teuthology.orchestra.run.smithi132.stdout: "compress-encrypted", 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: "placement_targets": [ 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "default-placement", 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: "tags": [], 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: "storage_classes": [ 2024-04-13T23:16:05.026 INFO:teuthology.orchestra.run.smithi132.stdout: "STANDARD" 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: "default_placement": "default-placement", 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_policy": { 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: "groups": [] 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled_features": [ 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: "resharding" 2024-04-13T23:16:05.027 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: "short_zone_ids": [ 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 555598186 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:16:05.028 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 782658364 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2041901071 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 2070976819 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.029 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zonegroup": "a", 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "period_config": { 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_quota": { 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:16:05.030 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "user_quota": { 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "check_on_raw": false, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size": -1, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "max_size_kb": 0, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "max_objects": -1 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.031 INFO:teuthology.orchestra.run.smithi132.stdout: "user_ratelimit": { 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "bucket_ratelimit": { 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:16:05.032 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "anonymous_ratelimit": { 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_ops": 0, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_ops": 0, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "max_read_bytes": 0, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "max_write_bytes": 0, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "enabled": false 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:05.033 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:16:05.034 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_name": "test-realm", 2024-04-13T23:16:05.034 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:05.034 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:05.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.035 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:05.084 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:05.084 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:05.170 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:16:05.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.171 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket vwnfns-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:05.230 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:05.230 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:05.402 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:16:05.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.403 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket vwnfns-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:05.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:05.473 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:05.596 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:16:05.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-76', '--source-bucket=*', '--dest-bucket=vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-76', '--source-bucket=*', '--dest-bucket=vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.597 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-vwnfns-76 --source-zones=a1 --dest-zones=a2 --bucket=vwnfns-76 '--source-bucket=*' --dest-bucket=vwnfns-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:05.663 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:05.663 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:05.786 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-13T23:16:05.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-77', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-76', '--source-bucket=*', '--dest-bucket=vwnfns-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-vwnfns-77', '--source-zones=a1', '--dest-zones=a2', '--bucket=vwnfns-76', '--source-bucket=*', '--dest-bucket=vwnfns-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.787 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-vwnfns-77 --source-zones=a1 --dest-zones=a2 --bucket=vwnfns-76 '--source-bucket=*' --dest-bucket=vwnfns-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:05.851 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:05.851 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:05.985 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-13T23:16:05.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:05.986 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket vwnfns-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:06.040 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:06.040 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:06.119 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-13T23:16:06.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:06.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:16:06.120 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:16:06.163 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:06.164 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:06.210 INFO:teuthology.orchestra.run.smithi132.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-vwnfns-76","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-vwnfns-77","source":{"bucket":"vwnfns-76","zones":["a1"]},"dest":{"bucket":"vwnfns-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:16:06.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:06.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:06.210 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:06.236 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T23:16:06.234+0000 7f5b022dd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-13T23:16:06.282 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:06.283 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:06.325 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:16:06.340 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:16:06.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050166.096948_1302.1", 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:16:06.096948Z" 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:06.341 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:16:06.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.342 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:06.342 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:16:06.342 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:06.342 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:16:06.346 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:06.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:16:06.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:06.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:06.346 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:06.395 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:06.395 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:06.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:06.440+0000 7fb22fe5f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:06.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:06.441+0000 7fb22fe5f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:06.458 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:06.458 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:06.459 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:06.460 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:06.461 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:06.462 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:06.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.464 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:06.464 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:06.464 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:06.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:06.464 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:06Z" 2024-04-13T23:16:06.464 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:06.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:06Z" } 2024-04-13T23:16:06.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:06.465 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:06.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:06.466 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050166.096948_1302.1 target=1_1713050063.751827_1254.1 2024-04-13T23:16:11.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:11.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:11.467 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:11.524 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:11.525 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:11.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:11.557+0000 7fb82c2d2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:11.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:11.558+0000 7fb82c2d2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:11.574 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:11.574 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:11.574 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:11.574 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:16:11.575 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:11.576 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:11.577 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:11.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:11.579 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:11Z" 2024-04-13T23:16:11.580 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:11.581 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:11Z" } 2024-04-13T23:16:11.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:11.581 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:11.582 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:11.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050166.096948_1302.1 target=1_1713050063.751827_1254.1 2024-04-13T23:16:16.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:16.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:16.583 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:16.639 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:16.640 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:16.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:16.680+0000 7f3f31299780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:16.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:16.682+0000 7f3f31299780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:16.703 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:16.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:16.707 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:16Z" 2024-04-13T23:16:16.708 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:16.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:16Z" } 2024-04-13T23:16:16.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:16.710 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:16.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:16.710 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050166.096948_1302.1 target=1_1713050063.751827_1254.1 2024-04-13T23:16:21.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:21.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:21.712 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:21.767 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:21.767 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:21.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:21.800+0000 7feb0b69c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:21.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:21.801+0000 7feb0b69c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:21.817 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:21.817 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:21.817 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:21.818 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:21.819 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:21.820 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:21.821 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:21.822 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:21Z" 2024-04-13T23:16:21.823 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:21.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:21Z" } 2024-04-13T23:16:21.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:21.825 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:21.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:21.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050166.096948_1302.1 target=1_1713050063.751827_1254.1 2024-04-13T23:16:26.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:26.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:26.826 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:26.881 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:26.882 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:26.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:26.931+0000 7f0c12fe2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:26.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:26.932+0000 7f0c12fe2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:26.953 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:26.953 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:26.953 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:26.954 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:26.955 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:26.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:26.957 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:26.958 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:26Z" 2024-04-13T23:16:26.959 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:26.960 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:26Z" } 2024-04-13T23:16:26.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:26.961 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:26.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:26.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050166.096948_1302.1 target=1_1713050063.751827_1254.1 2024-04-13T23:16:31.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:31.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:31.966 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:32.021 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:32.021 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:32.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:32.067+0000 7f1f14e21780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:32.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:32.068+0000 7f1f14e21780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:32.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050063.751827_1254.1", 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:23.751827Z", 2024-04-13T23:16:32.087 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:32.088 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:32.089 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:32.090 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:32.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:32.092 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:32Z" 2024-04-13T23:16:32.092 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:32.093 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050063.751827_1254.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:14:23.751827Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:32Z" } 2024-04-13T23:16:32.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050063.751827_1254.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:14:23.751827Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:32.093 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:32.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050063.751827_1254.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:32.093 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1713050166.096948_1302.1 target=1_1713050063.751827_1254.1 2024-04-13T23:16:37.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.095 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:37.152 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:37.152 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:37.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:37.201+0000 7fce5a895780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:37.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:37.202+0000 7fce5a895780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.220 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050166.096948_1302.1", 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:16:06.096948Z", 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.221 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:37.222 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.223 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.224 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:37.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.226 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:37Z" 2024-04-13T23:16:37.226 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:37.226 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050166.096948_1302.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:16:06.096948Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:37Z" } 2024-04-13T23:16:37.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050166.096948_1302.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:16:06.096948Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:37.227 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:37.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:37.227 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:16:37.227 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:16:37.228 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:16:37.228 DEBUG:boto:Method: HEAD 2024-04-13T23:16:37.228 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:16:37.228 DEBUG:boto:Data: 2024-04-13T23:16:37.228 DEBUG:boto:Headers: {} 2024-04-13T23:16:37.228 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:16:37.228 DEBUG:boto:Port: 8000 2024-04-13T23:16:37.228 DEBUG:boto:Params: {} 2024-04-13T23:16:37.229 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:16:37.229 DEBUG:boto:Token: None 2024-04-13T23:16:37.229 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:37 GMT /vwnfns-76/ 2024-04-13T23:16:37.229 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WyAFSX1Oe5kGdGapcnaQZLtFAxE= 2024-04-13T23:16:37.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WyAFSX1Oe5kGdGapcnaQZLtFAxE=', 'Content-Length': '0'} 2024-04-13T23:16:37.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbd5a1839d6b109c-00661b1255-3030-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:37.237 DEBUG:boto:path=/vwnfns-76/obj3 2024-04-13T23:16:37.237 DEBUG:boto:auth_path=/vwnfns-76/obj3 2024-04-13T23:16:37.237 DEBUG:boto:Method: PUT 2024-04-13T23:16:37.237 DEBUG:boto:Path: /vwnfns-76/obj3 2024-04-13T23:16:37.237 DEBUG:boto:Data: 2024-04-13T23:16:37.237 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:16:37.237 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:16:37.238 DEBUG:boto:Port: 8000 2024-04-13T23:16:37.238 DEBUG:boto:Params: {} 2024-04-13T23:16:37.238 DEBUG:boto:Token: None 2024-04-13T23:16:37.238 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:16:37 GMT /vwnfns-76/obj3 2024-04-13T23:16:37.238 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:kkuo8eg5GiMZKSO+Qys1x/E6I8M= 2024-04-13T23:16:37.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:16:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kkuo8eg5GiMZKSO+Qys1x/E6I8M='} 2024-04-13T23:16:37.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef18ced04f044128-00661b1255-3030-a1'), ('Date', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:37.246 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:16:37.246 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:16:37.246 DEBUG:boto:Method: HEAD 2024-04-13T23:16:37.246 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:16:37.246 DEBUG:boto:Data: 2024-04-13T23:16:37.247 DEBUG:boto:Headers: {} 2024-04-13T23:16:37.247 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:16:37.247 DEBUG:boto:Port: 8000 2024-04-13T23:16:37.247 DEBUG:boto:Params: {} 2024-04-13T23:16:37.247 DEBUG:boto:Token: None 2024-04-13T23:16:37.247 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:16:37 GMT /vwnfns-76/ 2024-04-13T23:16:37.247 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:WyAFSX1Oe5kGdGapcnaQZLtFAxE= 2024-04-13T23:16:37.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:16:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WyAFSX1Oe5kGdGapcnaQZLtFAxE=', 'Content-Length': '0'} 2024-04-13T23:16:37.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdca61dcec983757-00661b1255-3030-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:37.250 DEBUG:boto:path=/vwnfns-76/obj4 2024-04-13T23:16:37.250 DEBUG:boto:auth_path=/vwnfns-76/obj4 2024-04-13T23:16:37.250 DEBUG:boto:Method: PUT 2024-04-13T23:16:37.251 DEBUG:boto:Path: /vwnfns-76/obj4 2024-04-13T23:16:37.251 DEBUG:boto:Data: 2024-04-13T23:16:37.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:16:37.251 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:16:37.251 DEBUG:boto:Port: 8000 2024-04-13T23:16:37.251 DEBUG:boto:Params: {} 2024-04-13T23:16:37.251 DEBUG:boto:Token: None 2024-04-13T23:16:37.251 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 13 Apr 2024 23:16:37 GMT /vwnfns-76/obj4 2024-04-13T23:16:37.251 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:X7Pe0rQRe8YYD1TrDsYEqWhjuYg= 2024-04-13T23:16:37.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:16:37 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7Pe0rQRe8YYD1TrDsYEqWhjuYg='} 2024-04-13T23:16:37.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000184f26fd07d516b7-00661b1255-3030-a1'), ('Date', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:16:37.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.259 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:37.308 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:37.308 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:37.340 INFO:teuthology.orchestra.run.smithi132.stderr:No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:16:37.358 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:16:37.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.358 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050166.096948_1302.1", 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:16:06.096948Z" 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:16:37.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.360 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:37.360 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:16:37.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.360 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:16:37.361 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:37.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-13T23:16:37.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.362 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:37.411 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:37.411 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:37.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:37.448+0000 7f6649898780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:37.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-13T23:16:37.449+0000 7f6649898780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:16:37.472 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:37.472 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:37.472 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:37.473 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:37.473 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:37.473 INFO:teuthology.orchestra.run.smithi132.stdout: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:16:37.473 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.474 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:37.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.474 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:37.474 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.475 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.475 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713050166.096948_1302.1", 2024-04-13T23:16:37.476 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.476 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 1, 2024-04-13T23:16:37.476 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.477 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:16:06.096948Z", 2024-04-13T23:16:37.477 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.477 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.479 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "state": 1, 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:16:37.480 INFO:teuthology.orchestra.run.smithi132.stdout: "realm_epoch": 2 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 1, 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 1 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:37Z" 2024-04-13T23:16:37.481 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:37.483 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713050166.096948_1302.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-13T23:16:06.096948Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1713043708.018806_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T21:28:28.018806Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713047459.318155_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.318155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713047459.305268_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T22:30:59.305268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-13T23:16:37Z" } 2024-04-13T23:16:37.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713050166.096948_1302.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:16:06.096948Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:16:37.483 DEBUG:rgw_multi.tests:log_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:37.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1713050166.096948_1302.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:16:37.483 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-13T23:16:37.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.484 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:37.531 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:37.531 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:37.603 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.604 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000034032", 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:16:37.257412Z" 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054632", 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout: "last_update": "2024-04-13T23:16:37.244828Z" 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.605 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-04-13T23:16:37.606 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:16:37.606 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:16:37.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:37.607 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:37.656 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:37.657 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:37.713 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:16:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:16:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:37Z" 2024-04-13T23:16:37.719 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:37.720 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:16:37Z" } 2024-04-13T23:16:37.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:16:37.721 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:16:37.721 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:37.721 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:16:42.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:42.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:42.722 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:42.780 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:42.780 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:42.858 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:42.859 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:42.860 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:42.861 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:42.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:16:42.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:42Z" 2024-04-13T23:16:42.864 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:42.865 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:16:42Z" } 2024-04-13T23:16:42.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:16:42.866 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:16:42.866 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:42.866 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:16:47.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:47.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:47.867 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:47.924 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:47.925 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:47.976 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:16:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:16:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:47Z" 2024-04-13T23:16:47.982 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:47.983 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:16:47Z" } 2024-04-13T23:16:47.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:16:47.983 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:16:47.983 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:47.984 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:16:52.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:52.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:52.985 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:53.042 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:53.043 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:53.113 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:53.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:53.115 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:53.116 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:53.117 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:16:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:16:53.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:53.120 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:53Z" 2024-04-13T23:16:53.120 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:53.121 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:16:53Z" } 2024-04-13T23:16:53.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:16:53.121 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:16:53.121 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:53.121 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:16:58.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:58.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:16:58.123 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:16:58.180 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:16:58.180 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:16:58.232 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:16:58.232 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:16:58.232 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:16:58.232 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:16:58.233 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:16:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:58.235 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:16:58.236 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:16:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:16:58.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:16:58.238 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:16:58Z" 2024-04-13T23:16:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:16:58.239 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:16:58Z" } 2024-04-13T23:16:58.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:16:58.239 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:16:58.239 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:16:58.239 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:03.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:03.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:03.241 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:03.298 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:03.298 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:03.365 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:03.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:03.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:03.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:03.369 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:03.370 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:03Z" 2024-04-13T23:17:03.371 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:03.372 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:03Z" } 2024-04-13T23:17:03.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:03.373 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:03.373 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:03.373 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:08.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:08.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:08.375 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:08.433 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:08.433 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:08.486 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:08.486 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:08.486 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:08.487 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:08.488 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:08.489 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:08.490 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:08.492 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:08Z" 2024-04-13T23:17:08.492 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:08.493 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:08Z" } 2024-04-13T23:17:08.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:08.493 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:08.493 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:08.493 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:13.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:13.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:13.495 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:13.555 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:13.555 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:13.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:13.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:13.615 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:13.616 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:13.617 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:13Z" 2024-04-13T23:17:13.618 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:13.619 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:13Z" } 2024-04-13T23:17:13.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:13.620 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:13.620 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:13.620 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:18.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:18.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:18.621 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:18.680 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:18.680 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:18.741 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:18.741 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:18.741 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:18.742 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:18.743 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:18.744 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:18.745 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:18.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:18.747 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:18Z" 2024-04-13T23:17:18.747 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:18.748 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:18Z" } 2024-04-13T23:17:18.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:18.748 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:18.748 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:18.748 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:23.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:23.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:23.749 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:23.807 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:23.807 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:23.873 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:23.873 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:23.873 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:23.874 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:23.875 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:23.876 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:23.877 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:23.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:23.879 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:23Z" 2024-04-13T23:17:23.879 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:23.880 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:23Z" } 2024-04-13T23:17:23.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:23.880 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:23.880 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:23.880 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:28.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:28.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:28.881 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:28.940 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:28.940 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:29.009 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:29.009 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:29.010 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:29.011 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:29.012 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:29.013 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:29.014 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:29.015 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:29.015 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:29.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:29.015 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:28Z" 2024-04-13T23:17:29.015 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:29.016 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:28Z" } 2024-04-13T23:17:29.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:29.016 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:29.016 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:29.016 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:34.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:34.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:34.018 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:34.076 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:34.076 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:34.144 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:34.144 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:34.144 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:34.144 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:34.145 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:34.146 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:34.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:34.148 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:34.148 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:34.148 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:34.148 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:34.148 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:34.148 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:34.149 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:34.150 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:34.151 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:34.151 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:34.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:34.151 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:34Z" 2024-04-13T23:17:34.151 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:34.152 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:34Z" } 2024-04-13T23:17:34.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:34.152 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:34.152 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:34.153 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:39.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:39.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:39.154 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:39.213 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:39.213 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:39.274 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:39.275 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:39.276 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:39.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:39.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:39.279 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:39Z" 2024-04-13T23:17:39.279 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:39.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:39Z" } 2024-04-13T23:17:39.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:39.280 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:39.280 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:39.280 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:44.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:44.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:44.282 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:44.339 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:44.339 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:44.410 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:44Z" 2024-04-13T23:17:44.421 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:44.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:44Z" } 2024-04-13T23:17:44.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:44.422 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:44.422 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:44.422 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:49.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:49.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:49.424 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:49.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:49.481 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:49.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:49.542 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:49.543 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:49.544 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:49.545 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:49Z" 2024-04-13T23:17:49.547 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:49.547 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:49Z" } 2024-04-13T23:17:49.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:49.548 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:49.548 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:49.548 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:54.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:54.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:54.550 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:54.617 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:54.617 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:54.666 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:54.666 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:54.666 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:54.666 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:54.666 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:54.666 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:54.667 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:54.668 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:54.669 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:54.670 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:54.671 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:54Z" 2024-04-13T23:17:54.672 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:54.673 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:54Z" } 2024-04-13T23:17:54.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:54.673 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:54.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:54.674 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:17:59.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:59.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:17:59.676 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:17:59.733 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:17:59.733 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:17:59.807 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:59.808 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:59.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:59.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:59.810 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:17:59.810 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:17:59.811 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:17:59.812 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:59.813 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:17:59.814 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:17:59.814 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:17:59.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:17:59.814 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:17:59Z" 2024-04-13T23:17:59.814 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:17:59.815 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:17:59Z" } 2024-04-13T23:17:59.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:17:59.815 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:17:59.815 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:17:59.816 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:18:04.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:04.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:04.817 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:04.875 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:18:04.876 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:18:04.940 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:18:04.940 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:18:04.940 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:18:04.940 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:04.941 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:18:04.942 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:04.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000033891", 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:13:41.632949Z" 2024-04-13T23:18:04.944 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054491", 2024-04-13T23:18:04.945 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.329694Z" 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:18:04.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:04.947 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:18:04.947 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:18:04.947 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:18:04.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:04.947 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:18:04Z" 2024-04-13T23:18:04.947 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:18:04.948 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033891", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:13:41.632949Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054491", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:14:30.329694Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:18:04Z" } 2024-04-13T23:18:04.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033891', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:13:41.632949Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054491', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.329694Z'}}] 2024-04-13T23:18:04.948 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:18:04.949 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000033891', 3: '00000000000000000000:00000000000000054491'} 2024-04-13T23:18:04.949 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000034032 target=00000000000000000000:00000000000000033891, shard=3 master=00000000000000000000:00000000000000054632 target=00000000000000000000:00000000000000054491 2024-04-13T23:18:09.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:09.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:09.951 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.008 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.008 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.062 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-13T23:18:10.062 INFO:teuthology.orchestra.run.smithi132.stdout: "sync_status": { 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "info": { 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "sync", 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "num_shards": 4, 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "instance_id": 13387773392408425331 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "markers": [ 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 0, 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:10.063 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000046435", 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:08:58.013270Z" 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 1, 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:10.064 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000047915", 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 0, 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:14:30.361634Z" 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 2, 2024-04-13T23:18:10.065 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000034032", 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 9, 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:16:37.257412Z" 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-13T23:18:10.066 INFO:teuthology.orchestra.run.smithi132.stdout: "key": 3, 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "val": { 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "status": "incremental-sync", 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "marker": "00000000000000000000:00000000000000054632", 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "next_step_marker": "", 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "total_entries": 57, 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "pos": 0, 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: "timestamp": "2024-04-13T23:16:37.244828Z" 2024-04-13T23:18:10.067 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: "full_sync": { 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: "total": 66, 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: "complete": 66 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout: "current_time": "2024-04-13T23:18:10Z" 2024-04-13T23:18:10.068 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-13T23:18:10.069 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13387773392408425331 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:08:58.013270Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-13T23:14:30.361634Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-13T23:16:37.257412Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054632", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-13T23:16:37.244828Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-13T23:18:10Z" } 2024-04-13T23:18:10.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:08:58.013270Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:14:30.361634Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T23:16:37.257412Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054632', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:16:37.244828Z'}}] 2024-04-13T23:18:10.070 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:18:10.070 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046435', 1: '00000000000000000000:00000000000000047915', 2: '00000000000000000000:00000000000000034032', 3: '00000000000000000000:00000000000000054632'} 2024-04-13T23:18:10.070 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:18:10.070 DEBUG:boto:path=/vwnfns-76/ 2024-04-13T23:18:10.070 DEBUG:boto:auth_path=/vwnfns-76/ 2024-04-13T23:18:10.071 DEBUG:boto:Method: HEAD 2024-04-13T23:18:10.071 DEBUG:boto:Path: /vwnfns-76/ 2024-04-13T23:18:10.071 DEBUG:boto:Data: 2024-04-13T23:18:10.071 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.071 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.071 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.071 DEBUG:boto:Params: {} 2024-04-13T23:18:10.071 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:18:10.072 DEBUG:boto:Token: None 2024-04-13T23:18:10.072 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-76/ 2024-04-13T23:18:10.072 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VVnmusMa++4UZWoSRm8W933yUus= 2024-04-13T23:18:10.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VVnmusMa++4UZWoSRm8W933yUus=', 'Content-Length': '0'} 2024-04-13T23:18:10.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008889fd7d149c94ba-00661b12b2-303b-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.078 DEBUG:boto:path=/vwnfns-76/obj3 2024-04-13T23:18:10.078 DEBUG:boto:auth_path=/vwnfns-76/obj3 2024-04-13T23:18:10.079 DEBUG:boto:Method: HEAD 2024-04-13T23:18:10.079 DEBUG:boto:Path: /vwnfns-76/obj3 2024-04-13T23:18:10.079 DEBUG:boto:Data: 2024-04-13T23:18:10.079 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.079 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.079 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.079 DEBUG:boto:Params: {} 2024-04-13T23:18:10.079 DEBUG:boto:Token: None 2024-04-13T23:18:10.079 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-76/obj3 2024-04-13T23:18:10.079 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ydlHaa2+veZ6b/LwzRjHJy5GpT8= 2024-04-13T23:18:10.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ydlHaa2+veZ6b/LwzRjHJy5GpT8=', 'Content-Length': '0'} 2024-04-13T23:18:10.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051c6cddb5ed02e6d-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.082 DEBUG:boto:path=/vwnfns-76/obj3 2024-04-13T23:18:10.082 DEBUG:boto:auth_path=/vwnfns-76/obj3 2024-04-13T23:18:10.082 DEBUG:boto:Method: GET 2024-04-13T23:18:10.082 DEBUG:boto:Path: /vwnfns-76/obj3 2024-04-13T23:18:10.082 DEBUG:boto:Data: 2024-04-13T23:18:10.082 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.082 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.082 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.082 DEBUG:boto:Params: {} 2024-04-13T23:18:10.082 DEBUG:boto:Token: None 2024-04-13T23:18:10.083 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-76/obj3 2024-04-13T23:18:10.083 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:boFXO/GzhWpPgdEprD7ytXSVyVo= 2024-04-13T23:18:10.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:boFXO/GzhWpPgdEprD7ytXSVyVo=', 'Content-Length': '0'} 2024-04-13T23:18:10.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000580f4537c4d70b1c-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.085 DEBUG:boto:path=/vwnfns-76/obj4 2024-04-13T23:18:10.085 DEBUG:boto:auth_path=/vwnfns-76/obj4 2024-04-13T23:18:10.085 DEBUG:boto:Method: HEAD 2024-04-13T23:18:10.085 DEBUG:boto:Path: /vwnfns-76/obj4 2024-04-13T23:18:10.085 DEBUG:boto:Data: 2024-04-13T23:18:10.086 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.086 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.086 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.086 DEBUG:boto:Params: {} 2024-04-13T23:18:10.086 DEBUG:boto:Token: None 2024-04-13T23:18:10.086 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-76/obj4 2024-04-13T23:18:10.086 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:ytsff7ROdydcjuEPKJn90j0I4xs= 2024-04-13T23:18:10.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ytsff7ROdydcjuEPKJn90j0I4xs=', 'Content-Length': '0'} 2024-04-13T23:18:10.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000685e1b9653532d0-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.088 DEBUG:boto:path=/vwnfns-76/obj4 2024-04-13T23:18:10.088 DEBUG:boto:auth_path=/vwnfns-76/obj4 2024-04-13T23:18:10.088 DEBUG:boto:Method: GET 2024-04-13T23:18:10.088 DEBUG:boto:Path: /vwnfns-76/obj4 2024-04-13T23:18:10.088 DEBUG:boto:Data: 2024-04-13T23:18:10.088 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.088 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.089 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.089 DEBUG:boto:Params: {} 2024-04-13T23:18:10.089 DEBUG:boto:Token: None 2024-04-13T23:18:10.089 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-76/obj4 2024-04-13T23:18:10.089 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:/ZxF8GJyDkF7xTYMdGuu6F6EkPw= 2024-04-13T23:18:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/ZxF8GJyDkF7xTYMdGuu6F6EkPw=', 'Content-Length': '0'} 2024-04-13T23:18:10.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2beba42c475eb7f-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.091 DEBUG:boto:path=/vwnfns-77/ 2024-04-13T23:18:10.091 DEBUG:boto:auth_path=/vwnfns-77/ 2024-04-13T23:18:10.091 DEBUG:boto:Method: HEAD 2024-04-13T23:18:10.091 DEBUG:boto:Path: /vwnfns-77/ 2024-04-13T23:18:10.091 DEBUG:boto:Data: 2024-04-13T23:18:10.091 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.092 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.092 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.092 DEBUG:boto:Params: {} 2024-04-13T23:18:10.092 DEBUG:boto:Token: None 2024-04-13T23:18:10.092 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-77/ 2024-04-13T23:18:10.092 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:EQMzLDproh7fyDVxvVEZCEATeek= 2024-04-13T23:18:10.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EQMzLDproh7fyDVxvVEZCEATeek=', 'Content-Length': '0'} 2024-04-13T23:18:10.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f184d856cb427d2-00661b12b2-303b-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.095 DEBUG:boto:path=/vwnfns-77/obj3 2024-04-13T23:18:10.095 DEBUG:boto:auth_path=/vwnfns-77/obj3 2024-04-13T23:18:10.095 DEBUG:boto:Method: HEAD 2024-04-13T23:18:10.095 DEBUG:boto:Path: /vwnfns-77/obj3 2024-04-13T23:18:10.095 DEBUG:boto:Data: 2024-04-13T23:18:10.095 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.096 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.096 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.096 DEBUG:boto:Params: {} 2024-04-13T23:18:10.096 DEBUG:boto:Token: None 2024-04-13T23:18:10.096 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-77/obj3 2024-04-13T23:18:10.096 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:XUwcJk81BWFiI45DltAxMVl4QWM= 2024-04-13T23:18:10.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XUwcJk81BWFiI45DltAxMVl4QWM=', 'Content-Length': '0'} 2024-04-13T23:18:10.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4ef4ceec18d50de-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.098 DEBUG:boto:path=/vwnfns-77/obj3 2024-04-13T23:18:10.098 DEBUG:boto:auth_path=/vwnfns-77/obj3 2024-04-13T23:18:10.098 DEBUG:boto:Method: GET 2024-04-13T23:18:10.098 DEBUG:boto:Path: /vwnfns-77/obj3 2024-04-13T23:18:10.098 DEBUG:boto:Data: 2024-04-13T23:18:10.098 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.098 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.098 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.099 DEBUG:boto:Params: {} 2024-04-13T23:18:10.099 DEBUG:boto:Token: None 2024-04-13T23:18:10.099 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-77/obj3 2024-04-13T23:18:10.099 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:VKpfl4kGP2E8QURSEz/slITJxlg= 2024-04-13T23:18:10.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VKpfl4kGP2E8QURSEz/slITJxlg=', 'Content-Length': '0'} 2024-04-13T23:18:10.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b848c7a283356ba-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.101 DEBUG:boto:path=/vwnfns-77/obj4 2024-04-13T23:18:10.101 DEBUG:boto:auth_path=/vwnfns-77/obj4 2024-04-13T23:18:10.101 DEBUG:boto:Method: HEAD 2024-04-13T23:18:10.101 DEBUG:boto:Path: /vwnfns-77/obj4 2024-04-13T23:18:10.101 DEBUG:boto:Data: 2024-04-13T23:18:10.101 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.102 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.102 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.102 DEBUG:boto:Params: {} 2024-04-13T23:18:10.102 DEBUG:boto:Token: None 2024-04-13T23:18:10.102 DEBUG:boto:StringToSign: HEAD Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-77/obj4 2024-04-13T23:18:10.102 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:+UkcOCsfHxjBB2wPx6KGptyp2RY= 2024-04-13T23:18:10.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+UkcOCsfHxjBB2wPx6KGptyp2RY=', 'Content-Length': '0'} 2024-04-13T23:18:10.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004822b7bd9e1065dd-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.104 DEBUG:boto:path=/vwnfns-77/obj4 2024-04-13T23:18:10.104 DEBUG:boto:auth_path=/vwnfns-77/obj4 2024-04-13T23:18:10.104 DEBUG:boto:Method: GET 2024-04-13T23:18:10.104 DEBUG:boto:Path: /vwnfns-77/obj4 2024-04-13T23:18:10.104 DEBUG:boto:Data: 2024-04-13T23:18:10.104 DEBUG:boto:Headers: {} 2024-04-13T23:18:10.104 DEBUG:boto:Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.105 DEBUG:boto:Port: 8001 2024-04-13T23:18:10.105 DEBUG:boto:Params: {} 2024-04-13T23:18:10.105 DEBUG:boto:Token: None 2024-04-13T23:18:10.105 DEBUG:boto:StringToSign: GET Sat, 13 Apr 2024 23:18:10 GMT /vwnfns-77/obj4 2024-04-13T23:18:10.105 DEBUG:boto:Signature: AWS C81CVW06BOFYI3GBQ4YX:DwXn91awMnKj7Ga533P5EuKbS2E= 2024-04-13T23:18:10.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:18:10 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DwXn91awMnKj7Ga533P5EuKbS2E=', 'Content-Length': '0'} 2024-04-13T23:18:10.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:16:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-76:f6f208d6-f791-479b-bc3d-112512eec2d4.11636.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e604b4c6e5e192f3-00661b12b2-303b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:10.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'vwnfns-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:10.107 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket vwnfns-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:10.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.156 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.240 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:18:10.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:10.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:10.241 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:10.287 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.287 INFO:teuthology.orchestra.run.smithi132.stderr:ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.348 INFO:teuthology.orchestra.run.smithi132.stdout:{"sync_policy":{"groups":[]}} 2024-04-13T23:18:10.349 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2024-04-13T23:18:10.350 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.350 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:10.351 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-04-13T23:18:10.351 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:10.351 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:10.351 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:10.351 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=vwnfns-11 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.353 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:9cl4+lXbVTrhDB+4jA6u5jwCZ1E= 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9cl4+lXbVTrhDB+4jA6u5jwCZ1E=', 'Content-Length': '0'} 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000001090ccbbe29fb4fc-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=vwnfns-12 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.355 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:EYBRKpvCF5hNX9HoD+fZvaUJFU4= 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EYBRKpvCF5hNX9HoD+fZvaUJFU4=', 'Content-Length': '0'} 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002fec7f875f9fe81f-00661af935-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017e98b3bebc24d85-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj1 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj1 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj1 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.358 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj1 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rc6qA9vMNB9eEqI/1XmQ5zp3GZY= 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rc6qA9vMNB9eEqI/1XmQ5zp3GZY='} 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022a117557181e462-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aace2fd0e2160bb-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj2 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj2 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj2 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.361 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj2 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:1t77kGyEUeZf/rj6vXK6hTBK/II= 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1t77kGyEUeZf/rj6vXK6hTBK/II='} 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed1e253b77c7fb0e-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b9e0e841d1fc6d8-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj3 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj3 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj3 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.364 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj3 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FuBC2lixTTuFtzNN4HgDQv9dPDM= 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FuBC2lixTTuFtzNN4HgDQv9dPDM='} 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c154711e1850f3f-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032d118812ffbcfb5-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj4 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj4 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj4 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.367 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj4 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:aHfiJxg6eBYbW7Rx/tjlExE+BuE= 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aHfiJxg6eBYbW7Rx/tjlExE+BuE='} 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dfcd17d2a483799-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055a1be978549fcb9-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj5 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj5 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj5 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.370 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj5 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:09jlYKnbSt/f6PmNno5PRnCyYUE= 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:09jlYKnbSt/f6PmNno5PRnCyYUE='} 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052ff82dabdd34009-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89a2f0222a256b5-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj6 2024-04-13T23:18:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj6 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj6 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj6 2024-04-13T23:18:10.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:F+AN7SZusydjT+QnBrWzDi9CMKA= 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:F+AN7SZusydjT+QnBrWzDi9CMKA='} 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e098cead1dfbfdc-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b603802667998684-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj7 2024-04-13T23:18:10.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj7 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj7 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.376 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj7 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:aVDIwZQN8nloMsMRqAn5joULjfA= 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:aVDIwZQN8nloMsMRqAn5joULjfA='} 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088454787d7a35afc-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000900c08ac4bed26a9-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj8 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj8 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj8 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.379 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj8 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ahARxELLQSetAdgauxagMo9X4lg= 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ahARxELLQSetAdgauxagMo9X4lg='} 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b29dc9a55fcebebe-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b25c710f5c96611f-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj9 2024-04-13T23:18:10.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj9 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj9 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.382 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj9 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ei3BpzF6SpjG0nW5ez5Rwtf6zmc= 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ei3BpzF6SpjG0nW5ez5Rwtf6zmc='} 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e484eca9138aa063-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054983d2adb500ccc-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj10 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj10 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj10 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj10 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:x639aYamntLZApWARiXxjhit7xM= 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:x639aYamntLZApWARiXxjhit7xM='} 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f737870d602cc21e-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000204f90e30e24af0e-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj11 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj11 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj11 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj11 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2C9I8x59FteE4egJsXL0JOxFyU8= 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2C9I8x59FteE4egJsXL0JOxFyU8='} 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eda1e07886ac5237-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d22a8772039de504-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj12 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj12 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj12 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj12 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SeH26MTdBWgUo8qTgJdXzMoTEJY= 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SeH26MTdBWgUo8qTgJdXzMoTEJY='} 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bf1fde388e0853a-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.393 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098a58e6257b9f817-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj13 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj13 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj13 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj13 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oyBuaC4rgfuG3gm7OOFpRjPk1uY= 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oyBuaC4rgfuG3gm7OOFpRjPk1uY='} 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e95cf705074ab7f3-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.396 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002460738c5f02b313-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj14 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj14 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj14 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj14 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LIC9CxTDqk/qBuEMRvPdzKecxQU= 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LIC9CxTDqk/qBuEMRvPdzKecxQU='} 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b29059c7fbae1029-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.399 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ebae84286e2d422-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj15 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj15 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj15 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj15 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:70U3DnmGf9gnIa4O8uEu9iaaagg= 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:70U3DnmGf9gnIa4O8uEu9iaaagg='} 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003512bd956e6f0773-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c91556200b422de-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj16 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj16 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj16 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj16 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:D1+nI3qrggK/ENU2TeIEnd30bMA= 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D1+nI3qrggK/ENU2TeIEnd30bMA='} 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002650b57860318748-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007387a8ae5089ccb0-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj17 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj17 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj17 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj17 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LV/navtrVsJqY3UQTc1LVHMAewg= 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LV/navtrVsJqY3UQTc1LVHMAewg='} 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b380e8dbda6ea8e8-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.408 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c3bc0953cd28a53-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj18 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj18 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj18 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj18 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/GFranvcbchxDu9EWq0WFk3A09I= 2024-04-13T23:18:10.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/GFranvcbchxDu9EWq0WFk3A09I='} 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae658dad60dd7cba-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.411 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cd26dd02305df65-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj19 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj19 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj19 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj19 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Uqu6jNTGvrNsK07WyTvoAg1aUkY= 2024-04-13T23:18:10.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Uqu6jNTGvrNsK07WyTvoAg1aUkY='} 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcecea5dd86ca826-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098299bc5c6bfc0db-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj20 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj20 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj20 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj20 2024-04-13T23:18:10.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Tj0SC2+lQ6MrOAyp/1FooVzHRyE= 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Tj0SC2+lQ6MrOAyp/1FooVzHRyE='} 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f866785cbd3a32d0-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b13ba222bf4355e-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj21 2024-04-13T23:18:10.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj21 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj21 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.419 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj21 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/Cen8nwkYwF/0t2B5nOjVxCPv90= 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/Cen8nwkYwF/0t2B5nOjVxCPv90='} 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000545e64f6b0f4f189-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c46cf1fc8775908-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj22 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj22 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj22 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.422 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj22 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sXULMo1Ul7z0CTGwaaq4EYjFblg= 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sXULMo1Ul7z0CTGwaaq4EYjFblg='} 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dfec6e36b6e4574-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.424 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c2cda0a588b278e-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj23 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj23 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj23 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj23 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WD2qGzQCj6tfWwSMcK7i7FPV4uY= 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WD2qGzQCj6tfWwSMcK7i7FPV4uY='} 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdb8bf0c612d6ee8-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac511cc0d12bd0bb-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj24 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj24 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj24 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj24 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:BoGBjP5H76Kblei9MJygYkDdS+s= 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BoGBjP5H76Kblei9MJygYkDdS+s='} 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029af3207b33efab8-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.430 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab806ececbd6704b-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj25 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj25 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj25 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj25 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:1T3WKuV1OE8CBT6fmD3Ptcbri7A= 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1T3WKuV1OE8CBT6fmD3Ptcbri7A='} 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daeb2b4e9f2762e1-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.433 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f95d6c89638f8e2a-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj26 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj26 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj26 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj26 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:EvPxjcv5P0jqUrKO4f+ML2bxySQ= 2024-04-13T23:18:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EvPxjcv5P0jqUrKO4f+ML2bxySQ='} 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce199e49af6ac1a3-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.436 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY= 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CBj3XkPRWy30R6tfws34oclN2TY=', 'Content-Length': '0'} 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f11e0efe94605a86-00661af935-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj27 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj27 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj27 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:25 GMT 2024-04-13T23:18:10.438 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj27 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SqcW4HCpfZSIvl4gW7rPh3wrxKo= 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SqcW4HCpfZSIvl4gW7rPh3wrxKo='} 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc0813069c153526-00661af935-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044637087a7552a69-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj28 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj28 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj28 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.441 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj28 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ll23Rmws7apGCkRJZlIVYyBNsLw= 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ll23Rmws7apGCkRJZlIVYyBNsLw='} 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2049c3a31498a24-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096e03c88dbce00e7-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj29 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj29 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj29 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.444 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj29 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:h3WMdTswguhlrRBg5bVK9qcfxOg= 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h3WMdTswguhlrRBg5bVK9qcfxOg='} 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e15e90180915a83e-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.446 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009031537febff0272-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj30 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj30 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj30 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj30 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Oa9TOQ9zsvNcgcUe0fs4Nq43TK4= 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Oa9TOQ9zsvNcgcUe0fs4Nq43TK4='} 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027fa6f0ab53724d6-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e8f994a50f3d008-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj31 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj31 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj31 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj31 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DOEQV7nLEwq+wRP7cqe99wkWLOQ= 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOEQV7nLEwq+wRP7cqe99wkWLOQ='} 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ec0d62051514a8f-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.452 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835e0ca899ac347f-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj32 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj32 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj32 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj32 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:6CotuKHQPw5DVKZHo+dr6CKFZwA= 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6CotuKHQPw5DVKZHo+dr6CKFZwA='} 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000707823857ce950b2-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.455 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000230db2cca474fbdd-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj33 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj33 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj33 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj33 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:BhiAX6OvgxF76Px0EWLiDe97Z+E= 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BhiAX6OvgxF76Px0EWLiDe97Z+E='} 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f78877b562b84b01-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.458 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3247070ce15cd24-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj34 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj34 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj34 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj34 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CuxxhuO8c95wZgg2s1ItzI0cskQ= 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CuxxhuO8c95wZgg2s1ItzI0cskQ='} 2024-04-13T23:18:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd575bc40f3935c5-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.461 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1f9e27cb124e76a-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj35 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj35 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj35 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj35 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.463 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:QhQvvphQhyyl/dhRWFItSOj6RWs= 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QhQvvphQhyyl/dhRWFItSOj6RWs='} 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4fb2c89786896c9-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3de327dd2142a2c-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj36 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj36 2024-04-13T23:18:10.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj36 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.466 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj36 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CiSI57Fgw3oSJ0Otl83gqbbF1Uc= 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CiSI57Fgw3oSJ0Otl83gqbbF1Uc='} 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000351493995f7af09d-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a63c94b90da0602e-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj37 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj37 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj37 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.469 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj37 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fEbcEadGWr7kvEbkLfjZXQXJZeQ= 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fEbcEadGWr7kvEbkLfjZXQXJZeQ='} 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071490c2d205a4873-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b130922d92da4c97-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj38 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj38 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj38 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.472 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj38 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sksHMe7eJHo/BgiGbAAIy+rdloI= 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sksHMe7eJHo/BgiGbAAIy+rdloI='} 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c6bd62845c3b48b-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.474 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000101433ec7c695a01-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj39 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj39 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj39 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.475 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj39 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/dWa3RndVA1lIZy0EGl8qvV9giY= 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/dWa3RndVA1lIZy0EGl8qvV9giY='} 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081ba79296be95175-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006053e4e0c660b571-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj40 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj40 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj40 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.478 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj40 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:BmIQ0XLlfxbR9DXREzbfcAWGW6U= 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BmIQ0XLlfxbR9DXREzbfcAWGW6U='} 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be3e7cc8f7edbd6e-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000807bc2af786822ea-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj41 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj41 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj41 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.481 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj41 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lux2LmWiMK275PFnY7XiKdeUnDQ= 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lux2LmWiMK275PFnY7XiKdeUnDQ='} 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae9b41f6eba536b0-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae0c7917b194c702-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj42 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj42 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj42 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.484 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj42 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2eKIo72W9p3u1BaZgMhDynPyU8M= 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2eKIo72W9p3u1BaZgMhDynPyU8M='} 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f37365bdec6f6ec6-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6ebeaa69948d70a-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj43 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj43 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj43 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.487 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj43 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:N29q8g64GaEl+Ph7VT9rM3xxUDE= 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:N29q8g64GaEl+Ph7VT9rM3xxUDE='} 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e0e7fd67b9c0792-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f503f25e582e3846-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj44 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj44 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj44 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj44 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:k1UlD9yNn9t6+PoeXT2w95JrFM8= 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k1UlD9yNn9t6+PoeXT2w95JrFM8='} 2024-04-13T23:18:10.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6d8ff19c54788cb-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.492 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a4aeab5efc3573a-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj45 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj45 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj45 2024-04-13T23:18:10.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj45 2024-04-13T23:18:10.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ScYoLAqEIeRIb/zSnamg6YpLJEc= 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ScYoLAqEIeRIb/zSnamg6YpLJEc='} 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a127707ec3f9f28e-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003448de2966c02d6c-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj46 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj46 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj46 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.497 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj46 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NnazETLzd+4q8L54+1dN7C6V8vc= 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NnazETLzd+4q8L54+1dN7C6V8vc='} 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063e7c6587b804dcd-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069da71a5614e3f29-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj47 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj47 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj47 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.500 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj47 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:H2/+dSTbTmUGtrElste4/U7Coz0= 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:H2/+dSTbTmUGtrElste4/U7Coz0='} 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a704dc20440eb9c1-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ac0d34a67dd96b8-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj48 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj48 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj48 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.503 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj48 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0B3Lianij9u31zICLgRB8hLzvvo= 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0B3Lianij9u31zICLgRB8hLzvvo='} 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002717e6a487ef8472-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/ 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/ 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/ 2024-04-13T23:18:10.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:/vwnfns-11/ 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.505 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c= 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:INa5hSPubaKdHG0Y5pekzqeNL+c=', 'Content-Length': '0'} 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b07ca1eacae5547-00661af936-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-11/obj49 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-11/obj49 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-11/obj49 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:/vwnfns-11/obj49 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:1RXIULWZKXkAuzyO20n5dC9c6JY= 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1RXIULWZKXkAuzyO20n5dC9c6JY='} 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070548a503b38a696-00661af936-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.508 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080bea28c444af011-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj1 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj1 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj1 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj1 2024-04-13T23:18:10.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hHKkMXWnJ0FmqyeWJeWfIWgS97k= 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHKkMXWnJ0FmqyeWJeWfIWgS97k='} 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a80455b74920256d-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d86df8ba5614c3e6-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj2 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj2 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj2 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.513 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj2 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sNltT/1NcP+F5G8DBX0rqNs9SYg= 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNltT/1NcP+F5G8DBX0rqNs9SYg='} 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000385e0ad86abffea-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb3fe7f7343035d6-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj3 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj3 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj3 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.516 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj3 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:wuBvlJlh+TaxYxvtngl2qmpdIl8= 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wuBvlJlh+TaxYxvtngl2qmpdIl8='} 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce2b3e01393dd340-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a48ef6e8b9c76b4-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj4 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj4 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj4 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.519 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj4 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lCAtJUWpOLzg+DldD3pmX9OrrLM= 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lCAtJUWpOLzg+DldD3pmX9OrrLM='} 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f2a8f43c99d9746-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a407801ac0bb4c99-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj5 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj5 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj5 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj5 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LcUgZ3jiq2HhkejFAe56phXxA2M= 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LcUgZ3jiq2HhkejFAe56phXxA2M='} 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017ef1690c3a9131d-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.524 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c1307649a49931d-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj6 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj6 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj6 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj6 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:t0qVjcjptkynFkpvqF3X2I6V5Ck= 2024-04-13T23:18:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:t0qVjcjptkynFkpvqF3X2I6V5Ck='} 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a47e10d80f121c5a-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.527 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d64b26c068216e-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj7 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj7 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj7 2024-04-13T23:18:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.529 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj7 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:So+xo8TnLOyuWMhBb763X7vOTFQ= 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:So+xo8TnLOyuWMhBb763X7vOTFQ='} 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075f9e86efc2fbe57-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009248844ae4ab0045-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj8 2024-04-13T23:18:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj8 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj8 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.532 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj8 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:OH+iJzPD0a6RlKrktl8WGbJKQ9I= 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OH+iJzPD0a6RlKrktl8WGbJKQ9I='} 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a9c200395836874-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070371e9fa49c33f3-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj9 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj9 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj9 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj9 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:taQ0jt7G73zBqLGr6ZDLqBvc0Ps= 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:taQ0jt7G73zBqLGr6ZDLqBvc0Ps='} 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000363dca96cc888f95-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012e573bae08c7a16-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj10 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj10 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj10 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj10 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:3cMHQuQRF/kYGTU7cuS2+CzaSJI= 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3cMHQuQRF/kYGTU7cuS2+CzaSJI='} 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006514fbc574e88784-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.540 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000312d263ad19df3cb-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj11 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj11 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj11 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj11 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:W1tpjg+XjmIRa95Rkz6LOAqo8gY= 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W1tpjg+XjmIRa95Rkz6LOAqo8gY='} 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3f04ef39be0c65b-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.543 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dfc4355c161dfdc-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj12 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj12 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj12 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj12 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CvJUUaW3eFpEvZWcgnaGvow3snc= 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CvJUUaW3eFpEvZWcgnaGvow3snc='} 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5c1ad0d858ca59d-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.546 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000667b061613aeb5b8-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj13 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj13 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj13 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj13 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZspWn1KHTWlKqTpOjHrf8R7rmcs= 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZspWn1KHTWlKqTpOjHrf8R7rmcs='} 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d227a2bb07be231b-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.549 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e35de774c749ae-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj14 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj14 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj14 2024-04-13T23:18:10.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj14 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.551 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:UUWU5RwoPB+5rn9+blsJS4Y4PWs= 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UUWU5RwoPB+5rn9+blsJS4Y4PWs='} 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edf66b3262199a52-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0e4812a9028d86b-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj15 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj15 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj15 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj15 2024-04-13T23:18:10.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:m/l+SiRr9QKzUei80plqtr2EMNg= 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:m/l+SiRr9QKzUei80plqtr2EMNg='} 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000e3e8c161fc4d59-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d905ef0ac528db4-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj16 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj16 2024-04-13T23:18:10.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj16 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj16 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.557 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:e6lGATnbgaO2jXkx1r0OBDqajRg= 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e6lGATnbgaO2jXkx1r0OBDqajRg='} 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000262b5167332de738-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000483ade3080b85c34-00661af936-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj17 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj17 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj17 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj17 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.560 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:paKIelpSFnOuhT2D3YbdvHY3uYg= 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:paKIelpSFnOuhT2D3YbdvHY3uYg='} 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007aba4b10ce477ae-00661af936-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.561 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4= 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0e7jzbm+9eOCaesU6vg6EkpeHf4=', 'Content-Length': '0'} 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc11674204e2114c-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj18 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj18 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj18 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:26 GMT 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj18 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:q2k5bch1rpRzHtHXKDpAaV/SiSs= 2024-04-13T23:18:10.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q2k5bch1rpRzHtHXKDpAaV/SiSs='} 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008db130d9a139c840-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.564 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef88069758b53620-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj19 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj19 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj19 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj19 2024-04-13T23:18:10.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hyn3LCr61MMAMnOlIWby+4MPz6U= 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hyn3LCr61MMAMnOlIWby+4MPz6U='} 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f491bf6e471aae71-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4a40645c6a2bac7-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj20 2024-04-13T23:18:10.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj20 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj20 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.569 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj20 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lhYLGNuygmlVhiTARy3WFs0MZY0= 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lhYLGNuygmlVhiTARy3WFs0MZY0='} 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a643bad33830b1b9-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e9f60cc489bf7db-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj21 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj21 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj21 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.572 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj21 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:6+YwKs+kWQujGOu9NZnB7K5v724= 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:6+YwKs+kWQujGOu9NZnB7K5v724='} 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058efcb7b6a726605-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.574 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0c9f5ab331aeb14-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj22 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj22 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj22 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj22 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NENl1lsV0r8LzOGDom+SIn4w/oQ= 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NENl1lsV0r8LzOGDom+SIn4w/oQ='} 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088255cccf94c1173-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.577 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e9c51ab56fabf1f-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj23 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj23 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj23 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj23 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:j9P9aSVrdur+ePOocjWihqIEVTU= 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j9P9aSVrdur+ePOocjWihqIEVTU='} 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000405e8a1bfc014711-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e47e5dcf368bc758-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj24 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj24 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj24 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.581 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj24 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:UoRfdXDmGYPE6vOs3JYayL4CINI= 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UoRfdXDmGYPE6vOs3JYayL4CINI='} 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f817cd5d64fc0624-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003843efd2075c85b6-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj25 2024-04-13T23:18:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj25 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj25 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj25 2024-04-13T23:18:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:YHnv/u711BQ3VQd4J87Zb8jRGPg= 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YHnv/u711BQ3VQd4J87Zb8jRGPg='} 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e223360be2d93bea-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090d7d29dcf2c1bc9-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj26 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj26 2024-04-13T23:18:10.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj26 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.587 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj26 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VktOa4pih3WdiNy5SNOfe6dVkYg= 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VktOa4pih3WdiNy5SNOfe6dVkYg='} 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e7eb66f67fc8e21-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003860fde6edf756e6-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj27 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj27 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj27 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.590 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj27 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ihh/fATerveizLduuIZ9Q0yhQw4= 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ihh/fATerveizLduuIZ9Q0yhQw4='} 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047de0b1465dc01bd-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f360f920f998e0c-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj28 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj28 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj28 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.593 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj28 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8BegRVyln7zfkH1ULFCZAahv2i4= 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8BegRVyln7zfkH1ULFCZAahv2i4='} 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084536773ee0e5528-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f93b2d65536299a-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj29 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj29 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj29 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.596 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj29 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vwr+kSdLl1wKIBRA8gwpzdGZbrQ= 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vwr+kSdLl1wKIBRA8gwpzdGZbrQ='} 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002aa3f51039ccb5a8-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.598 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ec63ec797704818-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj30 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj30 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj30 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj30 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VNKAG4c3fBFMtpGPCZNYZML8vPg= 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VNKAG4c3fBFMtpGPCZNYZML8vPg='} 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d0e6374a30dac37-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008eec332c2e78ee8-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj31 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj31 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj31 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.602 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj31 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2In76eumQYN4qb/H2cT+zJ1bfh4= 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2In76eumQYN4qb/H2cT+zJ1bfh4='} 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000029e6fe3550dc4f3-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e88599499412ac63-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj32 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj32 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj32 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.605 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj32 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:dWsrZBmNNK+WUe7Uq5ygaJXBido= 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:dWsrZBmNNK+WUe7Uq5ygaJXBido='} 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9b931e954166a39-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bd55c3c79e16640-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj33 2024-04-13T23:18:10.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj33 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj33 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj33 2024-04-13T23:18:10.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Og4bIBCmw+KOWUW/xTQtp5UlFLY= 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Og4bIBCmw+KOWUW/xTQtp5UlFLY='} 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2100cf82f496b85-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020c393978d0fd532-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj34 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj34 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj34 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj34 2024-04-13T23:18:10.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Xnc6QFsAVFgeaGUZ4WFO+en374E= 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Xnc6QFsAVFgeaGUZ4WFO+en374E='} 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079318a39e20f6752-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.612 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008373236fea6c9f1a-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj35 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj35 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj35 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj35 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4XNV54uy36Bpno9x/ei7w9qxB/0= 2024-04-13T23:18:10.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4XNV54uy36Bpno9x/ei7w9qxB/0='} 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55df966e3cc41f0-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.615 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000662fbf8a78d2ee98-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj36 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj36 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj36 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj36 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/YEKOkTH2BusWPJBE7NY/l4j+t4= 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/YEKOkTH2BusWPJBE7NY/l4j+t4='} 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b7ce4cb9b304de1-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.618 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ced3a59817072c1-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj37 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj37 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj37 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj37 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:KI5y/uV+E2pUOo4zB7YaqVFbYNo= 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KI5y/uV+E2pUOo4zB7YaqVFbYNo='} 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f48a79b58c5b93f-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.621 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b41e29557ffc663-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj38 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj38 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj38 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj38 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PzuxdGNTVxIPlqDh06fd87tjPC0= 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PzuxdGNTVxIPlqDh06fd87tjPC0='} 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fdd6974a2f8d429-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077045dd62349642e-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj39 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj39 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj39 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.625 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj39 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:7MbuZ6bXurlW8Ut2J874KNqDG3Y= 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7MbuZ6bXurlW8Ut2J874KNqDG3Y='} 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010108dc89d56e795-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cea192ce68d77fa1-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj40 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj40 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj40 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj40 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.628 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:W6ixUNIYY4ftRF7DN9JVUc1fmk0= 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:W6ixUNIYY4ftRF7DN9JVUc1fmk0='} 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dee24e9ebba98c12-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015c38ba45afbe306-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj41 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj41 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj41 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.631 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj41 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:tgW7sLlcwEfR4JUVj6+OeSjRNtE= 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tgW7sLlcwEfR4JUVj6+OeSjRNtE='} 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028c04e76afbdfed9-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098c8450d253c46a9-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj42 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj42 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj42 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.634 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj42 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/pe4u15kd9RA0AabMnDHTkoTWLo= 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/pe4u15kd9RA0AabMnDHTkoTWLo='} 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000984db9abcc7d592-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a389bd66c1d223b-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj43 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj43 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj43 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj43 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LM+x/eG1yzESj5NQ+krH885KhBo= 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LM+x/eG1yzESj5NQ+krH885KhBo='} 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae02a5dc27a64545-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.639 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a9b4d4f29aac1e-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj44 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj44 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj44 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj44 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:RPcmPLA3mR3bYNPlwvq2kaxC9Gc= 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RPcmPLA3mR3bYNPlwvq2kaxC9Gc='} 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e1d0b91d152d7ce-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.642 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089e97b50d79a2fc5-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj45 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj45 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj45 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj45 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4yy2USFfcVUZuSZ7zTN40gMR4oE= 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4yy2USFfcVUZuSZ7zTN40gMR4oE='} 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046326d1d986af1a0-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.645 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec702628bbf585dd-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj46 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj46 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj46 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj46 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:MbEo7Zzu9Kjk4E/2oYlkXwkrdAk= 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MbEo7Zzu9Kjk4E/2oYlkXwkrdAk='} 2024-04-13T23:18:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ed3b36cbe040c5f-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.648 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f676425fc24d408-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj47 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj47 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj47 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj47 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.650 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PieaNwV1eeQPCZESvJ0wyhk6lVU= 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PieaNwV1eeQPCZESvJ0wyhk6lVU='} 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dc492a5d1de083a-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.651 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055312720473c4ee0-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj48 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj48 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj48 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj48 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.653 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZOcnGstaSAfIep7FFFVRUl9fHN0= 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZOcnGstaSAfIep7FFFVRUl9fHN0='} 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd750a152801c0d5-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/ 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/ 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/ 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:/vwnfns-12/ 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk= 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5g8S860aZegxlJsRIMng/gCILdk=', 'Content-Length': '0'} 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064b5e8edb2c88cb2-00661af937-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-12/obj49 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-12/obj49 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-12/obj49 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:29:27 GMT 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:/vwnfns-12/obj49 2024-04-13T23:18:10.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZZTl5Bh5SKoOQR84W1IjnzHFRyU= 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:29:27 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZZTl5Bh5SKoOQR84W1IjnzHFRyU='} 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003720876414b00abf-00661af937-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:29:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043765.266939_90.1", 2024-04-13T23:18:10.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:29:25.266939Z" 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:10.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713043765.266939_90.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:28.043+0000 7f0f6e706780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:28.044+0000 7f0f6e706780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043765.266939_90.1", 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:29:25.266939Z", 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:29:28Z" 2024-04-13T23:18:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043765.266939_90.1", 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:29:25.266939Z", 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.669 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:29:28Z" 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713043765.266939_90.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:29:25.266939Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713043765.266939_90.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713043765.266939_90.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:28.155+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:33.156+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:38.157+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:43.157+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:48.158+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:53.158+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:29:58.158+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:03.159+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:08.160+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:13.161+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:18.162+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:23.162+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:28.162+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:33.163+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:38.163+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:43.164+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:48.166+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:53.166+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:30:58.167+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:03.167+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:08.167+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:13.168+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:18.168+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:23.169+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:28.170+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:33.171+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:38.172+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:43.172+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:48.172+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:53.173+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:31:58.174+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:03.174+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:08.176+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:13.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:18.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:23.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:28.177+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:33.178+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:38.178+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:43.180+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:48.181+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:53.181+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:32:58.182+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:03.181+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:08.182+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:13.183+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:18.184+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:23.185+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:28.186+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:33.186+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:38.187+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:43.187+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:48.187+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:53.188+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:33:58.188+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:03.190+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:08.190+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:13.191+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:18.191+0000 7febb9dfa780 1 waiting to reach incremental sync.. 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:23.191+0000 7febb9dfa780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:23.191+0000 7febb9dfa780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:34:23.191+0000 7febb9dfa780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.675 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:10.676 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-04-13T23:18:10.676 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:10.677 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:10.677 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=vwnfns-33 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-33/ 2024-04-13T23:18:10.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-33/ 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-33/ 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:42:52 GMT 2024-04-13T23:18:10.679 INFO:tasks.rgw_multisite_tests:/vwnfns-33/ 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TwFUNM3CUNf6qQA5gl837IDak+g= 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TwFUNM3CUNf6qQA5gl837IDak+g=', 'Content-Length': '0'} 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002efe663443d6da87-00661afc5c-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=vwnfns-34 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-34/ 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-34/ 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-34/ 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:10.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:42:52 GMT 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:/vwnfns-34/ 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rTdpTtLOBP025EudLB50Sl6ofRk= 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:42:52 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rTdpTtLOBP025EudLB50Sl6ofRk=', 'Content-Length': '0'} 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000998467682132c6b8-00661afc5c-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:42:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044572.504898_335.1", 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:42:52.504898Z" 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:42:52.707+0000 7f623d2a4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:42:52.708+0000 7f623d2a4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:42:52Z" 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.691 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.692 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.693 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:42:52Z" 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:42:57.810+0000 7fe97fbcd780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:42:57.811+0000 7fe97fbcd780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:42:57Z" 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.700 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.703 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:42:57Z" 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:02.922+0000 7f43cbece780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:02.923+0000 7f43cbece780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:02Z" 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.713 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:02Z" 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:08.041+0000 7fb38270b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:08.042+0000 7fb38270b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:08Z" 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.721 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.722 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:08Z" 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:13.143+0000 7f4245489780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:13.144+0000 7f4245489780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:13Z" 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.732 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.735 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.736 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:13Z" 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:18.251+0000 7f16db0db780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:18.252+0000 7f16db0db780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:18Z" 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.743 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.746 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.747 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:18Z" 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:23.362+0000 7f7d48b4d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:23.363+0000 7f7d48b4d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:23Z" 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.754 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.757 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.758 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:23Z" 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:28.488+0000 7fc4b1626780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:28.489+0000 7fc4b1626780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:28Z" 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.765 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.768 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.769 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:28Z" 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:33.596+0000 7fbf67696780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:33.597+0000 7fbf67696780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.914750_265.1", 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.914750Z", 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:33Z" 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.914750_265.1", 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.914750Z", 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.777 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.778 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:33Z" 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.914750_265.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.914750Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.914750_265.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.914750_265.1 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:38.720+0000 7f47ddc26780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:38.721+0000 7f47ddc26780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:38Z" 2024-04-13T23:18:10.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.787 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.788 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.791 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:38Z" 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:43.828+0000 7f1fe2f61780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:43.829+0000 7f1fe2f61780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:43Z" 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.798 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.799 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.800 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.801 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.802 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:43Z" 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:48.936+0000 7fab47053780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:48.937+0000 7fab47053780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:48Z" 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.809 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.812 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.813 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:48Z" 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:54.045+0000 7f698181c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:54.046+0000 7f698181c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:54Z" 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.820 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.825 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:54Z" 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:59.163+0000 7f8f6a246780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:43:59.164+0000 7f8f6a246780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:43:59Z" 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.832 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.833 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.835 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.836 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:43:59Z" 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:04.274+0000 7f0e1134a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:04.276+0000 7f0e1134a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:04Z" 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.846 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.847 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:04Z" 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:09.381+0000 7f4c2b1b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:09.382+0000 7f4c2b1b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:09Z" 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.854 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.855 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.856 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.858 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:09Z" 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:14.492+0000 7f67e081f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:14.493+0000 7f67e081f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:14Z" 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.867 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.869 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:14Z" 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:19.600+0000 7f54ae26f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:19.601+0000 7f54ae26f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:19Z" 2024-04-13T23:18:10.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.878 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.879 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:19Z" 2024-04-13T23:18:10.880 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:24.707+0000 7fb16a85d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:24.708+0000 7fb16a85d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:24Z" 2024-04-13T23:18:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.888 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.891 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:24Z" 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:29.816+0000 7ff15743c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:29.817+0000 7ff15743c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:29Z" 2024-04-13T23:18:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.898 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.902 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:29Z" 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:34.925+0000 7f00da560780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:34.926+0000 7f00da560780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:34Z" 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.910 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:34Z" 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:40.093+0000 7f8f01924780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:40.094+0000 7f8f01924780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:40Z" 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.922 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.925 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:40Z" 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.926 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:45.204+0000 7f2064dd8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:45.205+0000 7f2064dd8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:45Z" 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.936 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.937 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:45Z" 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:50.318+0000 7f0bbd0fa780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:50.319+0000 7f0bbd0fa780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:50Z" 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.944 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.948 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:50Z" 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:55.431+0000 7f124ec27780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:44:55.432+0000 7f124ec27780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:44:55Z" 2024-04-13T23:18:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.956 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:44:55Z" 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.960 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:00.544+0000 7f0d12fe5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:00.545+0000 7f0d12fe5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:00Z" 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.966 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.967 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.970 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:00Z" 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:05.649+0000 7fb29c11c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:05.650+0000 7fb29c11c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:05Z" 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.977 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.978 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.980 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.981 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:05Z" 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.982 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:10.757+0000 7f14ed4ae780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:10.758+0000 7f14ed4ae780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:10Z" 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.988 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.989 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.990 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:10.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:10.992 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:10Z" 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:10.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:15.867+0000 7f638f372780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:15.867+0000 7f638f372780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:10.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:10.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:10.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:15Z" 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:10.999 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.000 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.001 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.002 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:15Z" 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:20.975+0000 7fb978b0c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:20.976+0000 7fb978b0c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.959035_270.1", 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.959035Z", 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:20Z" 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.010 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.959035_270.1", 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.959035Z", 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.011 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.013 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.014 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:20Z" 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.959035_270.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.959035Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.959035_270.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.959035_270.1 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:26.085+0000 7fb29b872780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:26.086+0000 7fb29b872780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.996146_276.1", 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.996146Z", 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:26Z" 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.996146_276.1", 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.996146Z", 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:26Z" 2024-04-13T23:18:11.026 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.996146_276.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.996146Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.996146_276.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.996146_276.1 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:31.193+0000 7f53f1352780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:31.194+0000 7f53f1352780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:31Z" 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.033 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.034 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.035 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.036 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.037 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:31Z" 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:36.324+0000 7f64c70c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:36.326+0000 7f64c70c2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:36Z" 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.044 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.047 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.048 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.049 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:36Z" 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:41.442+0000 7f73ae408780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:41.443+0000 7f73ae408780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:41Z" 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.057 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.058 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.059 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.060 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:41Z" 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:46.553+0000 7fc234291780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:46.554+0000 7fc234291780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:46Z" 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.067 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:46Z" 2024-04-13T23:18:11.071 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:51.663+0000 7f04ae962780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:51.664+0000 7f04ae962780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:51Z" 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.078 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.079 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:51Z" 2024-04-13T23:18:11.082 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:56.775+0000 7ff41c41f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:45:56.776+0000 7ff41c41f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:45:56Z" 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.089 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.090 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.091 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:45:56Z" 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:01.885+0000 7f83041c9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:01.886+0000 7f83041c9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:01Z" 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.100 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.104 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:01Z" 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:06.994+0000 7fc2220c5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:06.995+0000 7fc2220c5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:06Z" 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.114 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:06Z" 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.115 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:12.103+0000 7f4ac4065780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:12.104+0000 7f4ac4065780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:12Z" 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.121 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.122 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.125 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:12Z" 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:17.216+0000 7f7e74fd6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:17.217+0000 7f7e74fd6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:17Z" 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.133 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.134 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.136 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:17Z" 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:22.324+0000 7ff0587f9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:22.325+0000 7ff0587f9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:22Z" 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.144 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.145 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.146 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.147 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:22Z" 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:27.434+0000 7f6e8af4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:27.435+0000 7f6e8af4a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:27Z" 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.155 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.156 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.157 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.158 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:27Z" 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:32.543+0000 7fbfccf86780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:32.544+0000 7fbfccf86780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:32Z" 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.165 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.166 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.167 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.168 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.169 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:32Z" 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:37.650+0000 7fda5cb2f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:37.651+0000 7fda5cb2f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:37Z" 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.176 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.177 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.178 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.179 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.180 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:37Z" 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:42.763+0000 7f144b33a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:42.764+0000 7f144b33a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:42Z" 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.187 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.188 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.191 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:42Z" 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:47.875+0000 7f1d42a4e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:47.876+0000 7f1d42a4e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:47Z" 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.198 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.199 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.200 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.201 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:47Z" 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:52.985+0000 7f23cd769780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:52.986+0000 7f23cd769780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:52Z" 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.210 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.212 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.213 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:52Z" 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:58.095+0000 7fbe33ac1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:46:58.096+0000 7fbe33ac1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:46:58Z" 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.220 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.221 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.222 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.223 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:46:58Z" 2024-04-13T23:18:11.224 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:03.203+0000 7f6e4ae8d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:03.204+0000 7f6e4ae8d780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:03Z" 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.230 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.231 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.232 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.234 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:03Z" 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:08.310+0000 7f2690ed2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:08.311+0000 7f2690ed2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:08Z" 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.242 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.243 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.244 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.245 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:08Z" 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:13.420+0000 7f4b02ec9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:13.421+0000 7f4b02ec9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:13Z" 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.252 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.253 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.254 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.255 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.256 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:13Z" 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:18.530+0000 7fe1619f8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:18.531+0000 7fe1619f8780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:18Z" 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.263 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.264 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.265 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.266 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.267 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:18Z" 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:23.642+0000 7f465de99780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:23.643+0000 7f465de99780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:23Z" 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.274 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.276 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.277 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:23Z" 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.278 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:28.751+0000 7feff146c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:28.752+0000 7feff146c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:28Z" 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.285 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:28Z" 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.289 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:33.862+0000 7fa9d23d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:33.863+0000 7fa9d23d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:33Z" 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.295 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.298 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:33Z" 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:38.974+0000 7f54b12e6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:38.975+0000 7f54b12e6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:38Z" 2024-04-13T23:18:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.307 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.309 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.311 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:38Z" 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:44.081+0000 7f34ba7d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:44.082+0000 7f34ba7d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:44Z" 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.318 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.321 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.322 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:44Z" 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:49.185+0000 7f1fd9925780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:49.186+0000 7f1fd9925780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:49Z" 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.329 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.330 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.331 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:49Z" 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:54.301+0000 7f32f1452780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:54.302+0000 7f32f1452780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:54Z" 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.340 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.341 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.343 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.344 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:54Z" 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044572.504898_335.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044572.504898_335.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.345 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:11.346 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.346 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:11.346 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-04-13T23:18:11.346 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:11.347 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:11.347 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:11.347 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 1018, in test_datalog_autotrim 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=vwnfns-35 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-35/ 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-35/ 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-35/ 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:47:59 GMT 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:/vwnfns-35/ 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:YkP6OQzkGnFOwzIAr2QN+4tlpb0= 2024-04-13T23:18:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YkP6OQzkGnFOwzIAr2QN+4tlpb0=', 'Content-Length': '0'} 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b6c2de908c3b0fec-00661afd8f-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=vwnfns-36 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-36/ 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-36/ 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-36/ 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:47:59 GMT 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:/vwnfns-36/ 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LcncnaVOnKQBeFrru1rzIhQull0= 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LcncnaVOnKQBeFrru1rzIhQull0=', 'Content-Length': '0'} 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a7e587c5fcd64aa4-00661afd8f-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-35/ 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-35/ 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-35/ 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:47:59 GMT 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:/vwnfns-35/ 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:11.352 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uD6aFgnFJsP9CJrm07jKmvHOvDs= 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uD6aFgnFJsP9CJrm07jKmvHOvDs=', 'Content-Length': '0'} 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004564892b0141f534-00661afd8f-1114-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-35/key 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-35/key 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-35/key 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:47:59 GMT 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:/vwnfns-35/key 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qoJI+MPOzNdO1cx9XMFpcI8J6QM= 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qoJI+MPOzNdO1cx9XMFpcI8J6QM='} 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b14c72695ea7d20c-00661afd8f-1114-a1'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-36/ 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-36/ 2024-04-13T23:18:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-36/ 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:47:59 GMT 2024-04-13T23:18:11.355 INFO:tasks.rgw_multisite_tests:/vwnfns-36/ 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qJz86Cs+LDqbscXoua5RLtEkIaY= 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qJz86Cs+LDqbscXoua5RLtEkIaY=', 'Content-Length': '0'} 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003887dbd4dcc447da-00661afd8f-1110-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-36/key 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-36/key 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-36/key 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 21:47:59 GMT 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:/vwnfns-36/key 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:KiwxsnnBbuu6ywOY8b/81L8hPz4= 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 21:47:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KiwxsnnBbuu6ywOY8b/81L8hPz4='} 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2121689dac637cc-00661afd8f-1110-a2'), ('Date', 'Sat, 13 Apr 2024 21:47:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044879.371767_361.1", 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:47:59.371767Z" 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:59.622+0000 7f8fd204f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:47:59.623+0000 7f8fd204f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:47:59Z" 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.368 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:47:59Z" 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:04.735+0000 7f7efc3b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:04.748+0000 7f7efc3b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:04Z" 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.375 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.378 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.379 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:04Z" 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T21:48:08.258+0000 7f5ea15cd700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:09.858+0000 7f4da19cc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:09.859+0000 7f4da19cc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:09Z" 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.386 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.387 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.389 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.390 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:09Z" 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:14.966+0000 7f6b83602780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:14.967+0000 7f6b83602780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:14Z" 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.397 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.398 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.400 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.401 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:14Z" 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:20.075+0000 7f9e3ffe4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:20.076+0000 7f9e3ffe4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:20Z" 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.409 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.411 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:20Z" 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:25.187+0000 7fde632c6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:25.188+0000 7fde632c6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:25Z" 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.421 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.422 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.423 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:25Z" 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:30.295+0000 7f07af70c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:30.296+0000 7f07af70c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:30Z" 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.430 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.431 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:30Z" 2024-04-13T23:18:11.434 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:35.407+0000 7fb05ca10780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:35.408+0000 7fb05ca10780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:35Z" 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:35Z" 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.445 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:40.517+0000 7fa530f32780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:40.518+0000 7fa530f32780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:40Z" 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:40Z" 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.456 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:45.629+0000 7fb0199ba780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:45.630+0000 7fb0199ba780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:45Z" 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.463 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.466 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:45Z" 2024-04-13T23:18:11.467 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:50.739+0000 7f3741b1b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:50.740+0000 7f3741b1b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:50Z" 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.473 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.474 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.477 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:50Z" 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.478 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:55.850+0000 7f3926700780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:48:55.851+0000 7f3926700780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:48:55Z" 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.485 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.487 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.488 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:48:55Z" 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:00.957+0000 7f4564909780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:00.958+0000 7f4564909780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:00Z" 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.495 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.496 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.497 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.498 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.499 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:00Z" 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:06.066+0000 7f04201a2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:06.067+0000 7f04201a2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:06Z" 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.506 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.509 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.510 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:06Z" 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:11.177+0000 7ff70f6e5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:11.179+0000 7ff70f6e5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:11Z" 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.520 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.521 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:11Z" 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:11.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:16.297+0000 7f123768c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:16.298+0000 7f123768c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:16Z" 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.530 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.532 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.533 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:16Z" 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:11.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:21.405+0000 7fe3c5f68780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:21.406+0000 7fe3c5f68780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:21Z" 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.540 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.544 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:21Z" 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.545 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:26.514+0000 7f07f407c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:26.515+0000 7f07f407c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:26Z" 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.551 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.552 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.553 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.555 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:26Z" 2024-04-13T23:18:11.556 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:31.624+0000 7ff730ae2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:31.626+0000 7ff730ae2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:31Z" 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.563 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.565 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.566 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.567 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:31Z" 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:36.737+0000 7f2485157780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:36.738+0000 7f2485157780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.872147_258.1", 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.872147Z", 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:36Z" 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.574 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.872147_258.1", 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.872147Z", 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.575 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.576 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.577 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:36Z" 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.872147_258.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.872147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.872147_258.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.872147_258.1 2024-04-13T23:18:11.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:41.848+0000 7f3118dd4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:41.849+0000 7f3118dd4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:41Z" 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.586 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:41Z" 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:46.960+0000 7fe9b9a53780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:46.962+0000 7fe9b9a53780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:46Z" 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.596 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.599 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.600 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:46Z" 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:52.072+0000 7f3df91ec780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:52.074+0000 7f3df91ec780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:52Z" 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.609 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.611 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:52Z" 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:57.187+0000 7fca0e321780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:49:57.188+0000 7fca0e321780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:49:57Z" 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.618 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.620 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.622 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:49:57Z" 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:02.298+0000 7fb5e4b95780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:02.299+0000 7fb5e4b95780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:02Z" 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.629 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.630 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:02Z" 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:07.410+0000 7f1d587b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:07.412+0000 7f1d587b2780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:07Z" 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.641 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:07Z" 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:12.524+0000 7f8a078b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:12.525+0000 7f8a078b1780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:12Z" 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.651 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.654 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:12Z" 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:17.638+0000 7f0bfb063780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:17.639+0000 7f0bfb063780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:17Z" 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.662 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.665 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:17Z" 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:22.744+0000 7f8d53a79780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:22.745+0000 7f8d53a79780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:22Z" 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.676 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:22Z" 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:27.854+0000 7f16c5b97780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:27.854+0000 7f16c5b97780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:27Z" 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.684 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.686 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.687 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:27Z" 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:32.965+0000 7ff4df078780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:32.966+0000 7ff4df078780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:32Z" 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.697 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.698 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.699 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:32Z" 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:38.075+0000 7f7f5c1f4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:38.076+0000 7f7f5c1f4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:38Z" 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.706 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.707 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.708 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.710 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:38Z" 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:43.191+0000 7fa0337a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:43.192+0000 7fa0337a9780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:43Z" 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.717 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.718 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.719 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.721 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:43Z" 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:48.302+0000 7f87e0aab780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:48.303+0000 7f87e0aab780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:48Z" 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.732 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:48Z" 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:53.416+0000 7fa302afc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:53.417+0000 7fa302afc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:53Z" 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.739 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.740 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.742 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.743 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:53Z" 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:58.523+0000 7f9a9af69780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:50:58.524+0000 7f9a9af69780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044569.139905_297.1", 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:49.139905Z", 2024-04-13T23:18:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:50:58Z" 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044569.139905_297.1", 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:49.139905Z", 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.751 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.752 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.753 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.754 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:50:58Z" 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.139905_297.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.139905Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044569.139905_297.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044569.139905_297.1 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:03.633+0000 7f4c50d6e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:03.634+0000 7f4c50d6e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:03Z" 2024-04-13T23:18:11.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.763 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:03Z" 2024-04-13T23:18:11.765 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:08.746+0000 7f23caa34780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:08.747+0000 7f23caa34780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:08Z" 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:08Z" 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.776 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:13.850+0000 7f96cc461780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:13.851+0000 7f96cc461780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:13Z" 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.787 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:13Z" 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:18.966+0000 7ff439983780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:18.968+0000 7ff439983780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:18Z" 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.796 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.798 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:18Z" 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:24.076+0000 7f1b9e521780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:24.078+0000 7f1b9e521780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:24Z" 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.807 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.809 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:24Z" 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:29.187+0000 7fae5576a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:29.188+0000 7fae5576a780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:29Z" 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.817 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.820 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:29Z" 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:34.297+0000 7f1aadfab780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:34.298+0000 7f1aadfab780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:34Z" 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.827 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.831 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:34Z" 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:39.410+0000 7fcabd680780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:39.412+0000 7fcabd680780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:39Z" 2024-04-13T23:18:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.912549_264.1", 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.912549Z", 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:39Z" 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.912549_264.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.912549Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.912549_264.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.912549_264.1 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:44.530+0000 7fc6c387c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:44.531+0000 7fc6c387c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:44Z" 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.848 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.850 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.851 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.852 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:44Z" 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:49.634+0000 7f5d4ecf4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:49.635+0000 7f5d4ecf4780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:49Z" 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.860 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.861 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.862 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.863 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:49Z" 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:54.745+0000 7fc57b3d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:54.746+0000 7fc57b3d0780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:54Z" 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.870 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.871 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.872 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:54Z" 2024-04-13T23:18:11.874 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:59.854+0000 7f4725335780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:51:59.856+0000 7f4725335780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:51:59Z" 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.881 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.882 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.883 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:51:59Z" 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:04.966+0000 7f1437587780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:04.967+0000 7f1437587780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:04Z" 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.892 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.895 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:04Z" 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:10.077+0000 7f2c40115780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:10.078+0000 7f2c40115780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:10Z" 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.903 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:10Z" 2024-04-13T23:18:11.907 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:15.187+0000 7ff6e8f8f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:15.188+0000 7ff6e8f8f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044569.137566_296.1", 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:49.137566Z", 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:15Z" 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.913 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044569.137566_296.1", 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.914 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:49.137566Z", 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.916 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.917 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:15Z" 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044569.137566_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:49.137566Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044569.137566_296.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044569.137566_296.1 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:20.301+0000 7fb7f7bcc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:20.302+0000 7fb7f7bcc780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:20Z" 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.924 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.926 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.927 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.928 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.929 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.929 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.929 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.929 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.929 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:20Z" 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:25.409+0000 7fe09a1d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:25.410+0000 7fe09a1d6780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:25Z" 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.936 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.940 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:25Z" 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:30.521+0000 7fc9aaf36780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:30.522+0000 7fc9aaf36780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:30Z" 2024-04-13T23:18:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.947 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.948 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:30Z" 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.951 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:35.630+0000 7ff1271b5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:35.631+0000 7ff1271b5780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:35Z" 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.958 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.959 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.962 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:35Z" 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:40.743+0000 7f3c24a06780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:40.744+0000 7f3c24a06780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:40Z" 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.972 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:40Z" 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:45.862+0000 7fd2835df780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:45.863+0000 7fd2835df780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:45Z" 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.982 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.983 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:45Z" 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:50.989+0000 7f22e740f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:50.990+0000 7f22e740f780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:50Z" 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:11.990 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.907592_262.1", 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.991 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.907592Z", 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.993 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:11.994 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:50Z" 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:11.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.907592_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.907592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.907592_262.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.907592_262.1 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:56.101+0000 7f2501127780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:52:56.102+0000 7f2501127780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:11.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:11.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:52:56Z" 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.001 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.002 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.004 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:52:56Z" 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:53:01.207+0000 7f6dfc5e7780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T21:53:01.208+0000 7f6dfc5e7780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T21:53:01Z" 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.013 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1713044568.938952_266.1", 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:42:48.938952Z", 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.014 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.015 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.016 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.017 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T21:53:01Z" 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713044568.938952_266.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T21:42:48.938952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713044879.371767_361.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713044568.938952_266.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713044879.371767_361.1 target=1_1713044568.938952_266.1 2024-04-13T23:18:12.018 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:12.019 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.019 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:12.019 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-04-13T23:18:12.019 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:12.020 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:12.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:12.020 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:12.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 1451, in test_bucket_reshard_index_log_trim 2024-04-13T23:18:12.020 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2024-04-13T23:18:12.020 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=vwnfns-57 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ov85PUqSEDPlxopFG5ylrDiB8jg= 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ov85PUqSEDPlxopFG5ylrDiB8jg=', 'Content-Length': '0'} 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000dd8b2510be33dda6-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.023 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bc7da754c8fa49c-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.025 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:h7YSzn94GEN8azPqxooSz+jzoxs= 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h7YSzn94GEN8azPqxooSz+jzoxs='} 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ae220d1fc2a5685-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe7eaa3461ee382-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.028 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:asnqkugpRzoWoJrYPKYh1DnxrXo= 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:asnqkugpRzoWoJrYPKYh1DnxrXo='} 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cf2fbb8750ceea5-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e837cb57ad324e1f-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.031 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ER4Vanx0ImZEQZuWLSGy+M5kx8c= 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ER4Vanx0ImZEQZuWLSGy+M5kx8c='} 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfeebfd1e72f5ee3-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08= 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8uuLKubkPxu7XsdGi+RmfSCKm08=', 'Content-Length': '0'} 2024-04-13T23:18:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bbc43acdb0d751b-00661b0247-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.034 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:08:07 GMT 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZdgvwTWPT8kl0WyNUEProoE1FO8= 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:08:07 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZdgvwTWPT8kl0WyNUEProoE1FO8='} 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55c38057d55b8e5-00661b0247-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713046087.430279_694.1", 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:08:07.430279Z" 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "0.000000" 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:07.724+0000 7fe270341780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:07.725+0000 7fe270341780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713046025.344683_686.1", 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:07:05.344683Z", 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T22:08:07Z" 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1713046025.344683_686.1", 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:07:05.344683Z", 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T22:08:07Z" 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046025.344683_686.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:07:05.344683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:12.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713046025.344683_686.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713046087.430279_694.1 target=1_1713046025.344683_686.1 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:12.835+0000 7fbeca613780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:12.836+0000 7fbeca613780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713046087.430279_694.1", 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:08:07.430279Z", 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "", 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "0.000000", 2024-04-13T23:18:12.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T22:08:12Z" 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.053 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1713046087.430279_694.1", 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:08:07.430279Z", 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.057 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T22:08:12Z" 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713046087.430279_694.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:08:07.430279Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713046087.430279_694.1', 1: '1_1713043708.018806_80.1', 2: '', 3: ''} 2024-04-13T23:18:12.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-57 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:12.949+0000 7f398aca4780 1 bucket sync caught up with source: 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, 00000000001.47.6, , , , 00000000001.34.6, , , 00000000001.28.6, , 00000000001.23.6] 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, 00000000001.47.6, , , , 00000000001.34.6, , , 00000000001.28.6, , 00000000001.23.6] 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:12.949+0000 7f398aca4780 0 bucket checkpoint complete 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=vwnfns-57 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:13.051+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:18.052+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:23.052+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:28.053+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:33.055+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:38.055+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:43.056+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:48.055+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:53.056+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:08:58.056+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:03.057+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:08.059+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:13.059+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:18.060+0000 7fd75bbdd780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:23.062+0000 7fd75bbdd780 1 bucket sync caught up with source: 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:23.062+0000 7fd75bbdd780 0 bucket checkpoint complete 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=vwnfns-57 zones={a2, a1} 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE= 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE=', 'Content-Length': '0'} 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018f6e83d1036b3e3-00661b0293-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE= 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ClwkJUJ/7PzVN9oBYRes71hojgE=', 'Content-Length': '0'} 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2b93d2473d92528-00661b0293-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e64cc12fd926ff8-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-13T23:18:12.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c739d51e2ecce89-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.068 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T23:18:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8aed40b590df873-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920= 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4lowEhNvqU2bIelY9ch0rxil920=', 'Content-Length': '0'} 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177ad64d252bf4a3-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.073 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa592f9d0128ed4f-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T23:18:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000641b0d96c7cb370-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.076 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8= 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8=', 'Content-Length': '0'} 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a3316ecf787d4a-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.078 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8= 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Sk15wMEHrS8UL3SLKSytnvz9hM8=', 'Content-Length': '0'} 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100751f6e10f687d-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a59c3296cd4483-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.081 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU= 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sVv0boj79KfPllCmzPlTZz7j6UU=', 'Content-Length': '0'} 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f917cc4e2e710bd5-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd8f70eeb243c1b7-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bd124721035a6e-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.086 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw= 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw=', 'Content-Length': '0'} 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bfdc05510c6ce95-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw= 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxXM3fbTCpvTsuKKEOutHwJTcFw=', 'Content-Length': '0'} 2024-04-13T23:18:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c797a048c8dccbac-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.089 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE= 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE=', 'Content-Length': '0'} 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9a1ca71f39852d-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.091 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE= 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kLGSZvgupqXE7zkNNKGLuCHLYjE=', 'Content-Length': '0'} 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb1fa909cacbe34-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bba58ed5f5f9b3e-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.094 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d789aa53c00cad0-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.096 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e11f5df7a02a8918-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e8a693d07eea4f-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.099 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY= 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY=', 'Content-Length': '0'} 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000217c270935dd893e-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY= 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rMXUByeujj7NM75mEyMDQ9hsVqY=', 'Content-Length': '0'} 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec64f0ddfc72fec9-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.102 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d17aba8c5b0e268-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk= 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:edH9d9a2lDa1kopLod3X+i7aLzk=', 'Content-Length': '0'} 2024-04-13T23:18:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ed97308b0e3ce8-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2363773e224c26-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.107 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b4203cfc924aa2-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY= 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY=', 'Content-Length': '0'} 2024-04-13T23:18:12.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6130b7ee07f7ad-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.110 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY= 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D7IGNjCnGV8AwlZWP/YIKs0rTIY=', 'Content-Length': '0'} 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5991af13725eea-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34= 2024-04-13T23:18:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34=', 'Content-Length': '0'} 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075187101dae670ee-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.113 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34= 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Tof1nQrEA0rBOqjZDmV0Mz34G34=', 'Content-Length': '0'} 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0f347b6442c9d1-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.115 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076441db18ae42b5c-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8474c9a20d1ac50-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dfb3a39196a4f77-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.120 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e345c9a06880055-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.122 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI= 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI=', 'Content-Length': '0'} 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002adc289a8c6b7b6b-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI= 2024-04-13T23:18:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vFdVS4q2q/xn0Pm3cttDeUxtQLI=', 'Content-Length': '0'} 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f4fdb66ca5c78b9-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117ddc3f76167c72-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.127 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI= 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:q4fXPFGUmOzFphk0mNTCyJVIowI=', 'Content-Length': '0'} 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f97194d6ba02dd26-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e575a97689be5d41-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e866a6b334fd8942-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.133 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0= 2024-04-13T23:18:12.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0=', 'Content-Length': '0'} 2024-04-13T23:18:12.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adbc740c27de6edd-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.134 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0= 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+mtxZUoD0In2+pIrrlfD9RwTYr0=', 'Content-Length': '0'} 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c2f93e71ead2b6-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc= 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc=', 'Content-Length': '0'} 2024-04-13T23:18:12.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fcca34f88ba9e66-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.137 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc= 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y6fzDcv+CmloFBzKl3WgafTS8Sc=', 'Content-Length': '0'} 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ec2557176001f6-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b427db4c78d0777-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbcc7104e6f20edc-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.142 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003461cbdd2fa92e-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512eeeacfbb4dee0-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.145 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM= 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM=', 'Content-Length': '0'} 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da326be6dfe04cc-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM= 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pvYHI7gKnyoS5HYtikuZOGjiXvM=', 'Content-Length': '0'} 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b718339a4fae3b-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.147 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.148 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227dcefcd949b129-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ= 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWQ0o8o1JvF5f3vRqh3h54SY4UQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3d619212ae97a7-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.151 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ac377a1a2ddb4e-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9f67f8624c0d9e-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.154 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM= 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM=', 'Content-Length': '0'} 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27aa161780ae2a8-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM= 2024-04-13T23:18:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:peljp5r8rFVGtVp2wXapoM31czM=', 'Content-Length': '0'} 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56bbcf0f2d6a788-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.157 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0= 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0=', 'Content-Length': '0'} 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b3803fee8694f0-00661b0293-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0= 2024-04-13T23:18:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X1S9csIEw0c1lG5B0xlqpITVwx0=', 'Content-Length': '0'} 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f06752da5d478b6-00661b0293-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fade15e8298dfbc-00661b0293-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:23 GMT 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0= 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:23 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uhMUAMW/B76cxWfw91OXcz2Wal0=', 'Content-Length': '0'} 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb9c52f019a48d2f-00661b0293-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=vwnfns-57 zones={a2, a1} 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding": "None", 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_index": { 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "Normal", 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "normal": { 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 11, 2024-04-13T23:18:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "logs": [ 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 11, 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-57 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_id": "1#00000000001.47.6", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.6548832894401009285", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op": "write", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "object": "a", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance": "", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": "complete", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "index_ver": 1, 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:08:07.439673791Z", 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "ver": { 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pool": 9, 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "epoch": 36 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bilog_flags": 0, 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "versioned": false, 2024-04-13T23:18:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner": "", 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner_display_name": "", 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones_trace": [ 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_id": "5#00000000001.34.6", 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.11915460310344251621", 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op": "write", 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "object": "d", 2024-04-13T23:18:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance": "", 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": "complete", 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "index_ver": 1, 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:08:07.549016590Z", 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "ver": { 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pool": 9, 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "epoch": 11 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bilog_flags": 0, 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "versioned": false, 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner": "", 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner_display_name": "", 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones_trace": [ 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_id": "8#00000000001.28.6", 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.16136045423162777315", 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op": "write", 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "object": "c", 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance": "", 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": "complete", 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "index_ver": 1, 2024-04-13T23:18:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:08:07.498897639Z", 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "ver": { 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pool": 9, 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "epoch": 5 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bilog_flags": 0, 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "versioned": false, 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner": "", 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner_display_name": "", 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones_trace": [ 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_id": "10#00000000001.23.6", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op_tag": "f6f208d6-f791-479b-bc3d-112512eec2d4.6690.4391849356295990949", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "op": "write", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "object": "b", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance": "", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": "complete", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "index_ver": 1, 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:08:07.489738979Z", 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "ver": { 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pool": 9, 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "epoch": 11 2024-04-13T23:18:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bilog_flags": 0, 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "versioned": false, 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner": "", 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "owner_display_name": "", 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones_trace": [ 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "entry": "f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20" 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-57 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 4 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding": "None", 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_index": { 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "Normal", 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "normal": { 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "logs": [ 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 11, 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'vwnfns-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket vwnfns-57 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [] 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004eb49a91ab4e79e2-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.180 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:OUAB/fpetK/jnb8oBKCCCQpOdZo= 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OUAB/fpetK/jnb8oBKCCCQpOdZo='} 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001b07c0f4c7fccb7-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001edcd9ce6f4c027f-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2rUIYQUJoLvt2CZMxbRs6ZJwMeY= 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2rUIYQUJoLvt2CZMxbRs6ZJwMeY='} 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000618153d25d485cf5-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8101fb934e8863c-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.186 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rkcWU3NNbWfVvTiA5hEtgScz9Zo= 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rkcWU3NNbWfVvTiA5hEtgScz9Zo='} 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054c6dc90c45b5278-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU= 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VWUqi5S5LJoq35N+FyYtRMJ9ccU=', 'Content-Length': '0'} 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000280614d52e6bb0eb-00661b0294-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.189 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:24 GMT 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:YRCkaFSDh4OcbCyNd6wGUSfMKh0= 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:09:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YRCkaFSDh4OcbCyNd6wGUSfMKh0='} 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4217c2e143d2b6f-00661b0294-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-57 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:24.236+0000 7fd125eb8780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:29.236+0000 7fd125eb8780 1 waiting to reach latest gen 1, on 0.. 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:34.237+0000 7fd125eb8780 1 bucket sync caught up with source: 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [00000000003.39.6, 00000000002.76.6, 00000000002.26.6] 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.76.6, 00000000002.26.6] 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:34.237+0000 7fd125eb8780 0 bucket checkpoint complete 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=vwnfns-57 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:34.352+0000 7f9ea556a780 1 waiting for incremental sync to catch up: 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:39.353+0000 7f9ea556a780 1 waiting for incremental sync to catch up: 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, 00000000001.555.6, , , , 00000000001.2205.6, , , 00000000001.393.6, , 00000000001.1216.6] 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:44.355+0000 7f9ea556a780 1 bucket sync caught up with source: 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:44.355+0000 7f9ea556a780 0 bucket checkpoint complete 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=vwnfns-57 zones={a2, a1} 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.193 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE= 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE=', 'Content-Length': '0'} 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090bc245c1cfff698-00661b02a8-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE= 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:c9Y350n3F07is+7Iiv7TQWhVAgE=', 'Content-Length': '0'} 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924adc1b1847d267-00661b02a8-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-13T23:18:12.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.196 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7fb875f8500294-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-13T23:18:12.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b02a4e7a926c06b-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009748040814e3a907-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ= 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:skLRUckzSp3ks/6pvQ62MKrP8QQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e0ee2df8e70b65-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.204 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd2785b0671236f-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ce41814f24ed65-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.207 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk= 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk=', 'Content-Length': '0'} 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a24ec7033e5a41-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk= 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VBNusvKtYE3o3DR7BMC/j4iQXmk=', 'Content-Length': '0'} 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a34d7daddbd3be6-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.210 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075b14febc129642-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw= 2024-04-13T23:18:12.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:k26CkSv6qWuVz9WoIz/S0d2t3Gw=', 'Content-Length': '0'} 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc9dd875eb92e81-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e10d966b020a864-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.215 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da0875d75acbc09-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8= 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8=', 'Content-Length': '0'} 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066de601c0777cdda-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.218 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8= 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:P9KBKjt+UhKW98O22Fq8n2C45a8=', 'Content-Length': '0'} 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000048c6f23ad779113-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I= 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I=', 'Content-Length': '0'} 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eaeccc3d14fddc7-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I= 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qHwA2gisNbJFBO+EPzvEoIgvE9I=', 'Content-Length': '0'} 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2c9f413065f18d-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.223 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b874d5c89b2e2fd9-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008649a75081eae40e-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008166c99d80e4304d-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.228 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb9f03cd1af0ad3-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8= 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8=', 'Content-Length': '0'} 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef510ee3a841391-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.231 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8= 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SwSSi9S57UFmnqQodAgY/ONLfk8=', 'Content-Length': '0'} 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003409b020ffee8536-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f7ff236b4140bf-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.234 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE= 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZqR/Sr4EbowOL1rBbVkwiUWDHEE=', 'Content-Length': '0'} 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70e9dcea4fdf7eb-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f9ed0d9e8dbcc6-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f8567a6209aee15-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE= 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE=', 'Content-Length': '0'} 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06dc4fb2b8a23b6-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE= 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4Auwqau40SIAfvt96T2ePs43KsE=', 'Content-Length': '0'} 2024-04-13T23:18:12.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db504cbb7c92daf-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.242 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0= 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0=', 'Content-Length': '0'} 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009080dc5b3604a311-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0= 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wSDOoEep4NmfpJtLM6z6IWB04q0=', 'Content-Length': '0'} 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e5bbbdad7debd7-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.245 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbafd9673d179c20-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.247 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14e4dbf926edf09-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e73044c8873baad-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e92524ad668ddbf-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.252 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg= 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg=', 'Content-Length': '0'} 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396162906b972d6c-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg= 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:L/CtbqUtzLSPOkWai8ms2TB8igg=', 'Content-Length': '0'} 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d6fb3306b26f17-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.254 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.255 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f14101ac01dd45-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q= 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:s73MiDTH7dcKIzM4R7hTJX9nk7Q=', 'Content-Length': '0'} 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc07adc8e8569af-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c82fb13645945af-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e936ebd884a0068-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg= 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg=', 'Content-Length': '0'} 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2a230fa81f50beb-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg= 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f7fsaUxU/ncJS1VjzR5GaexZsSg=', 'Content-Length': '0'} 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d43c104215b7878-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs= 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs=', 'Content-Length': '0'} 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446a80ec46c354fd-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs= 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hwDhi0P38Hj8vBFwrd2bTc1BUxs=', 'Content-Length': '0'} 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999439e6be7c5ad9-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aad8350605c808f-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf6e8916661cf58d-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ffe126ec2e0838d-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.274 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001325b5d654f3a8cb-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY= 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY=', 'Content-Length': '0'} 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b93b9009089ab3-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.277 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY= 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e+xjrxtVGcuOJZ+nFg1UlISfabY=', 'Content-Length': '0'} 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233f33e507d3d6a5-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098739a39b958aaaf-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE= 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:g+k3P4pMqsoEg1xaWhlm0jhgzuE=', 'Content-Length': '0'} 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5cd6604a8b723d-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.282 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c517869882b04c3c-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.284 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ff5bc4b5c7c009-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0= 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0=', 'Content-Length': '0'} 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de995c8f52b5a4cc-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.287 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0= 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpP6xZr6x4AxHT7JV2+zgczWAy0=', 'Content-Length': '0'} 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897157f2bac6dc1e-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ= 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4531d4d08079500-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.290 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ= 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ut5ms77oa80kDnZAO3/aUUaZZVQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf19b52b852705fe-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000032385b495b0488b-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed952c1a02b935f-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e781cfa458df768-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.297 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4b679857cf5c4e-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w= 2024-04-13T23:18:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w=', 'Content-Length': '0'} 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5dec2760fe95fd9-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w= 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ayR3vaiV3Hot2OT2O4pLXKyjE4w=', 'Content-Length': '0'} 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c667af3068a319c-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.302 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b027c2b74448134-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU= 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:nuXo7xoTvGM2O0hA7HLL48rTMUU=', 'Content-Length': '0'} 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000875d8ad47f173551-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.305 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d952ece3c1349550-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083163c6d455b9082-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA= 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA=', 'Content-Length': '0'} 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf014a041f7ee81-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA= 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0DLSwJ6kc3UwzzVLPyOfT5mFXKA=', 'Content-Length': '0'} 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd2e7245f3f42d0-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q= 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q=', 'Content-Length': '0'} 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af7800c52127af5-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.313 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q= 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hhZnwp5qpnvLhyCrm0kzTQmDm0Q=', 'Content-Length': '0'} 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53d327d348a9e77-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2344ae7a597f2d-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.316 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077cde880ff834a4-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.318 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15b7bf9f4e53551-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T23:18:12.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000641402082ad45d35-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8= 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8=', 'Content-Length': '0'} 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478f91c3a7187d02-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.323 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8= 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:0uIz1mCEo6npot7eUPYbI9ytUL8=', 'Content-Length': '0'} 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1c73d81b84b64c-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.325 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da45824da9c904ed-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI= 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rWeZYh+yfye4g77KbYL8FauXeKI=', 'Content-Length': '0'} 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e510c661cc82501-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.328 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b5aed782fce0e83-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193c3d3f1ec18e64-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.331 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE= 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE=', 'Content-Length': '0'} 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b656c6091cb0cb-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE= 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CtaczqIiLsFEPFHXGUEe5AmA5fE=', 'Content-Length': '0'} 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c2690f7ff558045-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.334 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM= 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM=', 'Content-Length': '0'} 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031157ea5b00601fb-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM= 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:4t3KE2LDxsAHbn/gDm/lEq1rhlM=', 'Content-Length': '0'} 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7593fd4eb79dff3-00661b02a8-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.337 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b624457cb60dd527-00661b02a8-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.339 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA= 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iN5LAAoJzL3ekByHYVNAjR44mdA=', 'Content-Length': '0'} 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4ca600f11e553e-00661b02a8-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:44 GMT 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uzsHpYEprVPgB6gSO++E8AisZic= 2024-04-13T23:18:12.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:44 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uzsHpYEprVPgB6gSO++E8AisZic=', 'Content-Length': '0'} 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078d9e3eeeea0f44a-00661b02a8-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys= 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys=', 'Content-Length': '0'} 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce1eff5971aaecc-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.344 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI= 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI=', 'Content-Length': '0'} 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b30aa6e6348c0b-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI= 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kWT6zyXdbw/FVwx/RS5ShPhDRYI=', 'Content-Length': '0'} 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f570f531c415e4-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys= 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys=', 'Content-Length': '0'} 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe892cfec5b155ef-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.349 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys= 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DOTAuhPJ/rSi+P2ZuWPu4AFmCys=', 'Content-Length': '0'} 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2d15076acca3ec-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.351 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eef93cc01e2596c2-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005279c85fa8be254c-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8= 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8=', 'Content-Length': '0'} 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091790a52fd489ab-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.356 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8= 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:614AgmGC4fDOWbhKhrhNdl1jzN8=', 'Content-Length': '0'} 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4437c0e840a3f4-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE= 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE=', 'Content-Length': '0'} 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012412aa29fa7ced8-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.359 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE= 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:htlyXOr1ybWR1VJCm+814nhuDXE=', 'Content-Length': '0'} 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020e29cbd77b6aaa8-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2dda049bd2f998-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9879d59c6bbf9a2-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.364 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55d5abf126eaf05-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.366 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037de119bc934a7b2-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c= 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c=', 'Content-Length': '0'} 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f645beb6e0c494a5-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.369 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c= 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:f73rBPKAghfZ3b/bZt6VM6Prl8c=', 'Content-Length': '0'} 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b4444cdd118641-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6a99b8397dacdb-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M= 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7tOBKdjaz2O7AcSJkpVC6GR4w7M=', 'Content-Length': '0'} 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325760e443d7a67c-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.374 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005093f81802bb582f-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.376 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c627d667aa9e76-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ= 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000913c9f7c0125300-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.379 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ= 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VrtfiIEPQtOuQtlOEgQmw4uoOGQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f196ba81fad0d8-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM= 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM=', 'Content-Length': '0'} 2024-04-13T23:18:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ec52875a8b36b8-00661b02a9-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.382 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM= 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2kFgvCuq8kG9hoUGvY3rb2yQyIM=', 'Content-Length': '0'} 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b43c86ab14a817-00661b02a9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d23354c1073fd90-00661b02a9-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:45 GMT 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs= 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:45 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S5q/nVR4n36velA8+fldwxE8ezs=', 'Content-Length': '0'} 2024-04-13T23:18:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c67e729c156f3f-00661b02a9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=vwnfns-57 zones={a2, a1} 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'vwnfns-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket vwnfns-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-57 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 8 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding": "None", 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_index": { 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "Normal", 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "normal": { 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "logs": [ 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 0, 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 11, 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-57 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:45.477+0000 7f966cc2d780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:50.478+0000 7f966cc2d780 1 waiting to reach latest gen 2, on 1.. 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:55.479+0000 7f966cc2d780 1 bucket sync caught up with empty source 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:55.479+0000 7f966cc2d780 0 bucket checkpoint complete 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=vwnfns-57 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:55.598+0000 7fe3eb0ea780 1 bucket sync caught up with source: 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T23:18:12.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, 00000000002.561.6, , , 00000000001.1775.6, 00000000001.2205.6, , , 00000000002.399.6, , 00000000002.1222.6] 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:09:55.598+0000 7fe3eb0ea780 0 bucket checkpoint complete 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=vwnfns-57 zones={a2, a1} 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk= 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk=', 'Content-Length': '0'} 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000344b7eaf3fa7262b-00661b02b3-1a1e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/ 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk= 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fXXZVu2IRxamu7oIiD4DyAdKTjk=', 'Content-Length': '0'} 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a0f7ab814464fa0-00661b02b3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T23:18:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a1737686c47379-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.400 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000985c1302285a2e0c-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-13T23:18:12.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d389abce9b976a-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?versions&max-keys=999 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.404 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:/vwnfns-57/?versions 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo= 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TV1kcIzy1uo2NZrnUP90NCBv8Vo=', 'Content-Length': '0'} 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f6d7aaffb781a4-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57999falseanulltrue2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.406 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe43b39aa47e12c0-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T23:18:12.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d415cc6c75f791-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps= 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps=', 'Content-Length': '0'} 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068ba66a428694b12-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.411 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps= 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vGLvlYs6YVeDGttVG0Zs/Hpj6ps=', 'Content-Length': '0'} 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006238674cc12bae9a-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.413 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036179e73b8e9e450-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a?versionId=null 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a?versionId=null 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM= 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NkfZVjczFC50uEZ9rua72cpfJZM=', 'Content-Length': '0'} 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8eaf0c62d22fe57-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009831e6b9878005c6-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056547d283079465b-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.420 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk= 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk=', 'Content-Length': '0'} 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090bbc178ea4937fa-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk= 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WJSsBP30RtQjr2sDlCqGBsWKwOk=', 'Content-Length': '0'} 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007bc8388b3cd288a-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.422 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.423 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc= 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc=', 'Content-Length': '0'} 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a852e2ef13e092-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/a 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/a 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/a 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:/vwnfns-57/a 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc= 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZlB9NLK3vP0mOQSUt5MwQL5beSc=', 'Content-Length': '0'} 2024-04-13T23:18:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7da2b87c95afdae-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c0128f0b053040-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=a 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=a 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.428 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf19b66a65ca0b7e-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57a1000falsea2024-04-13T22:08:07.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.430 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ae9c1f9e5e9e16-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fe022c368c804b-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.433 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI= 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI=', 'Content-Length': '0'} 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe268aa4b01a5986-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI= 2024-04-13T23:18:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gWYwB01Yk7ErJs4oXk4ehiCHQTI=', 'Content-Length': '0'} 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df3841854fcdb68b-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007458315542bec2cc-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b?versionId=null 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.438 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b?versionId=null 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8= 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZxNrh6RA73J46j01on4E1kVfXP8=', 'Content-Length': '0'} 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b477d8d37eb7a19-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee5e43d52413caf-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f58a20112859c5e8-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.443 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4= 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4=', 'Content-Length': '0'} 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587599133f549b78-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4= 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UGySv/YWjVQEHPOeb/PfyMySEC4=', 'Content-Length': '0'} 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d07ea4ff167680-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-13T23:18:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.446 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI= 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI=', 'Content-Length': '0'} 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3011ab0aefcbce8-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/b 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/b 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/b 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:/vwnfns-57/b 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI= 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AZAKUiv2NIUllZBLH+5u9hau2uI=', 'Content-Length': '0'} 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3cc0332bec5323-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.449 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f47dc70089349490-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=b 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=b 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fa1fd158955cc75-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57b1000falseb2024-04-13T22:08:07.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T23:18:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e4674e3ce418ef-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.454 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088358d4d46cb9683-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs= 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs=', 'Content-Length': '0'} 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f8489f50ef063a-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs= 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/L0pPDROilcZt8ytoQGfbpRWRAs=', 'Content-Length': '0'} 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a366dea6dfd2f02-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.459 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a9ee4c260fefadc-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c?versionId=null 2024-04-13T23:18:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c?versionId=null 2024-04-13T23:18:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM= 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:G8JU+B0wGfZmi2Vp+xNUq9lQtoM=', 'Content-Length': '0'} 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4c5963ffd0c40d-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca55ed68755de98-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.464 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c4038c387b5b18-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.466 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0= 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0=', 'Content-Length': '0'} 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fe0833efa45595-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0= 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8Zw1cMoXQQS7Sp/P0xzC03+Lk/0=', 'Content-Length': '0'} 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8ccf720cdca525-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8= 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8=', 'Content-Length': '0'} 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de861fd2f517c84-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/c 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/c 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/c 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:/vwnfns-57/c 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8= 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D4ngLRRSURSv7YUlvm5UUeI+li8=', 'Content-Length': '0'} 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89c7bb4308c8e5a-00661b02b3-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.472 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007438af9c1a41568c-00661b02b3-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=c 2024-04-13T23:18:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=c 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.474 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y= 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:o5YbzUb0qy2vQyEep2RzHsICo5Y=', 'Content-Length': '0'} 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9109cedff1ef868-00661b02b3-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57c1000falsec2024-04-13T22:08:07.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:55 GMT 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.476 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fvWfpMZMBR5yP0w4Wo0TDGJwnoM= 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fvWfpMZMBR5yP0w4Wo0TDGJwnoM=', 'Content-Length': '0'} 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cfe47a8402db8a9-00661b02b3-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M= 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M=', 'Content-Length': '0'} 2024-04-13T23:18:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d846999ea75709-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.479 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc= 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc=', 'Content-Length': '0'} 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c5bfd17851bf718-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.481 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc= 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AyuApKdgY/z8IlJ4tamtFuf2tCc=', 'Content-Length': '0'} 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b671ed01095b2a8-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M= 2024-04-13T23:18:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M=', 'Content-Length': '0'} 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002504882484fab151-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d?versionId=null 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.484 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d?versionId=null 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M= 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:RJlEj2C8+cNsmdIdLFJxsm9Oj/M=', 'Content-Length': '0'} 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b23efec3c2bca8a-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6c471dabb36306-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1724680251e93ca-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.489 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU= 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU=', 'Content-Length': '0'} 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec792180d72c76fc-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.491 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU= 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qv1jegmZbsqOxklz38LQVkEe1NU=', 'Content-Length': '0'} 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc388f77f9df5d9-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM= 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM=', 'Content-Length': '0'} 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006392e09a716568e4-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/d 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/d 2024-04-13T23:18:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/d 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.494 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:/vwnfns-57/d 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM= 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fq/uhqBe4TKNbowHHJ4xmTnB/IM=', 'Content-Length': '0'} 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:08:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004be018731b4b3eaa-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002def083a184ec12d-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=d 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=d 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff2beded3d9a8a8-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57d1000falsed2024-04-13T22:08:07.549Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.498 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000076ccc8a521e0727-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.501 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a91283875c72a1-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.503 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY= 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY=', 'Content-Length': '0'} 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e183d8c7ab8e79-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY= 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vouwWCIX1f8UbraI2jMe/eQc1zY=', 'Content-Length': '0'} 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b290907997bab3f8-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.505 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ecde0ee01ba097-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e?versionId=null 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.508 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e?versionId=null 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4= 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:sNHXDFVSgym9OSAgAX/M5NBvyF4=', 'Content-Length': '0'} 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33a274fb0f06438-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.510 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e7e2ef3e2aef82-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c2a3958ff9821a-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.513 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA= 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA=', 'Content-Length': '0'} 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a2f01203f66638-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.515 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA= 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qcSdX0D2p8CAXi+4DHsR8di92zA=', 'Content-Length': '0'} 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a2c6b4f6bcba3f-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c= 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c=', 'Content-Length': '0'} 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a2790fb0ec3f8b-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/e 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/e 2024-04-13T23:18:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/e 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.518 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:/vwnfns-57/e 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c= 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ZcCdcrTzleOprZ2PQj81z1nkL4c=', 'Content-Length': '0'} 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000364464802f33899b-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.520 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c393cc4c2a474b11-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=e 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=e 2024-04-13T23:18:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.522 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000155c816482f42c1c-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57e1000falsee2024-04-13T22:09:24.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.524 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ee2cdf83d1a9f9-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07e70e93856a093-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I= 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I=', 'Content-Length': '0'} 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9760737820614f-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.529 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I= 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:66AQuI3FTfQsalQc3OCRTw+QD1I=', 'Content-Length': '0'} 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000721fdde74bc86c8-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab95a1e72b1cbef6-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f?versionId=null 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f?versionId=null 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ= 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ykcRvPtpLjd+O8SBXq+D8FMrLmQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed84ff4db71a3aab-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.534 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f71c45c2e5e2c2f9-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.536 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c3f5f8f983abb70-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.538 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ= 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c61b83835848898-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ= 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bvTHbDhk0+0c9dg3MlrI6X4v2oQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c9d2aeff26c988-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.541 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk= 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk=', 'Content-Length': '0'} 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf16734ef9bcaef-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/f 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/f 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/f 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:/vwnfns-57/f 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk= 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FApqksPA7YJL+KOFz6QgLZVYWzk=', 'Content-Length': '0'} 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33481b32b8edbbf-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87bdea1f6beaf5f-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=f 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=f 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b86a26eca09d51dc-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57f1000falsef2024-04-13T22:09:24.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-13T23:18:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.549 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0056aefa75ff86-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7298dd675c580a-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk= 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk=', 'Content-Length': '0'} 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080032f4e2222e565-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.554 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk= 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BqTLswk4xuuVZIlB5Nv9Kn/GEkk=', 'Content-Length': '0'} 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a536242c762bf104-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.556 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe82b25e7334a23-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g?versionId=null 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g?versionId=null 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc= 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:/3y0lGDEKFEHSuF9mvEgkfnSvnc=', 'Content-Length': '0'} 2024-04-13T23:18:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009293fe2419d6444c-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.559 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed6080b0ffc38bc9-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bace98ffb80606ce-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs= 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs=', 'Content-Length': '0'} 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e3fedb2d5367ee-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.564 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs= 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DYxqBi1XxvEw2To4i1oF1d+J+Zs=', 'Content-Length': '0'} 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000145fd31fecbcbdcc-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug= 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug=', 'Content-Length': '0'} 2024-04-13T23:18:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000913cf4802cbe8d84-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/g 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/g 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/g 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.567 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:/vwnfns-57/g 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug= 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SZ0mJbtWE/i+cq9miYnKfiGrAug=', 'Content-Length': '0'} 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea9a79f0dd2283e-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.569 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000181779977a0206a7-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=g 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=g 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6daaa62806914d8-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57g1000falseg2024-04-13T22:09:24.097Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T23:18:12.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022201e3f4cc3a182-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.577 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae8e4b5945cb2de-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ= 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78321d6263b54ee-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ= 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EnisM/klpsZJvOs+d2xS3vciZQQ=', 'Content-Length': '0'} 2024-04-13T23:18:12.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6fca6e80e74b621-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a6c89a64e078d3-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h?versionId=null 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.584 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h?versionId=null 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw= 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:E8HhG7kK4PxUpU7cSqCh1bb0Acw=', 'Content-Length': '0'} 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fcc6e4ded8b45d-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d21cc1e373b0909-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.587 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b7159e42ff72e9-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc= 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc=', 'Content-Length': '0'} 2024-04-13T23:18:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23360d02d682648-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.590 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc= 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Lbq+yHumC3Ml2I4Y/jPjykxb7Pc=', 'Content-Length': '0'} 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e817c8fbc2bc8b66-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4= 2024-04-13T23:18:12.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4=', 'Content-Length': '0'} 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-57:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf6c49bc5f3260d-00661b02b4-1a1e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/h 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/h 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/h 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.593 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:/vwnfns-57/h 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4= 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FxRusMpZB+pTCP0J+RTJuKy5CA4=', 'Content-Length': '0'} 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c700b7d064483062-00661b02b4-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000226f15eccc55027a-00661b02b4-1a1e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/ 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/ 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-57/?prefix=h 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-57/?prefix=h 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:09:56 GMT 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:/vwnfns-57/ 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI= 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:09:56 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LSqMaXcJkFd3OLmJpkH9z53ZqpI=', 'Content-Length': '0'} 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a3c046c2bd5420-00661b02b4-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:09:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-57h1000falseh2024-04-13T22:09:24.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.597 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=vwnfns-57 zones={a2, a1} 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.618+0000 7f5e0ffe8700 -1 req 13763461066637601712 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-33:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.33,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-33:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.33,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.618+0000 7f5deffa8700 -1 req 13368004055144513720 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-32:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.32,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-32:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.32,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.618+0000 7f5dae725700 -1 req 2416174129595301439 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-22:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.22,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-22:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.22,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.621+0000 7f5d8e6e5700 -1 req 13211626067910606235 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-30:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.30,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-30:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.30,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.622+0000 7f5d946f1700 -1 req 8741748541085336033 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-25:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.25,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-25:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.25,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.663+0000 7f5e48859700 -1 req 1938242242939681279 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-40:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.3,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-40:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.3,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.598 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.666+0000 7f5dfe7c5700 -1 req 10447478249211428170 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-39:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.2,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-39:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.2,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.675+0000 7f5e2a81d700 -1 req 14564861806786498179 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-23:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.23,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-23:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.23,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.705+0000 7f5e067d5700 -1 req 1710315130551123393 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-24:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.24,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-24:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.24,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.708+0000 7f5e1e805700 -1 req 11228260614169747268 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-5:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.5,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-5:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.5,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.718+0000 7f5dfbfc0700 -1 req 13852769238849506418 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-42:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.5,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-42:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.5,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding": "None", 2024-04-13T23:18:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_index": { 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "Normal", 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "normal": { 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "logs": [ 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.987+0000 7f5e1d002700 -1 req 13868092827117849892 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-38:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.1,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-38:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.1,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:56.990+0000 7f5e7d8c3700 -1 req 3477285246539159647 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-29:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.29,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-29:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.29,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:09:57.078+0000 7f5df27ad700 -1 req 3395898003046122895 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=vwnfns-34:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.34,z=f6f208d6-f791-479b-bc3d-112512eec2d4,az=0},d={b=vwnfns-34:f6f208d6-f791-479b-bc3d-112512eec2d4.4374.34,z=5da80fbc-5edc-4ba7-826d-471359a9fc92,az=0}} returned ret=-2 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'vwnfns-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket vwnfns-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding": "None", 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_index": { 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "Normal", 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "normal": { 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "logs": [ 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 1, 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "type": "InIndex", 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "in_index": { 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "gen": 2, 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "layout": { 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 3, 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hash_type": "Mod" 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:12.607 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 1809, in test_resharded_bucket_full_sync_after_data_sync_init 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-13T23:18:12.608 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:40 GMT 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+sCKc6S1+Aon+Pp2d0yT2I+3Evk= 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:40 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+sCKc6S1+Aon+Pp2d0yT2I+3Evk=', 'Content-Length': '0'} 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000233c73b11eadab9a-00661b08bc-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=vwnfns-63 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047740.325648_889.1", 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:35:40.325648Z" 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713047740.325648_889.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:40.499+0000 7f4b82a0b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:40.500+0000 7f4b82a0b780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047740.325648_889.1", 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:35:40.325648Z", 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T22:35:40Z" 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.618 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047740.325648_889.1", 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:35:40.325648Z", 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.619 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.620 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.622 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T22:35:40Z" 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713047740.325648_889.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:35:40.325648Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713047740.325648_889.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713047740.325648_889.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:35:40.522+0000 7f05630ed700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 112291) UID: 0 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:35:40.522+0000 7f05682c27c0 -1 shutting down 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T23:18:12.623 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.624 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003371a8a788292efb-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj11 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj11 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj11 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj11 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:mmtq0dYeXY1ps5YZpDTY/nboPhk= 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mmtq0dYeXY1ps5YZpDTY/nboPhk='} 2024-04-13T23:18:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac2f3c06f161acd4-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.627 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c22c3c9ac3db75f-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj22 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj22 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj22 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj22 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:mtmz1bypdMuJwfzmIrDXc4P9fZM= 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mtmz1bypdMuJwfzmIrDXc4P9fZM='} 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000616a0452e8dd2382-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.630 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065ea72056e1f76df-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj33 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj33 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj33 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj33 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:gva/bH4PI4Lcz0jzs9gVO0KkZUY= 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:gva/bH4PI4Lcz0jzs9gVO0KkZUY='} 2024-04-13T23:18:12.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b88811ecf8a9b0a-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.633 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b5215dfe6ee5d9c-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj44 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj44 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj44 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj44 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fj5xUQ438XDidh08Qov24bjjt2E= 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fj5xUQ438XDidh08Qov24bjjt2E='} 2024-04-13T23:18:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb8d4f6f068190a7-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.636 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000153a10c6595ec44d-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj55 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj55 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj55 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj55 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:di/SuBc3smUX2mps4uwiv7TlV9g= 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:di/SuBc3smUX2mps4uwiv7TlV9g='} 2024-04-13T23:18:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a547b43b7818571-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-63 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 5 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e7f1c87bc59f35f-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj17 2024-04-13T23:18:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj17 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj17 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.641 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj17 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XZsB1NZzCx39274IvLkuus8oKuQ= 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XZsB1NZzCx39274IvLkuus8oKuQ='} 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056d32a3927a700c7-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ace3f9b0a2b736b-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj34 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj34 2024-04-13T23:18:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj34 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.644 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj34 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oNfSXSsANEFhGBukAcCzn6n3AKk= 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oNfSXSsANEFhGBukAcCzn6n3AKk='} 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9b1a943f6fb74e8-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ee13baf0ea61e7f-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj51 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj51 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj51 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.647 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj51 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:SzWCXu/qwvN5Fo2QPYN4EysZ/eI= 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:SzWCXu/qwvN5Fo2QPYN4EysZ/eI='} 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000742e72a6da1a58b0-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.649 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097410f8c83c5cacd-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj68 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj68 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj68 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj68 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:A8sHLZt6VP23M7b9XY4ISYu7ri0= 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A8sHLZt6VP23M7b9XY4ISYu7ri0='} 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000472aa395e40a5af4-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.652 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU= 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:llOYUiTEeSrx+DEgQbGHxwObGiU=', 'Content-Length': '0'} 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b0e7f2a3d45ec60-00661b08c2-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj85 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj85 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj85 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:46 GMT 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj85 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oCT5y4ylBOhkDFkTMXyr8SfddC4= 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:46 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oCT5y4ylBOhkDFkTMXyr8SfddC4='} 2024-04-13T23:18:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a86410e9619719e9-00661b08c2-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-63 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 10 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2f9e4a33c3ed899-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj19 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj19 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj19 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.657 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj19 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:9Hovjhmb+adWuw8+jzykFmKWsU4= 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9Hovjhmb+adWuw8+jzykFmKWsU4='} 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000502d96fb01716e64-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092ef0682c96f4d20-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj38 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj38 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj38 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.660 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj38 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:72WCDr9f86mMNlO+2Tzs2gDSKsE= 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:72WCDr9f86mMNlO+2Tzs2gDSKsE='} 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe9f1f8c0727db6d-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057bd96ede10acb64-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj57 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj57 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj57 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.663 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj57 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:QULUYiSEKraGYhDYv0d+eS2dNIw= 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QULUYiSEKraGYhDYv0d+eS2dNIw='} 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1bda648acec9215-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.665 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddc90481796d05ea-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj76 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj76 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj76 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj76 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:pWPCH7IeL7mMZ5o3sItJRKslHwE= 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:pWPCH7IeL7mMZ5o3sItJRKslHwE='} 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6c35469a8ac21d5-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fbbe021500156fa-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj95 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj95 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj95 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.669 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj95 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Y4x8TL2mOSbx2jlDhh+WhiF+Q4Q= 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Y4x8TL2mOSbx2jlDhh+WhiF+Q4Q='} 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008129c375b519f07e-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-63 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 15 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.671 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efea4c40bec2d45e-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj23 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj23 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj23 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj23 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.673 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:huVK6boPbRggJBUJT86kLIVHItA= 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:huVK6boPbRggJBUJT86kLIVHItA='} 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c92eec90aaa1b21e-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c63274d61c62a3a5-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj46 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj46 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj46 2024-04-13T23:18:12.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj46 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.676 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:BRgk9yo/Hl91FTuo63mymN4reZg= 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:BRgk9yo/Hl91FTuo63mymN4reZg='} 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093209ddab83714c3-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.677 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1610a478e52e603-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj69 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj69 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj69 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj69 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:K4fVxpiQ1FGbVJGWRhF3tpi8A4Q= 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K4fVxpiQ1FGbVJGWRhF3tpi8A4Q='} 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000071d2e71fd340be6-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.680 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2abb2f68238a336-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj92 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj92 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj92 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.681 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj92 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WKC4HbQ4tYp6E2VBOl6mK/bVePA= 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WKC4HbQ4tYp6E2VBOl6mK/bVePA='} 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d7ecdc7c9f6eed8-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.683 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afa3b185092b20cb-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj115 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj115 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj115 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj115 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:jtONIxt8VrKL5xcyrgI7D45GSYI= 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jtONIxt8VrKL5xcyrgI7D45GSYI='} 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078e9b4848aefc011-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-63 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 20 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eafe0b61ed463381-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj29 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj29 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj29 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj29 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:moJz0QikIqg44Z6RngAqPEADj1A= 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:moJz0QikIqg44Z6RngAqPEADj1A='} 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9d844b8c3b1023c-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.690 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e5b0994bc3a1ffc-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj58 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj58 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj58 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj58 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.692 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ComYD1EXot2DS4Ug+1xMUzuiM0Q= 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ComYD1EXot2DS4Ug+1xMUzuiM0Q='} 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b07dcd246459bc46-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d098eb1a8c85eb9d-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj87 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj87 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj87 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj87 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iGpwRQRuv9/t65mpMZaqjlELQbA= 2024-04-13T23:18:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iGpwRQRuv9/t65mpMZaqjlELQbA='} 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f53da6777007a33-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.696 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c84e6dc008d85773-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj116 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj116 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj116 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj116 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PKj/fhkbPSaCrCYJsUSIsVonJPs= 2024-04-13T23:18:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PKj/fhkbPSaCrCYJsUSIsVonJPs='} 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093aa19bdba5a0dee-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.699 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0= 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FUvNZ4SRpnIOOecoDdgW5F1Y2J0=', 'Content-Length': '0'} 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000970ff83e56a8240f-00661b08c3-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj145 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj145 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj145 2024-04-13T23:18:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:47 GMT 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj145 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:j1wJ13s/UecB6MVNh0C+GSBo1Pk= 2024-04-13T23:18:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:47 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j1wJ13s/UecB6MVNh0C+GSBo1Pk='} 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d55d1d34d2dffbaa-00661b08c3-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-63 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 25 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.703 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e58ca65f53d76bb-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj31 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj31 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj31 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj31 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:D/NTW873XkF63kY8bfuPsboNYfY= 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:D/NTW873XkF63kY8bfuPsboNYfY='} 2024-04-13T23:18:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009891b8cf2bb40285-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.706 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a569628cbde6d75-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj62 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj62 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj62 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.708 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj62 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:JUTUWmXjQH19hunLUkOqBtP2n9Q= 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:JUTUWmXjQH19hunLUkOqBtP2n9Q='} 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000944954d6e39eeecb-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082841af653fdfe86-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj93 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj93 2024-04-13T23:18:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj93 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.711 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj93 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ftURUK9Xqap8KG0nued1QIW8Sxw= 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ftURUK9Xqap8KG0nued1QIW8Sxw='} 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9121d15c4227bfa-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0e7f1bd53c29bed-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj124 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj124 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj124 2024-04-13T23:18:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj124 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.714 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hzgsY7GSx8Mmirvwk/9D6ggNWFw= 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hzgsY7GSx8Mmirvwk/9D6ggNWFw='} 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071adb6d440c753c6-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004354f600707267a8-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj155 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj155 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj155 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj155 2024-04-13T23:18:12.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:HeYf37bqO7UMpilyhOH3twLB9Tc= 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HeYf37bqO7UMpilyhOH3twLB9Tc='} 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9a506b3fc24c73b-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-63 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 30 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe362e10b53fdace-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj37 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj37 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj37 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj37 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:n5VX74Hllq7qiIKdvw7abfbkhzs= 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:n5VX74Hllq7qiIKdvw7abfbkhzs='} 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b012a2e86edb2f1-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046eb9d7d11b3cc79-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj74 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj74 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj74 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj74 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:jXKDUcIh15J78ahthpAylFRZxzI= 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jXKDUcIh15J78ahthpAylFRZxzI='} 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006311d4a342f88dfa-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fabeba93149efad3-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj111 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj111 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj111 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj111 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:kgLsUOSop71HuWcrIUk8sqEToOQ= 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kgLsUOSop71HuWcrIUk8sqEToOQ='} 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a61f2bf1f8e9db-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.728 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000586461aeeb022ad9-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj148 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj148 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj148 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj148 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:OAyH5FBZY6eiZ4qbppHqiDGvqWI= 2024-04-13T23:18:12.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OAyH5FBZY6eiZ4qbppHqiDGvqWI='} 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6b6301bcb4ba8d2-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/ 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/ 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/ 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.731 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:/vwnfns-63/ 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4= 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8+UPb7IerxAFMJ5axVAIXGI40o4=', 'Content-Length': '0'} 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005da6686bb98ad9e0-00661b08c4-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-63/obj185 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-63/obj185 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-63/obj185 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:35:48 GMT 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:/vwnfns-63/obj185 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:CHg3hqPO3GyRlB4kA4msY3d/6Ts= 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:35:48 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:CHg3hqPO3GyRlB4kA4msY3d/6Ts='} 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce42496f0ac06b24-00661b08c4-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:35:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:48.616+0000 7f38ad606780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:48.617+0000 7f38ad606780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-13T23:18:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: anonymous 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-63 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:49.820+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-13T23:18:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:54.821+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:35:59.822+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:04.823+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:09.823+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:14.824+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:19.824+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:24.824+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:29.825+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:34.826+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:39.827+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:44.828+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:49.828+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:54.829+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:36:59.828+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:04.829+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:09.829+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:14.830+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:19.831+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:24.832+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:29.832+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:34.833+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:39.832+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:44.833+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:49.834+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:54.835+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:37:59.836+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:04.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:09.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:14.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:19.837+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:24.838+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:29.839+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:34.840+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:39.841+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:44.842+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:49.842+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:54.842+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:38:59.843+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:04.843+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:09.844+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:14.845+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:19.846+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:24.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:29.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:34.846+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:39.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:44.847+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:49.848+0000 7f339a15b780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:54.853+0000 7f339a15b780 1 bucket sync caught up with source: 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.66.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.111.6, , , , 00000000001.88.6] 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.66.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.111.6, , , , 00000000001.88.6] 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:54.853+0000 7f339a15b780 0 bucket checkpoint complete 2024-04-13T23:18:12.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=vwnfns-63 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:54.972+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:39:59.972+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:04.973+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:09.973+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:14.973+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:19.974+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:24.975+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:29.976+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:34.977+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:39.977+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:44.978+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:49.977+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:54.978+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:40:59.979+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:04.980+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:09.981+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:14.981+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:19.982+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:24.982+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:29.982+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:34.983+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:39.984+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:44.984+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:49.986+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:54.986+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:41:59.987+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:04.987+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:09.987+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:14.988+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:19.988+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:24.990+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:29.990+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:34.991+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:39.992+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:44.991+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:49.992+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:54.993+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:42:59.993+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:04.995+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:09.996+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:14.996+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:19.997+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:24.996+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:29.997+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:34.998+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:39.999+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:45.000+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:50.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:43:55.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:00.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:05.001+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:10.002+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:15.002+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:20.004+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:25.004+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:30.005+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:35.006+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:40.006+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:45.006+0000 7f0c67bc8780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:50.007+0000 7f0c67bc8780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:50.007+0000 7f0c67bc8780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:44:50.007+0000 7f0c67bc8780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-13T23:18:12.745 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-13T23:18:12.746 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:19 GMT 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:z+Ni6ZLpMarhjizv+G9mHRlnL6U= 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:19 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:z+Ni6ZLpMarhjizv+G9mHRlnL6U=', 'Content-Length': '0'} 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b7610b9eab028240-00661b0b77-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=vwnfns-65 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713048439.479509_937.1", 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:47:19.479509Z" 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:19.654+0000 7f439f56e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:19.655+0000 7f439f56e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713048439.473212_934.1", 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:47:19.473212Z", 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T22:47:19Z" 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.755 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1713048439.473212_934.1", 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.756 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:47:19.473212Z", 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.759 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T22:47:19Z" 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048439.473212_934.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:47:19.473212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713048439.473212_934.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1713048439.479509_937.1 target=1_1713048439.473212_934.1 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:24.769+0000 7f3ad801e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:24.770+0000 7f3ad801e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713048439.479509_937.1", 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:47:19.479509Z", 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T22:47:24Z" 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.766 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1713048439.479509_937.1", 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:47:19.479509Z", 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.767 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T22:47:24Z" 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048439.479509_937.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:47:19.479509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713048439.479509_937.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7afbc1a0610771e-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj11 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj11 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj11 2024-04-13T23:18:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj11 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:jMp7Mq0w+FXy1VJhE1WZiyidmVk= 2024-04-13T23:18:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jMp7Mq0w+FXy1VJhE1WZiyidmVk='} 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a28e513050b1bbcd-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f956683f4a68662-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj22 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj22 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj22 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj22 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.776 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5bO5dttzskQOiyCx1n1cljhG6vQ= 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5bO5dttzskQOiyCx1n1cljhG6vQ='} 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000274b85af013ddd3-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c074b2d259407c99-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj33 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj33 2024-04-13T23:18:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj33 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.779 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj33 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:j3VxkW0mEV3Sym37J4DMdO5lvmQ= 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:j3VxkW0mEV3Sym37J4DMdO5lvmQ='} 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0fa41aa77f4a1a1-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9738efadd9524ae-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj44 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj44 2024-04-13T23:18:12.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj44 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.782 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj44 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:kIJNMiGDqhyBmovGRtKCI1ksLuk= 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:kIJNMiGDqhyBmovGRtKCI1ksLuk='} 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eaf590b7940d7eaf-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo= 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:klsljaSM+t0V7NO1Ksk+Z7jJTvo=', 'Content-Length': '0'} 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bab2683542e84ca8-00661b0b7c-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj55 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj55 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj55 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.785 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:24 GMT 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj55 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:R/rHSHQRvNElrNDiauC9U7pQLPc= 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:24 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R/rHSHQRvNElrNDiauC9U7pQLPc='} 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9e1d5c56baf3b28-00661b0b7c-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-65 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 5 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.787 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc0e07ae0a0016f9-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj17 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj17 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj17 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj17 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DavugBFTPROR8GC318g+baBerrU= 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DavugBFTPROR8GC318g+baBerrU='} 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6a20774d4aa2168-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9ceb0a0da242ce8-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj34 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj34 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj34 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj34 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:h2Op0B01V8lZsafuTyO0Y0UoP+E= 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:h2Op0B01V8lZsafuTyO0Y0UoP+E='} 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c7afffe2c8d99ee-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.793 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059972ed5c2ac6e5a-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj51 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj51 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj51 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj51 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TuYZSa2AXtWD76x5K3H9fYPEZcw= 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TuYZSa2AXtWD76x5K3H9fYPEZcw='} 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088745bf37cb28665-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b33f3159268f453-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj68 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj68 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj68 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj68 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oxYHFMI65mqlbLKw5RQP5hN+NBU= 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oxYHFMI65mqlbLKw5RQP5hN+NBU='} 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048ee41ba00827f2b-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.799 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f0e9a12b3120d10-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj85 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj85 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj85 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj85 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:uf9qd4B8ikbbYlABnXzGObE59/o= 2024-04-13T23:18:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:uf9qd4B8ikbbYlABnXzGObE59/o='} 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f4004a20ae2493e-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-65 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 10 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000552f630f5f64eef4-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj19 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj19 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj19 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj19 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Q2OWFHfr9fd8J4ekb8vkdP+LGto= 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Q2OWFHfr9fd8J4ekb8vkdP+LGto='} 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036fe4754040155fc-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.806 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6c0b0c06e2861ca-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj38 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj38 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj38 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.807 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj38 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lHQ1sk3+7t4tqLDWRj/sB7aiIBc= 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lHQ1sk3+7t4tqLDWRj/sB7aiIBc='} 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce18a5762fce99ff-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000805f033dba21343-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj57 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj57 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj57 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.810 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj57 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vKBIKKmHRYpkjh0Nufwa7NI6H4w= 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vKBIKKmHRYpkjh0Nufwa7NI6H4w='} 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed56aeb7767a2d20-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.812 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003aa97d25b009e1c-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj76 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj76 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj76 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj76 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:jCCofFF6MR5PfJRJp4tEFIGh9Jk= 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jCCofFF6MR5PfJRJp4tEFIGh9Jk='} 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec3da2cb5422f5c8-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd746666740f705c-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj95 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj95 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj95 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.816 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj95 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:3JwaHSSQ2UeCroLpa2C9nGVwUf4= 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:3JwaHSSQ2UeCroLpa2C9nGVwUf4='} 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000216a73e9a216a391-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-65 2024-04-13T23:18:12.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 15 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.818 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000d4e912b0d01959-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj23 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj23 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj23 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj23 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:bFWEL4yKsOuIlNXu5qe1YFqp8vE= 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bFWEL4yKsOuIlNXu5qe1YFqp8vE='} 2024-04-13T23:18:12.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000118946a050d2df88-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.821 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042427b0d2c308351-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj46 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj46 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj46 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj46 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iOIWYrFs9czBx0PxfcBtvoVVk3A= 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iOIWYrFs9czBx0PxfcBtvoVVk3A='} 2024-04-13T23:18:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e94ea39b2e775d40-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.824 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfdf274fc7a2b2d6-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj69 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj69 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj69 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj69 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ivhoxUrZj00YkukeKx5w/glLW6g= 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ivhoxUrZj00YkukeKx5w/glLW6g='} 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1a04a009e370d27-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012740a2a36bf045d-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj92 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj92 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj92 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj92 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Aav2sj9R73hOmGm5Av/7/4RyHZ8= 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Aav2sj9R73hOmGm5Av/7/4RyHZ8='} 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b68d0525ee68b76c-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.830 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ac187c1248df93-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj115 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj115 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj115 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.831 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj115 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:tq3FSwNYMm1yIR17LmY6Gf81uMk= 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tq3FSwNYMm1yIR17LmY6Gf81uMk='} 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000973d31c0882849d1-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-65 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 20 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.833 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa3e994b3e8c6c7e-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj29 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj29 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj29 2024-04-13T23:18:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj29 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FJBRLUqA7AuO3KvH6No+g0suDhc= 2024-04-13T23:18:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FJBRLUqA7AuO3KvH6No+g0suDhc='} 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000647176f118ab63ef-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.836 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000612a702ecdade4d9-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj58 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj58 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj58 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj58 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:EUwyBq0Ff2lo+UfV59rgksY7yjI= 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EUwyBq0Ff2lo+UfV59rgksY7yjI='} 2024-04-13T23:18:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfc1261e045a1652-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.839 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo= 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:ebt0a1WZ/k/a17xEcZ2OPsURaAo=', 'Content-Length': '0'} 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae089809344b4d4d-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj87 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj87 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj87 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:25 GMT 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj87 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+HQfG/0bepLQXjABp5czaDbdVQw= 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:25 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+HQfG/0bepLQXjABp5czaDbdVQw='} 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bea773cbccabadb8-00661b0b7d-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.842 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113f1c19479f7f36-00661b0b7d-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj116 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj116 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj116 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj116 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:r+jpvl2S4JyfsB9vDMBi1ZjcDbI= 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r+jpvl2S4JyfsB9vDMBi1ZjcDbI='} 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db3ad9f37c8e0c3f-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.845 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e620d1cf50f08ca-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj145 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj145 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj145 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj145 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+H+7ilSz6qoJt4eEmhdkHS4ppaw= 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+H+7ilSz6qoJt4eEmhdkHS4ppaw='} 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000546324f22091e11b-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-65 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 25 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d75461d86cde47-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj31 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj31 2024-04-13T23:18:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj31 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.850 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj31 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hHx3ISGJgqxAtmtPdQWfxCk5YJ4= 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hHx3ISGJgqxAtmtPdQWfxCk5YJ4='} 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077f4efa58370b9a0-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdfb7c439b4b87b8-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj62 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj62 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj62 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj62 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.853 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:33sU5DtQPwATrlFPpSf7P1s8y+E= 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:33sU5DtQPwATrlFPpSf7P1s8y+E='} 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000556f7ae6be1f1a01-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073d8988252a89c4b-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj93 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj93 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj93 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj93 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.856 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:8ZX80yhh1PwcQlMvJqLeUBBWxks= 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:8ZX80yhh1PwcQlMvJqLeUBBWxks='} 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000698c1a07b3d3112c-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb794c16c3029123-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj124 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj124 2024-04-13T23:18:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj124 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj124 2024-04-13T23:18:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:T6L64Kvql+Kiuald6R3voPar1RY= 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:T6L64Kvql+Kiuald6R3voPar1RY='} 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c2251cc31c7a91f-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d53d1261e41ead5-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj155 2024-04-13T23:18:12.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj155 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj155 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.862 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj155 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:UvE+HrLy+zaWPzUE90E3vbU/ukM= 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:UvE+HrLy+zaWPzUE90E3vbU/ukM='} 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000faef6a208c87226c-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-65 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-65 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 30 2024-04-13T23:18:12.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.864 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000671f02ac2d567bbc-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj37 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj37 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj37 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj37 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:LbOpIUExzC+RctP0acdq3br5xHE= 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:LbOpIUExzC+RctP0acdq3br5xHE='} 2024-04-13T23:18:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e969143e3dc2aae-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.867 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b948692e1cc3fe2f-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj74 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj74 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj74 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj74 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2gjk/pQaDOF+wAawlYDi9FUpqFU= 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2gjk/pQaDOF+wAawlYDi9FUpqFU='} 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d98b528f2d88bb9d-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.870 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dadba39da83a96a7-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj111 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj111 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj111 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj111 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:qeePr5mh0RJV9RXPk+EYR8ugpvM= 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:qeePr5mh0RJV9RXPk+EYR8ugpvM='} 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006254f77967342114-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.873 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da280f3a0f628506-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj148 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj148 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj148 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj148 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:YUiBC/MioaF+OEvjmaJWaq2tlM4= 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YUiBC/MioaF+OEvjmaJWaq2tlM4='} 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062ea926d20bdc836-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/ 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/ 2024-04-13T23:18:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/ 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:/vwnfns-65/ 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.876 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo= 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:PjPZkQljseomnMWOv7XZyGANRJo=', 'Content-Length': '0'} 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f96241915df83a7f-00661b0b7e-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-65/obj185 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-65/obj185 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-65/obj185 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.877 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:47:26 GMT 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:/vwnfns-65/obj185 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:r3xWywo6I+OLAdU1iF4ts4HJfK0= 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:47:26 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:r3xWywo6I+OLAdU1iF4ts4HJfK0='} 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005abd1b57ffa9531b-00661b0b7e-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-65 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:26.768+0000 7f27da411780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:31.769+0000 7f27da411780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:36.773+0000 7f27da411780 1 bucket sync caught up with source: 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, , , , , , 00000000002.121.6, , , , , , 00000000002.186.6, , , , , 00000000002.170.6, , , , , , , , , , , , , , , 00000000001.101.6, , , , 00000000001.99.6] 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, , , , , , 00000000002.121.6, , , , , , 00000000002.186.6, , , , , 00000000002.170.6, , , , , , , , , , , , , , , 00000000001.101.6, , , , 00000000001.99.6] 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:36.773+0000 7f27da411780 0 bucket checkpoint complete 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=vwnfns-65 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:36.903+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:41.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:46.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:51.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:47:56.904+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:01.905+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:06.906+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:11.907+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:16.908+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:21.909+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:26.909+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:31.909+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:36.910+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:41.910+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:46.911+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:51.913+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:48:56.914+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:01.915+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:06.914+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:11.915+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:16.915+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:21.916+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:26.917+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:31.918+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:36.919+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:41.920+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:46.920+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:51.920+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:49:56.921+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:01.922+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:06.923+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:11.924+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:16.924+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:21.925+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:26.925+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:31.925+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:36.926+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:41.927+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:46.929+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:51.929+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:50:56.930+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:01.929+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:06.930+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:11.931+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:16.931+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:21.932+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:26.934+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:31.934+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:36.935+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:41.935+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:46.935+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:51.936+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:51:56.936+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:01.938+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:06.939+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:11.939+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:16.940+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:21.940+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:26.940+0000 7f4300684780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:31.941+0000 7f4300684780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:31.941+0000 7f4300684780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:31.941+0000 7f4300684780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-13T23:18:12.884 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:12.885 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.885 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:12.885 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-13T23:18:12.885 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 1937, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-13T23:18:12.886 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:31 GMT 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X+J7rKzo3/TAbfoI8dEw9U2QJHo= 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:31 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X+J7rKzo3/TAbfoI8dEw9U2QJHo=', 'Content-Length': '0'} 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000063dc36c78db19ad7-00661b0caf-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=vwnfns-66 2024-04-13T23:18:12.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713048751.983401_980.1", 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:52:31.983401Z" 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713048751.983401_980.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:12.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:32.171+0000 7f4bf9b9e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:32.172+0000 7f4bf9b9e780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713048751.983401_980.1", 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:52:31.983401Z", 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T22:52:32Z" 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:12.896 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1713048751.983401_980.1", 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:52:31.983401Z", 2024-04-13T23:18:12.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.898 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.899 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:12.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T22:52:32Z" 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713048751.983401_980.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:52:31.983401Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713048751.983401_980.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713048751.983401_980.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.902 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be163686d055a831-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:thvZOGq1TEREQsWO222jEKaslOw= 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:thvZOGq1TEREQsWO222jEKaslOw='} 2024-04-13T23:18:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085423f256b78c592-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.905 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaace3364e421275-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:MSMZqeXmFmcU+kgFjdAvKeqf1Xw= 2024-04-13T23:18:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:MSMZqeXmFmcU+kgFjdAvKeqf1Xw='} 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a10d95f281c1716-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.908 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000939551de1210a718-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:YJ3O+p7xy36T+yc8KELunvLYwQw= 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YJ3O+p7xy36T+yc8KELunvLYwQw='} 2024-04-13T23:18:12.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025f2a4e7a286cba3-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.911 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000775024217fa958cf-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Jd1bEbnWYu4Wf5NGvShgmaGh6iY= 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Jd1bEbnWYu4Wf5NGvShgmaGh6iY='} 2024-04-13T23:18:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c92c0096ab088dd7-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.914 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo= 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Z18OSJL6n9IPFT+rlMGWplFH9fo=', 'Content-Length': '0'} 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003289e36dab3d7d49-00661b0cb0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:32 GMT 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:K5zGLNLaDDJvYRDN1PzfNABpHjU= 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:52:32 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:K5zGLNLaDDJvYRDN1PzfNABpHjU='} 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ea6d4251ac8925f-00661b0cb0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-66 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:32.366+0000 7f7d296c9780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:37.368+0000 7f7d296c9780 1 bucket sync caught up with source: 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, , , , , , , , 00000000001.190.6, 00000000002.113.6, 00000000002.81.6] 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, , , , , , , , 00000000001.190.6, 00000000002.113.6, 00000000002.81.6] 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:37.368+0000 7f7d296c9780 0 bucket checkpoint complete 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=vwnfns-66 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-66 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:37.486+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:42.488+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:47.488+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:52.489+0000 7f821f887780 1 waiting to reach incremental sync.. 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:57.490+0000 7f821f887780 1 bucket sync caught up with source: 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: local status: [, , , , , , , , 00000000001.784.6, 00000000002.530.6, 00000000002.1162.6] 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: remote markers: [, , , , , , , , 00000000001.784.6, 00000000002.530.6, 00000000002.1162.6] 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:52:57.490+0000 7f821f887780 0 bucket checkpoint complete 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=vwnfns-66 zones={a2, a1} 2024-04-13T23:18:12.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.919 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA= 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA=', 'Content-Length': '0'} 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d37ab93aee516df-00661b0cc9-236e-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.921 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA= 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iYxcpWk6Gcuxi7+dy67F4+lMgfA=', 'Content-Length': '0'} 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e484ed6213fa2c0-00661b0cc9-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:/vwnfns-66/?versions 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b8dcd84a761d3b-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:/vwnfns-66/?versions 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T23:18:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf375c713a51f0b3-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.927 INFO:tasks.rgw_multisite_tests:/vwnfns-66/?versions 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c553da96f31bcf8b-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?versions&max-keys=999 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:/vwnfns-66/?versions 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY= 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S7jV1/GbaZmAfTDzZDazfgbwEGY=', 'Content-Length': '0'} 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1993a413f55062-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66999falseobj11nulltrue2024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ad551ec4ab33c8-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T23:18:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2f97146c1185ef-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.933 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA= 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA=', 'Content-Length': '0'} 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb983673be05eb8-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.935 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA= 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S9pJxnYs59tqbCpxuVizphzM6jA=', 'Content-Length': '0'} 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f624c4e7f76771ad-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6637a92007de2a8-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.938 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11?versionId=null 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns= 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:+JhyXy4IMYBBLT453S0ORu4YXns=', 'Content-Length': '0'} 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003301a549960f1b5c-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.940 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b32bd03cb63c2d9b-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d302c02f6749e3b-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM= 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM=', 'Content-Length': '0'} 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0a232ca0f1b396-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.945 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM= 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:KfjGE9jomkVLsXs/hPQhwcdXxkM=', 'Content-Length': '0'} 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b8d7602ba82acd-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11 2024-04-13T23:18:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.947 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI= 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI=', 'Content-Length': '0'} 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038cb12ffd56d3b8d-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj11 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj11 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj11 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj11 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI= 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:IzkqxsLwsCRoRSdq4Z5hRAsZrdI=', 'Content-Length': '0'} 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5fea8274bb5065-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.950 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006709d942d30786a7-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj11 2024-04-13T23:18:12.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.953 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.953 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002489f61da8d2e5cd-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj111000falseobj112024-04-13T22:52:32.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.955 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0855810229e5677-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T23:18:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec0997af236afbf-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo= 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo=', 'Content-Length': '0'} 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ff78562c6f1ed2-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.960 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo= 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:I0QzaTRYAE+G85qtLu93YywuwFo=', 'Content-Length': '0'} 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccaaf88ca69f2089-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a814de98c3f31b2d-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.963 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22?versionId=null 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4= 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oX5/2Ujysqm41KN3rp7xuqoXvr4=', 'Content-Length': '0'} 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006434dbb183b504d6-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e7ee4001f95f89-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2d41f66dd9f925-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.968 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A= 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A=', 'Content-Length': '0'} 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ef9078502f2a78-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A= 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5TbXrekHet6P8exQTw7eRlqlL/A=', 'Content-Length': '0'} 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ee6bdfda21e58a-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.970 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.971 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI= 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI=', 'Content-Length': '0'} 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42c0b7f1ea015b5-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj22 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj22 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj22 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj22 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI= 2024-04-13T23:18:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lEak6DMefdoIrce4EJ0bAh6tVUI=', 'Content-Length': '0'} 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153218e7433a00cf-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f2b988bb1e33d4-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj22 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.976 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764d6e2451c045a4-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj221000falseobj222024-04-13T22:52:32.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.978 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebb5853805d1377-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022706f577d7a9c6a-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.981 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8= 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8=', 'Content-Length': '0'} 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49c3dc408c7dcb1-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8= 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:1Clf962Q52WpYVm/bpjLumB4TU8=', 'Content-Length': '0'} 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090cf43b8eb939e8e-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.983 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.984 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b637b8b72fbcf38-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33?versionId=null 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.986 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c= 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:U0mmh5zMAbkcy/66G289EIU3E6c=', 'Content-Length': '0'} 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108375a68b348837-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000047a749bf511cf56-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.989 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e6acdac725dac6-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33 2024-04-13T23:18:12.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33 2024-04-13T23:18:12.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc= 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc=', 'Content-Length': '0'} 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe66c3afa00ec1a-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc= 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:S2q223M47zk7EIgcq9I1zMChyoc=', 'Content-Length': '0'} 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce84b2ad19cd9154-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.994 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE= 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE=', 'Content-Length': '0'} 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a9a5078a7c98aea-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj33 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj33 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj33 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj33 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE= 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NwbyaeyAjRXnEd7UchYRl2XcVRE=', 'Content-Length': '0'} 2024-04-13T23:18:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001049bb88f047a2f5-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2c41edf0880d49c-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj33 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:12.999 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4c0d5622d0e745-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj331000falseobj332024-04-13T22:52:32.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115d6ce453babb03-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2615cbfc29e8832-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.004 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8= 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8=', 'Content-Length': '0'} 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006adc549fa2ffca75-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8= 2024-04-13T23:18:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:VctJzTCFs8sFZqiZM/8LjnHdMO8=', 'Content-Length': '0'} 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc509b34aebbcaac-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162aebc5df0ed19f-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.009 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44?versionId=null 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ= 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:atf54qCobe+CLR5lhidf4aQ4nsQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d461b633e47eb15e-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.011 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000077ec7c00ae11f9-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794439cf774fa91c-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.014 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc= 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc=', 'Content-Length': '0'} 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d1fbdaf093eb74a-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc= 2024-04-13T23:18:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:OA6BihfrAuRuNlakmVPuekFY5Xc=', 'Content-Length': '0'} 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359c20f1e3fe163d-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A= 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A=', 'Content-Length': '0'} 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e927d6bf486b9180-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj44 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj44 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj44 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj44 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.019 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A= 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eKRdoFK+0B6YR8uRLHBBCmzAa1A=', 'Content-Length': '0'} 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c04cdec2ec3faaa-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f344c68e037fdab6-00661b0cc9-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj44 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU= 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:XFwgxRTWFvsUsCaEtlhMUCVkuXU=', 'Content-Length': '0'} 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b384a82ac67f708-00661b0cc9-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj441000falseobj442024-04-13T22:52:32.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s= 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s=', 'Content-Length': '0'} 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003540c34588a5bc82-00661b0cc9-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.026 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:57 GMT 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s= 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:57 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:URkc33JPPSjzExDAsAuekeejp9s=', 'Content-Length': '0'} 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2cc81a8d5a2808-00661b0cc9-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc= 2024-04-13T23:18:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc=', 'Content-Length': '0'} 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c56149a856d39d82-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc= 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NNa5N/aJwmWx8oayJLFNdgCmNmc=', 'Content-Length': '0'} 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d8c743a0a4ef3e-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.031 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8= 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8=', 'Content-Length': '0'} 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0cd36177e257d4-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.033 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55?versionId=null 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8= 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WLOFasf+25ykuaxaqFdFs9U4+j8=', 'Content-Length': '0'} 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3714b7918622559-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T23:18:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b68c2eee809fa08-00661b0cca-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180652a9ae213d6f-00661b0cca-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.038 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0= 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0=', 'Content-Length': '0'} 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5ede09950dc837-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0= 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:zalYU4yFITL+ax5uYMjZauR54t0=', 'Content-Length': '0'} 2024-04-13T23:18:13.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d315b6ef21003be-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.041 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ= 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-66:f6f208d6-f791-479b-bc3d-112512eec2d4.6692.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2b5bbe4a7ed456-00661b0cca-236e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj55 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj55 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj55 2024-04-13T23:18:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj55 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.043 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ= 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:mYAy4kkcDGpqPcAGEhq4hnKDjFQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 22:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197514d6c58266e9-00661b0cca-1a22-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004deb85208b35736a-00661b0cca-236e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/?prefix=obj55 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.046 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:52:58 GMT 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4= 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:52:58 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:w3GhoTiNdqc2RRsXCx4u70MPuh4=', 'Content-Length': '0'} 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e41f6033ff502729-00661b0cca-1a22-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 22:52:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vwnfns-66obj551000falseobj552024-04-13T22:52:32.276Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=vwnfns-66 zones={a2, a1} 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:52:58.085+0000 7fc70bec7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 116241) UID: 0 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi132.stdout: INFO: 2024-04-13T22:52:58.085+0000 7fc71109c7c0 -1 shutting down 2024-04-13T23:18:13.047 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-66 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 5 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.049 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de8c8d3a410a4bd-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj17 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj17 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj17 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.051 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj17 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:32iosuymcFZrIP2KY7QZOb8UQIc= 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:32iosuymcFZrIP2KY7QZOb8UQIc='} 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007457c6f10c2e943e-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000156b3cb79111cb29-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj34 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj34 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj34 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.054 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj34 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lc6mUMfBi+zfpZwY6Pw9NlkK90k= 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lc6mUMfBi+zfpZwY6Pw9NlkK90k='} 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf35cb17ec7d61f8-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.056 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df508f8846bfcc82-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj51 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj51 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj51 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj51 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:Ju1B2O/CgxZ9Vr6C9E+43k6soSA= 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:Ju1B2O/CgxZ9Vr6C9E+43k6soSA='} 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a50e1eda6fe17127-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.059 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbca12e170164b9e-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj68 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj68 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj68 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj68 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:YOXAFHwKCtLbz/kFxdXSW+WGXAs= 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:YOXAFHwKCtLbz/kFxdXSW+WGXAs='} 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000039f9eebba76b7f0-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.062 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baa00814d667e1b0-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj85 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj85 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj85 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj85 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:jfb8GMKxFrTRekHQ1q3y06otxRQ= 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:jfb8GMKxFrTRekHQ1q3y06otxRQ='} 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba9860bd698fbbdd-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-66 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 10 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ade0b18d3dd1394c-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj19 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj19 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj19 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj19 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:9JlSz+YNH9+a84xXHCuYwsl1ecE= 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9JlSz+YNH9+a84xXHCuYwsl1ecE='} 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47e9d33570e4640-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000358a7222d26da155-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj38 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj38 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj38 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj38 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:e8mwyVqgrlbkx+Y5JqCmpQa+SE4= 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:e8mwyVqgrlbkx+Y5JqCmpQa+SE4='} 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000415aaa5849618f11-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.072 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c6fcbfe0db5a8e4-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj57 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj57 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj57 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj57 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:WApo1ZL9XL0x2sxi2i0hiNYsgIY= 2024-04-13T23:18:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:WApo1ZL9XL0x2sxi2i0hiNYsgIY='} 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ecce265bf1404ba-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.075 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d4159260582e8b3-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj76 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj76 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj76 2024-04-13T23:18:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj76 2024-04-13T23:18:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:lGdECkOAabY8CgwhEgRNUR1McSQ= 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:lGdECkOAabY8CgwhEgRNUR1McSQ='} 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006027cdd029d76b2-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059eafd043d67d3a1-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj95 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj95 2024-04-13T23:18:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj95 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj95 2024-04-13T23:18:13.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:cDKxU5W29xmaSltrkXFCBqoT0nQ= 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:cDKxU5W29xmaSltrkXFCBqoT0nQ='} 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005acf70916692624d-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-66 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 15 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a1472b7009d2a4c-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj23 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj23 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj23 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj23 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:AcVz9pUbHfR27k9CMVf/xCfG0Xk= 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:AcVz9pUbHfR27k9CMVf/xCfG0Xk='} 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074f95932e8baa1f8-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.085 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009183b5f9f633e911-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj46 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj46 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj46 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj46 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:7uwhu9f0G+SFYY2hnoGiDySC1cs= 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:7uwhu9f0G+SFYY2hnoGiDySC1cs='} 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ca0ef2ea0ea8f9a-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.088 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000503903b35d614545-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj69 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj69 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj69 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj69 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5i42n6lpKmIcP1apQxfk/0D01qg= 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5i42n6lpKmIcP1apQxfk/0D01qg='} 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002322aa8ce54d21b-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.091 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.093 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.093 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.093 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY= 2024-04-13T23:18:13.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iC1hVc/aBSXVnL/JTNbjbzf8IvY=', 'Content-Length': '0'} 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8b942f1b2d1fa9f-00661b0cd0-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj92 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj92 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj92 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:04 GMT 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj92 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DpyvH5dUE231pH0/kRw549SM37g= 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:04 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DpyvH5dUE231pH0/kRw549SM37g='} 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027abc30edc7779ad-00661b0cd0-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000298d1319338dfe24-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj115 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj115 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj115 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj115 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:FzvS69U1MASxo4SchS5TV6D37HM= 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:FzvS69U1MASxo4SchS5TV6D37HM='} 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003845caa07add70e9-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-66 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 20 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.100 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a07d9a3aaa548c3-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj29 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj29 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj29 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.101 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj29 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:NEnT98IxW89zvnG9CEvZEmeGfqg= 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:NEnT98IxW89zvnG9CEvZEmeGfqg='} 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f00af851005d79b-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000152f16c938460a20-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj58 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj58 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj58 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.104 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj58 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:TCuFDiOiXDMG4GGvZisyLtEk0Is= 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:TCuFDiOiXDMG4GGvZisyLtEk0Is='} 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e148bab5d694922e-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.106 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be58b75eeb7d6ee5-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj87 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj87 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj87 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj87 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:9dJscfjC7aDipTQBz2L15omQaUk= 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:9dJscfjC7aDipTQBz2L15omQaUk='} 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a5393c1e0c765eb-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.109 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000223fca108b102d-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj116 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj116 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj116 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj116 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:vQRvluzu98ff1GFe5YoABoUe2/c= 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:vQRvluzu98ff1GFe5YoABoUe2/c='} 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075400ce5ab1f2cff-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.112 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a842f5d2e728c1c6-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj145 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj145 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj145 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj145 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:eCiQ5B6hcoN3ee+SWkbKBChgX5A= 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eCiQ5B6hcoN3ee+SWkbKBChgX5A='} 2024-04-13T23:18:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a67b50a139fc5ce-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-66 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 25 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85a127b41820905-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj31 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj31 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj31 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.117 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj31 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:736ACA/fLgQIUgTq5AHENfrPsQw= 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:736ACA/fLgQIUgTq5AHENfrPsQw='} 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1ab8b1183c3e12c-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8d562451efbc9f7-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj62 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj62 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj62 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj62 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:iT5OyyNbM1Sw8S8cViTT95PoKwQ= 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:iT5OyyNbM1Sw8S8cViTT95PoKwQ='} 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dcd27eeb8bd48b7-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.122 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092a5a6c9a83db4b5-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj93 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj93 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj93 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj93 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:A5KsK04X+7RZUEyZyECtWNRDftE= 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:A5KsK04X+7RZUEyZyECtWNRDftE='} 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ceabf06bb9c8ece-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020d8ebe6ff6c0ac3-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj124 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj124 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj124 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj124 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:2VF0bWtuvlTOCJWBx7ywbzF4ZZA= 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:2VF0bWtuvlTOCJWBx7ywbzF4ZZA='} 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd59dfc839821f77-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.128 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ada50197a1a9e40-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj155 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj155 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj155 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj155 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hyYc/OXyvDUoPhzndLItZZbhtIw= 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hyYc/OXyvDUoPhzndLItZZbhtIw='} 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee84af3458064978-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'vwnfns-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket vwnfns-66 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: tenant: 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: bucket name: vwnfns-66 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: total entries: 30 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8043e81bd3a160e-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj37 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj37 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj37 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.133 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj37 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:HdsBJTG++tgjNFORUPEl3xpIBK4= 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:HdsBJTG++tgjNFORUPEl3xpIBK4='} 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9df37f5942c2d05-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000860261f5f4d3881c-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj74 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj74 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj74 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj74 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:wYVPsi5lI43dd8r/ILMAJ2/Z6go= 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:wYVPsi5lI43dd8r/ILMAJ2/Z6go='} 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005551ccdd057bacfe-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.138 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d845acc62ba9d31-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj111 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj111 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj111 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj111 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:DYvDZnoBX+pNCHDC2pKvhZgVQjg= 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:DYvDZnoBX+pNCHDC2pKvhZgVQjg='} 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001636d8347c7db51c-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.141 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050704e233873d838-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj148 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj148 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj148 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj148 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:hZbbagdDsrjrAbBcgDeu3EI3ze8= 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:hZbbagdDsrjrAbBcgDeu3EI3ze8='} 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5b51908322b5d88-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/ 2024-04-13T23:18:13.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/ 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/ 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.144 INFO:tasks.rgw_multisite_tests:/vwnfns-66/ 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w= 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:X7+RDY3j+IYbhOxkmJEQpZiR8+w=', 'Content-Length': '0'} 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c6066b2486cceeb-00661b0cd1-1a22-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-66/obj185 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-66/obj185 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-66/obj185 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 22:53:05 GMT 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:/vwnfns-66/obj185 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fnSuFQcYrketdfcOmYz8/pOYXnI= 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 22:53:05 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fnSuFQcYrketdfcOmYz8/pOYXnI='} 2024-04-13T23:18:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6ba08cc444a9d5b-00661b0cd1-1a22-a1'), ('Date', 'Sat, 13 Apr 2024 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:06.050+0000 7fef185c3780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:06.053+0000 7fef185c3780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T23:18:13.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi132.front.sepia.ceph.com port 8001: Connection refused 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > curl http://smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: anonymous 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=vwnfns-66 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vwnfns-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vwnfns-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:07.258+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:12.259+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:17.260+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:22.261+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:27.262+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:32.263+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:37.262+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:42.263+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:47.264+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:52.264+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:53:57.266+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:02.266+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:07.267+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:12.268+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:17.268+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:22.268+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:27.269+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:32.270+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:37.271+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:42.272+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:47.273+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:52.273+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:54:57.273+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:02.274+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:07.274+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 0.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:12.275+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:17.277+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:22.277+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:27.278+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:32.278+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:37.279+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:42.279+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:47.279+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:52.280+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:55:57.281+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:02.281+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:07.282+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:12.282+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:17.282+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:22.283+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:27.284+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:32.285+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:37.286+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:42.287+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:47.287+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:52.287+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:56:57.288+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:02.289+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:07.289+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:12.291+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:17.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:22.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:27.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:32.292+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:37.293+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:42.294+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:47.295+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:52.296+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:57:57.296+0000 7fbf1ea8f780 1 waiting to reach latest gen 6, on 4.. 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:58:02.297+0000 7fbf1ea8f780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:58:02.297+0000 7fbf1ea8f780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T22:58:02.297+0000 7fbf1ea8f780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-13T23:18:13.153 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:13.154 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.154 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-13T23:18:13.154 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2024-04-13T23:18:13.154 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:13.155 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-13T23:18:13.155 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-13T23:18:13.155 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-13T23:18:13.155 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 2465, in test_sync_single_bucket 2024-04-13T23:18:13.155 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnames, content) 2024-04-13T23:18:13.155 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:13.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713049190.433608_1085.1", 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:59:50.433608Z" 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T23:01:54.239+0000 7f7d27c1c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:13.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T23:01:54.240+0000 7f7d27c1c780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713049190.433608_1085.1", 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.433608Z", 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:01:54Z" 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.163 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1713049190.433608_1085.1", 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.433608Z", 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.166 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:13.167 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:01:54Z" 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049190.433608_1085.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.433608Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713049190.433608_1085.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[]} 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[]} 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=vwnfns-70 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/ 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/ 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/ 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:54 GMT 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:/vwnfns-70/ 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.170 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:R9vqkRMcyn5kKPrD2+9+ougi9JI= 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:R9vqkRMcyn5kKPrD2+9+ougi9JI=', 'Content-Length': '0'} 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f09d5ea014ed0331-00661b0ee2-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=vwnfns-71 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-71/ 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-71/ 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-71/ 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.171 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:54 GMT 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:/vwnfns-71/ 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:fumts7JMLHEfidZHCR1oIK6BBBI= 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:54 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:fumts7JMLHEfidZHCR1oIK6BBBI=', 'Content-Length': '0'} 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000032e4b253bd5d3fc8-00661b0ee2-2678-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T23:01:54.558113Z" 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T23:01:54.766+0000 7f05b6661780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T23:01:54.767+0000 7f05b6661780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:01:54Z" 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.179 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:01:54Z" 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049314.558113_1103.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:01:54.558113Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=vwnfns-70', '--dest-bucket=vwnfns-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=vwnfns-70', '--dest-bucket=vwnfns-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=vwnfns-70 --dest-bucket=vwnfns-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"vwnfns-70","zones":["a2","a1"]},"dest":{"bucket":"vwnfns-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "epoch": 16, 2024-04-13T23:18:13.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "predecessor_uuid": "8913bd9b-e2fc-4bf8-8d46-2b27831a5cc4", 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": [], 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period_map": { 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zonegroups": [ 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "a", 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "a", 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "api_name": "", 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "is_master": true, 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "endpoints": [ 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:18:13.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hostnames": [], 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hostnames_s3website": [], 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones": [ 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "a2", 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "endpoints": [ 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "http://smithi132.front.sepia.ceph.com:8001" 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_meta": false, 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_data": true, 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-13T23:18:13.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "read_only": false, 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tier_type": "", 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from_all": true, 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from": [], 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "redirect_zone": "", 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "supported_features": [ 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "compress-encrypted", 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding" 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "a1", 2024-04-13T23:18:13.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "endpoints": [ 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "http://smithi132.front.sepia.ceph.com:8000" 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_meta": false, 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_data": true, 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "read_only": false, 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tier_type": "", 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from_all": true, 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from": [], 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "redirect_zone": "", 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "supported_features": [ 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "compress-encrypted", 2024-04-13T23:18:13.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding" 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "placement_targets": [ 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "default-placement", 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tags": [], 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "storage_classes": [ 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "STANDARD" 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "default_placement": "default-placement", 2024-04-13T23:18:13.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_policy": { 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "groups": [ 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "sync-group", 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "data_flow": { 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "symmetrical": [ 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "sync-flow1", 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones": [ 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pipes": [ 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "sync-pipe", 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "source": { 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket": "vwnfns-70", 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones": [ 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "dest": { 2024-04-13T23:18:13.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket": "vwnfns-70", 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones": [ 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "f6f208d6-f791-479b-bc3d-112512eec2d4" 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "params": { 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "source": { 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "filter": { 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tags": [] 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "dest": {}, 2024-04-13T23:18:13.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "priority": 0, 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "mode": "system", 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "user": "" 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "enabled" 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled_features": [ 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding" 2024-04-13T23:18:13.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "b", 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "b", 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "api_name": "", 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "is_master": false, 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "endpoints": [ 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hostnames": [], 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "hostnames_s3website": [], 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "master_zone": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:18:13.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "zones": [ 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "b2", 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "endpoints": [ 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8001" 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_meta": false, 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_data": true, 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "read_only": false, 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tier_type": "", 2024-04-13T23:18:13.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from_all": true, 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from": [], 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "redirect_zone": "", 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "supported_features": [ 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "compress-encrypted", 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding" 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "id": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "b1", 2024-04-13T23:18:13.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "endpoints": [ 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_meta": false, 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "log_data": true, 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "read_only": false, 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tier_type": "", 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from_all": true, 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_from": [], 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "redirect_zone": "", 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "supported_features": [ 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "compress-encrypted", 2024-04-13T23:18:13.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding" 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "placement_targets": [ 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "name": "default-placement", 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "tags": [], 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "storage_classes": [ 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "STANDARD" 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "default_placement": "default-placement", 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_policy": { 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "groups": [] 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled_features": [ 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "resharding" 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ], 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "short_zone_ids": [ 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": "4123dd49-f32f-49d9-98c0-83e8adaa8c84", 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": 555598186 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": "5da80fbc-5edc-4ba7-826d-471359a9fc92", 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": 782658364 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": "ba6de64d-3aa0-42be-83ea-cebdba376cd8", 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": 2041901071 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": 2070976819 2024-04-13T23:18:13.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "master_zonegroup": "a", 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "master_zone": "f6f208d6-f791-479b-bc3d-112512eec2d4", 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period_config": { 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket_quota": { 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled": false, 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "check_on_raw": false, 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_size": -1, 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_size_kb": 0, 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_objects": -1 2024-04-13T23:18:13.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "user_quota": { 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled": false, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "check_on_raw": false, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_size": -1, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_size_kb": 0, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_objects": -1 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "user_ratelimit": { 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_read_ops": 0, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_write_ops": 0, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_read_bytes": 0, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_write_bytes": 0, 2024-04-13T23:18:13.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled": false 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "bucket_ratelimit": { 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_read_ops": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_write_ops": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_read_bytes": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_write_bytes": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled": false 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "anonymous_ratelimit": { 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_read_ops": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_write_ops": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_read_bytes": 0, 2024-04-13T23:18:13.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "max_write_bytes": 0, 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "enabled": false 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_id": "645d07fe-13a2-4d07-bad9-a662f2f6c3cf", 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_name": "test-realm", 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: {"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"vwnfns-70"},"dest":{"zone":"a1","bucket":"vwnfns-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"vwnfns-70"},"dest":{"zone":"a2","bucket":"vwnfns-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/ 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/ 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/ 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.208 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:/vwnfns-70/ 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo= 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo=', 'Content-Length': '0'} 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068cfbeae1aa1dc0c-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/obj1 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/obj1 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/obj1 2024-04-13T23:18:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:/vwnfns-70/obj1 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.210 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:rRKU++zLXh9ZRCaWcMKq4Ufq57U= 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:rRKU++zLXh9ZRCaWcMKq4Ufq57U='} 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e328cd2f8dbebd-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/ 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/ 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/ 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.211 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:/vwnfns-70/ 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo= 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo=', 'Content-Length': '0'} 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcaf3ca3b6b348bb-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/obj2 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/obj2 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/obj2 2024-04-13T23:18:13.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:/vwnfns-70/obj2 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.213 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:EPEGU/UiwhyiHVQfb69dxFgs+bE= 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:EPEGU/UiwhyiHVQfb69dxFgs+bE='} 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b987d62623efa1c-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/ 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/ 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/ 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:/vwnfns-70/ 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo= 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:QlNWhTcc9+8o2QqZJAke8V/3Tuo=', 'Content-Length': '0'} 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004889fc88941e29f5-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/obj3 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/obj3 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/obj3 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:/vwnfns-70/obj3 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.216 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:bi3SQM/vlu5RcSAgo/9Ub41gPMQ= 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:bi3SQM/vlu5RcSAgo/9Ub41gPMQ='} 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001289099cfb88730b-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-71/ 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-71/ 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-71/ 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.217 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:/vwnfns-71/ 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:O+CJTCAvIcGw8yKFrd7Mc2Tt+gg= 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:O+CJTCAvIcGw8yKFrd7Mc2Tt+gg=', 'Content-Length': '0'} 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e5a11adbd153505-00661b0ee3-27ca-a1'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:18:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8000 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:PUT 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:01:55 GMT 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:/vwnfns-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:tsCuTQfXAHrED8h42W84xYVkrQY= 2024-04-13T23:18:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 13 Apr 2024 23:01:55 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tsCuTQfXAHrED8h42W84xYVkrQY='} 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d58eb918c3fcb7c5-00661b0ee3-27ca-a1'), ('Date', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: No --period given, using current period=5039ce81-80d5-4d03-a26f-34b9088aaab4 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T23:01:54.558113Z" 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T21:28:28.018806Z" 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.318155Z" 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:30:59.305268Z" 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T23:01:56.233+0000 7f6fd2709780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: 2024-04-13T23:01:56.237+0000 7f6fd2709780 20 RGW-SYNC:meta: read sync status 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 1, 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:18:13.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:13.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "state": 1, 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "realm_epoch": 2 2024-04-13T23:18:13.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 1, 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 1 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:01:56Z" 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.227 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "period": "5039ce81-80d5-4d03-a26f-34b9088aaab4", 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1713049314.558113_1103.1", 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T23:01:54.558113Z", 2024-04-13T23:18:13.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "marker": "1_1713043708.018806_80.1", 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T21:28:28.018806Z", 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.318155_197.1", 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.318155Z", 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.230 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "marker": "1_1713047459.305268_202.1", 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:30:59.305268Z", 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.231 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:01:56Z" 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713049314.558113_1103.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-13T23:01:54.558113Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713043708.018806_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T21:28:28.018806Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713047459.318155_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.318155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713047459.305268_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:30:59.305268Z', 'realm_epoch': 2}}] 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713049314.558113_1103.1', 1: '1_1713043708.018806_80.1', 2: '1_1713047459.318155_197.1', 3: '1_1713047459.305268_202.1'} 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: [ 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "last_update": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:01:56Z" 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.239 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.240 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.241 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:01:56Z" 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:01Z" 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.248 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.250 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.251 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:01Z" 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:06Z" 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.257 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.259 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:06Z" 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.261 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:11Z" 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.267 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.268 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.270 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:11Z" 2024-04-13T23:18:13.271 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:17Z" 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.277 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.278 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.280 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.281 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:17Z" 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:22Z" 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.287 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.290 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.291 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:22Z" 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:27Z" 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.298 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.299 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.300 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.301 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:27Z" 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:32Z" 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.309 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.311 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:32Z" 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:37Z" 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.317 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.318 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:37Z" 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:42Z" 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.327 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.330 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:42Z" 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:47Z" 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.339 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.339 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.339 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.339 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.339 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.340 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.341 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.342 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.343 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:47Z" 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:53Z" 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.350 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.351 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.352 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:53Z" 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:02:58Z" 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.359 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.361 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.362 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:02:58Z" 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:03Z" 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.371 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.372 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.373 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.374 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:03Z" 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:08Z" 2024-04-13T23:18:13.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.382 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:08Z" 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:13Z" 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.390 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.391 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:13Z" 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:18Z" 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.400 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.401 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:18Z" 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:23Z" 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.410 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.411 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.412 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:23Z" 2024-04-13T23:18:13.414 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:28Z" 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.421 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.423 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:28Z" 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.424 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:33Z" 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.431 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.432 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.433 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:33Z" 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:39Z" 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.440 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.442 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:39Z" 2024-04-13T23:18:13.444 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:44Z" 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.450 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.451 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.452 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.453 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:44Z" 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.454 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:49Z" 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.461 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.463 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:49Z" 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:54Z" 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046153", 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:05.852657Z" 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.472 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047071", 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:57.484330Z" 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.473 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053225", 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:59:50.402677Z" 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:54Z" 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046153', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:53:05.852657Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047071', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T22:59:57.484330Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053225', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T22:59:50.402677Z'}}] 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046153', 1: '00000000000000000000:00000000000000047071', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053225'} 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046295 target=00000000000000000000:00000000000000046153, shard=1 master=00000000000000000000:00000000000000047211 target=00000000000000000000:00000000000000047071, shard=3 master=00000000000000000000:00000000000000053505 target=00000000000000000000:00000000000000053225 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "sync_status": { 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "info": { 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "sync", 2024-04-13T23:18:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "num_shards": 4, 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "instance_id": 13387773392408425331 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "markers": [ 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 0, 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:18:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 1, 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 0, 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 2, 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 9, 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: { 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "key": 3, 2024-04-13T23:18:13.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "val": { 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "status": "incremental-sync", 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "next_step_marker": "", 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total_entries": 57, 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "pos": 0, 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: ] 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "full_sync": { 2024-04-13T23:18:13.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "total": 66, 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "complete": 66 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: }, 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: "current_time": "2024-04-13T23:03:59Z" 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi132.stdout: INFO: } 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: "instance_id": 13387773392408425331 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-13T23:18:13.481 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046295", 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T23:01:55.861427Z" 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-13T23:18:13.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047211", 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T23:01:55.789755Z" 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000033468", 2024-04-13T23:18:13.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T22:53:11.276425Z" 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: { 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053505", 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-13T23:18:13.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-13T23:01:55.836207Z" 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: } 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: ] 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-13T23:03:59Z" 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests:} 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046295', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.861427Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.789755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033468', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-13T22:53:11.276425Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053505', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-13T23:01:55.836207Z'}}] 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000046295', 1: '00000000000000000000:00000000000000047211', 2: '00000000000000000000:00000000000000033468', 3: '00000000000000000000:00000000000000053505'} 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/ 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/ 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/ 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:03:59 GMT 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:/vwnfns-70/ 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:tFOjMIDMNatUSv+2bBi2gslb7bM= 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:tFOjMIDMNatUSv+2bBi2gslb7bM=', 'Content-Length': '0'} 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003825a8407ffc6108-00661b0f5f-27d0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/obj1 2024-04-13T23:18:13.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/obj1 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/obj1 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.488 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:03:59 GMT 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:/vwnfns-70/obj1 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:oosvsvkrx9cpjs1o2Oqd59RRyJQ= 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:oosvsvkrx9cpjs1o2Oqd59RRyJQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-70:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aa9e0c8c323491b-00661b0f5f-27d0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/obj1 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/obj1 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/obj1 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:GET 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:03:59 GMT 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:/vwnfns-70/obj1 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:5EzvppZWRdIQFcw8GwtAElEbyx0= 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:5EzvppZWRdIQFcw8GwtAElEbyx0=', 'Content-Length': '0'} 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 13 Apr 2024 23:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'f6f208d6-f791-479b-bc3d-112512eec2d4:vwnfns-70:f6f208d6-f791-479b-bc3d-112512eec2d4.9850.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a62f44203992043-00661b0f5f-27d0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vwnfns-70/obj2 2024-04-13T23:18:13.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vwnfns-70/obj2 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vwnfns-70/obj2 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi132.front.sepia.ceph.com:8001 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.491 INFO:tasks.rgw_multisite_tests:Sat, 13 Apr 2024 23:03:59 GMT 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:/vwnfns-70/obj2 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:AWS C81CVW06BOFYI3GBQ4YX:eVTB+XAz03UfRDPTfFtH6+dOMGQ= 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 13 Apr 2024 23:03:59 GMT', 'Authorization': 'AWS C81CVW06BOFYI3GBQ4YX:eVTB+XAz03UfRDPTfFtH6+dOMGQ=', 'Content-Length': '0'} 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000022766e6910bc4bff-00661b0f5f-27d0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 13 Apr 2024 23:03:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-13T23:18:13.492 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 6821.132s 2024-04-13T23:18:13.493 INFO:tasks.rgw_multisite_tests: 2024-04-13T23:18:13.493 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=8) 2024-04-13T23:18:13.493 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-13T23:18:13.900 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6ea88206543243d29f7dad146446706d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_7d36322c373c73b184fd252cd881b122d8c51a3b/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-13T23:18:13.902 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-04-13T23:18:13.911 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-04-13T23:18:13.956 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-13T23:18:13.965 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-13T23:18:13.965 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:13.969 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T23:18:13.964+0000 7f5b20b1a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 48148) UID: 0 2024-04-13T23:18:13.969 INFO:tasks.rgw.c1.client.0.smithi132.stdout:2024-04-13T23:18:13.964+0000 7f5b25cef7c0 -1 shutting down 2024-04-13T23:18:15.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-13T23:18:20.067 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-13T23:18:20.067 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-04-13T23:18:20.088 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/vault-root-token 2024-04-13T23:18:20.175 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-13T23:18:20.175 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:20.176 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T23:18:20.175+0000 7fd95da60700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 119966) UID: 0 2024-04-13T23:18:20.176 INFO:tasks.rgw.c1.client.1.smithi132.stdout:2024-04-13T23:18:20.175+0000 7fd962c357c0 -1 shutting down 2024-04-13T23:18:20.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-13T23:18:20.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-13T23:18:26.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2024-04-13T23:18:26.277 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-13T23:18:26.278 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-04-13T23:18:26.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/vault-root-token 2024-04-13T23:18:26.372 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-13T23:18:26.372 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:26.373 INFO:tasks.rgw.c2.client.0.smithi175.stdout:2024-04-13T23:18:26.372+0000 7fa9c2a3f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 48159) UID: 0 2024-04-13T23:18:26.457 INFO:tasks.rgw.c2.client.0.smithi175.stdout:2024-04-13T23:18:26.372+0000 7fa9c7c147c0 -1 shutting down 2024-04-13T23:18:27.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-13T23:18:31.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-13T23:18:32.474 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-13T23:18:32.474 DEBUG:teuthology.orchestra.run.smithi175:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-04-13T23:18:32.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/vault-root-token 2024-04-13T23:18:32.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2024-04-13T23:18:32.730 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-13T23:18:32.730 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:32.731 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T23:18:32.729+0000 7fbf8cf1e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 49426) UID: 0 2024-04-13T23:18:32.736 INFO:tasks.rgw.c2.client.1.smithi175.stdout:2024-04-13T23:18:32.729+0000 7fbf920f37c0 -1 shutting down 2024-04-13T23:18:38.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-13T23:18:38.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-13T23:18:38.832 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-13T23:18:38.833 DEBUG:teuthology.orchestra.run.smithi175:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-04-13T23:18:38.854 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/vault-root-token 2024-04-13T23:18:38.926 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-13T23:18:38.936 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-04-13T23:18:39.691 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-13T23:18:39.692 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-13T23:18:39.692 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-04-13T23:18:39.692 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:39.755 INFO:tasks.ceph.c2.osd.0:Stopped 2024-04-13T23:18:39.755 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-04-13T23:18:39.755 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:39.817 INFO:tasks.ceph.c2.osd.1:Stopped 2024-04-13T23:18:39.818 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-04-13T23:18:39.818 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:39.880 INFO:tasks.ceph.c2.osd.2:Stopped 2024-04-13T23:18:39.881 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-13T23:18:39.881 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-04-13T23:18:39.881 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:39.906 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-04-13T23:18:39.906 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-13T23:18:39.906 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-04-13T23:18:39.906 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:39.966 INFO:tasks.ceph.c2.mon.a:Stopped 2024-04-13T23:18:39.967 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-13T23:18:39.967 DEBUG:teuthology.orchestra.run.smithi175:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-13T23:18:40.076 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T23:18:40.076 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-04-13T23:18:40.559 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T23:18:40.560 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-04-13T23:18:40.692 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T23:18:40.692 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-04-13T23:18:40.813 INFO:tasks.ceph:Archiving mon data... 2024-04-13T23:18:40.815 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/data/c2.mon.a.tgz 2024-04-13T23:18:40.815 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2024-04-13T23:18:40.832 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.qzZqPkSR1d 2024-04-13T23:18:40.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.qzZqPkSR1d 2024-04-13T23:18:41.015 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.qzZqPkSR1d 2024-04-13T23:18:41.151 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.qzZqPkSR1d is 412KB 2024-04-13T23:18:41.251 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.qzZqPkSR1d 2024-04-13T23:18:41.268 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-13T23:18:41.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T23:18:41.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T23:18:41.334 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-13T23:18:41.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-04-13T23:18:41.704 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-13T23:18:41.704 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-13T23:18:41.705 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-04-13T23:18:41.705 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:41.765 INFO:tasks.ceph.c1.osd.0:Stopped 2024-04-13T23:18:41.765 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-04-13T23:18:41.766 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:41.823 INFO:tasks.ceph.c1.osd.1:Stopped 2024-04-13T23:18:41.823 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-04-13T23:18:41.824 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:41.879 INFO:tasks.ceph.c1.osd.2:Stopped 2024-04-13T23:18:41.880 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-13T23:18:41.880 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-04-13T23:18:41.880 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:41.905 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-04-13T23:18:41.906 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-13T23:18:41.906 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-04-13T23:18:41.906 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T23:18:41.967 INFO:tasks.ceph.c1.mon.a:Stopped 2024-04-13T23:18:41.968 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-13T23:18:41.968 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-13T23:18:42.036 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T23:18:42.036 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-04-13T23:18:42.307 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T23:18:42.307 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-04-13T23:18:42.456 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T23:18:42.457 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-04-13T23:18:42.530 INFO:tasks.ceph:Archiving mon data... 2024-04-13T23:18:42.530 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/data/c1.mon.a.tgz 2024-04-13T23:18:42.531 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-04-13T23:18:42.547 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.KdCBz2VfWA 2024-04-13T23:18:42.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.KdCBz2VfWA 2024-04-13T23:18:42.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.KdCBz2VfWA 2024-04-13T23:18:42.795 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.KdCBz2VfWA is 379KB 2024-04-13T23:18:42.864 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.KdCBz2VfWA 2024-04-13T23:18:42.881 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-13T23:18:42.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T23:18:42.923 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T23:18:42.956 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-13T23:18:42.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-13T23:18:42.989 DEBUG:teuthology.orchestra.run.smithi175:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-13T23:18:43.014 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-13T23:18:43.027 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-13T23:18:43.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-13T23:18:43.182 INFO:tasks.ceph:Archiving crash dumps... 2024-04-13T23:18:43.184 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/remote/smithi132/crash 2024-04-13T23:18:43.185 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-13T23:18:43.212 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/remote/smithi175/crash 2024-04-13T23:18:43.213 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-13T23:18:43.252 INFO:tasks.ceph:Compressing logs... 2024-04-13T23:18:43.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T23:18:43.257 DEBUG:teuthology.orchestra.run.smithi175:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T23:34:07.236 INFO:tasks.ceph:Archiving logs... 2024-04-13T23:34:07.237 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/remote/smithi132/log 2024-04-13T23:34:07.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T23:34:30.061 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/remote/smithi175/log 2024-04-13T23:34:30.063 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T23:36:16.179 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-13T23:36:16.190 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-13T23:36:16.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T23:36:16.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T23:36:16.285 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-13T23:36:16.295 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-13T23:36:16.296 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T23:36:16.298 DEBUG:teuthology.orchestra.run.smithi175:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T23:36:16.314 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2024-04-13T23:36:16.317 INFO:teuthology.orchestra.run.smithi175.stderr:bash: ntpq: command not found 2024-04-13T23:36:16.319 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T23:36:16.319 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-04-13T23:36:16.319 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv01.front.sepia.ceph.com 3 9 377 221 +471us[ +394us] +/- 58ms 2024-04-13T23:36:16.319 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 79 -1102us[-1102us] +/- 53ms 2024-04-13T23:36:16.319 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +731us[ +731us] +/- 86ms 2024-04-13T23:36:16.319 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T23:36:16.346 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T23:36:16.347 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-04-13T23:36:16.347 INFO:teuthology.orchestra.run.smithi175.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 +555us[ +549us] +/- 61ms 2024-04-13T23:36:16.347 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 -1057us[-1057us] +/- 54ms 2024-04-13T23:36:16.347 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +815us[ +810us] +/- 86ms 2024-04-13T23:36:16.347 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T23:36:16.348 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-13T23:36:16.358 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-13T23:36:16.359 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-13T23:36:16.370 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T23:36:16.444 DEBUG:teuthology.orchestra.run.smithi175:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T23:36:16.555 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T23:36:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T23:36:16.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T23:36:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T23:36:16.619 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-13T23:36:16.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-13T23:36:16.638 INFO:teuthology.task.internal:Duration was 9208.404763 seconds 2024-04-13T23:36:16.639 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-13T23:36:16.648 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-13T23:36:16.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T23:36:16.650 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T23:36:16.697 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T23:36:16.766 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T23:36:16.927 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-13T23:36:16.928 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-04-13T23:36:16.928 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T23:36:16.950 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi175.front.sepia.ceph.com 2024-04-13T23:36:16.950 DEBUG:teuthology.orchestra.run.smithi175:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T23:36:16.972 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-13T23:36:16.973 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T23:36:16.993 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T23:36:17.138 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-13T23:36:17.138 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T23:36:17.141 DEBUG:teuthology.orchestra.run.smithi175:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T23:36:17.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-13T23:36:17.238 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-13T23:36:17.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T23:36:17.266 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T23:36:17.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-13T23:36:17.331 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T23:36:17.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T23:36:17.357 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-04-13T23:36:17.377 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = core 2024-04-13T23:36:17.394 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T23:36:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T23:36:17.438 DEBUG:teuthology.orchestra.run.smithi175:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T23:36:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T23:36:17.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-13T23:36:17.466 INFO:teuthology.task.internal:Transferring archived files... 2024-04-13T23:36:17.467 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/remote/smithi132 2024-04-13T23:36:17.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T23:36:17.516 DEBUG:teuthology.misc:Transferring archived files from smithi175:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822/remote/smithi175 2024-04-13T23:36:17.516 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T23:36:17.555 INFO:teuthology.task.internal:Removing archive directory... 2024-04-13T23:36:17.555 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-13T23:36:17.557 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-13T23:36:17.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-13T23:36:17.617 INFO:teuthology.task.internal:Not uploading archives. 2024-04-13T23:36:17.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-13T23:36:17.639 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-13T23:36:17.639 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T23:36:17.641 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T23:36:17.655 INFO:teuthology.orchestra.run.smithi132.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 23:36 /home/ubuntu/cephtest 2024-04-13T23:36:17.662 INFO:teuthology.orchestra.run.smithi175.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 23:36 /home/ubuntu/cephtest 2024-04-13T23:36:17.663 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-13T23:36:17.673 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-13T23:36:17.695 INFO:teuthology.nuke:Checking targets against current locks 2024-04-13T23:36:17.722 DEBUG:teuthology.nuke:shortname: smithi132 2024-04-13T23:36:17.722 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T23:36:17.748 DEBUG:teuthology.nuke:shortname: smithi175 2024-04-13T23:36:17.748 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T23:36:17.752 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 20:57:26.002487', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA/N/FKLBquRPdl60M0y1Az5c6woju1lkGZ2FTsAqgZWA3bDiyOusiaFVnYZpk0uy/nC9ouvTL4sTO6i7eJUTrk='} 2024-04-13T23:36:17.775 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_22:40:02-rgw-reef-distro-default-smithi/7654822', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 20:57:26.000943', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9iF1EUT4zEZ4kQHllVdkLfUpKC05i7vMlbtsRrcaTRcRWNj2GSkNeTg3+iKQ9jFBfrQGq3nEZemVeieYd/EBw='} 2024-04-13T23:36:17.803 INFO:teuthology.orchestra.console.smithi132:Power off 2024-04-13T23:36:17.804 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T23:36:17.833 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-04-13T23:36:17.833 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T23:36:17.851 INFO:teuthology.orchestra.console.smithi175:Power off 2024-04-13T23:36:17.852 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T23:36:17.863 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-04-13T23:36:17.876 DEBUG:teuthology.orchestra.console.smithi175:power off output: Chassis Power Control: Down/Off 2024-04-13T23:36:17.876 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T23:36:17.901 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2024-04-13T23:36:21.864 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T23:36:21.902 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T23:36:21.978 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-04-13T23:36:22.014 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2024-04-13T23:36:25.982 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T23:36:26.015 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T23:36:26.097 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-04-13T23:36:26.127 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off 2024-04-13T23:36:26.198 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-04-13T23:36:26.228 INFO:teuthology.orchestra.console.smithi175:Power off completed 2024-04-13T23:36:26.333 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} duration: 9208.404762506485 failure_reason: rgw multisite test failures flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6ea88206543243d29f7dad146446706d status: fail success: false 2024-04-13T23:36:26.333 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T23:36:26.424 INFO:teuthology.run:FAIL